Metrics for Project Management
An Experience Report from a Successful e-Government Software Project
Project: UKSMA Conference Wolverhampton Document–ID: Metrics V03 d Version: V1.0-08 Replaces: W0.3-02 Author: Dr. Thomas Fehlmann Confidentiality: Publication Distribution: ISBSG, UKSMA for Project Management
Issue of: 25. August 2003
Carbon Copy: SwiSMA Member of the
2 W0.2003 25. Thomas Fehlmann Dr.
Modification Notice New document Added acquisition story Submitted to UKSMA conference Reviewed and finalized
Author Dr. Thomas Fehlmann Dr. Thomas Fehlmann Dr.Change Control
The following table documents the actual development stage of this document. Every change made to this document requires a new issue.5.2003 21.5.DOC Copyright © 2003 Euro Project Office
Page 2 of 15
. Thomas Fehlmann
METRICS FOR PROJECT MANAGEMENT V10.3 V1.5.2003 29.8.0
Date 14.1 X0.
This case study explains the Six Step To Completion – method for progress tracking in projects. The critical success factor is to link software quality assurance to progress reporting. In this case it was the first time software people had to get used to metrics and that they draw some benefit out of it.DOC Copyright © 2003 Euro Project Office
Page 3 of 15
. Progress tracking is a project management task.
METRICS FOR PROJECT MANAGEMENT V10. We want the deliverables resulting from that task be reviewed and approved. Delivery rates become apparent from the beginning. but it has specific effects in software projects. Based on that data we track actual duration and effort spent in projects and compare with plans. A task is no longer finished when the planned effort had been spent. Teams stay focused on the planned tasks and keep to the deadline. Although bad effort estimation cannot be completely avoided. it is detected in the very early stages of the project. The effect is surprising: software projects suddenly behave “normal”.
1 4.1 3.1 2.4 Chapter 3: 3.Table of Contents
Chapter 1: 1.3 4.3 Chapter 2: 2.2 2.5 4.2 1.7
The Problem e-WorkPermits – an e-Government application The Project The Dilemma The Approach Taken The Taskforce The Six Step To Completion – Metrics Integrated Quality Assurance Progress Metrics for Consensus Build-up The Successful Outcome The Effects of Visualization Why This Works Tools And Methods The Project Database Visualization using Excel Data Capture with Personalized Tickets Data Capture in the War Room Automated Data Capture using a Project Repository Relation to Other Software Metrics Conclusion
5 5 5 6 7 7 7 8 8 9 9 9 10 10 10 12 13 14 14 15
Index of Figures
Figure 1: The Six Step To Completion Model for Measuring Objective Evidence Figure 2: Sample Project Management Overview Figure 3: Sample Progress Track with Issue History Figure 4: Sample Personalized Ticket Figure 5: Wall Sized Print of the Progress Track as Paper Tool 7 11 12 13 14
METRICS FOR PROJECT MANAGEMENT V10.3 2.2 4.1 1.6 4.4 4.2 Chapter 4: 4.DOC Copyright © 2003 Euro Project Office
Page 4 of 15
the need to exchange international experts is apparent. The first holder of the position left the project within the first two months.0-13.
The trouble started at the very beginning. A large consulting company won the contract and subcontracted several specialists for creating software and setting up the service on the Internet. Thus setting up a web site that allows applying for a work permit through the Internet and – if the application is all– inclusive for the needed documents and the applicant fulfils the law’s requirements – issuing the permits instantly is a major competitive advantage. Using the same data for external and internal users.UKSMA Conference Wolverhampton Metrics for Project Management
Dr. despite allocating around 30% of the total price for project management. Especially in the ICT industry. therefore the government department had no means of discrimination. the system should maximize effectiveness. Thus it was not required. by a total planned project
METRICS FOR PROJECT MANAGEMENT V10. the government department hired a consulting company to write an invitation to tender. This document did not mention project methodology or project management at all. Having no expertise in setting up a project. In many other aspects of economical life. 23 September. There was no project plan. but as well for banking and pharmaceuticals.DOC Copyright © 2003 Euro Project Office
Page 5 of 15
. the consulting company was not able to provide a skilled project manager. the second tried to manage the project by hiding the relevant information. All he had was a Gantt chart. having the ability to transfer experts from one country to another is decisive for the attractiveness of a site. and for sure no quality management or quality plan. However. mentioning activities lasting two to three month. and no details specified. no project organisation. 2003
e-WorkPermits – an e-Government application
The Department for Labour and Economy of the Swiss canton of Zurich decided to undertake a bold step towards e-Government in 2002 when starting a project to ease the issue of work permits. The same system was to provide workflow support for the civil servants of the several agencies involved in the review and approval of the applications. Thomas Fehlmann Version V1. no roles and responsibilities.
was to spend additional money for a second project manager who should take over after completion of the contract.UKSMA Conference Wolverhampton Metrics for Project Management
Dr. Obviously it was not. there was no other way than to insist on the schedule as stated in the contract but they had to let the supplier decide how to meet its obligations. All they could do. he did not accept the suggestion because it was not foreseen in the contract. this task must have been finished by this very afternoon. In reality not even the hosting was prepared.DOC Copyright © 2003 Euro Project Office
Page 6 of 15
. and for independent testing. We had to endure another three missed milestones before the supplier finally admitted that the delivery date had passed and the contract failed. and the software was buggy and very late. and a To-Do list that an impressing tendency to grow in size. It did neither mention a professional (PMI) certificate. For the government department. nor any specific project management method. 23 September. when this statement had been true the day before the meeting.0-13. 2003
duration of about seven month. to avoid having to rely on the supplier’s testing records.
METRICS FOR PROJECT MANAGEMENT V10. but not substantiate project management. Thus the project went the way all such projects must go: It hardly made the first milestone. and completely missed the second. and what they did. The contract did stipulate. When the author joined the project and proposed to use progress metrics to the supplier.
Obviously this was not state–of–the–art project management. already with a long list of To-Do’s. The author remembers his first project steering meeting where the project manager stated that one of these three-month tasks is “98% finished”. The author remarked that. Thomas Fehlmann Version V1.
Instead of arguing if the task or To-Do had been completed or not. each of the To-Do’s was connected to a distinctive deliverable. as every project manager does with tasks in his project plan.UKSMA Conference Wolverhampton Metrics for Project Management
Dr. We started with a To-Do list because that was a terminology people could follow. the evident status of the deliverable tells us without asking people’s opinion. Figure 1: The Six Step To Completion Model for Measuring Objective Evidence
Assignment Assignment Template Template
Draft version Draft version
Review report Review report
METRICS FOR PROJECT MANAGEMENT V10. Thomas Fehlmann Version V1. So how do you deal with people from whom you depend and who can listen neither to you nor to common lore nor even simply to good reason?
The Six Step To Completion – Metrics
The answer was a very simple metrics that was so simple those people from the supplier did not even understand that it is a metric indeed. That deliverable uniquely identifies the To-Do and gives objective evidence for completion of the task.DOC Copyright © 2003 Euro Project Office
Page 7 of 15
The Approach Taken
Thus the government department installed a taskforce to take over and lead the project to its goal. Professionals with an auditing background understand the term “objective evidence”. The primary objective was that the supplier finished the delivery of the web application as soon as possible. However. 23 September.0-13. thus it was not an option to stop the project.
The last stage is equivalent for making the deliverable available for developers who have to rely on it and does not necessarily refer to the end users. More questionable it is when deliverables are already in use while still under finalization. 23 September. Only tasks such as meetings – with the minutes as deliverable – and testing may have somewhat different percentages. see the last line in Figure 3: Sample Progress Track with Issue History. i.
Progress Metrics for Consensus Build-up
Originally these percentages were just a guess. Thus. Thus progress metrics are very important for management. the draft stage for another 30%. or missing approvals clearly indicate missing consensus about the delivered results. However. in the very first project where such completion metrics had been used (1999) we discovered that these percentages correspond quite neatly with the total task duration (not the effort!) such that we started using the completion percentages for Gantt chart tracking. The grading of the steps varies in nine degrees between “just begun” and “completed”. We have since 1999 a record of around forty projects in a dozen different organizations that followed the same pattern with a variation below ten percent. Thomas Fehlmann Version V1. training. Reviews and finalization work still going on.0-13. e. review reports. The percentage values set forth in Figure 1: The Six Step To Completion Model for Measuring Objective Evidence seem to fit pretty well for creating documents. but it happens too. We use the Six Steps to Completion – model to measure progress based on objective evidence for completion for each of the six steps. This worked well in the e-Government project. and evidence for operational use such as distribution lists for documents and build logs for software. For example. if the completion rate is 40%. preparing workshops. in object-oriented development. released versions.
METRICS FOR PROJECT MANAGEMENT V10.DOC Copyright © 2003 Euro Project Office
Page 8 of 15
. thus the familiar pattern of two green stages followed by two yellow ones. approval gives 15% and when put into operational use the last 10%. Thus it is possible to acknowledge work in progress within the Six Stages To Completion. or writing software. Also it often happens that review or testing activities and finalization go in parallel.g. The benefit of completion metrics based on reviews and approvals is that they reflect the degree of consensus reached for the deliverables. 2003
Integrated Quality Assurance
All we need is to provide a project repository where to post draft versions. completion rates give an accurate estimate for the actual task duration. it means that we still need 60% of the total duration. 1 ½ times the duration that had already passed. The idea stage accounts for 10% of the total completion metric. If that was not foreseen by the project plan then you better update the plan right now to make it consistent with reality and manage expectations.e. the finalization stage with 20%. Then comes the review stage with 15%.UKSMA Conference Wolverhampton Metrics for Project Management
ch went on-line 20 February 2003. Exactly that happened in the early stages of the e-Government project. The reason is very simple: The Six Steps To Completion represents nothing else than ordinary quality planning. The taskforce met then weekly in a room with the actual Six Steps To Completion displayed prominently at a wall. The effect of such missing quality assurance is not felt immediately. 23 September. And interestingly: even the taskforce cost remained within budget. causing significantly less bad surprises later on. Every serious Quality Plan must provide exactly that: a reviewing method and an approval procedure for every single deliverable in the project. The team silently agrees on the necessary quality assurance being done at the root. Often quality planning is done at the beginning of the project (although not in this instance) but then not executed because of other priorities shovelling up. It might even felt favourably for the moment because the QA resources are now free to do even more draft development. and was the reason why the supplier’s people were not able to listen.
Why This Works
Obviously this was more than just a psychological trick.DOC Copyright © 2003 Euro Project Office
Page 9 of 15
. The web site www. in contrary. There was not a single date missed.workpermits. or in this case. 2003
The Successful Outcome
The Effects of Visualization
You never forget when the supplier’s project manager first noted that his To-Do. the hosting strategies established. All To-Dos including all the quality issues discovered earlier were resolved. which he thought having finished by “85%”. but three weeks earlier than expected in the revised schedule after the taskforce took over.
METRICS FOR PROJECT MANAGEMENT V10. The Six Steps To Completion – metrics makes this impossible. to the To-Dos.zh.UKSMA Conference Wolverhampton Metrics for Project Management
Dr. It short-circuits the quality plan to the project plan. counted only for some 30%.0-13. Thomas Fehlmann Version V1. and the site started to receive very excited feedback. three months late with regard to the original schedule. This seemingly positive experience will then be iterated again and again until it is too late and the quality problems start becoming apparent.
and issues. such as deliverable specifications. since it is embedded in MS Office. Excel as GUI is possibly not the best choice. you need training. It was developed and refined during the sequence of projects that the Euro Project Office supported. Thomas Fehlmann Version V1.
2 Unadjusted Function Points according IFPUG 4. 2003
Tools And Methods
The Project Database
1 The tracking software is Open Source and can be found in the /ch/open project methodology
(www. Such a delivery rate outperforms some of the shortcomings of VBA. As already explained. This included the need to let the project manager use his tool without any restriction. or www. The basic philosophy was usability. However. planned and actual effort and cost. resource allocation and usage. or about six times more than Java or C++.swisma. We make more than one FP per hour. and is not self-explanatory.ch-open. Euro Project Office AG developed a tool1 based on MS–Project and Excel to integrate the Six Steps To Completion – method into project planning.UKSMA Conference Wolverhampton Metrics for Project Management
Dr. quality records.
Visualization using Excel
The choice of Excel for visualization makes it possible to easily adapt more functionality and analyze the project for later improvements. but using VBA and the (loop-free) formula programmability of Excel allows for fast implementation at a high Function Points2 (FP) delivery rate. on top of the data already present in MS–Project such as planned and actual schedule. This led to the solution to use the database of MS–Project to store all relevant project data.DOC Copyright © 2003 Euro Project Office
Page 10 of 15
. it does not install without proper support. At least. 23 September. and you need a project office to run the tool. All the MS–Project functionality is thus available but with significant enhancements. the completion rates collected by the tracking process become in turn available again showing the actual completion percentage of the planned tasks.ch). quality planning.1
METRICS FOR PROJECT MANAGEMENT V10.0-13.ch.
03.2003 10.10.2002 10.07.2003 28.01.2003
90% 80% 70% 60% 50% 40% 30% 20% 10% 0%
on Re vie w pr ov al Ap Dr aft Id ea Re ad yF or Us e Fin ali za ti
09.10. Because it is based on object evidence.2003 25.2003
Ap pro v
Completion Rate 100%
06. and saves cost in the project.06.2003 09.2003 28.09.2003 20.DOC Copyright © 2003 Euro Project Office
Page 11 of 15
.10.2003 06.0-13. 2003
The available Excel reports include Management overview of overall progress Specifications for each deliverable in the project Progress Track including issues per deliverable Quality Plan per deliverable Quality Records and access to deliverables Pricing per deliverable Test stories per deliverable Summary test results Efforts planned and actual per deliverable and per resource Summary planned and actual efforts per deliverable Progress data per project phase Costs and earned value per project phase Costs and earned value and expected trend per accounting period Figure 2: Sample Project Management Overview
25. 23 September.11. Januar 2003
Fin ali za tio n
Fo rU se Re ad y
Re vie w
Management Overview Completion Rate Planned Completion
Actual Date / Planned Date 25.2003 17.05.UKSMA Conference Wolverhampton Metrics for Project Management
Dr.12.01.08.2002 13.05.03.2002 01.2002 21.2003 20.2002 29. Thomas Fehlmann Version V1. Thus it replaces administrative work by a service.2002 21.2002 13.08.2002 01.2003 17.06.07.
METRICS FOR PROJECT MANAGEMENT V10.2003
These reports substitute all other project status reports.12.10.11.09.2003 29. such reports can be generated by the project office and need but comments by the project manager.
17. Result due by 20. www.2002: M arketing departm task force to provide available data until 20.1. RM. ent
PM. Switzerland. the project home page or any other suitable communication tool (including paper). QM. including time reporting.0 7.1
Figure 3: Sample Progress Track with Issue History
30% delivered Result ID Deliverable
Issu R o g e ep rtin
(D ay s)
Resp.0-13. G approval im ediately after 2n reviewon 20.1 1. et m d 17.2002: Prototype lacks the priority setting featu re
08.5 1.8 1.3. 23 September. Currently we investigate migration to a DBMS to improve the GUI. TPL1 Arch. 31-Mrz Exp.1 2.7 1. or as an add-on to Bugzilla2.11
Opportunity Memo Risk Analysis Customer Value Analysis Quality Plan Project Plan Statement of Work Project Calculation Business Plan Contractor Qualification Select Contractor
93% 6-Jan 6-Jan 6-Jan
100% 100% 100% 100% 100% 93% 70% 62% 100% 100% 100%
1 4 5 2 3 4 3 3 5 3 3
3 6 4 13 43 46 58 -1 44 50 46 17.0 7.2002: W is responsible for gathering m ho arketing data? 11. Thomas Fehlmann Version V1.gmc.3
Analysis & Design Requirements Engineering
Clickable Prototype Business Process Analysis Use Case Analysis
Dev2. either via the Intranet.2002: S e m points n clarification by the subcontractor om inor eed S ubco w provide statem by 18 Consolidate w project plan ill ent .2003
In the past project experience.1.3 to get agreem on th open poin e .UKSMA Conference Wolverhampton Metrics for Project Management
Dr. but also with the reporting process of each team member.com 2 Contact GMC Software Technology.
Fi n ali za tio n Re vie w
Ap pro va l Re ad yF o
mp le Ra tion te
H istory R ecords
1. RM. it was possible to integrate with an ERP tool such as SAP.1.0 7.scodi4p. TPL2. Final Project O rganziation m be ust com unicated to custom m er.2002: Project C lation com alcu plete. Dev3.4. W conduct a 2nd reviewby 18. Arch.4 1. The tool is also available as an option to SCODi4P from Triloga1. 17.3 1.
1 Contact Triloga AG. AM QM
PM & Team
PM & Team
K 14-Jan 5-Mrz
Proposed Solution & Commercial PM. Lucerne. www.2 1.0 7.0 4. Arch.1. The tickets integrate all of the reporting needed in a project. ent e ts.2002: M issing feature added 12. Every team member gets his own personalized ticket.net
METRICS FOR PROJECT MANAGEMENT V10.9 1. AM Offer
2 2.2002: Too m y open poin an ts.3. Form approval asap after project calculation is al final. Arch PM. PM. 24-Mrz Arch Dev1.6 1.3. ith .0 7.2002: D becau of resource problem elay se s. S earch for available skills.0 7.
Data Capture with Personalized Tickets
The progress-tracking tool is not only integrated with the project repository.0 6.2003: 2nd reviewplanned for 11.10 1.DOC Copyright © 2003 Euro Project Office
Page 12 of 15
. or with configuration management such as CVS.2 2. Customer
70% 58% 76% 45% 52% 3 3 3 5-Mai 5-Mai
66 61 68
21. TPL1. 17.0 7.
Usability Test passed. Okt 01 Nov .1.0-13. we made excellent experience with a more traditional way for data capture. 23 September. which they glue to the wallpaper. Thomas Fehlmann Version V1. 2003
Figure 4: Sample Personalized Ticket
17. a wallsized print of the progress track sheet (similar to Figure 3: Sample Progress Track with Issue History) serves the same purpose.
5. 1 No v 0.1
Dev2. The team members report progress using colored markers.
24. It requires more physical presence of the project office that has to input the data captured on the wallpaper into the Excel sheet but minimizes disruption for the develop-
METRICS FOR PROJECT MANAGEMENT V10.0 Reviews 1
Actual Vers Tool
Test Case 0
Average Test Success
Average Test Completion
Grade 0 .3. Requirements Engineering Priority
10% 30% 15% 20% 15% 10% 100%
Re vie w Fin ali za tio n Ap pro va l Re ad yF or U Co se mp le Ra tion te
2. 1 No v 5. Instead of distributing and collecting electronic tickets.5
07.DOC Copyright © 2003 Euro Project Office
Page 13 of 15
Quality Measurements Pilot Functionality
Release Identification Delivery item identification Version Under Test
Cycle 1 V02
Prototype Solution 1 3.3
30.UKSMA Conference Wolverhampton Metrics for Project Management
Dr. Found that requirements need to be refined for various user categories.
72 82 Working on this task
Arch Dev2 Dev3
2 32 48
1 5 2
0 7 0
3Hours 44 50 8 h/d 0 5 d/w 0 4 w/m
Data Capture in the War Room
Although electronic communication means are abundant and sometimes indispensable.
Date of Approval
Work Reported (in Hours)
Ba se lin e Bu W o r dg et k Ac tua Pe l Wo rfo r rm k ed Pe rfo Th rme is Pe d rio Es d tim Co ate T mp o let Ex e pe cte Co mp d At let i on Wo rk Ov e rr un
Rapid prototype displaying the functionality needed. Dev3.
UKSMA Conference Wolverhampton Metrics for Project Management
ers and maximizes their sense of responsiveness and ownership.0-13. Figure 5: Wall Sized Print of the Progress Track as Paper Tool
Automated Data Capture using a Project Repository
If a project repository is available.montana (www. The draft deliverables. errors in estimation are detected in the very early stages of the project. Team members still have to update time and issue reporting on the tickets and may want to set the draft or the review to some intermediary progress grade (yellow marker). data capture is even more automated.
1 For a project repository that integrates well see itp. This effect could be worth considering as a trade off. Thomas Fehlmann Version V1.DOC Copyright © 2003 Euro Project Office
Page 14 of 15
.itp-solutions. quality records and approvals can be tracked in the project repository and progress track relies less on people’s judgement. However they can no longer pretend be in finalization state without at least a preliminary review report being available in the project repository1. Using the Six Steps To Completion method. Delivery rates become apparent from the beginning.
Relation to Other Software Metrics
Bad effort estimation cannot be completely avoided. 23 September. in German)
METRICS FOR PROJECT MANAGEMENT V10.
Thus corrective actions can be taken immediately when needed. The metric establishes a short – circuit between plan and team. Sometimes it is possible to group development tasks such that they fit with logical boundaries. If planning and estimation was correct.UKSMA Conference Wolverhampton Metrics for Project Management
Dr. records test results and review finding. the effects are tremendous. sometimes two weeks as a reporting period. Progress metrics are the door opener for other metrics dearly needed for today’s complex tasks. The work breakdown structure of a software project does not automatically coincide with the software boundaries. 2003
When the team starts understanding that the progress metrics are not a threat but a valuable help for them. In the beginning. 23 September. many more sophisticated metrics such as software sizing using Function Points or bug statistics provide higher value when combined with progress metrics. The project manager perceives unrealistic assumptions and estimations within a single reporting period. Thus tracking project tasks and bug fixes becomes the same process. Moreover. and almost no fear against being measured. This is why we us no metrics without progress metrics in our software projects. the schedule will be respected. their reluctance against using metrics vanishes. Clearly we would like to add a software sizing metric to the tool.0-13. Metrics provide huge benefits for people and organizations. It provides progress metrics and bug statistics at once.
The Six Steps To Completion – metric is easy to use. such an approach is very promising and deserves further investigation. Thomas Fehlmann Version V1. it is often not even perceived as a metric. We most often use one week. with their specifications.
METRICS FOR PROJECT MANAGEMENT V10.DOC Copyright © 2003 Euro Project Office
Page 15 of 15
. On the other hand. with no pain. simple and straightforward. Nevertheless. However this yields some specific problems. test cases. Developers have only one tool that supplies them with new tasks. The GMC team in Hradec Králové (Czech Republic) integrated the method in the open source tool Bugzilla. it stirs only little opposition. and tracks effort and progress. but it make not always sense.