HM

TREASURY

Central Unit on Procurement

No. 43 Project Evaluation

This is one of a series of guidances prepared by CUP on the management of works projects. Its use is not mandatory but a statement of good professional practice. 1. INTRODUCTION 1.6 Many of the project controls (eg cost, time, quality and change), which must be in place on any well managed project, are subject to evaluation. These are set out at Annex A together with notes on some of the principal issues to be addressed. 1.7 Project evaluation is part of the total quality process. It contributes towards a successful outcome in terms of: - greater assurance of total performance in terms of cost, time and quality; - clearer definitions of responsibilities; - reduced exposure to risk; and - improved VFM and accountability. 2. SCOPE

1.1 Major projects are often complex and risky. An enormous number of decisions have to be made and actions taken during a projects life: often under tight time and budgetary constraints. Project evaluation enables participants to stand back from the detail of the project, review what has occurred and to take a forward look at the remainder of the project. 1.2 Project evaluation is a team effort. It enables individual members to consider aspects of the project from different points of view, identify possible conflicts or poor communication and improve working relationships benefiting the overall project. 1.3 Project sponsors and managers are also responsible for feedback from lessons learned, both good and bad, so where appropriate they can be used to benefit future projects. 1.4 Project evaluation techniques are an essential aid to improving project performance and achieving best value for money (VFM). This guidance sets out the principles of project evaluation explaining how, why and when they should be done. In carrying out such evaluations, project sponsors will need assistance from one or more of the following: - professional adviser; - project manager; and/or - other consultants. The extent to which they will be involved depends on the stage the project has reached and the scope of the evaluation. 1.5 This guidance assumes knowledge of guidance no 33 Project Sponsorship which deals with both the responsibilities and the activities of the sponsor and the professional team (see Annex 3 of guidance 33). These activities are also subject to evaluation techniques. Section 3 of guidance No 33 describes a number of key project tasks which form the core topics of project evaluation Core topics are by definition relevant at all stages in the project life cycle.

2.1 Although this guidance deals with evaluation of majorworksprojects in general terms, experience shows that it also applies to smaller projects. However, given the varied nature of projects, this guidance cannot be comprehensive. The principles of project evaluation can also be applied to projects other than works. 2.2 The guidance is in 9 sections with supporting Annexes. Section 1. Introduction 2. Scope 3. Definition of terms 4. General principles 5. Pre-project evaluation 6. In-project evaluation 7. Post-project evaluation 8. Occupancy review 9. Independent reviews A full list of contents is at Annex E. 3. DEFINITION OF TERMS

3.1 Project evaluation is a technique to review the current status of a project against plan and to provide practical, comprehensive and forward-looking recommendations for corrective action where necessary. In general terms it should consider: - project objectives in terms of cost, time and quality; - management;

- organisation; - systems and procedures; - suitability of contracts; - performance of consultants and contractors; and - work to date in terms of cost, time and quality meas-

appropriate to influence future projects. The aim is to use data which is of benefit to non-specialists so that it can be used to influence future decision making. This will help to ensure that past mistakes are not repeated, and that good practice is properly understood. 4. GENERAL PRINCIPLES 4.1 This guidance provides a summary of activities, procedures and controls which should be in place at each of the following stages of a well managed project: - pre-project stage covering appraisal and feasibility study; - in-project stage covering outline design to construction completion and commissioning; and - post-project stagecoveringrectificationofdefects, final payments, post-project evaluation and occupancy review. 4.2 Evaluations should: - assess the effectiveness of the project organisation; - review the project execution plan; - measure progress against planned cost, time, quality and performance objectives; - identify causes of deviation from plan; and

ured against plan.

3.2 Evaluation is often carried out by project participants (project manager, design and cost consultants with contributions from contractors and sub-contractors) and should be a routine performed at predetermined stages to establish that objectives are still achievable. Ad hoc evaluations may also be used as a means of focusing on specific problems. The starting point for ad hoc evaluations often arises from specific concerns of the project sponsor. 3.3 A difficulty with evaluation by project participants is that they often find it difficult to be objective about their own decisions. An evaluation or review by an independent consultant overcomes this problem and can bring in new ideas and useful outside experience. 3.4 Project evaluationshouldnot be confusedwith value engineering or economic appraisal:

3.5 When an investment decision is made, a project is born - though it may not necessarily pass later stages of approval and reach completion. This decision is the first stage of the Treasurys project approval procedure (see guidance no 38 Approval of Works Projects), ie the initial approval of the options and the assessment of the business case. The second stage is identifying the preferred option, establishing a preliminary budget and obtaining approval to take forward planning and design work on this option. 3.6 Project evaluation can be used from the second approval stage (by evaluating the chosen option against the objectives in the appraisal) throughout the projects life, ending with the post-project evaluation and occupancy review. It is important to note that the term evaluation in the Green Book is concerned with postproject evaluation only. 3.1 Post-project evaluation is a means of recording experience from past projects, so it can be used where

Value engineering is a pre-planned, formal review of the design philosophy and development, at one or - recommend remedial action. more stages of design development, sometimes led by an independent expert. These studies are short 4.3 Project evaluation is used to check that: exercises intended to review the delated design solutions against the objectives and to establish whether - methods for controlling the cost, time, quality and performance characteristics of a project, from the they can be achieved in a more cost effective manner. initial planning to post completion assessment, are Economic appraisal assesses the costs and benefits sound and effectively applied; of expenditure proposals. It is the process of defining - management information is prompt, adequate and objectives, examining options and weighing up the accurate; costs and benefits before a decision is made. The main source of guidance on appraisal is the Treasurys Green - assets and interests are properly controlled and Book (Economic Appraisal in Central Government: safeguarded from all losses; A Technical Guide for Government Departments - the cost of rechargeable works is recovered; HMSO 1991). It contains a checklist to ensure good appraisal techniques and notes some common errors. - there are adequate safeguards against fraud, error and impropriety; - material losses due to waste or inefficiency are identified and recovered where appropriate; - information on which costs and cash flow projections are based provide a sound basis for the funding of the project; and - internal guidance and other established policies and procedures are appropriate and are followed. 4.4 Project evaluation is independent of any established reporting requirements. It can be used to encourage the project team to stand back and review what it is doing and where it is going in relation to the stated project objectives. It should end with a formal written report presented to senior management by the sponsor based on data and reports from the project manager. 4.5 A disciplined and systematic approach to evaluation is recommended so that it becomes a routine project activity, heading off problems before they occur. 2

The project cycle 4.6 Every project can be visualised as a series of steps requiring a variety of technical skills to manage inherent risks. Initially objectives must be defined and options for achieving them generated and evaluated before deciding which option should be implemented. Project evaluation techniques provide the sponsor with an additional means for controlling and monitoring the progress of a project. 4 .7 The steps in the project cycle and the stages when a routine project evaluation should happen are at Figure 1.

Evaluations, project stages and approvals 4.8 Although projects and their procurement strategies vary, there is a pattern of definite stages common to most projects. Some of these stages overlap depending on the type of project and the method of procurement. Project evaluation is very effective when carried out at the end of the project stages which are immediately prior to the next approval stage. The link between project stages and approval stages is illustrated in the following table.

Figure 1: The Project Cycle

Definition

Generation of options

Evaluation of short-listed options

SELECTING CONTRACT

t
Monitoring and control

Decisions - Choice of options -Whether to implement

Monitor and con - Responsiblties Whether to implement

Timing of evaluations Project Stage there is an identification stage, when the Approval Stage

need is apparent, the benefits are measurable, the business case can be established, initial studies are carried out and options are appraised; the preferred option is taken forward and subjected to further feasibility studies (the preferred option may have established a location and even a site for the building) feasibility studies will examine the ways the building might fit the site, the environmental impact, planning and statutory requirements and estimated cost and funding;

Business case/option appraisal.

Submission for approval to take forward planning and design work.

planning and design work is taken forward through outline (concept) design and into scheme design; the project execution plan is further developed;

Approval Estimate leading to submission for final approval as soon as cost estimates of the required accuracy can be prepared (not linked to any specific design stage but usually around final sketch plan stage).

or overlapping depending on the contract strategy;

detail design, procurement and construction which may be sequential

Submission for re-approval if at any time forecast outturn cost is greater than approval and/or conditions established at final approval are likely to be broken. Occupancy review.

commissioning and occupation.

Timing

4.9 At intermediate stages, the natural points to consider evaluation are again at the end of one stage and prior to the commencement of the next, eg at the end of scheme design but before detail design or prior to the submission for final approval. 4.10 During construction, evaluations are recommended at the mid point of the construction programme or once a year for longer programmes. More frequent evaluations should be considered for particularly complex projects. Ad hoc evaluations to identify and solve specific problems may be called for at any time throughout the project.

greatest in the early stages decreasing rapidly as the project progresses as illustrated in Figure 2 below. However, uncertainties about the outcome of the project are also greater. Project managers should use risk analysis techniques to help evaluations and advise the sponsor to make plans and choose strategies which reduce risk, and where appropriate place remaining risks where they can be managed most effectively. Risks should be reflected in the level of contingency allowance so that outturn cost and time can be contained within forecasts. See guidance no 4 1 Managing Risk and Contingency for Works Projects.

4.11 The ability to control the outcome of a project is

Figure 2: Degree of Project Control

High

and systematic manner. Other participants may include members of the sponsors organisation, the professional adviser, the design team, the construction team and the intended users of the building. 4.18 If project sponsors suspect that project managers have not been fulfilling their responsibilities, they should consider either an evaluation by the professional adviser or an independent review. The professional adviser can provide guidance on this.

Low

Reporting 4.19 It is important that the evaluation should be documented clearly by all concerned so that if the need for a further review is identified the team can follow the process of planning, work, findings, conclusions and recommendations.

Topics for Evaluation

4.12 Annex A lists some of the topics to be evaluated at various stages of the project and, for each topic, several key issues to which the evaluation team should provide answers. Controls and procedures often have to envolve as the project progresses. The evaluation should check that a control mechanisms are: - established, appropriate and being used;

4.20 Each participant in the evaluation team should keep working files to support individual evaluation reports, recording the details of each stage from evaluation planning through to reporting. 4.2 1 The project manager will provide the sponsor with a written report, based on contributions from all the participants (but checked by the project manager as necessary) highlighting the key areas, This report should address in particular any aspects of control systems and procedures which were found wanting. The implications of these deficiencies should be explained together with recommendations for improvements. 4.22 The project sponsor reports the results of the evaluation to the project director on the basis of data and reports received from the project manager. The sponsors evaluation report should comprise:
- a short (one to two page) executive summary listing

- reviewed and amended to meet changing project needs; and - complied with. 4.13 Where existing controls are considered unsatisfactory, evaluations must provide the sponsor with recommendations for improvement. An independent review may be necessary if the evaluation either cannot establish the problems adequately or reveals a particularly serious problem for which the underlying cause or corrective action cannot be identified quickly. This is particularly applicable if poor performance, fraud or some other impropriety is suspected.
Duration

the reasons for the report, the principal conclusions and the actions to be taken; and identify any further action required.
PRE-PROJECT EVALUATION

the sponsor that the project is on course to meet its planned objectives or, if not, that the cause of any problem has been identified and proposals made for its resolution. 4.15 Typically evaluations should last one to two weeks, depending on the stage the project has reached and its size, importance and complexity. This is particularly important when the evaluation is carried out to deal with a perceived problem, so that corrective action can be taken quickly and the effects of the problem mitigated. 4.16 Evaluation of a specific problem will be achieved best by limiting its scope. For example, an evaluation to identity and reduce the causes of a cost overrun, may be limited to all changes made or planned. The review may be further limited where the problem is more specific, eg whether the cost effects of a particular design change have been fully assessed.
Management 4.17 The project sponsor is responsible for ensuring that evaluations take place at intervals recommended in this guidance. The project manager, when appointed, is responsible for managing the evaluations in a disciplined 5

4.14 Evaluations should be sufficiently detailed to satisfy

- sufficient detail on the findings of the evaluation to

5.

5.1 At this stage, prior to commencing design and construction, the objectives of the project sponsors department and the project itself are established. Project objectives should be defied clearly and prioritised in terms of time, cost, quality and performance. A preproject evaluation should be carried out to confirm that this is so by investigating: - the clarity of the project objectives; - the viability of achieving those objectives; - whether sufficient options for achieving the objectives have been generated; - that these options have been properly evaluated; - the degrees of influence in the client organisation; - that the main risks and constraints to the project have been identified and assessed; and - that the project has appropriate funding.

The purpose of a pre-project evaluation is to provide a checkthat these matters are being or have been resolved satisfactory. 5.2 If the evaluation reveals serious flaws in the project or its proposed implementation, it may be necessary to delay moving to the design stage until further investigation by a more detailed evaluation or independent review is complete and the implications have been assessed. Timing 5.3 The best time for a pre-project evaluation is shortly after appointment of the project manager, and before appointing the permanent design team. It may be necessary to seek specialist advice, for example on risk identification and analysis, during the course of the evaluation. In this event temporary appointments may have to be considered. Management 5.4 The project sponsor is responsible for ensuring that the pre-project evaluation takes place. The project manager should manage the evaluation. Contents 5.5 Annex A indicates some of the topics to be reviewed. The evaluation should establish that the necessary systems and controls exist to proceed to design stage or identify the means of achieving this. The project objectives, preliminary contract strategy and the project organisation should be identified and procedures should exist for the appointment of consultants, and for design and cost control to the pre-tender stage. Many of the control documents at this stage are the initial versions for the project and will be amended by the professional team as the project progresses. However, they should be appropriate for this stage. Reporting 5.6 The project managers evaluation report should state clearly whether the project should proceed as currently proposed and if not, the actions necessary for the project to be successfully continued. The report must also state whether the evaluation has been adequate for its intended purpose or whether a more detailed evaluation is recommended for all or part of the project. 5.1 An indication of the issues to be covered in the project sponsorss evaluation summary is at Annex B. However, the summary will need to be tailored to suit the particular project. Benefits 5.8 Evaluation plays a valuable role at this stage in ensuring that the department has defined the project objectives clearly and that the chosen preliminary contract strategy is consistent with those objectives prior to entering contractual relationships and commitment of funds.

5.9 Most crucial decisions are taken prior to commencing design (see paragraph 4.11). The evaluation should identify where action needs to be taken. Changes at this stage are rarely as costly or as disruptive to the project as those implemented later. 5.10 When the pre-project evaluation confirms that the project objectives are achievable and that the necessary controls are in place, the project can proceed to the design stages. 6. IN-PROJECT EVALUATION

6.1 During the design stages of a major project, interaction between the various parties is intense and complex. Oversights and mistakes can easily occur. As the project design progresses the opportunity for introducing significant changes (without incurring costs for abortive design work) reduces rapidly. A preplanned evaluation should be carried out at the pre-tender stage to establish that all is well with the project before entering into the most significant financial commitment (ie construction of the project). 6.2 Duringthe constructionphase, the project sponsors department incurs the major part of the project expenditure. Changes introduced at this stage are likely to incur disproportionate cost increases and result in delay to the project. 6.3 As projects can be subject to significant risks during both the design and the construction phases, in-project evaluations provide the project team with the opportunity to stand back from the detail of the project and:
- evaluate how well they are working together; - assess the status of the project against cost, time and

quality objectives;

- identify existing or potential problems; - make forward-looking, practical recommendations

for improvements which can be implemented immediately; aged; and

- check that the risks to the project are being man- ensure a commitment of funds to the project.

6.4 It is good practice to carry out in-project evaluations during:

- design: the project manager should advise the

sponsor on the timing, but typically this would be prior to commencing detailed design and/or prior to tender and construction; and

- construction: the project manager should advise the sponsor on the frequency, which might typically be at 6-9 month intervals. Management 6.5 The project manager, reporting to the sponsor, is responsible for managing the evaluation, the participants and for providing the report.

6.13 The advantage of conducting an evaluation at the outline and/or scheme design stages is that changes are 6.6 Annex A summarises some of the topics whichshould relatively easy and economic to introduce. This is not the be evaluated during the design and construction stages. case once detailed design has taken place. The evaluation process should check that: 6.14 Evaluation at mid-construction provides the final - control procedures for the core topics are appropricheck that the project is on course. Whilst significant ate to meet current and future project needs and are changes to project scope at this stage will be costly to being complied with; and implement, changes in control procedures are not and - new controlmechanisms specific to in-project phases may be necessary if forecast time, cost or quality objecexist, are appropriate and are being complied with. tives are not being met. Contents Reporting 6.7 Evaluations should be summarised and presented to the sponsor in a report prepared by the project manager based on contributions from project participants. The report should: - state clearly whether the project remains on course to meet its plan - consider risks and how they are being managed; - provide fully evaluated recommendations to deal with any existing or potential problems; and - improve the effectiveness of project execution. 6.8 Design stage evaluations should recommend whether to proceed to detailed design and/or construction They must state whether the evaluation is sufficient or whether a more detailed independent review is required. Duration 6.9 Outline design stage and scheme design stage evaluations should be completed normally within 2 weeks. If serious problems are revealed, moving to the next design stage should be delayed until a further detailed evaluation has been undertaken, the implications assessed and remedial measures taken. 6.10 Construction stage evaluations should proceed in parallel with construction and normally be completed within 2 weeks. 6.11 Delaying the construction process is likely to prove very costly and should only be considered if: - the evaluation reveals serious problems which significantly affect the achievement of project objectives; and - these problems will worsen significantly if construction is allowed to continue before taking remedial action. In these cases a more detailed evaluation or independent review may be necessary to provide recommendations for corrective action. Benefits 6.12 Evaluations during in-project stages should identify the causes of any problems and provide recommendations for remedial action. The sooner this is done, the more likely the project will be completed successfully. It serves to limit the risk to which the project sponsors department is exposed. 7
7.

POST-PROJECT EVALUATION

7.1 This should be carried out soon after completion of construction to:
- measure the success of the project in achieving its

planned objectives; arisen: and

- identify the reasons for any problems that have - determine what remedial actions should be taken: - record the lessons which have been learnt to

improve performance on subsequent projects.

The objective is to learn lessons for application to future projects and to pass them onto other departmentalprojects or departments. The post-project evaluation report for a building project may be supplemented by an occupancy review (see Section 8). 7.2 There is a minimum level of post-project evaluation which should be carried out on all major projects. The aim ofpost-project evaluationis to improve project appraisal, design, management and implementation, Its objective is to get maximum benefit from accrued experience, not to apportion blame. Timing 7.3 Time is not usually critical at this stage. However, the earlier the post-project evaluation is done, the earlier the lessons are available for subsequent projects. In particular, the evaluation should be completed sufficiently early to ensure that: - knowledge is still live; - those involvedwiththeproject are still available; and - outstanding obligations are identified and steps taken to ensure they are fulfilled. Management 7.4 The project manager is usually responsible for directing the post-project evaluation. The evaluation requires the active participation of all parties to the project as part of their final duties and this should be set out in their terms of appointment. Contents 7.5 The evaluation should assess whether the objectives of the project were realised and the established controls complied with. The effectiveness of these controls should

also be reviewed and recommendations should be made for improvement of controls on future projects.
Reporting 7.6 Post-project evaluationshouldconcludewithareport prepared by the project manager using contributions fromtheprojectparticipants. It shouldstateclearlywhether the project met its objectives in terms of time, cost, quality and performance. Recommendations for reducing the possibilityofproblemsoccurringonfutureprojectsshould be made. It should also highlight aspects of project management which have worked particularly well or not and state that the evaluation was sufficient for its intended purpose or whether a more detailed evaluation is required.

anticipated to change duringthe course oftheproject, to assess whether those changes were accommodated: - confirm that the building functions in use as planned; - ensure that any outstanding work, including defects, have been remedied; and - ensure that key lessons learnt on occupation are put into practice without delay for the benefit of future and continuing major projects,
Management 8.3 The occupying business manager responsible for the property should arrange for the occupancy review to be conducted. Assistance will be required from the senior users and preferably also the project manager. Contents 8.4 An occupancy review should include:

7.7 The sponsor adds a report on the project managers performance. The sponsor submits the final post-project evaluation report to the project director who uses it to check whether the departments objectives and requirements have been met. 7.8 Typical issues to be addressed in the report are at Annex B but the report should be tailored to particular projects and evaluations. 7.9 Departments who use the Contractor Management Information System (CMIS) and the Consultants Register (CONREG) operated by Construction Industry Sponsorship (CIS), DOE, 2 Marsham Street, London SW1P 3EB, are required to submit performance reports on contractors and consultants as a matter of course.
Benefits 7.10 A post-project evaluation provides an opportunity to review the project with the benefit of hindsight. It reviews all project stages and the problems that have arisen thereby providing lessons for the future and identifying how problems arose. It is also valuable to conduct postproject evaluations on major projects that have gone particularly well, in order to identify ingredients for success.

- an assessment of whether the users requirements have been met. If not, consideration should be given to the briefing process and the extent to which users were consulted and involved as the design developed; - costs in use - how the planned operating cost of the building compares with actual cost and how the planned efficiency and effectiveness of the equipment and facilities compare with actuals; and - any further recommendations the users can make to improve VFM performance of future projects. For example, the users may consider that some of the facilities provided were unnecessary.
Reporting

review report are at Annex C. The report should be tailored to suit the individual project.
Benefits

8.5 Typical issues to be addressed in the occupancy

7.11 The evaluation can also provide the sponsor with assurance that the works have been completed in accordance with the contracts, that consultants and contractors have discharged their responsibilities and that all potential liabilities have been recognised and quantified. It will expose any shortfall in these areas and enable the sponsor to recognise where commitments still exist and/ or damages might be recoverable.

8.6 The occupancyreviewis the finalfeedbacklinkfrom the users which closes the loop of the project cycle. The review also identifies the existence of any outstanding obligations, with which the project sponsor must deal.

9.

INDEPENDENT REVIEWS

a.

OCCUPANCY REVIEW

8.1 When the project relates to a building, an occupancy review should be performed one to two years after the building has been occupied. The result of this review is an addendum to the post-project evaluation. It records how successfully the completed project meets the requirements of the occupiers and identifies any further work required.

9.1 Project sponsors should commission independent detailed reviews if they are concerned about either the general status and progress of the project or, more usually, about specific issues. 9.2 Its aims should not be limited to identifying the root causes of any problem but should also make practical recommendations for corrective action and aim to generate direct and significant time, cost and quality improvements.
When an independent review is necessary 9.3 The need for an independent review may arise at 8

8.2 An occupancy review has the following objectives to: - consider whether the objectives of the users have been met. If those objectives changed, or were

any time when the project sponsor considers that certain problems are serious enough to require separate investigation However, the project manager should also advise the project sponsor when one is necessary. There are a number of typical warning signs: - cost issues: - large increases between successive estimates; - tenders for construction being significantly above or below estimate; - a significant overspend or underspend compared with predicted cash flow; - unpredicted changes resulting in high additional costs; - large contractual claims for additional payments or extensions of time; and - a final contract sum which is much higher than the tender sum; -time issues: - progress falling significantly behind programme eg failure to meet milestone dates; - delays in providing the contractor with necessary information, which results in delays and/or additional construction costs; and - late completion; - quality and performance issues: - a pattern of failings in design or workmanship; and - a finished product which fails to meet the users requirements in certain important respects; - management/organisation issues: - delays, cost increases or reduced quality standards arising from failure to provide the right people with the necessary information to make decisions or take actions at the right time; and - lack of clarity of roles and responsibilities, Procedures 9.5 Reviews will be capable of identifying problems and providing recommendations for their resolution. The reviewers approach should be objective. Subjective judgements, for example relating to truth or fairness, should be expressed specifically as an opinion as distinct from a finding. 9.6 The project sponsor should make reviewers aware of the circumstances which make the project unique. Although reviews may cover the same topics as evaluations (see Annex A) they are often in greater depth and therefore detailedprocedures will need to be established individually for each review. Review of project management controls 9.1 Project management controls can be audited as follows:
9

- systems evaluation;

- compliance testing; and - tests of detail. The flowchart at Figure 3 (page 10) indicates this process. Systems evaluation 9.8 This comments on the adequacy of control system and offers recommendations for improvement. Th approach is broadly based rather than detailed and i conducted in two stages. First, the system is recorded in terms of its operation, preferably in flowchart forn Second, the system is evaluated by defining its control objectives and assessing the strengths and weaknesses of the system in meeting those objectives.
9.9

A typical evaluation might cover the following:

- is the procedure clearly defined? - is someone nominated to carry out the procedure - is this person fully aware of the responsibility fc

carrying out the procedure?

- is the person required to demonstrate that th

procedure has been followed?

- are there any further procedures providing for a

independent witness or check that the procedure has been carried out? and

- does the procedure provide the intended control

Compliance testing 9.10 This is used to check that the system is operating a intended. It focuses on the key controls of the system an the principal management actions which must be carried out if the system is to operate as intended. The tests are i whatever form is appropriate to determine whether the key controls are operating in practice. Examples of these tests include:
- observation of the procedure in operation; - examination of working papers and formal docu-

mentation; and

- interviews with project participants to determine th

actual procedures in use.

9.11 If systems evaluation has proved that the system is itself inherently weak, compliance testing is usually inappropriate. Tests of detail 9.12 These may be necessary when systems evaluatio and/or compliance testing have revealed weaknesse and deficiencies. There are two types of tests of detail: - those concerned with financial values; and - those concerned with the quality of managemer decisions. 9.13 Financial tests of detail either start at:

Figure 3: Audit Flowchart

Select system to be audited I

Does system exist?

) 7

No I

1
Systems evaluation

Gather data on how system operates

I I

I I

Flowchart the system

Define what the system is intended to achieve

t
Evaluate strengths and weakmesses of system

Report to management

-_-----

1 1 Debekey contr;kthesystem

I I I I I I

Devise sampling procedures and compliance tests

~~ I I I I
I

Compliance testing

I I
I I I

Reporttort

- - - -

I
I I I I I I I I I I I I I
-I

Tests of detail

Report to management

+--_---_____----------

10

- a figure in an account - then trace the data or

evidence validating that figure (eg expenses or costs); or

the data or evidence - then check that the correct figure arising from that information is included in the account (eg income or deductions from expenses).

9.14 During the course of a project, some management decisions fall outside the scope of systems and control. These decisions may have to deal with unforeseen circumstances eg poor ground conditions, bad weather or poor interaction between the parties to the project. Objective tests of detail consider the quality of these decisions by: - setting down clearly the relevant history and facts relating to the decision;
- considering all options available to management;

facts during their investigation, They should have relevant experience and a successful record of conducting construction project reviews. Suitable candidates can be found in construction and property management consultants, project and cost management consultants and consulting engineers. The professional adviser and/or project manager can advise the project sponsor on selection. 9.20 Details of the form of the review, the scope of the work and the final report should be agreed between the project sponsor and the review team before work starts. 9.21 The reviewers will need the co-operation of the project participants in gathering information. However, when the project sponsor has commissioned a review rather than an evaluation of the project, participants may feel threatened and may be reluctant or unwilling to cooperate. This may reduce the effectiveness of the review. The obligation to co-operate must therefore be included in the terms and conditions of engagement of the project participants. When fraud or negligence are being investigated, it may be necessary for the sponsor to remind participants of their contractual obligations on co-operation.
Contents

- analysing how well each option relates to the achievement of the project objectives; and
- recording the reasons for selecting a particular

option.

9.15 Good decision makers should have followed the process described above. However, it is stillnecessary to check that the process has been completed correctly and that the associated working papers are on file.
Scope of review

9.16 The scope of an independent review will depend on the nature of the problem under consideration, its associated risk and why it is being carried out. If the objective is simply to assess the effectiveness of systems and controls and to provide recommendations for improvement, the review may be limited to systems evaluation and compliance testing. If a judgement on a specific issue is required, compliance and detailed testing will be necessary. If fraud is suspected, all three levels of review are likely to be necessary to reveal the full extent of the problem. 9.17 However, the reviews size may be limited by sampling techniques. Judgmental sampling relies on the reviewers judgement to pick out the major or most contentious items for investigation. Whilst this method is quick and cost-effective, the project sponsor should recognise that oversights may occur. 9.18 Statistical sampling is likely to be more accurate but more time-consuming. The reviewer selects the size and type of sample based on an assessment of risk and importance. The reviewer will advise the sponsor on how and why the sample was chosen and the confidence that can be placed in the results.
Management 9.19 The project sponsor is responsible for commissioning independent reviews and appointing the review team to undertake the investigation. The team should be independent, without any conflict of interest, unbiased and have no reason to protect any individual or conceal any

9.22 Annex A summarises some of the topics which may be included at different stages of the project. The project sponsor should define the scope of the review and prepare a brief for the review team. - 9.23 The level of detail of the review depends on: - the project sponsors brief to the review team; - the project stage; - the matter under investigation: and - discoveries made during the review process. The reviewers should investigate whether the controls are clearly defined and are being satisfied.
November 1993

11

ANNEX A OUTLINE CHECK LIST OF TOPICS FOR EVALUATION

This Annex lists some of the topics (with their key issues and problem areas) which may be reviewed during project evaluation. Where appropriate, evaluation teams should both examine and provide answers to each. However, the list is not exhaustive and will need to be reviewed depending on the projects stage and its contract strategy. In addition, the list may also need amendment to reflect the scope and requirements of the particular project. The project sponsor, on the advice of the project manager and/or professional adviser, should define the scope of partial or specific evaluations. Controls andprocedures evolve as theprojectprogresses and evaluations should establish that they are: - established and appropriate to the stage of the project; - being reviewed and amended to suit the changing needs of the project; and - complied with. Where procedures and controls are considered unsatisfactory evaluations must provide recommendations for improvement. The suggested topics for evaluation are as follows: Project Definition user requirements project objectives project brief planning, social and environmental impact assessments Project organisation and management departments client organisation project management team project organisation structure risk assessment approved budget cost, time, quality and change control procedures contract strategy project execution plan procurement procedures detail design reporting systems contractual claims Availability of funds Statutory and other External Factors planning approval building regulation approval health and safety regulations site access inflation interest rates currency exchange rates VAT

Force Majeure catastrophic events Design development designers design team responsibilities design brief design process design programme design elements the design Estimating Data cost estimate for professional services (resources) works cost estimate other cost estimate project programme Site Conditions the site conditions above and below ground susceptibilities - flooding, bad weather labour availability existing services existing buildings environmental impact assessment Construction and Commissioning contractors site management responsibilities health and safety measures emergency procedures permits to work construction methods long delivery, short supply items commissioning methods handover procedures operating manuals as built drawings

12

ANNEX A - continued PROJECT DEFINITION User requirements Adequately defined? Incomplete or unclear? Conflicts between different user requirements? Likely to change? Properly defied? Incomplete or unclear? Conflicts between primary/ secondary objectives? Possibility of change? Complete? Inadequate or unclear? Conflicts between different requirements? Possibility of change? Are the following unclear, in conflict or likely to change: - site; - scope: - extent; - quality and performance: - design concept; - cost and time; or - completion date? Sufficiently output orientated to provide design flexibility? Carried out? Issues of concern identified? External authorities consulted? Their requirements met? Outstanding matters? Project Organisation and Management -continued Project management team Project manager pro-active? Continuity in project management Project managers links with participants effective? Project manager administers contracts efficiently? Queries dealt with promptly? Clearly defined? Clear direction and communication? Project team responsibilities clear? Gaps or overlaps? Arbitrary external interference in project team? Clear communication and identified responsibility with interested organisations not directly involved? Carried out at important stages? Used to determine contingency, tolerance and approved budget? Control budget within approved budget? Conditions attached to authorisation to proceed? Effective and working? Control budget regularly reviewed? Costs monitored and exceptions acted upon? Time control effective and working? Project programme regularly reviewed? Progress monitored and exceptions acted upon? Quality control effective and working, exceptions acted upon? Change control effective and working? Sponsors approval to changes?

continued

Project objectives

Project organisation structure

Project brief

Risk assessment

Planning, social and environmental impact assessments

Cost control procedures

PROJECT ORGANISATION AND MANAGEMENT Departments client organisation Clearly defined? Clear direction and communication? Sponsor and client team responsibilities clearly defined? Appropriate? Gaps or overlaps? Arbitrary external interference in client team? Sponsor administers contract efficiently? Notes and decisions issued promptly? Correct balance of resources and expertise? Effective? Responsibilities clear? Appropriate? Project manager exercises adequate authority?

Contract strategy

Takes account of project

Project management team

objectives? Project execution plan current? Defines the work packaging of design, procurement, construction and commissioning? Defines how activities are coordinated and managed? Effective and working? Comply with EC/GATT rules? Tender documents adequate? Tender periods adequate? Information reliable?

Procedures for procurement

13

Project Organisation and Management -continued Procedures for procurement

STATUTORY AND OTHER EXTERNAL FACTORS Planning approval, building approval, licences and permits

continued

Tender documents protect department from low tenders with potential for claims? Tender evaluation appropriate? Approval and award of contracts effective and working? Contracts clearly define scope of work and responsibility of contracting party? Any nominated sub-contractors? Departments responsibilities recognised and taken into account? Complete before construction Design requirements properly coordinated? Level of design detail appropriate and adequate? Likely to change after start of construction? Design information complete and timely? Queries and conflicts resolved quickly? Effective and working? Include narratives on current status of cost, time, change and quality? Shortfalls in meeting objectives highlighted? Corrective action identified and initiated? Identify actual and anticipated critical areas? Essential information provided in a coherent format? Information used to prompt action and take decisions? Likely to arise? Performance likely to prompt claims? Adequate records maintained to counter claims? Validity and potential exposure to actual claims independently audited?
starts?

Detailed design

Possibility of delay in obtaining? Project compliant with building and health and safety regulations? Necessary statutory approvals obtained? Access to site subject to approval of external organisations or authorities? Bureaucratic procedures likely to cause delay? Anticipated changes in regulations or legislation affect project? Inflation taken into account? Affected by interest rates? - taken into account? Affected by rates of exchange? - taken into account? VAT taken into account?

Inflation, interest rates, rates of exchange and VAT

FORCE MAJEURE Catastrophic events

Reporting systems

Contractual claims

Departments investment protected from extreme weather, flood, landslips, earth quakes, fire, crashing aircraft or other catastrophic events? Adequate plans for dealing with them? Adequate precautions against war, revolution, riot, insurrection, hijack, piracy, pestilence, disease, etc? Protected from claims by consultants and contractors under terms of force majeure clauses?

DESIGN DEVELOPMENT Design team

AVAILABILITY OF FUNDS Availability of funds

Subject to approval or annuality? Dependent on other departments or agencies? Allocation adequate? Funding policy (or funding policy of other departments or agencies) likely to change? Timing of funds match cashflow forecasts? Timing or scale of funds likely to change? 14

Adequate vetting before short listing? Selected using weighted quality and performance criteria for bid evaluation? Properly experienced and competent? Financially secure?

Adequately defined? Design team responsibilities Clear?

Clear organisation structure? Effective? Match technical requirements? Correct balance of resources and expertise? Gaps or overlaps?

Design Development -continued Design briefs Complete? Issued early enough to design team members? Adequate and clear? Conflicts between different requirements? Any omissions? Approved by sponsor? Managed and coordinated effectively? Exchange of technical and design information effective? Design optimisation accounts for relevant objectives? Delivery of information ahead of construction? Design resources deployed adequate? Meets all requirements? Selected to meet requirements of contract strategy and execution plan? Costed as developed? Design within constraints of control budget and design prog-ramme? Appropriate? In accordance with design brief? Design data appropriate? Makes use of new or untried technology? Procedures for designs to be reviewed by sponsor?

Estimating Data -continued Works cost estimate Reliable? Based on quantity surveyors approach? How accurate are quantities? Historical all-in data reliable? Provisional lump sum items reliable? Proportion of work costed using lump sum guestimates? Reliable percentage applied to cover contractors preliminaries, staff, plant, accommodation, overheads and profit? Figures reflect current market forces? Realistic and reliable? Based on contractors tenders? Includes allowances for costs arising from terms and conditions of contractors contracts? Allowances realistic? Work for health and safety, security, site preparation, temporary works and facilities, permanent materials, plant and equipment, construction and installation, connection to services, commissioning and handover included? Reliable? Provide adequately for land purchase? Handover and occupancy costs included and reliable? Department supplied furniture fittings and equipment reliably identified and costed? Works of art included? Reliable? Level of development? Produced using CPM or PERT methodology? Properly reflect objectives of contract strategy and execution plan? Regularly updated?

Design process

Design programme

Design elements

The design

Other costs estimates

ESTIMATING DATA Cost estimate for professional services Reliable for resources? Based on ad valorem approach? How applied percentages determined? Reflects current market forces? Realistic and reliable? Resources cost estimate based on consultants costed proposals? Includes allowances for costs arising from terms and conditions of consultants contracts? Allowances realistic? All foreseeable work of external advisers, project managers, architects, design and specialist consultants, quantity surveyors (and in-house staff) included?

Project programme

SITE CONDITIONS The site Greenfield or brownfield? Data adequate on archeological remains, underground obstructions, contamination and ground conditions? Adjacent structures or works likely to be affected? Special precautions necessary? Departments interest protected?

I.5

Site Conditions -continued The site

Construction and Commissioning -continued Health and safety measures

continued

Liable to flooding? Access adequate? Any additional infrastructure? Liable to unusual weather conditions? Seasonal weather pattern affect programme? Remote? Adequate local labour and accommodation? Will local customs, holidays or religious restrictions affect programme?
All identified locations known

Adequate? Individual and company responsibilities clear? Clients responsibilities recognised? Emergency procedures established? Updated regularly? Permits to work? Effective and working? Methods defined and agreed? Realistic programme drawn and agreed? New, unusual or untried construction processes? Special skills required? Special plant or equipment required? Permanent materials, fixed plant or equipment in short supply? Approval subject to special testing, design or storage? Points when management passes from construction to commissioning defined? Methods defined and agreed? Scope of work defined? Individual items of equipment, the systems and the whole system identified? Adequate performance specifications? Realistic programme drawn and agreed? Coordinated with achievable dates for construction/ completion? Ongoing construction and partial operation? Points when management responsibility passes from construction to project handover defined? Effective and working? Completion of contractual obligations ensured? Operating manuals and as-built drawings complete and of required quality? Scope of required documentation adequately defined? Includes scope of suppliers documentation?

The construction

Existing services

accurately? Present a hazard? Require future maintenance? Available for use during construction and installation?

Existing installations

inaccessible? Locations of asbestos and hazardous material identified? Environmental impact assessment carried out? Any matters arising?

Any locations currently

Commissioning

CONSTRUCTION AND COMMISSIONING Contractors

Adequately vetted before short listing? Experienced and competent to undertake contracted works? Quality of management and coordination adequate? Contractors, sub-contractors or specialist suppliers to provide design services? Sub-contractors and specialist suppliers experienced and competent to undertake works? Quality of coordination and supervision works adequate? Financially secure? Responsibilities clear? Representation on site of the various functions? On site organisations have correct balance of resources and expertise? Clear individual management responsibilities? Interfaces between organisations clearly defined and understood?

Handover procedures

Management

PROJECT EVALUATION SUMMARY

Project name: Project sponsor:

Evaluation date: Department:

Project stage: Pre-Project/In-Project/Post-Project * * delete as applicable

Project status Conclusions The project sponsor should provide information on les-

Theprojectsponsorsreportmustbe basedontheproject managers evaluation report and record: - whether the objectives of the evaluation were met; - whether the systems, procedures and controls for the areas under evaluation: - have been established; - are or were appropriate; and - are or were being used; - whether the project objectives in terms of time, cost and performance have been defined and are being met at the time of the evaluation; and - an explanation of deviations from those objectives,
Recommendations The project sponsor should provide the following recom-

sons learned during the course of the evaluation so that future projects may benefit from: - what went particularly well and why; and - where problems were experienced, the reasons for these and recommendations on how they could be avoided.

Attachment The project sponsor should also attach the brief issued to

the project manager for the evaluation.

mendations, based on the project managers report:

- whether the evaluation was sufficient and, if not: - what particular aspects should be subject to further evaluation or independent review; - whether the project should proceed as currently proposed and, if not: - an explanation of the reasons; - proposed actions to allow the project to proceed; and - implications, timescales and responsibilities of these actions.

17

OCCUPANCY REVIEW SUMMARY Project name: Project sponsor: Occupying manager: Date of occupation: Review date: Department: Department:

Project status The report must address: - whether the users original requirements have been met. If not:
- what were the shortfalls; - explanations of these; - if the users requirements changed significantly

Recommendations The report should consider from the point of view of the users requirements those aspects of the completed building that are particularly: - useful; - efficient; - surplus to needs; - uneconomic to operate; - and under performing. Recommendations and what should be included and avoided on future projects should be provided in the light of this assessment.

during the course of the project: - what these changes were;

- whether they were accommodated and, if not: - are they still required;
- whether the facilities provided are adequate, fall

short of or exceed requirements, appropriate supporting evidence should be provided; works; and

- identification of outstanding defects and remedial - an assessment of whether the building is economic

to maintain and operate, with details of any unsatisfactory aspects,

18

OTHER GUIDANCES

Applicable CUP Guidances

No 1
No7 No 11 No 12 No 13 No 15 No 17 No 19 No 20 No 25 No 26a No 26b No 33.

Post Tender Negotiation Project Sponsorship: Planning and Progress Monitoring Debriefing Contracts and Contract Management for Construction Works The Selection and Appointment of Works Consultants Estimating for Works Projects Quality Assurance in Building and Construction PTN Update The P&S Function and Works Projects Cost Management for Works Projects Selection of Works Contractors: Prequalification and Tendering Procedures Selection of Works Contractors: Bid Evaluation and Award Project Sponsorship Contract Strategy selection for Major Projects Approval of Works Projects Managing Risk and Contingency for Works Projects.

Produced by Property and Buildings Directorate, Department of the Environment, St Christopher House, Southwark Street, London SE1 OTE Contracting for Works Services

Contractor Management Information System Handbook Consultants Register Guide Guidelines for the Design of Government Buildings Environmental Action Guide (including series of six notes) Historic Buildings Conservation Guide Fire Precautions Guide The Government Property and MOD Fire Standards Standard Fire Precautions for Contractors Better Briefing means Better Buildings by J J W OReilIy published by the Building Research Establishment of the Department of the Environment.

No 36
No 38 No41

To be published

Procurement of Project Management and other consultancy services (excluding design) including model form of contact.

19

CONTENTS 1. INTRODUCTION 2. SCOPE 3. DEFINITION OF TERMS 4. GENERAL PRINCIPLES The project cycle Evaluations, project stages and approvals Timing Topics for evaluation Duration Management Reporting 5. PRE-PRO JECT EVALUATION Timing Management Contents Reporting Benefits 6. IN-PROJECT EVALUATION Management Contents Reporting Duration Benefits 7. POST-PROJECT EVALUATION Timing Management Contents Reporting Benefits

Commencing paragraph 1.1 2.2 3.1 4.1 4.6 4.8 4.9 4.12 4.14 4.17 4.19 5.1 5.3 5.4 5.5 5.6 5.8 6.1 6.6 6.7 6.8 6.10 6.13 7.1 7.3 7.4 7.5 7.6 7.10

8. OCCUPANCY REVIEW Management Contents Reporting Benefits 9. INDEPENDENT REVIEWS When an independent review is necessary Procedures Review of project management controls Systems evaluation Compliance testing Tests of detail Scope of review Management Contents ANNEX A Topics for evaluation B Project evaluation summary C Occupancy review summary D Other guidances E Contents

8.1 8.3 8.4 8.5 8.6 9.1 9.3 9.5 9.7 9.8 9.10 9.12 9.16 9.19 9.22 Page 12 17 18 19 20

20