You are on page 1of 20

HM TREASURY

Central Unit on Procurement


No. 43 Project Evaluation
This is one of a series of guidances prepared by 1.6 Many of the project controls (eg cost, time, quality
CUP on the management of works projects. Its use and change), which must be in place on any well man-
is not mandatory but a statement of good profes- aged project, are subject to evaluation. These are set out
sional practice. at Annex A together with notes on some of the principal
issues to be addressed.
1. INTRODUCTION
1.7 Project evaluation is part of the total quality process.
1.1 Major projects are often complex and risky. An It contributes towards a successful outcome in terms of:
enormous number of decisions have to be made and
- greater assurance of total performance in terms of
actions taken during a project’s life: often under tight time
cost, time and quality;
and budgetary constraints. Project evaluation enables
participants to stand back from the detail of the project, - clearer definitions of responsibilities;
review what has occurred and to take a forward look at the
- reduced exposure to risk; and
remainder of the project.
- improved VFM and accountability.
1.2 Project evaluation is a team effort. It enables indi-
vidual members to consider aspects of the project from
different points of view, identify possible conflicts or poor 2. SCOPE
communication and improve working relationships 2.1 Although this guidance deals with evaluation of
benefiting the overall project. majorworksprojects in general terms, experience shows
1.3 Project sponsors and managers are also that it also applies to smaller projects. However, given
responsible for feedback from lessons learned, both the varied nature of projects, this guidance cannot be
good and bad, so where appropriate they can be used to comprehensive. The principles of project evaluation can
benefit future projects. also be applied to projects other than works.
1.4 Project evaluation techniques are an essential aid to 2.2 The guidance is in 9 sections with supporting
improving project performance and achieving best value Annexes.
for money (VFM). This guidance sets out the principles of Section
project evaluation explaining how, why and when they 1. Introduction
should be done. In carrying out such evaluations, project 2. Scope
sponsors will need assistance from one or more of the 3. Definition of terms
following: 4. General principles
- professional adviser; 5. Pre-project evaluation
6. In-project evaluation
- project manager; and/or 7. Post-project evaluation
- other consultants. 8. Occupancy review
9. Independent reviews
The extent to which they will be involved depends on the
stage the project has reached and the scope of the evalu- A full list of contents is at Annex E.
ation.
1.5 This guidance assumes knowledge of guidance no 3. DEFINITION OF TERMS
33 “Project Sponsorship” which deals with both the 3.1 Project evaluation is a technique to review the
responsibilities and the activities of the sponsor and the current status of a project against plan and to provide
professional team (see Annex 3 of guidance 33). These practical, comprehensive and forward-looking recom-
activities are also subject to evaluation techniques. Sec- mendations for corrective action where necessary. In
tion 3 of guidance No 33 describes a number of key general terms it should consider:
project tasks which form the core topics of project evalu-
ation Core topics are by definition relevant at all stages in - project objectives in terms of cost, time and quality;
the project life cycle. - management;
- organisation; appropriate to influence future projects. The aim is to use
- systems and procedures; data which is of benefit to non-specialists so that it can be
used to influence future decision making. This will help to
- suitability of contracts; ensure that past mistakes are not repeated, and that
- performance of consultants and contractors; and “good practice” is properly understood.

- work to date in terms of cost, time and quality meas- 4. GENERAL PRINCIPLES
ured against plan.
4.1 This guidance provides a summary of activities,
3.2 Evaluation is often carried out by project partici- procedures and controls which should be in place at each
pants (project manager, design and cost consultants with of the following stages of a well managed project:
contributions from contractors and sub-contractors) and
- pre-project stage covering appraisal and
should be a routine performed at predetermined stages
feasibility study;
to establish that objectives are still achievable. Ad hoc
evaluations may also be used as a means of focusing on - in-project stage covering outline design to
specific problems. The starting point for ad hoc evalua- construction completion and commissioning; and
tions often arises from specific concerns of the project - post-project stagecoveringrectificationofdefects,
sponsor. final payments, post-project evaluation and
3.3 A difficulty with evaluation by project participants is occupancy review.
that they often find it difficult to be objective about their 4.2 Evaluations should:
own decisions. An evaluation or review by an independ-
ent consultant overcomes this problem and can bring in - assess the effectiveness of the project organisation;
new ideas and useful outside experience. - review the project execution plan;
3.4 Project evaluationshouldnot be confusedwith ‘value - measure progress against planned cost, time,
engineering’ or ‘economic appraisal’:
quality and performance objectives;
Value engineering is a pre-planned, formal review - identify causes of deviation from plan; and
of the design philosophy and development, at one or
more stages of design development, sometimes led by - recommend remedial action.
an independent expert. These studies are short 4.3
Project evaluation is used to check that:
exercises intended to review the delated design
solutions against the objectives and to establish whether - methods for controlling the cost, time, quality and
they can be achieved in a more cost effective manner. performance characteristics of a project, from the
initial planning to post completion assessment, are
Economic appraisal assesses the costs and benefits
sound and effectively applied;
of expenditure proposals. It is the process of defining
objectives, examining options and weighing up the - management information is prompt, adequate and
costs and benefits before a decision is made. The main accurate;
source of guidance on appraisal is the Treasury’s “Green - assets and interests are properly controlled and
Book” (Economic Appraisal in Central Government:
safeguarded from all losses;
A Technical Guide for Government Departments
HMSO 1991). It contains a checklist to ensure good - the cost of rechargeable works is recovered;
appraisal techniques and notes some common errors. - there are adequate safeguards against fraud, error
3.5 When an investment decision is made, a project is and impropriety;
“born” - though it may not necessarily pass later stages of - material losses due to waste or inefficiency are
approval and reach completion. This decision is the first identified and recovered where appropriate;
stage of the Treasury’s project approval procedure (see
guidance no 38 “Approval of Works Projects”), ie the - information on which costs and cash flow projections
initial approval of the options and the assessment of are based provide a sound basis for the funding of
the business case. The second stage is identifying the the project; and
preferred option, establishing a preliminary budget and - internal guidance and other established policies and
obtaining approval to take forward planning and design procedures are appropriate and are followed.
work on this option.
4.4 Project evaluation is independent of any established
3.6 Project evaluation can be used from the second reporting requirements. It can be used to encourage the
approval stage (by evaluating the chosen option against project team to stand back and review what it is doing and
the objectives in the appraisal) throughout the project’s where it is going in relation to the stated project objec-
life, ending with the post-project evaluation and occu- tives. It should end with a formal written report presented
pancy review. It is important to note that the term to senior management by the sponsor based on data and
‘evaluation” in the ‘Green Book’ is concerned with post-
reports from the project manager.
project evaluation only.
4.5 A disciplined and systematic approach to evaluation
3.1 Post-project evaluation is a means of recording
is recommended so that it becomes a routine project
experience from past projects, so it can be used where
activity, heading off problems before they occur.
2
The project cycle Evaluations, project stages and approvals
4.6 Every project can be visualised as a series of steps 4.8 Although projects and their procurement strategies
requiring a variety of technical skills to manage inherent vary, there is a pattern of definite stages common to most
risks. Initially objectives must be defined and options for projects. Some of these stages overlap depending on the
achieving them generated and evaluated before decid- type of project and the method of procurement. Project
ing which option should be implemented. Project evalu- evaluation is very effective when carried out at the end of
ation techniques provide the sponsor with an additional the project stages which are immediately prior to the next
means for controlling and monitoring the progress of a approval stage. The link between project stages and
project. approval stages is illustrated in the following table.
4 .7 The steps in the project cycle and the stages when a
routine project evaluation should happen are at Figure 1.

Figure 1: The Project Cycle

Definition

Generation of
options

Evaluation of
short-listed options

SELECTING
CONTRACT

t
Decisions
Monitoring - Choice of options
and control -Whether to implement

Monitor
and con

- Responsiblties
Whether to implement
Timing of evaluations
1
Project Stage Approval Stage

there is an identification stage, when the


need is apparent, the benefits are measurable,
the business case can be established, initial
studies are carried out and options are
appraised;
Business case/option appraisal.

the preferred option is taken forward and


subjected to further feasibility studies
(the preferred option may have established
a location and even a site for the building)
feasibility studies will examine the ways
the building might fit the site, the
environmental impact, planning and
statutory requirements and estimated
cost and funding;
Submission for approval to take forward planning
and design work.

planning and design work is taken forward


through outline (concept) design and into
scheme design; the project execution
plan is further developed;
Approval Estimate leading to submission for final
approval as soon as cost estimates of the
required accuracy can be prepared
(not linked to any specific design stage but
usually around final sketch plan stage).

detail design, procurement and


construction which may be sequential
or overlapping depending on the
contract strategy;
Submission for re-approval if at any time
forecast outturn cost is greater than approval
and/or conditions established at
final approval are likely to be broken.

commissioning and occupation. Occupancy review.

Timing 4.11 The ability to control the outcome of a project is


greatest in the early stages decreasing rapidly as the
4.9 At intermediate stages, the natural points to consider project progresses as illustrated in Figure 2 below. How-
evaluation are again at the end of one stage and prior to ever, uncertainties about the outcome of the project are
the commencement of the next, eg at the end of scheme also greater. Project managers should use risk analysis
design but before detail design or prior to the submission techniques to help evaluations and advise the sponsor to
for final approval. make plans and choose strategies which reduce risk, and
where appropriate place remaining risks where
4.10 During construction, evaluations are recommended they can be managed most effectively. Risks should be
at the mid point of the construction programme or once a reflected in the level of contingency allowance so that
year for longer programmes. More frequent evaluations outturn cost and time can be contained within forecasts.
should be considered for particularly complex projects. See guidance no 4 1 ‘Managing Risk and Contingency for
Ad hoc evaluations to identify and solve specific prob- Works Projects’.
lems may be called for at any time throughout the project.
Figure 2: Degree of Project Control and systematic manner. Other participants may include
members of the sponsor’s organisation, the professional
High
adviser, the design team, the construction team and the
intended users of the building.
4.18 If project sponsors suspect that project managers
have not been fulfilling their responsibilities, they should
consider either an evaluation by the professional adviser
or an independent review. The professional adviser can
provide guidance on this.

Reporting
Low
4.19 It is important that the evaluation should be docu-
mented clearly by all concerned so that if the need for a
Topics for Evaluation further review is identified the team can follow the proc-
4.12 Annex A lists some of the topics to be evaluated at ess of planning, work, findings, conclusions and recom-
various stages of the project and, for each topic, several mendations.
key issues to which the evaluation team should provide 4.20 Each participant in the evaluation team should keep
answers. Controls and procedures often have to envolve working files to support individual evaluation reports,
as the project progresses. The evaluation should check recording the details of each stage from evaluation plan-
that a control mechanisms are: ning through to reporting.
- established, appropriate and being used; 4.2 1 The project manager will provide the sponsor with a
- reviewed and amended to meet changing project written report, based on contributions from all the partici-
needs; and pants (but checked by the project manager as necessary)
highlighting the key areas, This report should address in
- complied with. particular any aspects of control systems and procedures
4.13 Where existing controls are considered unsatisfac- which were found wanting. The implications of these
tory, evaluations must provide the sponsor with recom- deficiencies should be explained together with recom-
mendations for improvement. An independent review mendations for improvements.
may be necessary if the evaluation either cannot establish 4.22 The project sponsor reports the results of the evalu-
the problems adequately or reveals a particularly serious ation to the project director on the basis of data and
problem for which the underlying cause or corrective reports received from the project manager. The spon-
action cannot be identified quickly. This is particularly sor’s evaluation report should comprise:
applicable if poor performance, fraud or some other - a short (one to two page) executive summary listing
impropriety is suspected.
the reasons for the report, the principal conclusions
Duration and the actions to be taken; and
- sufficient detail on the findings of the evaluation to
4.14 Evaluations should be sufficiently detailed to satisfy
the sponsor that the project is on course to meet its identify any further action required.
planned objectives or, if not, that the cause of any problem
has been identified and proposals made for its resolution. 5. PRE-PROJECT EVALUATION
4.15 Typically evaluations should last one to two weeks, 5.1 At this stage, prior to commencing design and
depending on the stage the project has reached and its construction, the objectives of the project sponsor’s
size, importance and complexity. This is particularly im- department and the project itself are established. Project
portant when the evaluation is carried out to deal with a objectives should be defied clearly and prioritised
perceived problem, so that corrective action can be in terms of time, cost, quality and performance. A pre-
taken quickly and the effects of the problem mitigated. project evaluation should be carried out to confirm that
4.16 Evaluation of a specific problem will be achieved this is so by investigating:
best by limiting its scope. For example, an evaluation to - the clarity of the project objectives;
identity and reduce the causes of a cost overrun, may be
limited to all changes made or planned. The review may - the viability of achieving those objectives;
be further limited where the problem is more specific, eg - whether sufficient options for achieving the objec-
whether the cost effects of a particular design change tives have been generated;
have been fully assessed.
- that these options have been properly evaluated;
Management - the degrees of influence in the client organisation;
4.17 The project sponsor is responsible for ensuring that - that the main risks and constraints to the project have
evaluations take place at intervals recommended in this been identified and assessed; and
guidance. The project manager, when appointed, is re-
sponsible for managing the evaluations in a disciplined - that the project has appropriate funding.

5
The purpose of a pre-project evaluation is to provide a 5.9 Most crucial decisions are taken prior to commenc-
checkthat these matters are being or have been resolved ing design (see paragraph 4.11). The evaluation should
satisfactory. identify where action needs to be taken. Changes at this
stage are rarely as costly or as disruptive to the project as
5.2 If the evaluation reveals serious flaws in the project those implemented later.
or its proposed implementation, it may be necessary to
delay moving to the design stage until further investiga- 5.10 When the pre-project evaluation confirms that the
tion by a more detailed evaluation or independent review project objectives are achievable and that the necessary
is complete and the implications have been assessed. controls are in place, the project can proceed to the
design stages.
Timing
6. IN-PROJECT EVALUATION
5.3 The best time for a pre-project evaluation is shortly
after appointment of the project manager, and before 6.1 During the design stages of a major project, interac-
appointing the permanent design team. It may be neces- tion between the various parties is intense and complex.
sary to seek specialist advice, for example on risk identi- Oversights and mistakes can easily occur. As the project
fication and analysis, during the course of the evaluation. design progresses the opportunity for introducing
In this event temporary appointments may have to be significant changes (without incurring costs for abortive
considered. design work) reduces rapidly. A preplanned evaluation
should be carried out at the pre-tender stage to establish
Management that all is well with the project before entering into the
most significant financial commitment (ie construction of
5.4 The project sponsor is responsible for ensuring that the project).
the pre-project evaluation takes place. The project man-
ager should manage the evaluation. 6.2 Duringthe constructionphase, the project sponsor’s
department incurs the major part of the project
Contents expenditure. Changes introduced at this stage are likely
to incur disproportionate cost increases and result in
5.5 Annex A indicates some of the topics to be reviewed. delay to the project.
The evaluation should establish that the necessary systems
and controls exist to proceed to design stage or identify 6.3 As projects can be subject to significant risks during
the means of achieving this. The project objectives, both the design and the construction phases, in-project
preliminary contract strategy and the project organisa- evaluations provide the project team with the opportunity
tion should be identified and procedures should exist for to stand back from the detail of the project and:
the appointment of consultants, and for design and cost - evaluate how well they are working together;
control to the pre-tender stage. Many of the control docu-
ments at this stage are the initial versions for the project - assess the status of the project against cost, time and
and will be amended by the professional team quality objectives;
as the project progresses. However, they should be - identify existing or potential problems;
appropriate for this stage.
- make forward-looking, practical recommendations
Reporting for improvements which can be implemented
immediately;
5.6 The project manager’s evaluation report should state
- check that the risks to the project are being man-
clearly whether the project should proceed as currently
proposed and if not, the actions necessary for the project aged; and
to be successfully continued. The report must also state - ensure a commitment of funds to the project.
whether the evaluation has been adequate for its intended
purpose or whether a more detailed evaluation is 6.4 It is good practice to carry out in-project evaluations
recommended for all or part of the project. during:
- design: the project manager should advise the
5.1 An indication of the issues to be covered in the
project sponsors’s evaluation summary is at Annex B. sponsor on the timing, but typically this would be
However, the summary will need to be tailored to suit the prior to commencing detailed design and/or prior to
particular project. tender and construction; and
- construction: the project manager should advise
Benefits the sponsor on the frequency, which might typically
5.8 Evaluation plays a valuable role at this stage in be at 6-9 month intervals.
ensuring that the department has defined the project
objectives clearly and that the chosen preliminary con- Management
tract strategy is consistent with those objectives prior to 6.5 The project manager, reporting to the sponsor, is
entering contractual relationships and commitment of responsible for managing the evaluation, the participants
funds. and for providing the report.

6
Contents 6.13 The advantage of conducting an evaluation at the
outline and/or scheme design stages is that changes are
6.6 Annex A summarises some of the topics whichshould relatively easy and economic to introduce. This is not the
be evaluated during the design and construction stages. case once detailed design has taken place.
The evaluation process should check that:
6.14 Evaluation at mid-construction provides the final
- control procedures for the core topics are appropri-
check that the project is on course. Whilst significant
ate to meet current and future project needs and are changes to project scope at this stage will be costly to
being complied with; and implement, changes in control procedures are not and
- new controlmechanisms specific to in-project phases may be necessary if forecast time, cost or quality objec-
exist, are appropriate and are being complied with. tives are not being met.

Reporting 7. POST-PROJECT EVALUATION


6.7 Evaluations should be summarised and presented 7.1 This should be carried out soon after completion of
to the sponsor in a report prepared by the project man- construction to:
ager based on contributions from project participants. - measure the success of the project in achieving its
The report should: planned objectives;
- state clearly whether the project remains on course - identify the reasons for any problems that have
to meet its plan arisen:
- consider risks and how they are being managed; - determine what remedial actions should be taken:
- provide fully evaluated recommendations to deal and
with any existing or potential problems; and - record the lessons which have been learnt to
- improve the effectiveness of project execution. improve performance on subsequent projects.
6.8 Design stage evaluations should recommend The objective is to learn lessons for application to future
whether to proceed to detailed design and/or construc- projects and to pass them onto other departmentalprojects
tion They must state whether the evaluation is sufficient or or departments. The post-project evaluation report for a
whether a more detailed independent review is required. building project may be supplemented by an occupancy
review (see Section 8).
Duration 7.2 There is a minimum level of post-project evaluation
6.9 Outline design stage and scheme design stage which should be carried out on all major projects. The aim
evaluations should be completed normally within 2 weeks. ofpost-project evaluationis to improve project appraisal,
If serious problems are revealed, moving to the next design, management and implementation, Its objective is
design stage should be delayed until a further detailed to get maximum benefit from accrued experience, not to
evaluation has been undertaken, the implications apportion blame.
assessed and remedial measures taken.
Timing
6.10 Construction stage evaluations should proceed in
parallel with construction and normally be completed 7.3 Time is not usually critical at this stage. However, the
within 2 weeks. earlier the post-project evaluation is done, the earlier the
lessons are available for subsequent projects. In particular,
6.11 Delaying the construction process is likely to prove the evaluation should be completed sufficiently early to
very costly and should only be considered if:
ensure that:
- the evaluation reveals serious problems which
- knowledge is still live;
significantly affect the achievement of project
objectives; and - those involvedwiththeproject are still available; and
- these problems will worsen significantly if con- - outstanding obligations are identified and steps taken
struction is allowed to continue before taking to ensure they are fulfilled.
remedial action.
Management
In these cases a more detailed evaluation or independent
review may be necessary to provide recommendations 7.4 The project manager is usually responsible for di-
for corrective action. recting the post-project evaluation. The evaluation re-
quires the active participation of all parties to the project
Benefits as part of their final duties and this should be set out in their
terms of appointment.
6.12 Evaluations during in-project stages should identify
the causes of any problems and provide recommenda-
Contents
tions for remedial action. The sooner this is done, the
more likely the project will be completed successfully. 7.5 The evaluation should assess whether the objectives
It serves to limit the risk to which the project sponsor’s of the project were realised and the established controls
department is exposed. complied with. The effectiveness of these controls should

7
also be reviewed and recommendations should be made anticipated to change duringthe course oftheproject,
for improvement of controls on future projects. to assess whether those changes were accommo-
dated:
Reporting - confirm that the building functions in use as planned;
7.6 Post-project evaluationshouldconcludewithareport - ensure that any outstanding work, including defects,
prepared by the project manager using contributions have been remedied; and
fromtheprojectparticipants. It shouldstateclearlywhether
the project met its objectives in terms of time, cost, quality - ensure that key lessons learnt on occupation are put
and performance. Recommendations for reducing the into practice without delay for the benefit of future
possibilityofproblemsoccurringonfutureprojectsshould and continuing major projects,
be made. It should also highlight aspects of project man-
agement which have worked particularly well or not and Management
state that the evaluation was sufficient for its intended 8.3 The occupying business manager responsible for
purpose or whether a more detailed evaluation is required. the property should arrange for the occupancy review to
7.7 The sponsor adds a report on the project manager’s be conducted. Assistance will be required from the senior
performance. The sponsor submits the final post-project users and preferably also the project manager.
evaluation report to the project director who uses it
to check whether the department’s objectives and Contents
requirements have been met. 8.4 An occupancy review should include:
7.8 Typical issues to be addressed in the report are at - an assessment of whether the users’ requirements
Annex B but the report should be tailored to particular have been met. If not, consideration should be given
projects and evaluations. to the briefing process and the extent to which
7.9 Departments who use the Contractor Management users were consulted and involved as the design
Information System (CMIS) and the Consultants Register developed;
(CONREG) operated by Construction Industry Sponsor- - costs in use - how the planned operating cost of the
ship (CIS), DOE, 2 Marsham Street, London SW1P 3EB, building compares with actual cost and how the
are required to submit performance reports on contractors planned efficiency and effectiveness of the equip-
and consultants as a matter of course. ment and facilities compare with actuals; and
Benefits - any further recommendations the users can make to
improve VFM performance of future projects. For
7.10 A post-project evaluation provides an opportunity to example, the users may consider that some of the
review the project with the benefit of hindsight. It reviews facilities provided were unnecessary.
all project stages and the problems that have arisen
thereby providing lessons for the future and identifying Reporting
how problems arose. It is also valuable to conduct post-
project evaluations on major projects that have gone 8.5 Typical issues to be addressed in the occupancy
particularly well, in order to identify ingredients for review report are at Annex C. The report should be
success. tailored to suit the individual project.
7.11 The evaluation can also provide the sponsor with Benefits
assurance that the works have been completed in
accordance with the contracts, that consultants and con- 8.6 The occupancyreviewis the finalfeedbacklinkfrom
tractors have discharged their responsibilities and that all the users which closes the loop of the project cycle. The
potential liabilities have been recognised and quantified. review also identifies the existence of any outstanding
It will expose any shortfall in these areas and enable the obligations, with which the project sponsor must deal.
sponsor to recognise where commitments still exist and/
or damages might be recoverable. 9. INDEPENDENT REVIEWS
9.1 Project sponsors should commission independent
a. OCCUPANCY REVIEW detailed reviews if they are concerned about either
8.1 When the project relates to a building, an occupancy the general status and progress of the project or, more
review should be performed one to two years after the usually, about specific issues.
building has been occupied. The result of this review is an 9.2 Its aims should not be limited to identifying the root
addendum to the post-project evaluation. It records how causes of any problem but should also make practical
successfully the completed project meets the require- recommendations for corrective action and aim to
ments of the occupiers and identifies any further work generate direct and significant time, cost and quality
required. improvements.
8.2 An occupancy review has the following objectives to:
When an independent review is necessary
- consider whether the objectives of the users have
been met. If those objectives changed, or were 9.3 The need for an independent review may arise at

8
any time when the project sponsor considers that certain - systems evaluation;
problems are serious enough to require separate inves-
- compliance testing; and
tigation However, the project manager should also
advise the project sponsor when one is necessary. There - tests of detail.
are a number of typical warning signs:
The flowchart at Figure 3 (page 10) indicates this process.
- cost issues:
Systems evaluation
- large increases between successive estimates;
- tenders for construction being significantly above or 9.8 This comments on the adequacy of control system
and offers recommendations for improvement. Th
below estimate;
approach is broadly based rather than detailed and i
- a significant overspend or underspend compared conducted in two stages. First, the system is recorded
with predicted cash flow; in terms of its operation, preferably in flowchart forn
- unpredicted changes resulting in high additional Second, the system is evaluated by defining its control
objectives and assessing the strengths and weaknesses
costs;
of the system in meeting those objectives.
- large contractual claims for additional payments or
9.9 A typical evaluation might cover the following:
extensions of time; and
- is the procedure clearly defined?
- a final contract sum which is much higher than the
tender sum; - is someone nominated to carry out the procedure
- is this person fully aware of the responsibility fc
-time issues:
carrying out the procedure?
- progress falling significantly behind programme eg - is the person required to demonstrate that th
failure to meet milestone dates;
procedure has been followed?
- delays in providing the contractor with necessary
- are there any further procedures providing for a
information, which results in delays and/or addi-
tional construction costs; and independent witness or check that the procedure
has been carried out? and
- late completion;
- does the procedure provide the intended control
- quality and performance issues:
- a pattern of failings in design or workmanship; and Compliance testing
- a finished product which fails to meet the users 9.10 This is used to check that the system is operating a
intended. It focuses on the key controls of the system an
requirements in certain important respects;
the principal management actions which must be carried
- management/organisation issues: out if the system is to operate as intended. The tests are i
whatever form is appropriate to determine whether the
- delays, cost increases or reduced quality standards key controls are operating in practice. Examples of these
arising from failure to provide the right people with tests include:
the necessary information to make decisions or take
- observation of the procedure in operation;
actions at the right time; and
- examination of working papers and formal docu-
- lack of clarity of roles and responsibilities,
mentation; and
Procedures - interviews with project participants to determine th
9.5 Reviews will be capable of identifying problems and actual procedures in use.
providing recommendations for their resolution. The 9.11 If systems evaluation has proved that the system is
reviewer’s approach should be objective. Subjective itself inherently weak, compliance testing is usually
judgements, for example relating to truth or fairness, inappropriate.
should be expressed specifically as an ‘opinion’ as distinct
from a ‘finding’. Tests of detail
9.6 The project sponsor should make reviewers aware 9.12 These may be necessary when systems evaluatio
of the circumstances which make the project unique. and/or compliance testing have revealed weaknesse
Although reviews may cover the same topics as evalua- and deficiencies. There are two types of tests of detail:
tions (see Annex A) they are often in greater depth and
therefore detailedprocedures will need to be established - those concerned with financial values; and
individually for each review. - those concerned with the quality of managemer
Review of project management controls decisions.

9.1 Project management controls can be audited as 9.13 Financial tests of detail either start at:
follows:

9
Figure 3: Audit Flowchart

Select system to be audited


I

Does system exist? ) No


7 I

I Gather data on how I


1 system operates
I

I
Systems
evaluation I Flowchart the system
I

I Define what the system is


intended to achieve I

t
Evaluate strengths and
weakmesses of system

I Report to management
I
- - - - - - - - - - - - -_-----
I
I
I
I
1 1 Debekey contr;kthesystem I

I
Devise sampling procedures
and compliance tests I ~~
Compliance
testing I I
I I
I I
I I
I I
I
/ Reporttort I + - - - - - - - - - - - - - - - - I I
I
I
I
I
I
I
I
I
Tests of detail I
I
I
I
Report to management +--_---_____---------- -I

10
- a figure in an account - then trace the data or facts during their investigation, They should have rel-
evidence validating that figure (eg expenses or evant experience and a successful record of conducting
costs); or construction project reviews. Suitable candidates can be
- found in construction and property management consult-
the data or evidence - then check that the correct
ants, project and cost management consultants and
figure arising from that information is included in the
consulting engineers. The professional adviser and/or
account (eg income or deductions from expenses).
project manager can advise the project sponsor on
9.14 During the course of a project, some management selection.
decisions fall outside the scope of systems and control.
9.20 Details of the form of the review, the scope of the
These decisions may have to deal with unforeseen
work and the final report should be agreed between the
circumstances eg poor ground conditions, bad weather
project sponsor and the review team before work starts.
or poor interaction between the parties to the project.
Objective tests of detail consider the quality of these 9.21 The reviewers will need the co-operation of the
decisions by: project participants in gathering information. However,
- setting down clearly the relevant history and facts when the project sponsor has commissioned a review
rather than an evaluation of the project, participants may
relating to the decision;
feel threatened and may be reluctant or unwilling to co-
- considering all options available to management; operate. This may reduce the effectiveness of the review.
- analysing how well each option relates to the The obligation to co-operate must therefore be included
in the terms and conditions of engagement of the
achievement of the project objectives; and
project participants. When fraud or negligence are being
- recording the reasons for selecting a particular investigated, it may be necessary for the sponsor to
option. remind participants of their contractual obligations on
co-operation.
9.15 Good decision makers should have followed the
process described above. However, it is stillnecessary to Contents
check that the process has been completed correctly and
9.22 Annex A summarises some of the topics which may
that the associated working papers are on file.
be included at different stages of the project. The project
sponsor should define the scope of the review and
Scope of review
prepare a brief for the review team.
9.16 The scope of an independent review will depend on - -
9.23 The level of detail of the review depends on:
the nature of the problem under consideration, its associ-
ated risk and why it is being carried out. If the objective is - the project sponsor’s brief to the review team;
simply to assess the effectiveness of systems and controls
- the project stage;
and to provide recommendations for improvement,
the review may be limited to systems evaluation and - the matter under investigation: and
compliance testing. If a judgement on a specific issue is
- discoveries made during the review process.
required, compliance and detailed testing will be neces-
sary. If fraud is suspected, all three levels of review are The reviewers should investigate whether the controls
likely to be necessary to reveal the full extent of the are clearly defined and are being satisfied.
problem.
November 1993
9.17 However, the review’s size may be limited by
sampling techniques. Judgmental sampling relies on the
reviewer’s judgement to pick out the major or most
contentious items for investigation. Whilst this method is
quick and cost-effective, the project sponsor should
recognise that oversights may occur.
9.18 Statistical sampling is likely to be more accurate but
more time-consuming. The reviewer selects the size and
type of sample based on an assessment of risk and
importance. The reviewer will advise the sponsor on how
and why the sample was chosen and the confidence that
can be placed in the results.

Management
9.19 The project sponsor is responsible for commission-
ing independent reviews and appointing the review team
to undertake the investigation. The team should be inde-
pendent, without any conflict of interest, unbiased and
have no reason to protect any individual or conceal any

11
ANNEX A

OUTLINE CHECK LIST OF TOPICS FOR EVALUATION

This Annex lists some of the topics (with their key issues Force Majeure
and problem areas) which may be reviewed during catastrophic events
project evaluation. Where appropriate, evaluation teams
should both examine and provide answers to each. Design development
However, the list is not exhaustive and will need to be designers
reviewed depending on the project’s stage and its design team responsibilities
contract strategy. In addition, the list may also need design brief
amendment to reflect the scope and requirements of the design process
particular project. The project sponsor, on the advice of design programme
the project manager and/or professional adviser, should design elements
define the scope of partial or specific evaluations. the design
Controls andprocedures evolve as theprojectprogresses Estimating Data
and evaluations should establish that they are:
cost estimate for professional services (resources)
- established and appropriate to the stage of the
works cost estimate
project; other cost estimate
- being reviewed and amended to suit the changing project programme
needs of the project; and Site Conditions
- complied with.
the site
Where procedures and controls are considered unsatis- conditions above and below ground
factory evaluations must provide recommendations for susceptibilities - flooding, bad weather
improvement. labour availability
existing services
The suggested topics for evaluation are as follows: existing buildings
Project Definition environmental impact assessment
user requirements Construction and Commissioning
project objectives contractors
project brief site management responsibilities
planning, social and environmental impact assessments health and safety measures
Project organisation and management emergency procedures
permits to work
department’s client organisation construction methods
project management team long delivery, short supply items
project organisation structure commissioning methods
risk assessment handover procedures
approved budget operating manuals
cost, time, quality and change control procedures as built drawings
contract strategy
project execution plan
procurement procedures
detail design
reporting systems
contractual claims
Availability of funds
Statutory and other External Factors
planning approval
building regulation approval
health and safety regulations
site access
inflation
interest rates
currency exchange rates
VAT

12
ANNEX A - continued

PROJECT DEFINITION Project Organisation and Management -continued

User Adequately defined? Project Project manager pro-active?


requirements Incomplete or unclear? management Continuity in project management’
Conflicts between different user team Project manager’s links with
requirements? continued participants effective?
Likely to change? Project manager administers
contracts efficiently?
Project Properly defied? Queries dealt with promptly?
objectives Incomplete or unclear?
Conflicts between primary/ Project Clearly defined?
secondary objectives? organisation Clear direction and
Possibility of change? structure communication?
Project team responsibilities clear?
Project brief Complete? Gaps or overlaps?
Inadequate or unclear? Arbitrary external interference
Conflicts between different in project team?
requirements? Clear communication and
Possibility of change? identified responsibility with
Are the following unclear, in interested organisations not
conflict or likely to change: directly involved?
- site;
- scope: Risk Carried out at important stages?
- extent; assessment Used to determine contingency,
- quality and performance: tolerance and approved budget?
- design concept; Control budget within approved
- cost and time; or budget?
- completion date? Conditions attached to
Sufficiently output orientated to authorisation to proceed?
provide design flexibility?
Cost Effective and working?
Planning, Carried out? control Control budget regularly
social and Issues of concern identified? procedures reviewed?
environmental External authorities consulted? Costs monitored and exceptions
impact Their requirements met? acted upon?
assessments Outstanding matters? Time control effective and
working?
Project programme regularly
PROJECT ORGANISATION AND MANAGEMENT reviewed?
Progress monitored and
Department’s Clearly defined? exceptions acted upon?
client Clear direction and Quality control effective and
organisation communication? working, exceptions acted upon?
Sponsor and client team Change control effective and
responsibilities clearly defined? working?
Appropriate? Sponsor’s approval to changes?
Gaps or overlaps?
Arbitrary external interference Contract Takes account of project
in client team? strategy objectives?
Sponsor administers contract Project execution plan current?
efficiently? Defines the work packaging of
Notes and decisions issued design, procurement,
promptly? construction and commissioning?
Defines how activities are
Project Correct balance of resources coordinated and managed?
management and expertise?
team Effective? Procedures for Effective and working?
Responsibilities clear? procurement Comply with EC/GATT rules?
Appropriate? Tender documents adequate?
Project manager exercises Tender periods adequate?
adequate authority? Information reliable?

13
Project Organisation and Management -continued STATUTORY AND OTHER EXTERNAL FACTORS

Procedures for Tender documents protect Planning Possibility of delay in obtaining?


procurement department from low tenders approval, Project compliant with building
continued with potential for claims? building and health and safety
Tender evaluation appropriate? approval, regulations?
Approval and award of contracts licences and Necessary statutory approvals
effective and working? permits obtained?
Contracts clearly define scope Access to site subject to approval
of work and responsibility of of external organisations or
contracting party? authorities?
Any nominated sub-contractors? Bureaucratic procedures likely to
Department’s responsibilities cause delay?
recognised and taken into account? Anticipated changes in
regulations or legislation affect
Detailed Complete before construction project?
design starts?
Design requirements properly Inflation, Inflation taken into account?
coordinated? interest rates, Affected by interest rates?
Level of design detail appropriate rates of - taken into account?
and adequate? exchange Affected by rates of exchange?
Likely to change after start of and VAT - taken into account?
construction? VAT taken into account?
Design information complete and
timely?
Queries and conflicts resolved FORCE MAJEURE
quickly?
Catastrophic Department’s investment
Reporting Effective and working? events protected from extreme
systems Include narratives on current status weather, flood, landslips, earth
of cost, time, change and quality? quakes, fire, crashing aircraft or
Shortfalls in meeting objectives other catastrophic events?
highlighted? Adequate plans for dealing with
Corrective action identified and them?
initiated? Adequate precautions against
Identify actual and anticipated war, revolution, riot, insurrection,
critical areas? hijack, piracy, pestilence,
Essential information provided in disease, etc?
a coherent format? Protected from claims by
Information used to prompt action consultants and contractors
and take decisions? under terms of force majeure
clauses?
Contractual Likely to arise?
claims Performance likely to prompt
claims? DESIGN DEVELOPMENT
Adequate records maintained to
counter claims? Design team Adequate vetting before short
Validity and potential exposure to listing?
actual claims independently Selected using weighted quality
audited? and performance criteria for bid
evaluation?
Properly experienced and
AVAILABILITY OF FUNDS competent?
Financially secure?
Availability Subject to approval or annuality?
of funds Dependent on other departments Design team Adequately defined?
or agencies? responsibilities Clear?
Allocation adequate? Clear organisation structure?
Funding policy (or funding policy Effective?
of other departments or Match technical requirements?
agencies) likely to change? Correct balance of resources and
Timing of funds match cashflow expertise?
forecasts? Gaps or overlaps?
Timing or scale of funds likely to
change?

14
Design Development -continued Estimating Data -continued

Design briefs Complete? Works cost Reliable?


Issued early enough to design estimate Based on quantity surveyors’
team members? approach?
Adequate and clear? How accurate are quantities?
Conflicts between different Historical all-in data reliable?
requirements? Provisional lump sum items
Any omissions? reliable?
Approved by sponsor? Proportion of work costed using
lump sum guestimates?
Design process Managed and coordinated Reliable percentage applied to
effectively? cover contractors’ preliminaries,
Exchange of technical and design staff, plant, accommodation,
information effective? overheads and profit?
Design optimisation accounts for Figures reflect current market
relevant objectives? forces?
Realistic and reliable?
Design Delivery of information ahead of Based on contractors’ tenders?
programme construction? Includes allowances for costs
Design resources deployed arising from terms and
adequate? conditions of contractors’
Meets all requirements? contracts?
Allowances realistic?
Design Selected to meet requirements of Work for health and safety,
elements contract strategy and execution security, site preparation,
plan? temporary works and facilities,
Costed as developed? permanent materials, plant and
Design within constraints of equipment, construction and
control budget and design installation, connection to
prog-ramme? services, commissioning and
handover included?
The design Appropriate?
In accordance with design Other costs Reliable?
brief? estimates Provide adequately for land
Design data appropriate? purchase?
Makes use of new or untried Handover and occupancy costs
technology? included and reliable?
Procedures for designs to be Department supplied furniture
reviewed by sponsor? fittings and equipment
reliably identified and costed?
Works of art included?
ESTIMATING DATA
Project Reliable?
Cost estimate Reliable for resources? programme Level of development?
for professional Based on ad valorem approach? Produced using CPM or PERT
services How applied percentages methodology?
determined? Properly reflect objectives of
Reflects current market forces? contract strategy and execution
Realistic and reliable? plan?
Resources cost estimate based Regularly updated?
on consultants’ costed
proposals?
Includes allowances for costs SITE CONDITIONS
arising from terms and
conditions of consultants’ The site Greenfield or brownfield?
contracts? Data adequate on archeological
Allowances realistic? remains, underground
All foreseeable work of external obstructions, contamination and
advisers, project managers, ground conditions?
architects, design and specialist Adjacent structures or works
consultants, quantity surveyors likely to be affected?
(and in-house staff) included? Special precautions necessary?
Department’s interest protected?

I.5
Site Conditions -continued Construction and Commissioning -continued

The site Liable to flooding? Health and Adequate?


continued Access adequate? safety Individual and company
Any additional infrastructure? measures responsibilities clear?
Liable to unusual weather Client’s responsibilities
conditions? recognised?
Seasonal weather pattern affect Emergency procedures
programme? established?
Remote? Updated regularly?
Adequate local labour and Permits to work?
accommodation? Effective and working?
Will local customs, holidays or
religious restrictions affect The Methods defined and agreed?
programme? construction Realistic programme drawn and
agreed?
Existing All identified locations known New, unusual or untried
services accurately? construction processes?
Present a hazard? Special skills required?
Require future maintenance? Special plant or equipment
Available for use during required?
construction and installation? Permanent materials, fixed plant
or equipment in short supply?
Existing Approval subject to special
installations Any locations currently testing, design or storage?
inaccessible? Points when management passes
Locations of asbestos and from construction to
hazardous material identified? commissioning defined?
Environmental impact
assessment carried out? Commissioning Methods defined and agreed?
Any matters arising? Scope of work defined?
Individual items of equipment, the
systems and the whole system
CONSTRUCTION AND COMMISSIONING identified?
Adequate performance
Contractors Adequately vetted before short specifications?
listing? Realistic programme drawn and
Experienced and competent to agreed?
undertake contracted works? Coordinated with achievable
Quality of management and dates for construction/
coordination adequate? completion?
Contractors, sub-contractors or Ongoing construction and partial
specialist suppliers to provide operation?
design services? Points when management
Sub-contractors and specialist responsibility passes from
suppliers experienced and construction to project handover
competent to undertake works? defined?
Quality of coordination and
supervision works adequate? Handover Effective and working?
Financially secure? procedures Completion of contractual
obligations ensured?
Management Responsibilities clear? Operating manuals and as-built
Representation on site of the drawings complete and of
various functions? required quality?
On site organisations have Scope of required documentation
correct balance of resources adequately defined?
and expertise? Includes scope of supplier’s
Clear individual management documentation?
responsibilities?
Interfaces between organisations
clearly defined and understood?

1
PROJECT EVALUATION SUMMARY

Project name: Evaluation date:

Project sponsor: Department:

Project stage: Pre-Project/In-Project/Post-Project *


* delete as applicable

Project status Conclusions

Theprojectsponsor’sreportmustbe basedontheproject The project sponsor should provide information on les-


manager’s evaluation report and record: sons learned during the course of the evaluation so that
future projects may benefit from:
- whether the objectives of the evaluation were met;
- what went particularly well and why; and
- whether the systems, procedures and controls for
the areas under evaluation: - where problems were experienced, the reasons for
these and recommendations on how they could be
- have been established; avoided.

- are or were appropriate; and Attachment

- are or were being used; The project sponsor should also attach the brief issued to
the project manager for the evaluation.
- whether the project objectives in terms of time, cost
and performance have been defined and are being
met at the time of the evaluation; and
- an explanation of deviations from those objectives,

Recommendations

The project sponsor should provide the following recom-


mendations, based on the project manager’s report:
- whether the evaluation was sufficient and, if not:

- what particular aspects should be subject to


further evaluation or independent review;

- whether the project should ‘proceed as currently


proposed and, if not:
- an explanation of the reasons;

- proposed actions to allow the project to


proceed; and

- implications, timescales and responsibilities of


these actions.

17
OCCUPANCY REVIEW SUMMARY

Project name: Review date:

Project sponsor: Department:

Occupying manager: Department:

Date of occupation:

Project status Recommendations

The report must address: The report should consider from the point of view of the
user’s requirements those aspects of the completed
building that are particularly:
- whether the user’s original requirements have been
met. If not: - useful;

- what were the shortfalls; - efficient;

- explanations of these; - surplus to needs;

- if the user’s requirements changed significantly - uneconomic to operate;


during the course of the project:
- and under performing.
- what these changes were;
Recommendations and what should be included and
- whether they were accommodated and, if not: avoided on future projects should be provided in the light
of this assessment.
- are they still required;

- whether the facilities provided are adequate, fall


short of or exceed requirements, appropriate
supporting evidence should be provided;

- identification of outstanding defects and remedial


works; and

- an assessment of whether the building is economic


to maintain and operate, with details of any unsatis-
factory aspects,

18
OTHER GUIDANCES

Applicable CUP Guidances Produced by Property and Buildings Directorate,


Department of the Environment, St Christopher
No 1 Post Tender Negotiation House, Southwark Street, London SE1 OTE

No7 Project Sponsorship: Planning and Progress Contracting for Works Services
Monitoring
Contractor Management
No 11 Debriefing Information System Handbook

No 12 Contracts and Contract Management for Consultants Register Guide


Construction Works
Guidelines for the Design of Government Buildings
No 13 The Selection and Appointment of Works
Consultants Environmental Action Guide
(including series of six notes)
No 15 Estimating for Works Projects
Historic Buildings Conservation Guide
No 17 Quality Assurance in Building and
Construction Fire Precautions Guide

No 19 PTN Update The Government Property and MOD Fire Standards

No 20 The P&S Function and Works Projects Standard Fire Precautions for Contractors

No 25 Cost Management for Works Projects Better Briefing means Better Buildings by J J W O’ReilIy
published by the Building Research Establishment of the
No 26a Selection of Works Contractors: Department of the Environment.
Prequalification and Tendering Procedures

No 26b Selection of Works Contractors:


Bid Evaluation and Award

No 33. Project Sponsorship

No 36 Contract Strategy selection for Major Projects

No 38 Approval of Works Projects

No41 Managing Risk and Contingency for Works


Projects.

To be published

Procurement of Project Management and other consul-


tancy services (excluding design) including model form
of contact.

19
CONTENTS Commencing 8. OCCUPANCY REVIEW 8.1
paragraph
Management 8.3
1. INTRODUCTION 1.1 Contents 8.4
Reporting 8.5
2. SCOPE 2.2 Benefits 8.6

3. DEFINITION OF TERMS 3.1 9. INDEPENDENT REVIEWS 9.1

4. GENERAL PRINCIPLES 4.1 When an independent review is necessary 9.3


Procedures 9.5
The project cycle 4.6 Review of project management controls 9.7
Evaluations, project Systems evaluation 9.8
stages and approvals 4.8 Compliance testing 9.10
Timing 4.9 Tests of detail 9.12
Topics for evaluation 4.12 Scope of review 9.16
Duration 4.14 Management 9.19
Management 4.17 Contents 9.22
Reporting 4.19

5. PRE-PRO JECT EVALUATION 5.1 ANNEX Page

Timing 5.3 A Topics for evaluation 12


Management 5.4
Contents 5.5 B Project evaluation summary 17
Reporting 5.6
Benefits 5.8 C Occupancy review summary 18

6. IN-PROJECT EVALUATION 6.1 D Other guidances 19

Management 6.6 E Contents 20


Contents 6.7
Reporting 6.8
Duration 6.10
Benefits 6.13

7. POST-PROJECT EVALUATION 7.1

Timing 7.3
Management 7.4
Contents 7.5
Reporting 7.6
Benefits 7.10

20