COASTAL Project - Case No.

80068 Monitoring and Evaluation Plan

Objectives
Monitoring: The Monitoring process in respect of the COASTAL Project will be based upon the collection, verification, collation and reporting of data in respect of all relevant aspects of financial and service/participant activity and performance. In respect of monitoring the latter, the following indicators will be used:

Outputs
Total Participants Female Participants Participants Economically inactive and unemployed Economically inactive Unemployed NEET participants BME participants Older participants Participants with work-limiting health condition or disability Lone parents Projects using soft outcome measurement systems Projects integrating sustainable development into awareness raising, education and training programmes Employers assisted or financially supported

Results
Participants gaining qualifications - Economically inactive and unemployed Qualification levels: Basic Skills At Level 2 At Level 3 Level 4 and above Participants entering employment - Economically inactive and unemployed Participants entering further learning - Economically inactive and unemployed Participants gaining other positive outcomes - Economically inactive and unemployed Employers adopting or improving equality and diversity strategies and monitoring systems

The Project will develop an on-line reporting facility, linked to the Project website. All joint sponsors and external delivery agents will have direct access to this process. The system will be an Access database, based upon a previous established system developed by one of our Joint Sponsors, for use in recording monitoring information in respect of Objective 1 project activity. The format, fields etc within this database will mirror the reporting requirements of WEFO. Discussions are underway to plan for the establishment of the project website and for conversion of the existing database into SQL format for on-line use. It is

recognised that we will be unable to commission any work to be undertaken to develop or establish this system before the agreed start date of the project. Therefore, we are planning to introduce an interim paper-based system, to capture all activity in the early months of the project. This paper-based system will be such that the data contained will be easily transferable, retrospectively, to the electronic system once established. The information gathered within the envisaged system will provide all of the category breakdown data both at participant and enterprise level, to meet the WEFO monitoring requirements. Each joint sponsor, i.e. Local Authority, will be responsible for the collection of all relevant data in respect of activity within their area, including that of any locally procured external services. This information will then be submitted to the Regional Management Team, who will be responsible for the collation of the data received from each sponsor, plus that of any regionally procured external services, for submission to WEFO. All data collection will be undertaken in line with the reporting timescales required by WEFO. Evaluation: In a major project such as COASTAL, it is recognised that, in addition to a range of on-going, internal quantitative and qualitative monitoring and evaluation, including soft-outcome measurement, formal evaluation, undertaken by an independent, external evaluator will be an essential requirement. Accordingly, appropriate levels of funding have been included in the project costs to include this. It is anticipated that this evaluation process will combine both formative and summative techniques in order to deliver a full qualitative and quantitative evaluation. In conducting the evaluation, we would anticipate that an initial baseline position will be established, wherever possible, and agreed with the contracted evaluator. At appropriate and agreed periods during the lifetime of the project, the following evaluation techniques are examples of those to be undertaken: Scoping depth of engagement Defining performance indicators in detail Management and officer interviews Full Case Studies Observation of pockets of activity Direct participant contact through; Questionnaires, Focus Groups, Individual contact interview (agreed % random sample) Reports received from agreed % of project activity areas to assess activity Review and monitoring of project Web site activity and content Periodic interim summary reports Final Report Procurement of an appropriately experienced evaluator will be one of the first procurement exercises to be undertaken, in order to ensure that, wherever required, an accurate baseline assessment can be identifed and agreed.

Lessons Learned from Previous Projects

Each of the Project Joint Sponsors has had varying degrees of project delivery and monitoring. In particular, the experience of the Lead Sponsor, City & County of Swansea, is substantial, having experience of the continuous delivery and monitoring of the Partnership for Employment (P4E) series of Objective 1, ESF funded projects from 2001 to date. It is this series of projects that have formed the basis upon which the COASTAL Project has been developed.

As Lead Sponsor, City & County of Swansea will be responsible for the establishment of the Regional Management Team, which will in turn develop and implement the monitoring mechanisms for use by all of the participating organisations, whether joint sponsors, or external delivery agents. The P4E Projects were all based upon the principles of partnership working, and the monitoring procedures adopted were robust, yet flexible enough to accommodate the varying needs of partners from different sectors. The external evaluation of the P4E Projects highlighted the project management and its processes and procedures as exceptionaly good practice, which had made a major contribution to the successes demonstrated by the projects. The COASTAL project will adopt and further develop these same procedures. There has also been a range of experience in respect of project evaluation across a number of the sponsors. The planning and undertaking of an evaluation, in respect of a project serving individuals experiencing serious illness, needs to take a number of additional factors into account. The breadth of knowledge and understanding required by the evaluating team is often much greater than that required to evaluate a mainstream project. The needs of the participants must be fully recognised, in particular in respect of issues of confidentiality. When considering participant engagement and interaction, careful selection of the evaluation team can make the difference between an indifferent and poorly informed response and an enthusiatic, honest and open response. Participant feedback and satisfaction is one of the crucial elements in a project of this nature. Other notable lessons learned from previous evaluation exercises are that the evaluation team should not be viewed as having a regulatory or inspectorial role, but one of a critical friend. Experience has shown that good progressive evaluation, at regular, appropriate periods throughout the life of the project, can be an invaluable tool in maintaining drive, enthusiasm and morale whilst also ensuring that the project remains on target.

Implementation Monitoring
Indicators The indicators selected for the project and estimated targets are as follows:

What Outputs will be achieved by the project proposal?

Total Participants Female Participants Participants Economically inactive and unemployed Economically inactive Unemployed NEET participants BME participants Older participants Participants with work-limiting health condition or disability Lone parents Projects using soft outcome measurement systems Projects integrating sustainable development into awareness raising, education and training programmes Employers assisted or financially supported

Estimated number to be achieved 9,020 4,100 9,020 9,020 0 740 225 2,542 8,610 220 1 1 20

What Results will be achieved by the project proposal? Participants gaining qualifications - Economically inactive and unemployed Qualification levels to be gained Basic Skills At Level 2 At Level 3 Level 4 and above Participants entering employment - Economically inactive and unemployed Participants entering further learning - Economically inactive and unemployed Participants gaining other positive outcomes Economically inactive and unemployed Employers adopting or improving equality and diversity strategies and monitoring systems

Estimated number to be achieved 5,412

3,116 1,866 307 123 2,870 1,763 6,724

20

The indicators were selected, having intimate knowledge of the range of services involved and the needs of the target participant group. This knowledge has been derived from an extremely wide range of specialist advisors and service providers across the full spectrum of illness and disability services. The detailed service knowledge has been further enhanced by 7 years continuous experience of project management and delivery, of a similar nature, throughout the preceding Objective 1 Programme. In adition to the Framework indicators identified above, the project will also deliver against the projected, headline Programme Level Indicators, within the Convergence ESF Operational Programme. By bringing six Local Authorities together in a collaborative Regional Project, the COASTAL Project will deliver against a further indicator, at this level, delivering 5% of the total Programme performance against the Indicator Collaborative Agreements Between Public Service Bodies which has a 2015 Target of 20. Forecasting The forecasting of outputs against the selected indicators has been calculated through the extrapolation of actual results achieved by previous projects of a similar nature, which had been managed by the Lead Sponsor. The forecasting exercise included recognition of the high level of rehabilitation and preparatory work likely to be required by those participants with particularly complex needs. Therefore the profile shows a relatively slow progress in the first year, building to a peak output in year three. Also taken into account is the time required for some participants to achieve qualifications, particularly at the higher levels. (ALSO SEE ANNEX 1 DELIVERY PROFILE) Monitoring system In the early stages of the Project, all project monitoring data will be recorded and collected manually, until such time as the electronic system has been established. The format for data recording and reporting will be based upon systems tried and tested as part of the project management developed under the preceding Objective 1 activity and will follow all of the WEFO requirements as identified within the guidance. In this initial stage, we will utilise a combination of manual recording, supported by Excel spreadsheets and Access database. Development, in terms of planning, is already well underway to design the

electronic, on-line reporting facility, which will be linked to the project website. All of the relevant forms required for each delivery agent to record and report their participant and financial activity will be provided as templates on the system. This will ensure that all of the delivery agents will record and report their data in a consistent and standardised format, which will allow for collation by the Regional Management Team, with the minimum of difficulty. The Access database that will form the basis for the reporting and collection of participant activity is being adopted from a tried & tested system that was purposebuilt by one of the joint sponsors for use under Objective 1. This system, which is currently free-standing will be converted to SQL format for on-line use. Hard copy backing documentation will be retained, on-site, by all delivery agents, with all relevant documentation and data being forwarded either via the Access database, or in scanned format, to the Regional Management Team. On receipt, all data submitted electronically will be securely stored within a dedicated electronic storage facility held at the Regional office base. This electronic storage will be routinely backed-up. Data quality The project will implement a range of validation measures to ensure data quality. Electronically submitted data will be provided in a predetermined template format, which will ensure consistency and quality. Hard copy backing documentation, relating both to participant and financial activity will be regularly validated both as random manual and electronic checks and as part of the routine checks on data submitted as part of the claims and output reporting procedures. Members of the Regional Management Team will be engaged in regular visits to all delivery sites. Such visits will include a monitoring function, to ensure that data quality is maintained. Participant and enterprise data collection forms will be designed in line with the WEFO guidance and will comply with regard to inclusion of all monitoring requirements, as identified in the WEFO Category Breakdown Guidance, (December 2007). Reporting arrangements Annex 1 shows details of the projected timetable for delivery against the selected indicators. Regular Output Reports will be submitted to WEFO at quarterly periods. With regard to the reporting of progress to key stakeholders, each of the joint sponsors has well-established Senior Officer Project Steering Groups which have been fully involved in the development of the project. These groups will continue throughout the life of the project and will be the main vehicle through which the central Regional Management Team will engage with services in each of the Local Authority areas. It is envisaged that each of these groups would establish a local Operational Group which will meet monthly, with the senior officers forming a Strategic Group which would meet on a quarterly basis. The Regional Management Team will be represented at each of these meetings, to receive feedback from the local services and to provide regional progress reports in return. Additional monitoring information In addition to the wide range of quantitative data that the project will routinely collect, a range of qualitative data will also be collected. This will be gathered in a variety of ways, often linked in with the on-going evaluation process. Mechanisms such as participant satisfaction questionnaires, focus groups and feedback from employers and placement providers will inform project development and planning. A standardised format will also be used, by all delivery agents to assess, measure and monitor Distance Travelled in terms of Soft Outcomes achieved by participants.

Evaluation
In addition to the above range of on-going, internal quantitative and qualitative monitoring and evaluation, including soft-outcome measurement, appropriate costings for a full Independent, External Evaluation have been built into the project. It is anticipated that this evaluation process will combine both formative and summative techniques and will contribute to the on-going planning and project review process, assisting in the appropriate progression and development of the project over its lifetime. Evaluation questions to be addressed and proposed methodology As identified above, the proposed evaluation will be a combination of methodologies, both formative and summative and will entail the use of a number of techniques, including observation, use of focus groups and a range of interview and survey mechanisms and eventual reporting and impact studies. The specific questions to be addressed will cover issues such as: Did the project achieve the stated objectives? If not, why? Were the projected ouputs achieved? If not, why? How has the project impacted upon its target participant group? How has the project impacted upon the local community and economy? Have wider socio-economic factors impacted upon the project and either enhanced or restricted its performance? How efficient and effective were the processes of the project planned and implemented?

In addition to the above criteria to be addressed, the evaluation will also specifically examine how well the project has addressed issues of Equal Opportunities and Environmental Sustainability. Given the nature of the target participant group, i.e. those individuals experiencing barriers related to their serious illness and/or disability, the issues of equality of opportunity are central to the project ethos.

Potential use of evaluation In addition to the potential six joint sponsors, the project will engage with a wide range of external delivery partners, plus a variety of user representative groups. Each of these organisations, agencies and groups will have a stakeholder interest in the project. Without dissemination and implementation of the recommendations, the process of evaluation is meaningless. Dissemination is seen as more than a single event, but more as a consistent process throughout the life of the project. Paying heed to recommendations and acting on them is crucial and evidences the fact that the project is receptive to external feedback and has the capacity to modify its services, delivery and processes as a result. As stated, evaluation feedback will be regularly disseminated to stakeholders as a matter of routine, through the engagement of the Regional Management Team with local project managers and steering groups.

Indicative timetable Detailed planning of a tailored evaluation strategy will be carried out in conjunction with the external evaluator, once identified and contracted. This strategy will cover a full programme and process of evaluation events, as opposed to a distinct start, middle and end scenario and will include its own discrete aims and objectives. It is proposed that the procurement exercise to engage an external evaluator be

one of the first to be conducted. This is seen as important, as one of the initial tasks would be to confirm a baseline assessment of the current position, against which future evaluation can be made. It is hoped that this process, including the production of the detailed terms of reference and the evaluation strategy, be completed by June 2009. It is proposed that the evaluation will provide the Baseline Statement before the end of 2009, an Interim Report in mid 2011, and a Final Evaluation Report in May 2013. Throughout the life of the project, it is anticipated that a number of public events will be organised in a variety of ways. Each of these will have the purpose of highlighting the work of the project and disseminating feedback from the evaluation process. External or internal evaluation Given the complexity and range of the project, it is seen as essential that evaluation be included as an integral part of its work, to be carried out as a combination of internal and external responsibility. Internal evaluation is seen as a crucial part of the work of the local project managers and will dovetail closely with their responsibilities for the setting-up and operation of the monitoring and data collection processes. The consistent, direct contact of the project managers, with delivery agents and participants will enable the building of an on-going formative picture of project activity, whilst the periodic involvement of the external evaluator will allow the capture of appropriate and relevant snapshots of activity. The combination of both these information sources will ensure the widest and most informative basis for evaluation. Our experience in commissioning external evaluation of previous projects has shown the value of having very clear criteria for the evaluation. The process of procurement will need to ensure that the eventual provider has the capacity to undertake the evaluation of a complex major project and has all of the relevant background knowledge and experience, both of structural funded projects and of the diverse participant groups being supported. In order to engage and fully understand the issues involved, ranging from; multiple disabilities and the attendant barriers to learning and employment that they produce, equal opportunities, environmental sustainability etc. it will be essential that the evaluator selected has a breadth of knowledge and experience, or the capacity to source such specialist background knowledge. We would also see a role for the external evaluator in advising on methodology and engagement with project participants, partners and contractors.

Resources planned for each evaluation An indicative, discrete costing of 100,000 has been included, within the Project Management budget for the procurement of an external evaluator. This will cover all directly related costs for the process. In addition to this specific budget, all key staff within the Project Management structure, both at Regional and Local levels will be tasked with facilitating the process of internal evaluation and for liaison and joint working with the externally contracted evaluator.

Monitoring arrangements It is proposed that an Evaluation and Dissemination Group be established, as part of the wider management structure. The decision to amalgamate the two tasks of Evaluation and Dissemination within one group was taken in order to encapsulate both the capturing and sharing of the project story as part of the same process.

The group will primarily be comprised of the lead officers and Project Managers, from each of the local authority areas, the external evaluator, representatives of the external delivery agents, participant representatives and will be chaired by the Monitoring & Compliance Manager from the Regional Management Team. This Group will be initially tasked with the development of the terms of reference, agreement of the Baseline position and development of the Evaluation Strategy. On-going work will then include: the collation and reporting of additional monitoring information gathered internally, receiving of feedback and reports from; o The external evaluator o Stakeholders o Participant representative groups Planning and organisation of public awareness and dissemination events Planning and preparation for the production and dissemination of final evaluation and project closure reports Meetings of this group will take place monthly, in the earlier stages of the project, reducing to quarterly as the project progresses. Frequency may need to then be increased towards project closure in order to manage the project-end reports.

ANNEX 1 Delivery Profile - Cumulative Projected Outputs

Output Projected total Year 1 Year 2 Year 3 Year 4 Year 5 (6 months)

Total Participants Female Participants Economically inactive Unemployed NEET participants BME participants Older participants Participants with worklimiting health condition/ disability Lone parents

9020 4100 9020 0 740 225 2542 8610

950 432 950 0 78 24 268 907

3034 1379 3034 0 249 76 855 2896

5250 2386 5250 0 431 131 1479 5011

7525 3421 7525 0 617 188 2121 7183

9020 4100 9020 0 740 225 2542 8610

220

23

74

128

184

220

Delivery Profile Cumulative Projected Results

Results Participants gaining qualifications Levels gained Basic Skills At Level 2 At Level 3 Level 4 and above Participants entering employment Participants entering further learning Participants gaining other positive outcomes Employers adopting or improving equality and diversity strategies and monitoring systems Projected total 5412 Year 1 390 Year 2 1342 Year 3 2674 Year 4 4043 Year 5 (6 months) 5412

3116 1866 307 123 2870

224 119 0 0 169

772 464 20 10 712

1539 922 143 75 1292

2328 1394 204 82 1883

3116 1866 307 123 2870

1763

98

275

871

1317

1763

6724

484

1667

3322

5023

6724

20

10

15

18

20

20