This action might not be possible to undo. Are you sure you want to continue?
MERIT Program Evaluation & Monitoring Framework
Revised August 2005
Published by the Crime Prevention Division NSW Attorney General’s Department Level 19 Goodsell Building 8-12 Chifley Square Sydney NSW 2000 Telephone 9228 7206 Fax 9228 8559 Website: www.lawlink.nsw.gov.au/cpd
ISBN: 0 7347 2823 9
TABLE OF CONTENTS
1.0 1.1 1.2 1.3 1.4
Introduction Purpose of program evaluation and monitoring Program description MERIT program evaluation and monitoring group MERIT database
Data quality and availability Program resources
4 4 6 6
2.0 2.1 2.2 2.3
2.3.1 2.3.2 2.3.3 2.3.4 2.3.5
Monitoring Purpose of monitoring program process and outcomes Summary of monitoring reports Monitoring process
Operations and activities Performance Target participation Service utilisation Court coverage
9 9 10
11 11 11 12 12
Program, post program and unintended outcomes
3.0 3.1 3.2 3.3
3.3.1 3.3.2 3.3.3 3.3.4 3.3.5
Evaluation The purpose of evaluating program process and outcome Summary of evaluation reports Evaluating process
Program fidelity Program equity Stakeholder attitudes Resources Follow up care
12 12 13
13 13 13 14 14
List of Tables Table 1.0 Summary of monitoring strategies and reports Table 2.0 Summary of evaluation strategies and reports
List of Figures Figure 1 Evaluation map
1.1 Purpose of program evaluation and monitoring The measure of performance is a key element in the NSW Government’s commitment to managing results in the public sector. A program’s performance can be effectively measured and assessed through ongoing monitoring of the program and by formal evaluation. Social programs such as MERIT aim to:
ameliorate a social problem or improve social conditions. It follows that is appropriate for the parties who invest in social programs to hold them accountable for their contribution to the social good. Correspondingly, any evaluation of such programs that is worthy of the name must evaluate – that is, judge – the quality of a program’s performance as it relates to some aspects of its effectiveness in producing social benefits (Rossi, Lipsey and Freeman, 2004:17-18).
Evaluation and monitoring are crucial in distinguishing effective social programs from ineffective ones. The commitment to engage in evaluation of the MERIT program is reflected in the existence of a formal evaluation strategy and a working party (the MERIT Program Evaluation Group) that exists to specifically develop and implement the evaluation strategy. This document provides an overview of the MERIT program evaluation strategy.
1.2 MERIT Program description As part of the Government Plan of Action to implement the resolutions of the NSW Drug Summit, a commitment was made for a range of innovative schemes to be introduced and trialled. These schemes allow police and courts to exercise discretion and divert offenders from prosecution and custodial sentences. The schemes aim to break the drugs-crime cycle by trying to engage offenders at various stages in treatment and rehabilitation programs as a way of addressing their drug dependency. The MERIT program is one initiative that responds to the recommendations of the NSW Drug Summit. MERIT is an inter agency initiative between the NSW Attorney General’s Department (lead agency), the NSW Department of Health (Centre for Drug and Alcohol) and NSW Police. MERIT scheme has two levels of management: central and regional. At both levels of management the membership involves a range of key stakeholders that may vary between sites and over time. The NSW Attorney General’s Department convenes the MERIT Statewide Steering Group which is responsible for the overall policy and design of MERIT as well as supervising the implementation of MERIT across the state. MERIT targets defendants with a demonstrable drug problem who are eligible and suitable for release on bail and who are motivated to engage in treatment and rehabilitation for their illicit drug problems. An important feature of the program is that participation is voluntary and that the defendant is not required to enter a plea to the
charges that bring them before the court in order to participate in the program. A person’s agreement to become involved in MERIT is not an admission of guilt to the offence(s) for which s/he has been charged. Once charged, the defendant is eligible for the MERIT program if they:
• • • • • • • • • are eligible for release on bail have a demonstrable illicit drug problem are an adult consent to voluntarily participate in the MERIT program are not involved in offences related to violence, sexual offences or wholly indictable drug offences have no matters pending of a violent or sexual assault nature have a treatable problem have been deemed suitable for the program usually reside in the defined treatment catchment area, and are approved to participate in the program by the Magistrate
MERIT provides for referrals from police at the time of arrest, from solicitors following arrest, or at court by the Magistrate. Referrals from other sources are considered. The outcomes of the MERIT program may be divided into specific program outcomes, post program outcomes and unintended outcomes. Program Outcomes The intended outcomes for the MERIT program are to:
• Decrease drug related crime by participating defendants for the duration of the program • Decrease drug related crime by participating defendants following program completion • Increase community protection • Improve health and social functioning for the duration of the program and in the post program period • Reduce sentences due to better rehabilitation prospects
Post-Program Outcomes Once a defendant has completed the MERIT program, it is hoped they will:
• • • Cease their drug related crime Cease their illicit drug use Make lifestyle changes that support a drug-free existence, and Improve their health and social functioning
Unintended Outcomes Possible unintended program outcomes are:
• • Increased remand numbers if offenders fail to comply with bail conditions Increased number of court appearances by MERIT clients due the provision of judicial supervision Added strain on some of the Area Health Services to provide the full range of drug treatment services
1.3 The MERIT Program Evaluation Group The MERIT Program Evaluation Group was established to oversee MERIT evaluation and monitoring. The Evaluation Group meets on a quarterly basis with representatives from the NSW Attorney General’s Department, NSW Health and NSW Police. This groups reports to MERIT Statewide Steering Group. The role of the Group is to develop and implement the evaluation strategy. The specific activities are: 1. Strategic issues To share recent research literature in the field of drug and alcohol diversion To identify strategic challenges in MERIT evaluation strategy and develop strategies to oversee these challenges To identify challenges in current evaluation projects and propose ways of overcoming them 2. Monitoring To monitor the MERIT evaluation strategy To monitor existing evaluation projects To monitor the health and criminal justice outcomes of MERIT across all MERIT Teams To monitor data collection across NSW Area Health Service MERIT sites To review and revise the MERIT data dictionary
1.4 The MERIT database A specific database, the ‘MERIT Information Management System’ (MIMS) provides data for the measurement of process and outcome evaluation indicators. A data dictionary provides a description of each item recorded. The database provides the basis for the monitoring reports as well as fulfilling reporting requirements to the Commonwealth. Each year a complete set of program data from the time of program inception is downloaded by the MERIT database manager in order that a whole of program analysis might be made. This data is analysed and included in the MERIT Annual Report completed by the MERIT project officer. The MERIT Evaluation Group monitors the extent to which the data measures may effectively be used to address evaluation questions. The continual monitoring of data in this way means that ideas concerning better measurement are identified and addressed. At times supplementary information from external sources must be used in conjunction with MIMS data to adequately answer evaluation questions. For example, the Bureau of Crime Statistics and Research annually match their data on criminal records with MERIT client data in order to answer evaluation questions concerning recidivism patterns.
1.4.1 Data quality and availability The quality of data is affected by a number of factors including the consistency of recording of data between MERIT program sites. Training sessions have been held to promote consistency. Ongoing training and updates are provided by the fulltime MERIT database manager. All stakeholders in the MERIT program have a responsibility to ensure that there is provision for program evaluation that includes measurement of the program’s intended health and criminal justice outcomes. The data collected for this purpose are largely undertaken by health workers and may generally be made available to other stakeholders with adequate individual privacy safeguards.
1.4.2 Program resources Program resources include a budget allocation, special purpose funding and additional monies approved on an ad hoc basis following from budgetary roll overs. There are three identified positions across the NSW Attorney General’s Department and NSW Health working on MERIT evaluation.
1.5 Evaluation map An activities map of the evaluation strategy is presented in Figure 1. This map is used to report on progress for current studies and is updated on a regular basis to reflect new evaluation strategies.
MERIT EVALUATION ACTIVITIES
Operations and activities Activity & Quarterly reports*
Program Outcomes Annual Reports***
Program fidelity Process study***
Program outcomes Lismore evaluation** Health outcomes study*** Criminal justice outcomes study**** Efficiency Cost-Benefit Study (Lismore Evalution)** Cost-effectiveness Study*** Costing Study***
Performance Key performance indciators ***
Post Program Outcomes Annual Reports***
Equity Aboriginal Participation Study** Women**** Cultural background & Mental Health**** Stakeholder attitudes Magistrates Survey** Police Survey*** Legal Practitioner Survey**** Follow up care Aftercare study***
Target participation Activity reports*
Unintended Outcomes ****
Service utilisation Residential rehabilitation*
Participant satisfaction Lismore Evaluation (participant survey) Health Outcomes Study (questions to participants)
* Ongoing projects ** Completed projects *** Projects in progress **** To be developed
2.1 The purpose of monitoring Monitoring reports seek to convey timely and relevant data regarding particular indicators of the program’s outputs. These serve to highlight areas of variation in the program implementation between different health areas as well as monitor critical indicators of the program’s operation. Ongoing monitoring will lead to the establishment of key performance indicators and benchmarks.
2.2 Summary of monitoring reports Table 1.0 presents a summary of the monitoring components, indicators and reporting formats for the MERIT program.
Table 1.0: Summary of monitoring strategy and reports
Monitoring component Monitoring of process Indicators Operations and activities Indicators Referral numbers and rate Source of referrals Acceptances into MERIT Program completion Breaches Data issues (data dictionary, NGO involvement) Target participation Program availability Monitoring of outcomes Court coverage Format Monthly activity & Quarterly reports Monthly activity & Quarterly reports Monthly activity & Quarterly reports Monthly activity & Quarterly reports Monthly activity & Quarterly reports Ongoing reports Monthly activity & Quarterly reports On request Annual reports
Program, post Decrease drug related crime by program and unintend participating defendants for the duration outcomes of the program prospects Decrease drug related crime by participating defendants following program completion Increase community protection Improve health and social functioning for the duration of the program and in the post program period Reduce sentences due to better rehabilitation Cease their drug related crime; Cease their illicit drug use;
Annual reports Annual reports
Annual reports Annual reports Annual reports
Improve their health and social Annual reports functioning. Health outcomes study Make lifestyle changes that support a drug-free Annual reports existence Increased remand numbers if offenders Annual reports fail to comply with bail conditions Increased number of court appearances Annual reports by MERIT clients due the provision of judicial supervision; Added strain on some of the Area Health Servic Annual reports to provide the full range of drug treatment servic
Monthly Reports The Centre for Drug and Alcohol, NSW Department of Health produce monthly and quarterly reports after individual sites have sent their data at the end of each month. The aim of these reports is to measure the level of program activities across health areas in the period. Measures include:
• • • • • • Number referred for assessment Number partially assessed Number accepted Number ineligible, declined or did not appear Number breached, removed or withdrew voluntarily Number remaining on program Number completed program/court requirements
The monthly report provides data on the cumulative numbers who have been involved in the program at the State and individual Area Health Service level. It is possible to derive rates of referral acceptance and program completion rates. These comparisons need to be interpreted with care as the MERIT program in each area varies in terms of development, resources, demographics, and offender profiles.
Quarterly Reports Quarterly monitoring reports include a more extensive and detailed coverage of program measures. Indicators include the source of referral, gender, Aboriginality, principle drug problem, offence type, number with previous gaol sentences, numbers receiving particular types of health services and court findings. As well each health area reports on its MERIT program implementation.
Annual Reports The NSW Attorney General’s Project Officer (Monitoring and Evaluation) produces an Annual Report on MERIT. As well as summarising and discussing monthly and quarterly reports, the Annual Report includes data collected for outcome monitoring. Differences between area programs are analysed as are MERIT outcomes for particular groups such as women and Aboriginal and Torres Strait Islanders. The Annual Report presents an overview of trends in the MERIT program and includes some statistical analyses associating indicators with outcomes. The NSW Health Department produces an Annual Report for the Commonwealth focusing on health outcomes.
2.3 Monitoring process In order to ascertain whether a program is operating as it intended it is necessary to continually and systematically monitoring how the program is actually working in practice, that is, the program process. The monitoring of these indicators will, over time, lead to the development of key performance indicators that may then become a standard for which to compare performances. Thus process monitoring involves the repeated measurement of performance indicators over time.
Monitoring operations and activities
Referral Numbers and Rate Referral rates are key indicators of demand for MERIT. Areas where there are particularly low numbers of referrals may indicate problems with the process. This could reflect on the level training or non-acceptance of the scheme by referral agencies within an area. Alternatively it may be a true reflection of low demand due to differing patterns of drug use and offending in different areas. Where significant differences appear in the monitoring reports, further investigation may require a more qualitative approach involving in-depth interviews with the parties concerned, such as magistrates or police. This may highlight attitudes, practices and resource issues that are producing a pattern of low referral rates. Source of Referrals The source of referrals is a potentially important process indicator. The source of referral to the program can provide an indicator of the way the program is working within an area. Low rates of referral from a particular source may indicate that further information or training is required. Acceptances into MERIT As well as the acceptance rate it is important to monitor reasons for non-acceptance onto the program. This may highlight areas where resources or training are required to make the program more efficient. The proportion of acceptances and nonacceptances is monitored by health area and recorded in monthly and quarterly reports. Further analysis of demographic characteristics, drug use and criminal history provides a description of MERIT participants. This can be used to identify particular groups who are not being accepted into the program as well as those that the program is reaching. Breaches Breaches are monitored as part of the standard reporting system. Significant variability in the relative proportion of breaches in each area could indicate operational program differences requiring further investigation. Higher rates of breaches may be the result of the nature of participants referred to the program in a particular area, for example differing criminal histories and/or levels of drug offending. They may also indicate significant variation in practice.
2.3.2 Performance As well as indicating area differences in the program, the monitoring process can provide information on program quality and standards. Key performance indicators are currently being devised for the MERIT program.
2.3.3 Target participation The characteristics of clients referred, participating and completing MERIT is monitored by the collection of data on demographic characteristics. These are reported in the monthly and quarterly data reports.
2.3.4 Service utilisation and MERIT expansion The number and nature of residential rehabilitation services is monitored. The number of courts operating the MERIT program is monitored and reported on at the MERIT Statewide Steering Group.
2.4 Monitoring outcomes While it is important to monitor process related indicators, it is also essential to systematically monitor program outcomes. Monitoring of program outcomes allows the documentation of progress in meeting goals. Monitoring outcomes helps to generate information for program managers in order to efficiently administer and improve programs. The monitoring of outcomes is not a substitute for formal impact evaluation but rather a gauge of progress to date.
2.4.1 Monitoring program, post program and unintended outcomes Progress on meeting selected program outcomes are reported periodically in the MERIT Annual Report. The data collected in the Annual Report may then be compared to data collected in prior years in order to gain a sense of change over time. Trends in the meeting of program outcomes may be identified. The trends in data may highlight specific issues that can be acted upon, or may highlight areas of future evaluation research.
3.1 The purpose of evaluation The systematic and continual monitoring of program process and outcomes is essential in providing a gauge of the working nature of a program and procedural issues arising on an everyday basis. However to assess whether a program is achieving its intended outcomes it is essential to conduct more formal impact evaluations. It is only then that a reliable and valid assessment of the worth of the program may be made. Ensuing decisions, with significant social implications, may then be justifiable as they are based on a legitimate foundation of research.
3.2 Summary of impact evaluation reports Table 2.0 provides a summary of the components of both process and outcome based MERIT evaluation. While impact studies will reflect evaluation of each of the program’s outlined objectives, formal evaluation of a program’s process often cover issues such as program fidelity, equity, attitudes, financial considerations and follow up care. Other process evaluations may occur premised on the identification of specific issues in the monitoring reports, or premised on specific requests from stakeholders.
Table 2.0: Summary of impact evaluation strategy and reports
Evaluation component Evaluating process Indicators Program fidelity Key Question Format Process Study Aboriginal participation study What does the program look like in practice? Does it match the intended process? Program equity Are clients accessing the program in a fair and equitable manner? In particular, given research on low referral and completion rates of Aboriginal clients in diversion program, what is Aboriginal participation and completion like in the MERIT program? Stakeholder attitudes What are the attitudes of key stakeholder groups (in particular Magistrates and Police) to the MERIT program? Resources What are the financial issues in the practice of MERIT? Do they match the intended costs? Follow up care What are the follow up needs for MERIT clients? Are these being met in the current process? Program, post program To what extent does the MERIT program lead unintended outcomes to the intended criminal justice outcomes? To what extent does the MERIT program lead to the intended health outcomes?
Magistrates, Police and Legal Practitioner survey Cost effectiveness study Aftercare study
Lismore evaluation report Health outcomes study
3.3 Evaluating process Evaluation of process related variables is important in helping determine whether a program matches the program model and therefore will likely achieve its program goals and objectives.
3.3.1 Evaluating program fidelity To complement existing process evaluation indicators an evaluation of MERIT at a local level is being undertaken. The MERIT Process/Standards Study will assist in identifying current process and practice challenges being experienced by MERIT across the State, and will assist in developing program standards and performance indicators. This information will also assist in the review of the MERIT Operational Manual to best meet the current operational needs of MERIT.
3.3.2 Evaluating program equity- Aboriginal participation Part of any process evaluation strategy should be the specific study of participation. This data will be used in particular to answer questions about the equitable participation of groups traditionally under represented in diversion programs. The Aboriginal Participation Study investigates the participation and completion of Aboriginal clients in the MERIT program. The findings from this study will highlight areas of the program that might be improved in order to increase access and completion rates.
3.3.3 Evaluating stakeholder attitudes – Police and Magistrates It is important in process evaluation to study the views of those stakeholders who have a specified role in the program’s implementation. In the MERIT program both Magistrates and Police play important roles in referring, and for Magistrates overseeing, the program. A study into Magistrates attitudes to MERIT has been completed, while the Police attitudes survey is in progress.
3.3.4 Evaluating costs and benefits It is important to examine whether each of the program’s benefits exceed their costs, and whether diversion into treatment offers economic benefits to the community in terms of reduced crime and improved health and social functioning as compared with the costs of the status quo where there was no diversion option for eligible offenders. A cost benefit analysis was undertaken as part of the Lismore evaluation and provided an assessment of the cost savings of diversion into treatment. NSW Health is currently developing a Health Costing study. 3.3.5 Evaluating follow up care Process evaluation of programs with significant social implications should also include study of the follow up care available to clients. This will help determine the likely long term effects of program outcomes. The MERIT Aftercare Study addresses this part of the process evaluation. This study will examine evidence on best practice in aftercare for offenders and non offenders following drug treatment. Following this review a trial model of aftercare may be developed, implemented in one or two regions and evaluated.
3.4 Evaluating outcomes Formal impact evaluation of all program outcomes is important in gauging whether the program meets its stated objectives. Evaluation of outcomes attempts to discern the extent to which the change in outcomes is attributable to the program itself.
3.4.1 Evaluating program outcomes A full impact evaluation was carried out on the MERIT pilot program in Lismore (Passey, 2003). The evaluation examined a number of program outcomes and included study of: program and participant profiles, court outcomes and recidivism, health and social functioning, participant perspectives, stakeholder views, economic assessment and a review of legal issues. A separate Health Outcomes Study is in the process of being conducted by NSW Health and will focus on improved health and social functioning for the duration of the program and in the post program period.
Passey, M (2003). Evaluation of the Lismore MERIT Pilot Program. NSW Attorney General’s Department. Rossi, P.H., Lipsey, M.W., & Freeman, H.E. (2004). Evaluation: A Systematic Approach. 7th Edition, Sage Publications, Thousand Oaks.