You are on page 1of 46

Foundations in Monitoring and Evaluation

Niamh Barry February 2010

Session outline
1. What is M&E? 2. Why do M&E? 3. What happens without M&E? 4. Tools and Approaches to M&E 5. Logical frameworks 6. Performance indicators 7. Practical work 8. Data collection and MOV’s 9. Reporting and reviewing 10.Documentation

What is M & E?
• Monitoring – routine & regular collection , analysis & use of information to track progress towards goals • Evaluation : assessment of the extent to which a project is achieving or has achieved its stated goals  Or • Monitoring : Improvement and development of the community. • Evaluation : A process design to show the relationship

 

What's the difference? – Timing – Analysis What do we M & E?  Inputs  Process  Progress towards goals  Impact of actions against stated goals

What M&E is can not do
• M&E ‘s fundamental tenet is to inform judgments about project performance

• Measure things that are immeasurable! i.e. if all your activities do not have a goal then how can progress be measured???

• The key is in program design if your program design is weak and inconsistent then you will not have a quality M&E framework

• First question should look at your program ‘is this the best way to achieve our goals’ then you can answer the question of how to best M&E the program

• M&E should always be built in – it must work with the program

Why do M&E?
• M&E of development activities provides organizations with a better means for learning from past experience, improving service delivery, planning and allocating resources, and demonstrating results as part of accountability to donors & partners.

• Within the development community there is a strong focus on results— this helps explain the growing interest in M&E.

Why do M&E, when we should focus on implementation?
   

“If your not keeping score, your only practicing”
 

What happens without M&E?
• • • • • If you do not monitor progress, you do not know if you can or are succeeding If you do not monitor progress, you can never recognize program failures OR success If you do not evaluate Impact, you do not know if you have succeed or failed You prevent learning and sharing If you never M&E you may repeat programs that have no impact and are wasting organizational and target participants time, resources and donor funds.

• •

Tools and Approaches to M&E
• Various tools and methods exist to begin to help us M&E, the choice depends what is being being M&E (ed)

Progress/impact indicators  The logical framework approach  Formal surveys – Baselines  Needs assessments  Participatory methods (FGDs) Cost-benefit and cost-effectiveness analysis  Impact evaluation

• Some of these tools & approaches are complementary; some are substitutes. Some have broad applicability, while others are quite narrow in their uses.


What do we use?


  

A combination of logical framework and performance indicators is the most commonly used in long term programming and planning


Causal Pathway (same but with out indicators)

Causal Pathways
• This is the initial step in creating the logical framework • It is a program planning tool that ensure that program activities will cause the desired impact • This is a very important step – all inputs must be enough to produce activities and these activities must eventually produce the goal • It’s logic is based on cause and effect or the ‘theory of change’ but you must be careful to think of challenges during this exercise i.e. you can not assume condom distribution will lead to condom use.

Simple Logical Framework
Goal Objectives Indicators MoVs Assumptions





• •

Input-resources needed in the project Activities- process of using inputs to achieve results Output –the results of the completed project activities
O u t si d Outcome/objectives-takes place when the project targets uses e p outputs as anticipated r o j e c Impact/goal-long term t c change/effectiveness of the project o n t r Many other terms are used-but effectively mean the same thing …goalso l objectives-results-effects-purpose…the point is to know when to

 

 

  

  

• •

use them in the process

M&E Components

The Soup Analogy

HIV Program


Chicken, vegetables, Broth, Spices Condoms, Testing Kits, Staff time, Pot, Stove Transport, Funding

Activities Outputs

Chopping vegetables, cooking chicken, boiling and stirring Soup has been made

Training Delivery of service VCT available Condoms available Knowledge on transmission Trained staff Increase in condom use Increase in uptake of VCT Changes in sexual behavior Decrease in STI trends Decrease in HIV transmission, morbidity and mortality Change in social norms Economic impact


Appeasement of hunger Satisfaction of eating soup Improved nutritional status of person eating soup


Important Notes on Log Frames
• You must know what you want to set out to achieve before doing the log frame, is your programme: • Causal: strategies aimed at directly changing a situation • Persuasive: strategies aimed at influencing a situation or thinking about a situation • Supportive: strategies aimed at influencing the environment with which a focal problem is situated • If you do not know exactly what you want to do then you can’t realistically expect it to be M&E(ed)

Important Notes on Log Frames
Remember that change happens of a period of time (Theory of Change) That is, how a particular project is anticipated to contribute to social change through time. Often and incorrectly the vertical logic is used to systematically disaggregate the problem – without consideration of time

Right and Wrong Approaches
Goal :a wall Objectives :the bricks that make up the wall Outputs :the sand and cement that make up the bricks Activities :the molecules that makes up the grains of sand and cement
 

Goal :significant and lasting changes anticipated in the lives of the ultimate beneficiaries (e.g. households with improved livelihoods) Objectives : Change to action – peer educators using knowledge and increased capacity Outcomes :changes in knowledge /attitude /practice influenced among intermediary stakeholders (e.g. Peer educators with better capacity) Outputs :deliverables of the project implementing team aimed to initiate the change process (e.g. training of peer educators) Activities : Trainings, follow ups



Performance Indicators
• Performance indicators are measures of inputs, processes, outputs, outcomes, and impacts for development projects, programs, or strategies. • Only beneficial when supported with sound data collection— perhaps involving formal surveys—analysis and reporting, • Indicators enable us to track progress, demonstrate results, and take corrective action to improve service delivery. • Participation of key stakeholders in defining indicators is important because they are then more likely to understand and use indicators for management decision-making.

Developing indicators
• Working with indicators is not difficult • Indicators are no more than criteria that will help us monitor and evaluate our project • To develop them, we have to ask ourselves:

What aspect will tell us whether or not something went well?

Whether we achieved our objectives or not?

Whether we followed our plans?

Example of Performance Indicators
• A program has the goal to increase community knowledge of HIV prevention and access to HIV testing treatment and care through targeted health talks • What could the performance indicators of this programme be?
• Number of community members reached • Improvement in community members knowledge on HIV • Improvement in number of people testing • Improvement in the number of people receiving treatment • Increase in condom use • Reduction in HIV transmission

Important Note on Indicators
• When we talk about any of the below type of indicators:  Increase  Decrease  Improvement  Change  Etc • We can only measure these if we have a BASELINE to compare it with • A baseline is information on whatever we wish to impact prior to our program intervention.

Important note on Indicators

Keep them SMART

S – Specific M - Measurable A – Achievable R – Realistic T – Timely
 

Resources to Help on Indicator Development


Devising your own Logical frame work


Case study
• Country XX has one of the highest burdens of TB worldwide. XXX has quite a number of challenges that correspond to this high burden of TB:

v Limited clinical infrastructure v Poorly trained clinic staff v Low health seeking behavior in the community v Stigma regarding TB v Low knowledge of TB v Low adherence/Lost to follow up for those on medications

• An TB NGO establishes itself in country XX and wants to decrease the incidence of TB. Of course to do this it most address all of the challenges. • The NGO’s overall Goal is the decrease the incidence of TB mortality and morbidity in country XX • • Design a logical framework with activities, Outputs, Objectives and the stated goal. • Start with your goal and then work with activities up! • Draft a number of indicators for each of these steps.

Key Questions to Consider
• How will you track or measure the success of this project?

• What key indicators will allow to measure results of the work?

• What are some sources and/or means of data collection for these indicators?

• What challenges do you anticipate? What assumptions do you have? •


Means of Verification (MOV)
• For every indicator you create you must be able to verify it. • That is you must be able to collect data to prove that it is true.

Example Indicator: Decrease in the number of TB patients lost to follow up MOV: National TB registers Indicator: Increase in TB/HIV knowledge among clinic staff MOV: TB/HIV Knowledge questionnaire • • Essentially these are the data collection tools you will use to monitor progress and evaluate impact .
 

Data collection Tools
• • • • • • • These are integral to any M&E Framework with them you can not monitor progress or evaluate impact. You must ensure all data collection tools capture the indicators you have established When designing data collection tools – Less is more Do not ask for unnecessary information Do ensure the tool is user friendly Do pilot test the data collection tool (if possible) before you finalise it Do design the tools as you are also designing the indicators – this will tell you if your indicator is SMART

Important Notes
• • Does your data collection tool also need to collect baseline information? Does each indicator require a new data collection tool? Are there MOVs such as National Registers, KAP surveys already available with information on your target population. • Remember: data collection tools are anything that collects information on your indicators

Attendance lists Training evaluations

National registers

Questionnaires Monthly reports



• One of the key reasons for implementing an M&E framework is because of reporting requirements • Most funding agencies require this annually • Additionally the organization should have its own reporting structure, not only to managers but to the beneficiaries and all stakeholders

Reporting ctd
• Monthly reports on progress are recommended to ensure that the programme is progressing without challenges and to review progress made against targets • Monthly reports are more desirable as they will detect any challenges early • Additionally data collected should be entered into a program data base

• All programs need an internal review and replanning process at least yearly. • Recommended that this take place as a prelude and guide to implementing new work plans • Reviewing should focus on program’s overall achievements, challenges and emerging issues, based on information obtained through the M&E processes described in the preceding slides. • Through this process, work plans are reviewed • The process should include Project partners and beneficiaries


• • Information provided by a strong M&E framework is invaluable Organizations should always documenting: good practices, success stories and case studies. • • Case studies will facilitate the sharing of ideas for action. They will provide information on what people are doing and what's working, what communities have learned from their experience and how it made a difference. They will provide inspiration and show what's possible. • Disseminating the M&E findings including the documentation should form a critical part of the overall M&E activities

Concluding Messages
• • • • • • Keep it simple Build it in Meaningful as well as measurable Develop evaluation capacity Learn and do it yourself!

Thank you!