MONITORING AND EVALUATION HPAD 201 Session 8

At the end of the session, the graduate students should be able to:
1. 2. Review the management cycle and major processes in health programs and projects; Review Total Quality Management principles including systems approach to health care and the Voice of the Customer-Process Model; Describe the hierarchy of objectives in relation to monitoring and evaluation including performance appraisal; Discuss the concepts and principles of control, monitoring and evaluation as supervisory functions; Explain and discuss the following: charts, flow charts, problem tree and other tools and Discuss the salient features of field monitoring visits and outline the major parts of monitoring reports and examples of monitoring and evaluation criteria

Session Plan

3.

4.

5. 6.

3:00 The Management Cycle and Total Quality Management 3:15 Monitoring and Evaluation as to Hierarchy of Objectives and Supervisory Functions including Performance Appraisal 3:45 Management tools for monitoring and evaluation 4:15 Field Monitoring Visits and Developing Monitoring & Evaluation Criteria and Reports 4:45 RECAP 4:55 Preparing for Next Session

MONITORING AND EVALUATION HPAD 201 Session 8

At the end of the session, the graduate students should be able to:
1. 2. Review the management cycle and major processes in health programs and projects; Review Total Quality Management principles including systems approach to health care and the Voice of the Customer-Process Model; Describe the hierarchy of objectives in relation to monitoring and evaluation including performance appraisal; Discuss the concepts and principles of control, monitoring and evaluation as supervisory functions; Explain and discuss the following: charts, flow charts, problem tree and other tools and Discuss the salient features of field monitoring visits and outline the major parts of monitoring reports and examples of monitoring and evaluation criteria

Session Plan

3.

4.

5. 6.

3:00 The Management Cycle and Total Quality Management 3:15 Monitoring and Evaluation as to Hierarchy of Objectives and Supervisory Functions including Performance Appraisal 3:45 Management tools for monitoring and evaluation 4:15 Field Monitoring Visits and Developing Monitoring & Evaluation Criteria and Reports 4:45 RECAP 4:55 Preparing for Next Session

MONITORING AND EVALUATION: HPAD 201 Session 8

Management Cycle: Planning, Implementation and Evaluation and Planning, Organizing, Staffing, Coordinating, Reporting, Budgeting and Evaluation Objectives->Strategies->Targets->Activities Goal (Impact) ---->Purpose---->Outputs (results, deliverables) --->Strategic Activities Monitoring and Evaluation: Inputs -->Processes-->Outputs Outcomes Impact Objectively Verifiable Indicators and Means Of Verification

The Cost of Poor Quality

The tip of the iceberg

Inc ng o re as it d ol r e il m P d b or ta e isa y lit d ed yong st l a W ugs Pro dr ed ess illn

The rest of the iceberg

Wa Dissatisfi tim sted ed e e Prol patients tiv infe onged ga uni ctio Ne m usn m r e ss co ty ssa ce s ne vice Insufficient Frustrate n U ser funds d workers y
O v d erc fa r ci ow lit d ie e s
4

Total Quality Management

Refers to activities that establish the objectives and requirements for quality.

Quality Management System

A set of coordinated activities to direct and control an organization in order to continually improve the effectiveness and efficiency of its performance.

Main Thrust
Defining the processes which will result in the production of quality products and services not detecting the defective products and services after it has been

Universal Dimensions of Quality Care

Efficacious Safe Efficient Accessible Satisfying Socially acceptable Affordable

7

Voice of the Customer (VOC)

Stated and unstated customers needs and requirements Source and methods dependent on: 1.Type of relationship 2.Products and services provided 3.Available IT and other resources 4.Timing (pre-, during and post-) 5.Expected outputs of management

Key Points on VOC

Never monolithic (multiple customer voices); hence need to reconcile and prioritize Product/service development not only marketing experts concerns Customer Representative VS FGDs, surveys Types of customers: current, potential and lead What customers need in terms of

Systems Framework

Public Health Management Course

Structure ( (Input) Process

........

Resources, Infrastructure, Organization, Facilities

....... Activities ........

Objectives Attained?

Outcome

Inputs/Processes:
What goes in to the

program or work

Training of the personnel Funds Equipment/Materials Salary of personnel Organizational set up

Outputs ( deliverables):
Direct services Referral services Number of classes given

health education lectures

Outcome/Impact:
Decrease in gastrointestinal

problems Decrease in the number of absences due to the intestinal parasitism problems Increase in the number of referrals complied with Decrease in the prevalence of infectious diseases Improvement in the utilization of the clinic

Shewart PDCA model: Voice of the Customer Process Model

PDCA cycle - Plan

Plan to improve your operations first by finding out what things are going wrong and come up with ideas for solving these problems.
Public Health Management Course

PDCA- Do
Do changes designed to solve the problems on a small or experimental scale first. This minimises disruption to routine activity while testing whether the changes will work or not.

Public Health Management Course

PDCA cycle- Check

Check whether the small scale or experimental changes are achieving the desired result or not. Also, continuously Check nominated key activities (regardless of any experimentation going on) to ensure that you know what the quality of the output is at all times to identify any new problems when they crop up.
Public Health Management Course

Public Health Management Course

What is a problem
A gap between the ideal and the existing A gap between the standards and actual performance A gap between expectations and the actual performance
Public Health Management Course

Performance Review and identification of Problem

A Problem is a gap between the ideal and the existing

Performance Review of a)Targets b) Standards (inputs, Process, Outcomes) c) Expectations of Clients

Public Health Management Course

How do we identify management problems?

1. Listen to the customers 2. Monitor intended activities 3. Assess Performance 4. Utilize available tools
Public Health Management Course

Customer needs, wants, and expectations

LISTEN TO YOUR CUSTOMERS

Public Health Management Course

Llyod (2004)
BEFORE services assessment : preservice

WHILE they are experiencing the service or care. : point of service assessment. AFTER they have been discharged or after care.
Public Health Management Course

Pre service Solicit point of service feedbac k Post service

Identify customer expectations Designer customer friendly system Identify opportunities for further improvement Manage unsolicited feedback

Measure Organizational Performance

How often do you listen to the VOC

Tools to listen to customer

Tools Pre POS Post

Survey Focus Group Discussion Observation Personal Interview Unsolicited advise High-tech tools Experienced shopper

Public Health Management Course

Customers
Customers 1. 2. 3. 4. Expectations Are expectations met

Public Health Management Course

How do we identify management problems?

1. Listen to the customers 1. Monitor intended activities 1. Assess Performance 2. Utilize available tools
Public Health Management Course

Monitoring

Staying in touch with reality

Monitoring
Periodic Targeted Systematic collection and analysis of data and information during the course of an activity

Monitoring

Are we doing the program/ project right ? Focus on the output and activities

Structure (Input) Process

........

Output

Resources, Infrastructures, Organization, Facilities

..... Activities .. Objectives Attained? Outcome ........

Purpose of monitoring Allows managers to determine whether key activities are carried out as plan? To implement corrective actions

Activities in terms Quality Quantity Time Budget

staying on course

Process
Components of the program

intended recipients (eligibility requirements, selection process) intended context: e.g.. multisectoral collaboration intended Delivery: Activities, staffing, materials needed for delivery, information system Intended scope of the program

Outcome focuses on the accomplishments and impact of the services or program logical end point analysis

How do we identify management problems?

1. Listen to the customers 2. Monitor intended activities 1. Assess Performance 1. Utilize available tools
Public Health Management Course

How do we assess performance?

Use of Indicators Indicators must be linked to the activities- activities to the objectives Use of valid reliable metrics Objectively verifiable indicators

Public Health Management Course

Public Health Management Course

Exercise : What are the standards for the Expanded Program on Immunization
Standards Are they being met? Inputs Processes Outputs Outcomes

Public Health Management Course

Exercise
What are the indicators of performance for the Hospital X on EPI ?

Public Health Management Course

Example
Service Target:

A Fully A FIC of 60 Immunized per cent Child coverage of 90 per cent of all targeted
Public Health Management Course

Actual Hospital X Performance:

How do we identify management problems?

1. Listen to the customers 2. Monitor intended activities 3. Assess Performance 1. Utilize available tools
Public Health Management Course

MONITORING AND EVALUATION: Control & Monitoring

HPAD 201 Session 8

Control
Alter ego to planning; regulatory in nature within the managerial system Dependent on management style Monitoring, feedback, feedforward, MIS Impact on effectiveness and efficiency Establishing standards, comparing performance & standards and correcting deviations Accountability & Responsibility

Monitoring
Process of routine periodic measurement of program & office inputs, processes and outputs Focus on operations Like evaluation, baselines and documentation needed Performance monitoring assesses use of inputs as to approved budget and schedule Process monitoring provides feedback to management operational perceptions of clientele and effectiveness of communication and linkages

Monitoring and Evaluation

HPAD 201 Session 8
Performance Appraisal (PA)
Performance assessment of the quantity and quality of job output against standards Effective performance appraisal accurately measures current performance levels through observation and identification with mechanisms for reinforcements and feedback Effectiveness-based appraisal provides objective indicators (MBO/MBR) of employees contribution; what they produce and not what they do nor how they spend their time. Pitfalls of PA human judgment, inadequate information of rater, ambiguous criteria & format and inflexibility & mismatch

1. 2. 3. 4. 5. 6. 7. 8. 9.

Tips in Giving Performance Feedbacks Have mutual trust: confidentiality, fairness and objective; Make discussions lead to solutions; Observe two-way listening Support and enable ratee to express freely; State feelings clearly then move on!; Be descriptive and not judgmental; Be specific about events and time; Give feedback as soon as appropriate; Share ideas and information rather than advice

Tools that may be used to review performance

Murphys Law and Paretos Principle Pareto Charts: Histogram Flowchart Scatter Plot Run Chart Fish bone
Public Health Management Course

Pareto Charts
Vital few, trivial many Cumulative Percentages 80 -20 rule

Public Health Management Course

Run Chart

Public Health Management Course

Flow Chart

Public Health Management Course

Affinity Chart

Public Health Management Course

Affinity Chart

Public Health Management Course

Relationship Diagram

Public Health Management Course

Relationship diagram

Public Health Management Course

Problem Tree

Public Health Management Course

Ishikawa Diagram

Public Health Management Course

Public Health Management Course

Monitoring Checklist/ sheets

Also known as the defect concentration diagram When to Use a Check Sheet
When data can be observed and collected repeatedly by the same person or at the same location. When collecting data on the frequency or patterns of events, problems, defects, defect location, defect causes, etc. When collecting data from a production process. Public Health Management Course

Public Health Management Course

1. In adequate or unavailable resources 2. Poor communication strategy 3. Poorly defined action required to execute the strategy 4. Unclear accountability for execution 5. Organizational silos (underground storage) 6. A culture that blocks execution 7. Inadequate performance monitoring 8. Inadequate consequences for failure or success 9. Poor , uncommitted senior leadership
Public Health Management Course

Mankins and Steele(2005) Nine factors that contribute to poor performance

MONITORING AND EVALUATION: Field Monitoring Visits HPAD 201 Session 8

Field Monitoring Visits

Advanced communication, courtesy calls and exit presentation Use of monitoring checklist Brief and systematic conduct using records review, observation and interviews (staff, clients, community leaders) Report: Major accomplishments, issues & concerns and recommendations

MONITORING AND EVALUATION: Best Practices HPAD 201 Session 8

Consistent with policy and priority of Office Cost effectiveness and available resources Good governance Sustainability and community participation In pursuit of global, national and regional developmental and public health goals and objectives

MONITORING AND EVALUATION: Evaluation HPAD 201 SESSION 8 Outcomes and impact i.e. mortality and morbidity reduction; program milestones/strategies Formulative (project feasibility study) formative and summative Special types: costing studies, cost effectiveness cost benefit analysis, burden of diseases, validation surveys Program/project/activity, HRD (personnel audit), organizational development, financial and fiscal

HPAD 201 SESSION 8

MONITORING AND EVALUATION: RECAP HPAD 201 Session 8

Management Cyle: PIE and POSCoRBE; Objectives->Strategies->Targets>Activities; TQM and VOCP model Monitoring and Evaluation: Inputs -->Processes--->Outputs Outcomes Impact Control: alter ego to planning; regulatory in nature within the managerial system; dependent on management style; accountability & responsibility Monitoring: process of routine periodic measurement of program & office inputs, processes and outputs; focus on operations Field Monitoring Visits: Advanced communication, courtesy calls and exit presentation; Use of monitoring checklist; Brief and systematic conduct using records review, observation and interviews (staff, clients, community leaders);Report: Major accomplishments, issues & concerns and recommendations Best Practices: consistent with Office policy and priority, cost effectiveness, good governance, sustainability & community participation, equity in health, magnitude and available technology Evaluation: Outcomes and impact i.e. mortality and morbidity reduction; program milestones/strategies; Formulative (project feasibility study) formative and summative; Special types: costing studies, cost effectiveness cost benefit analysis, burden of diseases, validation surveys and Program/project/activity, HRD (personnel audit), organizational development, financial and fiscal