You are on page 1of 20
Monitoring and evaluation of community systems strengthening Ms. Nathalie Zorzi Manager, Monitoring and Evaluation Support

Monitoring and evaluation of community systems strengthening

Ms. Nathalie Zorzi Manager, Monitoring and Evaluation Support Team The Global Fund 18/19 March 2010

CSS indicator development process

August 2008: The Global Fund commissioned a review exercise in Pretoria to define indicators for community level service delivery and systems strengthening;

•

January 2009: UNAIDS developed a guidance tool for including Community Systems Strengthening in Global Fund proposals which includes a number of CSS indicators;

•

November-December 2009: A multi-partner Technical Working Groups oversaw the implementation of 9 field exercises which aimed at capturing best practices in community level service delivery and systems strengthening including the set of CSS indicators used at country level;

•

February 2010: a harmonization workshop was organized in Geneva where M&E experts from various organizations reviewed existing CSS indicators and developed an updated list which was aligned with the proposed building blocks and SDAs of the new CSS framework;

•

March 2010: An internal review and further definition of the indicators was done by the Global Fund. Comments from the Technical Working Group and other stakeholders were integrated into this process.

Challenges - Review of CSS case studies

• Challenges at community level and CSOs

- Community level workers have limited language

skills,

multiple reporting, and high turnover.

- Electricity & internet have limited availability

- Ability to analyze the collected data for decision making is limited – resulting in a perception of “purposeless data collection”

- When you can’t make sense of your own data how can you explain it to others in the community for their meaningful involvement?

Challenges

- Limited capacity at sub-national & national level for written feedback & supervision to show purpose in data

- Poor planning for trainings for M&E at all levels

- Poor correlation of program performance, logistics and finance (more purposelessness)

- Thus, loss of opportunity of learning at community level

- Poor data analysis and strategy at national level for systematic involvement of CSOs

- CSOs are registered with departments other than health causing counting difficulties

for systematic involvement of CSOs - CSOs are registered with departments other than health causing counting

Approach

• Include both M&E for service delivery and CSS

• Community level M&E means community level of both CSOs and public health system

• Large CSOs follow M&E for both community level and health facility level according to the level of application

• Community level M&E to be integrated with national level at the planning stage itself

– Design jointly by various stakeholders

– Harmonized list of indicators

– Common reporting formats and data collection tools

jointly by various stakeholders – Harmonized list of indicators – Common reporting formats and data collection

Approach

• MoUs between disease program and CSOs

– Avoids double counting

– Gives baseline for calculation of % for indicators

– Better understanding of roles & responsibilities

– Facilitated by provision of guidelines from the national level for involvement of CSOs

• Build capacity of community level in sharing information with the community

• Include meetings at community level and program review at various levels

• Include operational research at community level

• Include CSOs in evaluations and joint reviews and - learning opportunities for health facilities and CSOs

Approach

• Assuring data quality

– Publication of M&E guidelines giving standardized data collection tools and reporting formats

– Simple forms with minimal information at peripheral level

– Training & re-trainings of staff on M&E

– Guidelines on frequency and content of written feedback and supervision

– Ensuring dedicated trained M&E staff at all levels for supervision (and their travel); ensure data entry staff

– Standardization of data base

Current status of CSS indicators (March 2010)

Currently we have a list of 21 indicators with a description of the rationale, definition, numerator, denominator, proposed method of measurement, data sources and data collection frequency defined

•

Page 21 of the CSS indicator document contains a number of additional indicators for consideration

The list of CSS indicators should be updated and finalized to be ready for Global Fund Round 10 (May 2010)

consideration The list of CSS indicators should be updated and finalized to be ready for Global

Building Blocks, SDAs and indicators (1)

   

# and % of organizations working at the community level with a staff member responsible for monitoring and evaluation (CSS-ME1.1)

SDA 1:

# and % of organizations working at the community level using standard data collection tools and reporting formats that enable to report to the national reporting system (CSS-ME1.2)

Monitoring

 

& evaluation

# and % of organizations working at the community level submitting timely, complete and accurate program reports according to nationally recommended guidelines (CSS-ME1.3)

BB1: Monitoring & Evaluation and Planning

# and % of organizations working at the community level conducting reviews of their program performance in the past 3/6 months (CSS-

ME1.4)

 

# and % of organizations working at the community level with a costed annual work plan in place which includes monitoring and evaluation activities (CSS-SP2.1)

SDA 2: Strategic planning

# and % of organizations working at the community level with at least one staff member who received training or re-training in planning and M&E according to nationally recommended guidelines in the past 2 years (CSS-SP2.2)

Building Blocks, SDAs and indicators (2)

   

# and % of community health workers currently employed who received training or re-training in HIV, TB or malaria service delivery according to national guidelines in the past 2 years (CSS-

SB3.1)

SDA 3: Community actors: skills building for service delivery

# and % of currently employed staff members of organizations working at the community level that have been recruited more than 1 year ago (CSS-SB3.2)

BB2: Resources and capacity building

# and % of organizations working at the community level that received supervision in accordance with national guidelines in last 3/6 months (CSS-SB3.3)

 

# and % of organizations working at the community level with a budget coming from more than one source (CSS-FR4.1)

 

SDA 4: Financial resources

# and % of organizations working at the community level that submit timely, complete and accurate financial reports according to nationally recommended standards and guidelines (CSS-FR4.2)

# and % of organizations working at the community level that are planning to implement resource mobilization activities (CSS-

FR4.3)

Building Blocks, SDAs and indicators (3)

   

# and % of organizations working at the community level with the minimum infrastructure in place to deliver quality HIV, TB and malaria services (CSS-MR5.1)

SDA 5: Material resources - infrastructure; information; essential medical and other products & technologies

# and % of organizations working at the community level that have not experienced stock-outs of essential commodities in the last 3/6 months (CSS-MR5.2)

BB2: Resources and capacity building

% of organizations working at the community level that keep adequate logistics data for inventory management (CSS-MR5.3)

# and % of organizations working at the community level with staff trained or re-trained in stock management in the past 2 years

 

(CSS-MR5.4)

# and % of organizations working at the community level that maintain acceptable storage conditions and handling procedures (CSS-

MR5.5)

Building Blocks, SDAs and indicators (4)

 

SDA 6:

# and % of organizations working at the community level with staff who have received training or re-training in the past 2 years on management, leadership and accountability (CSS-MA6.1)

BB3:

Organisational

Management, accountability and leadership

strengthening

 

BB4: Community networks, linkages & partnerships

SDA 7: Building community linkages and collaboration for service delivery

# and % of organizations working at the community level that are implementing activities in partnership with other organizations working at the community level or with the national health system (CSS-CL7.1)

 

SDA 8:

# and % of organizations working at the community level that are planning to implement communication or advocacy activities to address barriers to equitable access for target populations

(CSS-CA8.1)

BB5: Enabling environment

Communicatio n and advocacy for enabling environments and community engagement

# of legal arrangements amended or put in place to build an enabling environment for community engagement (CSS-CA8.2)

Issues for discussion

•

Are the indicators useful for actors involved with CSS that aim at strengthening a limited number of organizations at the community level?

•

Are there any considerations regarding the feasibility of data collection for the proposed indicators?

•

If the list of building blocks and SDAs will be amended will this have an impact on the proposed indicators and how do we address this?

•

How do we ensure that all actors involved with CSS have access to the indicators?

•

Are there any suggestions on the next steps in the CSS indicator development process?

access to the indicators? • Are there any suggestions on the next steps in the CSS

Asia feedback

1. Make the definitions more concise

1. Change from Community Systems Strengthening to Community SECTOR Strengthening (treat CSS as a strategy & not as a system)

2. Community Actors be changed to mean Key Affected Populations and grass-root community members.

2. The framework should focus on GF programming

3. A new set of 7 building blocks with 14 SDAs have been proposed

4. The framework should be included in the ‘expanded’ version of the M&E toolkit

5. Technical support & assistance should be more local & peer- based

Proposed BB & SDAs

BB1:

SDA1: Sustainable financial resources

Sustainable funding

BB2:

SDA 2: Build community workforce and strengthen overall capacity of new and existing staff and volunteers

Community sector workforce

 

SDA 3: Build technical skills for service delivery (of CBO staff and volunteers)

Proposed BB & SDAs

BB3:

SDA 4:

Organizational & management Strengthening

Organizational/management support and training for small NGOs/CBOs

 

SDA 5: Transparent financial management structures

BB 4:

SDA 6: Needs assessment, gap analysis, mapping

Planning and M&E

 

SDA 7: Evidence-based strategic planning among community groups, linked to M&E systems. (including OR)

  SDA 7: Evidence-based strategic planning among community groups, linked to M&E systems. (including OR)

Proposed BB & SDAs

BB 5:

SDA 8: Partnership building & collaboration, including networks of KAP

Communication, Information & Networking

 

SDA 9: Information and knowledge sharing (including mechanism for information sharing, material development and resources)

BB 6:

SDA 10: Models & tools for community accountability & transparency to its constituency

Accountability and transparency

& tools for community accountability & transparency to its constituency Accountability and transparency

Proposed BB & SDAs

BB 7:

SDA 11: Monitoring and documentation (of both community and government interventions)

Social Mobilization & Advocacy

 

SDA 12: Advocacy, Campaigning & Community mobilization

 

SDA 13: Enabling environment (policy, legal, governance)

 

SDA 14: Reducing stigma & discrimination in all settings

Enabling environment (policy, legal, governance)   SDA 14 : Reducing stigma & discrimination in all settings

Thank you

Covered in this presentation

1. Indicators for community systems strengthening

Challenges of community level monitoring and evaluation

Approaches to overcome these challenges

Feedback on the CSS framework from Asia