End of Project Evaluation Report

Policy Influence through Community Empowerment

November 2010 - May 2011

May 2011

TABLE OF CONTENTS Table of Contents........................................................................................................ 2 Acknowledgments ...................................................................................................... 3 Affirmations ............................................................................................................... 3 Glossary/Acronyms and Abbreviations ........................................................................4 Executive Summary.................................................................................................... 5 Evaluation Introduction/Background............................................................................7

Policy Influence through Citizen Empowerment: Project Designs Document, p 4.......................8

Evaluation methodology .............................................................................................9 Limitations................................................................................................................14 Evaluation findings....................................................................................................14 Evaluation objective 1...............................................................................................14

Policy Influence through Citizen Empowerment: Project Designs Document, p 2.....................16

Evaluation objective 2...............................................................................................22 Evaluation objective 3...............................................................................................30 Evaluation objective 4...............................................................................................31 Evaluation objective 5...............................................................................................37 Conclusions and Recommendations..........................................................................42 Lessons learned from the Evaluation Process............................................................44 Appendices............................................................................................................... 45

Acknowledgments Policy Influence through Community Empowerment Project, project summative evaluation exercise and report is the result of contributions by WVA DME M&E officer Anna Pokhsraryan, WVA DME M&E officer Artak Saghatelyan, WVA DME Manager Astghik Movsisyan, WV UK PPA Programme Manager Robert Gillen, WVI CV&A learning Specialist Keren Winterford. Stepanavan and Alaverdi TDFs did a professional job in collecting data. Special thanks to WVA DME M&E Officer Mikayel Hambardzumyan for provided expertise during the analysis. Affirmations Except as acknowledged by the references in this paper to other authors and publications, the end of project evaluation described herein consists of our own work. It was undertaken to determine the projects effectiveness, and results measured against goals, objectives and outputs set forth in the project Primary quantitative and qualitative data collected throughout the evaluation will remain the property of the communities and families described in this document. Anna Pokhsraryan, WVA Monitoring and Evaluation Officer March, 2011

Glossary/Acronyms and Abbreviations ADP CAG CBO CBON CVA CWB CWBO DME FG KII M&E LG NO SO TST WV Area Development Programme Community Active Group Community-based Organization Community-based Organizations Network Citizen Voice and Action Child Well-being Child Well-being Outcomes Design, Monitoring and Evaluation Focus group Key Informant Interview Monitoring and Evaluation Local Government National Office Support Office WV Armenia NO Technical Support Team World Vision

Executive Summary Application of Policy Influence and Citizen Empowerment project which was based on the CV&A methodology in Stepanavan and Alaverdi ADPs for the last three years resulted in establishment of a more structured approach and social accountability mechanisms towards promotion of citizen empowerment and mobilization towards the common goal of improved public services. According to community residents who participated in evaluation process there is significant improvement of the quality of services if compared with the situation existing before application of CV&A tool. According to residents the project established a platform for dialogue between the ordinary citizens and public service providers as well engaging in discussions with decision makers at regional and national levels through community general meetings, town hall meetings and roundtable discussions. Another achievement of the project implementation is significant increase of awareness on entitlements for public services among target community residents and service providers. In depth discussion of entitlements among different service receivers groups, even among children groups resulted in several positive changes initiated by these groups. Community residents can require exact reform referring to an exact law, legal norm or decree. Application of the CV&A also contributed to the enhancement of facilitation skills among the ADP staff and strengthened the capacity in the National Office in terms of carrying out evidence based advocacy. Compared to other ADPs where CV&A method was not applied one can say that the participation of ordinary citizens is ensured in a more structured way. The majority of active groups established by other ADPs (not applying CV&A methodology) convene meetings upon availability of exact project opportunities or if there is a need to address any specific issue. Community action plans are good examples of participatory planning tool which has potential of replication while discussing and planning community budgets. Nevertheless, several barriers and limitations were identified as the result of the project evaluation. The low capacities among community representatives in identifying concrete steps that are needed for improvement of public services in their communities hinder the quality of action plans. Also given this low capacity of communities, service providers are in a more advantageous position and always try to push their own agendas. Low capacity comes also from the fact that the majority of community members are not aware of community budgetary funds, distribution of functions and responsibilities among LG, Marz level administration, etc. There is a risk that service providers try to push their own agenda (e.g. some of interviewed pupils mentioned that during the meeting they were asked by teachers or even by principal to raise certain issues). The data entered into the CV&A database is not so practical and sometimes remains unutilized. Not all ADP staff has the sufficient capacity to effectively facilitate community discussions and meetings. The main partners of WV for facilitating the participatory process were selected CBOs and community active members. Meanwhile, according to the National legislation on Local Governance community council is the body that is mandated to ensure community participation in decision making and represent the community best interest in decision making. Given the current situation these community councils were least involved in the CVA processes, which causes limited interest

among the councils to participate and take over, thus ensuring institutional sustainability of the approach. It must be stressed that lack of awareness among community members about community budgets results in unrealistic action plans. Meanwhile, in some cases community councils and heads are reluctant to conduct open discussions of the community budgets. According to almost all interviewees the only real guarantee of the project sustainability is WV. That is why, it is especially important to facilitate the involvement of community councils into the process.

Evaluation Introduction/Background Programme/Project 185461 Policy Influence through Citizen Empowerment project Project Phase FY08-11 Evaluation Type Summative Evaluation Evaluation Purpose To determine the projects effectiveness, and results measured against goals, objectives, outputs set forth in the project. The evaluation will offer strategic and operational recommendations to ensure future direction, sustainability and effectiveness of this project. Primary Quantitative and qualitative methods of data collection, Methodologies secondary data review Evaluation Start and 4 months End Dates Anticipated End of March, 2011 Evaluation Report Release Date This report presents the summative evaluation of the project Policy Influence through Citizen Empowerment project. The aim of the research was to collect data on the following items: the relevance of the project to local needs, its effectiveness, unintended outcomes, project model impact, integration with ADP practices and sustainability. The project was designed to address the needs existing in Armenian society in terms of low civil participation in decision making, lack of ownership towards community resources and development and low level of delivery of basic services. The project initially started with the tool of Community-based Performance Monitoring (CBPM), which then, largely contributed by the analysis of the project implementation, was developed into the tool of Civic Voice and Action (CVA). The major difference between these tools is that the CBPM was more focused on public monitoring of the state services, while the CVA has a broader advocacy goal of community empowerment and participation. To promote democratic processes, citizens of Armenia need both expanded opportunities to participate in civic life and more open governance institutions to channel participation into a stronger, more democratic system of governance. Many local citizens are often excluded from engaging with, influencing and benefiting from the planning and implementation of poverty reduction and development programmes. Meanwhile, effective involvement of civil society in poverty reduction will be enhanced by increased use of social accountability mechanisms, such as Community-Based Performance Monitoring (CBPM), to increase the influence of, and accountability to, poor people in PRS monitoring and the delivery of health, education and other essential services.1 The goal of this project was to contribute to the empowerment of citizens of Stepanavan and Alaverdi ADP communities to be able to influence good governance through ensuring the civil society participation in elaboration,
1

World Vision, Armenia PRSP Guide; A guide to World Vision engagement with the Poverty

Reduction Strategy Process in Armenia

implementation and monitoring of the community development processes and public services, as well as to use the data collected from the field for the national level policy influence2. To serve the above goal the following outcomes were set for the project: Outcome 1: Increased civil society mobilization and engagement in local/regional level development processes Outcome 2: Effective relationships between the LG, CBO, communities and other development actors operating in the project sites ensured for further promotion of local self-government decentralization Outcome 3: Improved public service delivery at local level through community empowerment Outcome 4: Development of good governance promoted at national level Evaluation Type This is summative evaluation to investigate the whole impact chain of efficiency, effectiveness and consistency. Evaluation Purpose To determine the projects effectiveness, and results measured against goals, objectives, results and outputs set forth in the project. The evaluation will offer strategic and operational recommendations to ensure future direction, sustainability and effectiveness of this project. The evaluation objectives are: Evaluation objectives Key evaluation questions To what extent was/is the project strategy relevant 1. Analyse the and aligned with the priorities of target areas? relevance of the project How was the program design process conducted? design to the existing Were any unplanned outputs achieved? needs How did assumptions affect the program?

2. Analyse the project

effectiveness, i.e. to what extent the project objectives have been achieved and how, including revealing the possible unintended outcomes

3. Analyse

the integration of the project with the overall ADP design considering the ways in which the
2

To what extent has the project been effective in achieving the objectives and outcomes as stated in the strategy? How practical and doable is CV&A methodology? What unintended outcomes (both positive and negative) have occurred? How did the community benefit from the project? What major changes have been implemented as a result of the project? How the project supplements the overall ADP design? Did the project objectives contradict with/challenge the overall ADP design? Did the project objectives overlap with the overall

Policy Influence through Citizen Empowerment: Project Designs Document, p 4

project has linked with ADP designs and ADP programming practice 4. Analyse the impact of the project model on citizen empowerment

ADP design?

How the project did enhance civic participation in

decision making process? (Comparing CVA and CAG models)3 What barriers were there to participation and how were these addressed? What forms of community participation have been increased and in what sectors of social activity the level of civic participation has especially increased as the result of the project? (Comparing CVA and CAG models) What mechanisms of social accountability were established through the project? (Comparing CVA and CAG models) What was the socio-demographic distribution of people who took part in the project? (gender and age equality in community participation) (Comparing CVA and CAG models) To what extent has the project established/enhanced capacity, processes and systems that are likely to be sustained? What capabilities does the ADP/CBO staff have for facilitating further application of CV&A What elements of CV&A can be applied as a general model in all ADPs To what extent is the LG supportive to the CVA model and willing to sustain the project achievements

5. Analyse sustainability project model

of

the the

Evaluation methodology a) Research methods Evaluation will be based on both qualitative and quantitative research methodologies selecting appropriate research methods for each criterion, as well as secondary data review. Objective 1. Analyse the relevance of the project design to the existing needs The evaluation of objective 1 started with analysis of project documentation package, namely project design document, progress reports and monitoring reports, as well as the documentation package for the target ADPs. Document review took place in parallel with the field stage. After completing questionnaire approbation and interviewers training key informant interviews (KIIs) with CBO/ADP managers and local government representatives were done. Focus-groups with community members were conducted as well.

CAG, Civic Action Groups, are the community engagement/mobilization model applied by WV Armenia ADPs prior to application of the CVA model and currently utilized in some of the ADPs.

Objective 2. Analyse the project effectiveness, i.e. to what extent the project objectives have been achieved and how The evaluation of objective 2 started with analysis of project documentation package, namely program log frame was analyzed in order to find out to what extend the program objectives were addressed. Data review included CV&A database to find out the level of its usefulness in overall project prospective. Project design document, progress reports and monitoring reports, as well as the documentation package for the target ADPs were analyzed as well. Document review took place simultaneously with the field stage. Focus group discussions (FGD) took place with ADP/CBO staffs. FGD were organized with community members as well. KIIs were conducted with LG representatives and service providers. Objective 3. Analyse the integration of the project with the overall ADP design The evaluation of objective 3 started with analysis of project documentation package, namely project design document, progress reports and monitoring reports, as well as the documentation package for the target ADPs. Document review took place prior to the field stage. KIIs with ADP /CBON managers and staff were conducted as well. Objective 4. Analyse the impact of the project model on citizen empowerment The evaluation of objective 4 was based on quantitative interviews with community members, using structured interview as a tool. Document review of participants lists and community meetings reports were implemented. FGD with community members were conducted. KIIs with CBO and ADP managers were done as well. Objective 5. Analyse the sustainability of the project model The evaluation of objective 5 started with analysis project documentation package, namely project design document, progress reports and monitoring reports, as well as the documentation package for the target ADPs. Document review took take place simultaneously with the field stage. FGD with community members were conducted. KIIs with CBO and ADP managers were done as well. b) Research tools Quantitative research: Quantitative interviews were conducted using semi-standardized questionnaire (For LG representatives in target ADPs communities) as a research tool. Questionnaires contained 21 core questions stressing such points as LGs perceptions of citizen participation in general and CVA tool in particular, the degree of their openness for civil society, the way how the role of civil society increased in community decision making process, etc. Quantitative research: Key informant interviews were conducted using KII guide (Service providers, including LG representatives of surveyed ADP communities) as a research tool. This research tool includes open-qualitative questions such as perceptions of citizen participation in general and CVA tool in particular, the way how community-service providers relations and services accessibility/quality were changed as a result of the project, etc.

10

Group discussions were conducted with FGD guide (with ADP, CBON, CAG staff and community residents4). In all research tools evaluation objectives were highlighted as separate research question blocks. The questionnaires used for each of target groups are presented in Appendices. Document review: Evaluation team reviewed the documentation of implementation process to define whether activities completed and outputs achieved. Secondary data included such data sources as package, namely project design document, progress reports and monitoring reports, ITT, CVA database as well as the documentation package for the target ADPs. Sampling strategy The quantitative part of the survey was conducted on sampling criteria of census as a procedure of systematically acquiring and recording information about all members of a given social group. That means that all community mayors from target ADPs communities were interviewed, except 3 mayors of surveyed communities per each ADP, who were already included in qualitative part of the research to get indepth information over evaluation questions. Table 1: Sample scheme for quantitative research with community mayors STRUCTURED INTERVIEWS Target ADPs # of ADP Number of interviews communities Stepanavan 20 17 Alaverdi Tavush Total 10 20 30 7 5 24

Within qualitative part of the research KIIs and FGDs were conducted with different target groups. Quantitative research was based on purposive sample strategy. In case of community residents group interviews, the sample is based on snowball technique using social networks as group composition mechanisms. During the research the following methodology of communities selection was used. Together with ADPs and CBONs staff, the CVA databases for each of ADP were reviewed and community rates calculated based on a comparison between number of identified/ planned activities in CVA action plans and number of improvement cases. Three communities per ADP were selected which are marked as the best, middle and the worst according to obtained scores. In each of selected communities 4 FGDs were conducted with the following social groups. 1. Mixed group: In middle level communities mixed groups consist of the poor members of society. 2. Non-participants 3. Women 4. School age children

The same guide were used as a tool with ADP, CBON managers interviews

11

KIIs were conducted with the following local service providers: 1. LG 2. Education (secondary school, kindergarten) 3. Health (medical posts) As for project implementers, the following groups were included in the survey sample: 1. ADP managers/staff 2. CBON managers/staff 3. CAG staff Four interviews were conducted with Lori marz level authorities from: Head of education department, head of LG department, Deputy Manager of the Department of Health and Social Affairs; Deputy Manager of the Department of Analysis and Development Table 2: Sample scheme for qualitative research (FGD-community residents) Focus group interviews (community residents) Target # of # of Surveyed # of surveyed # of ADPs ADP surv communiti social groups intervie com eyed es wed mun com commun ities mun ity ities member s Stepan 20 3 1.the best- Mixed group 3 36 avan Vardablur; 2.middle Non3 34 Pushkino participants 3.the Women 3 32 worstSchool age 3 32 Agarak children Alaverd i 30 3 1. The best Haghpat 2. middle Qarinj 3.the worstArevatsag 6 Mixed group 3 36 32 32 36 270

participants 3 Nonparticipants Women 3 School age 3 children 8 24

Total

50

Table 3. Sample scheme for qualitative research (FGD and KII-project implementers) Focus group interviews Key informant interviews Target Target # of Target # of surveyed ADPs group surveyed group social groups social groups Stepanav CBON 1 ADP 1 an manager CAG 3 ADP 1

12

Alaverdi

CBON CAG ADP CAG ADP

1 3 1 1 1 12

ADP manager

Tavush Total

ADP manager

1 3

Table 4. Sample scheme for qualitative research (KII- service providers) Key informant interviews Target Target group # of Target # of ADPs surveye group surveyed d social social groups groups Stepanav LG 3 ADP 1 an manager Secondary 3 Educati school on Kindergart 15 en Health 3 Alaverdi LG 3 ADP 1 manager Educatio Secondary 3 n Health Total school Kindergart en 3 3 22 2

c) Analysis strategy
Because of the limited sample, the quantitative research data were entered into Excel database developed by WVA M&E officer and analyzed using descriptive statistics commands. Analysis of the collected qualitative data was carried out using analytical sketches driven from Grounded theory supplemented by categorical-formal models of data combining and generating6. Upon the preliminary analysis of the evaluation findings the latter were validated with respective people, including the WV Armenia Operations Manager, Managers of the target ADPs, Manager of Sisian ADP where the CVA tool is applied, Programme Development and Quality Manager, DME Manager and two DME M&E Officers, who

No kindergarten was operating in 2 of Stepanavan ADP communities.

Grounded Theory is a research method in which the theory is developed from the data, rather than the other way

around. That makes this is an inductive approach, meaning that it moves from the specific to the more general. The method of study is essentially based on three elements: concepts, categories and propositions. Grounded Theory is used as a method of qualitative data analysis.

13

were responsible for leading the evaluation exercise. During the validation meeting the initial findings were discussed and agreed upon and respective recommendations drawn. In the course of the discussion several new factors affecting the quality and success of the CVA tool application were identified and agreed upon. An illustration to the latter can be the staff capacity to facilitate that is deemed among the key success factors by the respective leadership. Limitations Target ADP staff were very helpful in providing all necessary information and organizing field works. Nevertheless, in case of outside advisers engagement in evaluation process it will be reasonable to have more translators involved in the process in order to provide gain data circulation between evaluation team members during the field stage Another important limitation is that NO technical staff was not available through the whole duration of the evaluation. Whilst they were able to be interviewed, this was shortened, and only limited involvement in the exercise. They also did not participate in the finalisation of this report. Evaluation findings Evaluation objective 1 The relevance of the project design to the existing needs Under evaluation objective 1 the following questions were discussed. To what extent was/is the project strategy relevant and aligned with the priorities of target areas? How was the program design process conducted? How did assumptions affect the project? Were any unplanned outputs achieved? What major changes have been implemented as a result of the project? The extent of the project effectiveness in achieving the objectives and outcomes as stated in the strategy To reveal the extent to which project strategic objectives are aligned with community needs one should refer to problems and important objectives raised and programmatic achievements stressed by community itself.

(Problem) Low level of public service Increased delivery of public services

delivery:

(Achievement)

It should be mentioned that both Stepanavan and Alaverdi residents involved in the survey were quoted as saying that the general situation with social services improved in recent years. The majority of residents think that during the past three years there were numerous significant changes in their communities regarding services both in terms of physical access and quality. According to surveyed social groups main achievements are seen in the following areas: - Water supply and gasification - Improved preschool and secondary school facilities

14

Increased access to health services Improved waste management

It should be noted that In terms of evaluation process the common needs of the communities also have been discussed. Low employment level, high poverty rate and intensive migration problems are mentioned among other issues which according to residents all derive from poor socioeconomic development level of the communities. Although the problem of socioeconomic development per se is not listed in programmatic aims and objectives it should be mentioned that some targeted social groups believe that communities first of all need to create economic development opportunities. People do not have either workplace or money for food, but they always call us and ask us about our priorities you can help us create jobs in the village so that people can understand something from their lives instead of chattering if we work and earn money, we can mobilize and put our village in its own feet Extracts from FGs with Pushkino and Vardablur community members (the poor)

(Problem) Low level of community Increased community mobilization

mobilization:

(Achievement)

Increased level of community mobilization is considered as one of the most important achievements of the project by all targeted social group representatives participating in the survey. The life of villager is complicated everyone deals with his/her own problems to meet ends. During these community meetings we start thinking together and find solutions Before we used to complain in the street with no result but now we realized that if we act togheter our voices will be heard Extracts from FGs with Agarak and Qarinj community members (participants groups) (Problem) Lack of knowledge of rights and responsibilities towards public services: (Achievement) knowledgeable community residents

According to the majority of target community residents there was a significant need of information on their rights regarding this or that service. The knowledge existing among the questioned residents was fragmented and of an incomplete nature and connected to particular service which residents were using at a given moment. Throughout project implementation residents received significant amount of information on state entitlements and obligations on concrete public services and their rights. Nevertheless it should be mentioned that increased awareness about rights not always leads to proper implementation of obligations. During these meetings people attack us all around with their demands. But are they paying, for example, the land tax? Pay taxes so that village municipality has funds to cover existing issues. When you explain their rights please tell them about their responsibilities as well Extracts from FG with Vardablur community mayor (Stepanavan)

15

Nevertheless, in Alaverdi communities there are several cases when residents paid land taxes couple of days after participating in community meetings. Problem (Lack of civic dialogue, accountability): (Achievement) Community residents-LG-service providers improved relations In Armenian context rural environment is generally characterized by informal social relations that result in private dialogue between residents and service providers. Non formal relationships on one hand can enlarge the scope of connections with service providers (people can easily reach them and approach local level service providers). On the other hand, non formal relationships, as a rule, decrease the level of social responsibility mechanisms. When you approach the village mayor with your problem he says, Today or tomorrow it will be solved.... But that today never would come. Community meeting is quite another matter, when something is agreed during the meeting it will be done. Extracts from FG with Vardablur community residents (the poor) The analysis of the evaluation findings allows to conclude that the project design had been done based on the actual community needs and in line with the five components of the CVA tool, namely service, information, voice, dialogue and accountability7. Program design process Policy influence through Citizen Empowerment project design document provides detailed analysis of the project overall strategy and its main components. The review of Project design document show that the main scope of problems lying in the framework of project proposal is based on target ADPs assessment reports as well as on social-economic and political situation analysis of transition Armenian society. The overall goal of the project is to contribute to the empowerment of citizens to influence good governance, which should enhance civil society model and bring to poverty reduction in target communities. To contribute to the achievement of the goal the following components were stressed within the project: a) institutional framework enabling community participation and the delivery of basic services; b) adequate capacity of local authorities to accept and promote civil society participation in decision making, implementing and monitoring processes; c) adequate capacity of community and civil society to participate in these processes8. The integration of project objectives into both ADPs overall design (for details, please, see Objective 3, p 26.), collaboration with appropriate government entities and local organizations and in depth economic and socio-demographic analysis of the area, with a special emphasis on the youth and women, resulted in accurate and reasonable compilation of specific areas for intervention in design document.

Policy Influence through Citizen Empowerment: Project Designs Document, p 2

Policy Influence through Citizen Empowerment: Project Designs Document, pp.1-2

16

Project Design Document paid special attention to the evaluation of voluntary and community active groups activities implemented in target communities which served as basis for Community Based Organizations Network models development and implementation as part of the project design?. Group discussion results with CBO and CAG and ADP members involved in current evaluation exercise show that establishment of CBO network significantly reinforced CAGs operations, voluntary initiatives and overall community cohesion. With regards to the stakeholders participation in design stage it should be mentioned that all the questioned CAG, CBO as well as ADP representatives stated that they had an opportunity to participate in discussions aimed at outlining possible programmatic directions. Service providers participation in design process is relatively low. The majority of questioned service providers stated that they have participated in implementation stage and had rather passive role in design process. Log frame review showed that in general it had SMART indicators which enabled evaluation team to overcome financial and time costs needed for measurement. Nevertheless, there are two main problems which, to some extent, limited evaluation process: 1. No M&E plan was designed 2. No baseline evaluation was conducted The extend of assumptions effect on the program To assess whether the planned outputs led to achievement of outcomes, the vertical logic of the log frame has been analyzed in the framework of the listed assumptions and risks. The analysis of the rationale underlying causal linkages between the outputs and outcome of the projects and the factors that may influence the program or project beyond the management control of the project have been analyzed. The effects of outcome and goal level assumptions on the program have been discussed with ADP/CBON, CAG staff. Discussions show that the majority of underlying assumptions included in aforementioned levels of log frame has been proven as true which ensured output and outcome achievement. Goal: Contribute to the empowerment of citizens to influence good governance Assumption: Local Self governance representatives are open for civil society According to quantitative survey results conducted among village mayors the majority of respondents (70.5%) highlight the role of citizens participation in making public services accessible. Total of 80% of village mayors state that community priorities are defined through joint meetings with community residents. Moreover qualitative interviews also show that village mayors have positive attitude towards citizens initiatives promoted in the scope of current project particularly stressing the importance of such programmatic outcomes as increase of social cohesion, development of community potential and active participation of citizens not only in statement of priorities but in seeking solutions. Group discussions with CBON managers and members show that due to the fact of organized trainings they possess information which allows them to influence local level decision making. Both Alaverdi and Stepanavan CBO members think that they have constructive relationships with LG representatives and via systematic

17

discussions are able promoting community priorities revealed through CVA methodology into LG agenda. Village mayors are well aware about us; they cooperate with us and in some cases consult with us Extracts from FG CBON staff (Stepanavan) Nevertheless 2010 first quarter monitoring visits conducted by WVA DME M&E officer revealed some gaps in cooperation between CBON and LG bodies in Stepanavan. During the reporting period CBO officially approached the mayors for the participation in budget planning, but the letter was not responded. Later, it turned out that there was misinterpretation of the letter sent by CBO: it was supposed by mayors, that CBO/ADP wanted to have delegation to intervene in budget planning procedure. Together with ADP the CBO decided to meet with mayors during budget revision process to be able to see the community development plan and the budget to be assured that the issues discussed have been reflected in both plans and to understand what can be the ADP's intentional interference for the given community. The evaluation process revealed some issues regarding abilities of CBON to influence policy making in higher level than communities. CBON representatives state that their organization is still in early development stage and as yet has no appropriate potential, including cooperation experience, fundraising abilities, to influence higher level policy and decision making. Specific cases of cooperation have taken place under the umbrella of WV Armenia. Interviews show that Stepanavan CBON has lesser level of cooperation with Marz level state structures compared to Alaverdi. Though it should be mentioned that in case of CBON-Authorities cooperation Alaverdi ADP had mentoring role which is a positive phenomenon when it comes to supporting new established organizations but on the other hand that kind of cooperation is mostly initiated by the ADP leaving CBON in a passive role. Mentoring role could be valuable in the initial stages but from the perspective of CVA ideology and sustainability CBON should act as independent actor in the field. Outcome 1 Increased civil society mobilization and engagement local/regional level development processes Assumption: Local Self governance representatives are open for civil society Please, see previous section. Output 1.1: CBO Network (CBON) promoted in targeted communities Assumption: Stable legislation on civil society. The only major change registered in the legislation regulating the Civil Society actors is the amendment of the law on NGOs. According to the amendment the NGO having annual budget exceeding one million drams shall publish information on their funds, expenditures, description and geography of implemented projects etc. which cannot have significant impact within the project scope. Output 2.2 LG - CBON participatory community development planning contributing to the child well being Assumption: Willingness of LG to involve CBOs in community development planning In the course of the entire process there has been no formal engagement of CBONs with Community Development Plans. However, both in Alaverdi and Stepanavan in

18

CBONs representatives have participated in number of Community Council meetings and maintained regular communication with their representatives. Also, participatory planning and budgeting was one of the topics covered during the capacity building training for LGs held in April 2009, where the LG representatives and CBONs discussed the opportunities of engagement in the process of development of Community Development Plans. As it was mentioned by CBON and ADP staff during the training, representatives of LG agreed that CBON engagement in that process would be generally beneficial for their communities.9 CAGs and CBONs will cooperate increasingly with regional authorities. Action Plans will become fully integrated with Community Development Plans and aligned with regional processes (i.e. fully institutionalised). WV will also explore the incorporation of other stakeholders such as local businesses in future Extracts from KII with Alaverdi ADP Manager Outcome 3: Improved Public Service Delivery at Local Level through Community Empowerment Assumption: Public Service providers are open to dialogue CVA, which was applied in the project communities throughout the past 3 years, has enabled community members to raise important issues and find solutions for many of them. CVA has proven to be an effective tool for solving community issues locally. FG discussions with community social groups show that education, health and municipal service providers are open enough for joint meetings with residents aiming at prioritization of community problems and outlining solutions. The interviews with service providers also indicate such willingness and ardour. Moreover all interviewed service providers from all spheres in the communities have been involved in discussions and design of joint action plans throughout the project. Output 3.3 Increased influence on local level policy and decision making processes Assumption: LGs and Service providers are open to public monitoring Please, see previous section. Major changes identified as a result of the project Survey results show that the vast majority of target social groups value the end results of the programmatic activities rather than the process. In other words, different social groups were more desirable to see final project results (mainly improved service delivery) and were less interested in the process of how those results would happen. During FGDs when residents were asked to distinguish the main problems they faced three years ago, along with infrastructure problems such as drinking water or poor condition of school buildings they have been stressing the factors like lack of information or weak mobilization of community hindering civil society development. When asking to describe the main programmatic achievements

It must be mentioned that due to September 8 of 10 Alaverdi communities have updated 4

year community development plans according to CVA action plans

19

throughout these years the residents basically concentrate on actual improvements registered in this or that service and mention changes in the sphere of social solidarity, civil dialogue, etc as secondary changes.

The vast majority of survey participants belive that the main changes taken place due to the application of CVA methodology refer to local level projects in education, health and municipal servic spheres. The increased level of accountability of service providers is the next important change noted by survey participants.

It is no longer required to wait for hours to receive an information or document from village municipality. Municipality staff already works until 17:00 p.m. since they know that people are already demanding. Extract from FG with Haghpat community members (the poor) It should be stressed that one of the targeted and achieved results of the project is increased awareness on state standards and obligations existing in the local level social services which has crucial role in conducting community based monitoring of abovementioned services. Yet some CBO and ADP representatives during interviews suggested considering differentiated dissemination of various state entitlements. Some of the interviewees mentioned that not all entitlements could be used to evaluate the quality of accessibility of certain social services on rural areas. For example the standard provides that the classroom should be lightened from appropriate angle or tables and chairs should be situated in fixed distance from each other, etc. There are standards according to which we shall completely suspend the classes or terminate the services Extract from FG with CBON staff (Stepanavan)

The growth of intensity of social contacts is mentioned among the key identified changes. As stated by many residents the community meetings promote both the communication between the residents themselves and service providers and those who receive the service which actually eases the solution or clarifies many problems at service level.

I always wanted to meet with school headmaster but it was not easy later participated in community meeting and could discuss with him my childs problems there in any case it is an opportunity for people to gather and discuss different issues. Some people think that rural people are not like urban ones and communicate more actively with each other, but things changed and now each person lives alone with its family problems. Extract from FG with Vardablur community members (women)

Though many survey participants prove that community participation initiatives promote social trust, it should be mentioned that social trust in rural environment remain conditioned by individual interactions. In other words, confidence has emphasized personalised nature and confidence towards any social service means confidence towards the person who provides that service. Thus while discussing the overall poor quality of given service the FG participants were trying to avoid pointing exact issues and presenting the service staff personal qualities in positive light.

20

There are many medicaments that our health post does not posess and we must get them from Stepanavan or Yerevan, but our doctor is a very good person and always helps us ... Extract from FG with Pushkino community members (pupils)

Prevalence of positive attitudes towards future and increased social cohesion are seen as important derivatives of CVA initiatives.

Village residents participated and will participate in improvement of daycare center services. Peoples participation is increased, people use the slogan not mine but everyones Extract from FG with Arevatsag community education service provider (kindergarten) To sum up the social perceptions on impact of CVA application one can conclude that it is about changes aimed at solving concrete problems and strengthening social capital (aggregate of social trust and mutual assistance norms) which certainly has crucial role in shaping community empowerment and development perspectives.

21

Evaluation objective 2 Project effectiveness (to what extent the project objectives have been achieved and how, including revealing the possible unintended outcomes) Under evaluation objective 2 the following questions are discussed. To what extent has the project been effective in achieving the objectives and outcomes as stated in the strategy? How did the community benefit from the project? How practical and doable is CV&A methodology? Project effectiveness in achieving strategy objectives and outcomes In this section the evaluation report analyses the level of achievement of the project goal and outcome level indicators to show to what extent the project was effective in achieving objectives and outcomes stated in Project log frame. Goal: Contribute to the empowerment of citizens to influence good governance Indicators: *At least 50% of surveyed local self governmant bodies of target communities state increased role of civil society in community decision making processes The vast majority of village mayors say that before project implementation community priorities were set based on opinions heard during sporadically organized meetings with the participation of community active group members and service providers. Community participation in local level community underwent significant changes after application of CVA methodology. As a result, 63% (15 out of 24 communities of Alaverdi and Stepanavan) of surveyed community mayors state increased role of civil society in community decision making processes. * The CBO Network states its capacity and access to information sufficient for influencing decision making and policy changes. According to survey results CBO Network rates its capacity and access to information sufficient for influencing decision making. The majority of CBON members in both target communities stressed that they need more capacities to be involved in policy making and changing processes. (For more details, please, see Objective 1, p 6) Outcome 1. Increased civil society mobilization and engagement in local/regional level development processes Indicator: # of community development issues raised and addressed with active participation of civil society actors KIIs and FGs with community members, ADP, CBON staff as well as Stepanavan and Alaverdi ADPs CVA databases reviews showed the following statistics regarding the ratio of raised and solved community issues.
ADP Total numbe r of identifi ed issues with CVA techniq ue 355 276 Total num ber of issue s solve d 177 92 Number of issues solved with community participation / % out of total number of identified issues Number of issues solved with community participation and external organizations (including WV) % out of total number of identified issues Number of issues addressed only by WV / % out of total number of identified issues

Alaverdi Stepana van Total

38 / 21.4% 12 / 13%

98 / 55.3% 5 / 5.4%

41 / 23.1% 75 / 81.5%

22

631

269

50 / 18.5%

103 / 38.2%

116 / 43.1%

Hence, end of project value of the above mentioned indicator is 114. Outcome 2. Effective relationships between the LG, CBOs, communities and other development actors operating in the project sites ensured for further promotion of local self-government decentralization Indicator: # of CBON staff, community members participating at LG operations (e.g. community strategy development, budget hearings, etc.) Alaverdi and Stepanavan ADPs have been actively facilitating dialogue and communication between the LGs of 30 project communities and the community members. ADPs have been working closely with LGs through regular meetings to identify the need of LGs for capacity building and tools necessary for LG staff to address problems identified by communities. In order to ensure engagement of CBONs in community development planning processes Alaverdi and Stepanavan ADPs have been working with heads of LGs to highlight the importance of community development plans and the role of community participation in those processes. As per Project Annual reports about 40 member of CBON staff and 60 community members participated at LG operations. Outcome 3: Improved Public Service Delivery at Local Level through Community Empowerment Indicator: # of public services delivered to community stated as improved by surveyed community members CVA, which was applied in the project communities, has enabled community members to raise important issues and find solutions to many of them. CVA has proven to be an effective tool for solving community issues locally. Overall, 269 issues out of the raised 631 were resolved in the project communities. This data is taken from CVA database review. Qualitative research tools were used in order to identify community residents opinions and perceptions over services improvement. Number of interviewed community members is around 270 (4 FG per community with 8-10 participants). The number of solved issues listed by FGs participants is about 24. 24 of public services delivered to community stated as improved by surveyed community members.

Social

services communities

delivery

and

quality

assessment

by

surveyed

While referring to the issue of the CVA project efficiency, the analysis of the situation with the quality of the services, their impact on community welfare should be taken

23

into consideration. Since the communities subject to evaluation have been selected based on the level of the CVA application effectiveness, the assessment of services should be presented separately for each community indicating existing similarities and differences. 1. The healthcare and education services have been indicated as vital for community life by participants in all communities and received the following rates: healthcare and education services. In Stepanavan ADP three target communities rated significantly high the healthcare service operations effectiveness (4.7) while in the case of Alaverdi ADP communities the education services have been considered as the highest quality services (4-4.5 points mean). The rate of education services in Stepanavan ADP communities vary from 3 to 3.5 points which according to survey participants is conditioned by expensiveness or lack of textbooks, scarcity of laboratory equipment, infrastructure for sport events and in some exceptional cases by absence of quality specialists. The assessment of Alaverdi region health care services resulted in almost the same rating points as in Stepanavans education case which in its turn is based on personal negative experience. Figure 1. Social services participatory assessment scores10 (Arevatsag, Alaverdi ADP)11

Figure 2. Social services participatory assessment scores (Vardablur, Stepanavan ADP)

10

Figures 1-6 represent data on community services assessment by community target groups. 5-

point scale was used to identify the level of services accessibility, where 1 means absolutely not accessible and 5 means absolutely accessible.
11

The blue line of non-participants cannot be observed on the graph since it repeats the values of the

poor presented in lilac.

24

Figure 3. Social services participatory assessment scores (Qarinj, Alaverdi ADP)

1. As for the assessment of NGOs the only organization mentioned by target

groups is the World Vision. Political parties rating points are mainly situated in 0-1 range which according to target groups is conditioned by virtual abscense of any influence on community life from their side. Meanwhile poor contingent evaluated the political parties relatively higher than other groups. The phenomenon stems from the fact that the operations of parties are limited to various donations for the poor population during preelections. 2. There are a number of differences between Stepanavan and Alaverdi target communities residents perceptions towards central and self-government bodies. Particularly central government body (Marzpetaran) received 2.5 points (mean) by Alaverdi residents while Stepanavan residents evaluated the activities of the same instance with 1.6 points (mean). The same trend is seen in evaluation of local self government bodies. Alaverdi residents evaluated their municipalities with 4.25 points while in Stepanavan communities it was 3.7. According to the survey results there is correlation between the fact of participation in community meetings and evaluation of municipal services. Thus the groups of people who ever participated in community meetings evaluate municipal services higher compared to those who never participated in any meeting (Nonparticipants: Stepanavan-1.3 Alaverdi-2, Participants Stepanavan-2 Alaverdi-3). Existing social dispositions towards LG bodies are in rather high levels of tolerance scale. Even putting lower grades, people want to emphasize that their village officials do their best. Moreover any initiative of LG officials are perceived and interpreted by people as very important. Compared to other services central governments activities are evaluated with the lowest points. Survey participants condition the fact by such factors as minimum presence of Marzpetaran officials in community life, tendency to ignore local level issues and unawareness of residents on Marzpetaran activities We have seen the governor only once when they need us they come and call a meeting but when we need them we never see them .... we have applied to support us with kindergarten with no results Extract from FG with Vardablur community members (Women)

Service assessment results of the womens and nonparticipants groups in three target communities of Stapanavan ADP are mainly similar. While according to Project DD one of the main objectives of the project is to promote

25

participation of women, however the evaluation could not answer the question regarding the level of actual participation of women in project implementation to the full extent. The reason is that the qualitative findings received contradict to the quantitative data not allowing for further analysis without an additional field exercise with a specific focus and design. Figure 4. Social services participatory assessment scores (Pushkino, Stepanavan ADP)12

Figure 5. Social services participatory assessment scores (Agarak, Stepanavan ADP)

Assessment results of the poors and nonparticipants in Alaverdi communities also correspond to each other which in its turn may mean that nonparticipant groups are mainly consisted by poor segment of the population. Figure 6.Social services participatory assessment scores (Haghpat, Alaverdi ADP)

12

The blue line of non-participants cannot be observed on the graph since it repeats the

values of the poor presented in lilac.

26

Overall nonparticipant group members tend to put lower points while assessing community services compared to those who participated in this or that community meeting. This can bring to conclusion that the higher is the awareness level on state standards or obligations regarding community services, the higher is the level of constructive criticism approach while evaluating social services. If nonparticipants condition their lower assessment points by negative personal experience (for example one applied to mayor for job but rejected) the participants assess community services taking into consideration qualitative aspects of the service and material features. I would assess physical conditions of the school with 4 points since we lack sport hall and appropriate laboratories but the quality should be assessed by highest point 5 since our school has high quality specialists Extract from FG with Agark community members (Women)

First of all, it should be mentiond that according to survey results there was some misunderstanding among all interviewed social groups about the format of community meetings and discussions. At the beginning of interviews social group representatives mixed the community meetings organized in the scope of CVA project with other similar meetings (in some cases organized by other organizations). It seems like the series of CVA discussions are not considered as separate and more important ones than others. Referring the peculiarities of CVA application in Alaverdi and Stepanavan communities, one can say that Alaverdi communities are more optimistic towards the process and the participation level is comparatively higher. During the evaluation results analysis it turned out that there are several information sources through we community residents learned of CVA discussion. 1. Community people get information on upcoming meeting from ADP staff during recurrent site visits. No single interviewee pointed any local stakeholder of the process as responsible for convening meeting or conducting discussion. WVA is seen as main responsible of the process. The issue is further discussed in Objective 5 of this report (p. 31). 2. Some FG participants told that they have seen announcements about the meeting in their municipalities or municipality staff informed them about the meeting. 3. Informing the community members through school administration is the most widespread method of informing residents about meetings. This method can be considered as effective since it ensures participation of wide scope of stakeholders in discussions about educational issues but the participants selection process in some cases has been distorted.

27

No one tells us that we have right to participate in that discussion. Teacher or headmaster comes and says You, you and you go to that meeting Extract from FG with Pushkino community members (pupils) The success of any initiative on participation is conditioned by the target groups perception and interpretation of the roles of key components. According to survey results the vast majority of target community residents think that civic participation process is limited to participation in meetings to define priorities and find solutions. Surveyed community residents and service providers opinions over CVA objectives are combined in the following two main types: 1. Community based- problem oriented: the vast majority of interviewed residents think that CVA is designed to help people identify priorities and solve them through participatory mechanisms. 2. Community based- mobilizing: some part of interviewees think that CVA is to raise community mobilization and make decisions jointly. The majority of surveyed project implementers stated that CVA methodology is quite practical and doable. Nevertheless, there are several problems relating to citizen motivation to participate in community gatherings, meetings There are two sides of the problem. On the one hand, people often complain that decisions have been forced upon them by the local authorities or that they have not been consulted in planning at local government level. On the other hand, there are evidences to suggest that on many occasions people do not participate actively in local planning even if they have been given the opportunity to do so. A number of factors are important in determining whether people really want to participate in planning. It seems that participation in local planning is usually restricted to those citizens who feel directly threatened or affected by the local government or personally involved in its endeavours. It also seems that people are unlikely to participate willingly in planning if they feel that their participation will have no significant effect on the final outcome. People are also reluctant to participate in local activities in which they have no particular interest, or which are unlikely to affect them directly. People tend to participate only in planning when there is some specific interest for them in its outcome. Another factor is ignorance. It seems that the average citizen has very little knowledge of the range of options (and of the implications of these options) open to him or her in terms of forums which could be used to express his or her views and desires. A substantial number of citizens therefore do not avail themselves of these opportunities to shape policy directly because of inertia or indifference. The third factor is the communication problems. These problems amount to practical difficulties such as language problems, differences in attitudes and expectations, and mutual feelings of mistrust, suspicion or resentment. Another issue related to CVA methodology practicality refers to the CVA database. As it was stated by the managers of Stepanavan and Alaverdi ADPs, the CV&A database is less operational and more time consuming exercise. It was suggested to revise the format of the base and make it more applicable.

28

Evaluation objective 3 Integration of the project with the overall ADP design considering the ways in which the project has linked with ADP designs and ADP programming practice Under evaluation objective 3 the following questions are discussed How the project supplements the overall ADP design? Did the project objectives contradict with/challenge the overall ADP design? Did the project objectives overlap with the overall ADP design? Projects linkage with ADP overall design Survey results show that there are several links to which the project supplements the overall ADP designs. Community empowerment and citizen empowerment is one of the main strategic objectives of WV Armenia strategy to ensure sustainability of projects and programs as well as part of their transition to communities. This approach is applied in all WV Armenia ADPs, including Stepanavan and Alaverdi ADP. The CBPM tool is a crosscutting for all projects and is integrated in all other projects of the ADP (Education, Health, Economic development). Collaboration with CBON is not limited by the scopes of the proposed project. On both Stepanavan and Alaverdi CBONs are the pone of the main partners of the ADP in general, and the sponsorship management project in particular. CBPM as one of mechanisms of social accountability is being widely used by Stepanavan ADP operated in target communities. CBPM application was also widely applied during the design phase of Alaverdi ADP. Facilitation of establishment of a CBO Network, their capacity building along with capacity building of other role players, including local authorities through the proposed project added value to the effectiveness of the mentioned ADP community development projects aimed at reduction of poverty in targeted communities. The ADPs supplement the project by the long term presence of WV Armenia in targeted ensuring continuity of efforts to build further the dialogue between all community stakeholders. In both Alaverdi and Stepanavan ADPs the project contributes to the Civil Society project mainly, by empowering citizens to be able to influence the decision making and, if needed, policy changes in their communities. According to interviewees, CVA definitely helps and complements the project sectoral goals. No significant contradictions and overlaps were identified with the overall ADPs design. Moreover, each year target ADPs amend the programs DIP to address issues included in local Action Plans. in FGDs with service users we cover training on entitlements and encourage discussion of decisions. People reflect on the quality of public services (in schools and kindergartens for example). They dont always have these opportunities with conventional WV service delivery programming approaches Good facilitators + CVA = Good results. CVA may not be as relevant in its current

29

form in urban settings as communities are harder to distinguish Extracts from KII with Alaverdi ADP manager

Evaluation objective 4 The impact of the project model on the level of community participation (comparing CAG13 and CVA models) Under evaluation objective 4 the following questions are discussed: How the project did enhance civic participation in decision making process? What barriers were there to participation and how were these addressed? What forms of community participation have been increased and in what sectors of social activity the level of civic participation has especially increased as the result of the project? What mechanisms of social accountability were established through the project? What was the socio-demographic distribution of people who took part in the project? (gender and age equality in community participation) Community participation in decision making process. Civic participation is one of the principal means by which individuals exercise their empowerment for the benefit of the locality, often at the same time increasing their own level of empowerment. It helps to make civic institutions more representative of and accountable to the local population. It also contributes to concentrating local decision making more effectively on the issues all members of community believe are important, as well as strengthening ties between such bodies and the people they serve, so building trust. Survey results show that community involvement in decision-making was promoted through CVA so that people know what should be done, by whom and when. People have an opportunity to gather and discuss priorities in the past we used to gather from event to event but now it is done in a more coordinated way Extract from KII with Community council (Metz Ayrum, Alaverdi) A show that arguments in favour of enhancing citizen participation frequently focus on the benefits of the process itself, taking into consideration the point that citizen participation is a transformative tool to social change. Nevertheless, there are such opinions that citizen involvement is intended to produce better decisions and thus more efficiency in benefits to the rest of community. Hence, we can conclude that there are two types of benefits and advantages of citizen participation

13

CAG, Civic Action Groups, are the community engagement/mobilization model applied by

WV Armenia ADPs prior to application of the CVA model and currently utilized in some of the ADPs.

30

Table 6. Community participation: advantages for citizens and LG14 Advantages for citizen: Process: Educated citizens: informed and involved citizens to some extent became expert understating technical difficulties of decision making process as well as making community wide solutions. Persuade and enlighten LGs: informed and involved citizens have large possibilities to raise the most important and urgent issues for community and persuade LGs to do certain steps to solve issues. Thus, community activists may have regular contact with key decision makers and can persuasively convey their viewpoints. Gained skills for activist citizenship: due to participation practices community members became more active and can make further collaborations to create civil society institutions. Outcomes: Priority based decision making: citizen participation ensure the process of making local level decisions based on community priorities and not on local authorities capriciousness or self-will. Community based monitoring: citizen participation helps implementing monitoring of services and policies. Advantages for LG: Process Feedback mechanisms: through regular contact with community residents LG representatives learn which activities/projects/policies are likely to have negative feedback from community and how to avoid such failures. Political persuasion: when the government truly collaborates with citizens or when it works to win over citizen sentiment, a key assumption of successful political suasion is the social influence of citizen participants. Outcomes: Better policy and implementation decisions: by ensuring citizen participation in decision making process government representatives creates bases to elaborate effective policies and make decisions: Decision making legitimacy: the most important outcome especially for Armenian reality is legitimate political decisions. When community have an opportunity to participate in decision making process and when their priorities are reflected in policies, legitimate perception of authorities will be more expended.

Socio-demographic picture of community participation modes Referring to the analysis of CVA participants socio-demographic characteristics the following should be taken into consideration. As noted both by CVA process participant residents and organizers /CBON, ADP/ almost in all communities there are groups of people always participating in such discussions and groups of people avoiding any community initiatives in vast majority of cases.

14

The information mentioned in Table 6, was incorporated on the bases of ADP staff, CBON

staff, CAG members as well as some LG representatives ideas, opinions and perceptions.

31

The group behaviour is conditioned by its members perceptions. Those who do not participate are led by non-intervention strategy. Based on FGDs with ADP staff as well as with community residents Non-intervention strategy can be explained by the following reasons:

Low wellbeing: Tha vast majority of interviewees believe that as a rule vulnerable groups such as poor and lonely women do not participate in community level initiatives. Need to address basic needs: FG discussions with non participants show that these people need to meet their basic needs on daily basis and do not want to waste their time participating in such discussions. Low intensivity of social interactions, limited social capital: These families are described with passive interactions with other families and are mainly isolated from social life. Lack of awareness: Some FG participants noted that these families do not participate due to lack of information on upcoming events. Indifference towards community initiatives: These groups are mainly indifferent towards community initiatives which may be explained by low level of trust towards latters. Intermediate, nondifferentiated position: Interviewes with the representatives of nonparticipants groups show that these people as a rule do not have exact requirement to quality of this or that service and do not see the way they influence any development in their community. Figure 7. Causal linkage between factors of non-participation social tactic

To raise the effectiveness of CVA process it is expedient to make the process maximise inclusion particularly ensuring involvement of vulnerable groups. Youngsters and middle-age groups representatives are actively involved in discussion. The picture is a bit contradictory from gender perspective. As it presented in Evaluation objective 2 the opinions of nonparticipants and women are similar. According to part of FG participants women are participating in discussions more actively. Women have more free time thus they participate more actively men cannot leave their job for such a meetings

32

Extract from FG with Vardablur community members (the poor) Another part of FG participants think that women participation is limited due to sociocultural stereotypes on women behavioural norms. The data collected through various evaluation tools on the level of women participation in the community discussions is rather contradictory. This fact limits the ability to analyze it without additional field exercise with specific focus on the issue. However, at this point additional field exercise is deemed not cost-effective. This issue will be addressed through the next project cycle (to be funded through the PPA funding) with gender disaggregated indicators. Almost all the beneficiary group representatives stated that as a rule the same people participate in all discussions (predominantly representatives of school staff and student councils, and community proactive residents who participate in every event). Survey participants pointed some issues hindering the effective application of the CVA tool to support civic participation. Accordingly they suggested some changes in this respect. Limitations The frequency of meetings is limited and not all the people have opportunity to participate Nearly the same people participate in all discussions Some documents designed during discussions have a form of wishlist with no practical implications Unawareness of community budget possibilities hinder the process of making realistic suggestions Suggestions Increase the frequency of meetings Promote participation of all people and make the process more transparent15 Emphasize the practicality of decisions made Make the process of community budgeting transparent and more participatory

The following types of participation, varying from one another by intensively and impact have been established due to implementation of CVA methodology.
1. Community

contribution to the development effort, i.e. to the implementation of the decisions. In-kind contribution of beneficiaries - local residents contribute their labor in implementation process, along with an assistance of external agency. Besides WV is seen as a main external assistant agency particularly when it comes to financial assistance. A financial contribution of the community - A lot of projects have been implemented with the financial contribution of LG bodies. It should be mentioned that these forms of participation are not applied separately. In vast majority of cases implementation of project implies community residents labour force partial financial contribution from LG bodies and assistance from abroad in terms of funds and assets.

15

For example, Alaverdi ADP manager suggested to put certain criteria of participation, e.g. in

each of new community discussion 40% of FG participants should be previously non participants.

33

2. Community-based monitoring: The monitoring of social services also can be

considered as a form of participation embedded by CVA methodology. The awareness on state standards on social services is of major importance to sustain this form of participation. Social accountability mechanism established through the project Essential to the well-being of all people is the effective delivery of basic services such as health, education, water and sanitation. Yet, widespread evidence shows that services are failing poor people in a large number of countries with negative impacts on human development outcomes. In addressing the failure of services, one key point is that the failure of services is not just technical, it is the result of the lack of accountability of public, private and non-profit organizations to poor people. All interviewed project implementers stressed the point that CVA is one of the most effective tools in which community participation processes and mechanisms can strengthen accountability and also affect service delivery outcomes. Citizens can exert their collective voice to influence policy, strategies and expenditure priorities at local level according to community preferences. According to interviewed target groups within CVA technique the following model of social accountability was established. This model could be mentioned as short route of accountability that includes 2 groups of stakeholders: (i) citizens, as clients, directly influence service providers (ii) to improve the quality and accessibility of different services. As the result citizens became more empowered and service delivery is improved. In terms of the long and short routes, depending on the context, different routes will be more or less amenable to various forms of community participation. The long route includes electoral politics, citizen oversight boards, use of media, social accountability movements that bring together citizen direct action with more formal government mechanisms. The long route is more difficult for community organizations, without some types of enabling environment and functional public institutions that can incorporate or engage with community initiatives. In each context, the state/society nexus shapes the room for access on the long route. External Non-Governmental Organizations, especially international NGOs, are not the short route, but mechanisms for supporting either the short or long route. The short route includes direct provision of services, community monitoring and management of services, contracting with service providers, and projects funded by external NGOs but managed locally. In relation to service delivery, there are a range of issues related to the role of community participation and stakeholder involvement in service provision. The short route of accountability provides for direct community action, both through community provision of services and through communities holding providers accountable at the point of service delivery. The evaluation research results show that there are several groups of perceptions over the role and possible impact of civic participation on social services accessibility. 1. Some of interviewed service providers as well as ADP and CBON representatives think that civic participation and community-based interventions have the potential to make service providers be more responsive to the needs and priorities of beneficiaries (this could be called allocative efficiency). 2. Other interviewed evaluation research participants are sure that civic participation provides designing of community-based projects that are

34

comparatively cost effective (productive efficiency) because of lower levels of bureaucracy and better knowledge of local costs. From another point of view, almost all research participants agree that because of the lack in knowledge of local budgetary funds the joint action plans that community participants come up with wish lists rather than realistic and focused steps. ADP/CBO managers and staffs as well as service providers and LG representatives mentioned several suggestions to provide improved accountability: overcoming information asymmetry- service providers possess in-depth knowledge on the subject matter compared to ordinary citizens and as a result during discussions they easily promote their agenda; providing communities with information on service quality through various forms of Monitoring and Evaluation; improving various dimensions of allocative efficiency includes increased transparency on budgets and public resources through such mechanisms as community hall meetings, public budgeting.

35

Evaluation objective 5 Sustainability of the project model Under evaluation objective 5 the following questions are discussed: What elements of CV&A can be applied as a general model in all ADPs? To what extent has the project established/enhanced capacity, processes and systems that are likely to be sustained? What capabilities does the ADP/CBO staff have for facilitating further application of CV&A? What elements of CV&A can be applied as a general model in all ADPs? To reveal what elements of CVA process can be replicated by other ADPs it is necessary to refer to activities of experimental groups, i.e. Tavush ADP areas community active groups. According to interviews with different active group members, CAGs in Tavush apply different mechanisms for identification and prioritization of community problems.

CAG and community council cooperation. According to Tavush regions CAG members their main aim is to reveal community problems, vulnerable groups and cooperate with community councils for solutions. Some CAGs have community council members in their group. Other CAG members used to participate in community councils meetings. Before some CAGs gathered information on community problems through personal inquiries or request sent by individuals.

We have director of post office, teacher, and veterinarian in our group. We convene meetings and inform our villagers by announcements Extract from FG with CAG staff (Aygehovit, Tavsuh ADP)

Currently these CAGs also convene meetings but more sporadically compared to CVA community CAGs. Community priorities are set by CAG members and sent to community councils for discussions. We try to meet the people in all districts and ask them about their problems. Later we prioritise issues based on the spread of the issue Extract from FG with CAG staff (Aygehovit, Tavsuh ADP)

Revealing of problems through social network. Ditavan CAG members think that communities are small enough and information sharing is intense so they easily get information.

CAG group member live in different districts and hear about existing problems in daily basis. Community is small and people are connected to each other Extract from FG with CAG staff (Ditavan, Tavsuh ADP)

36

CAG-Community council - Village mayor cooperation. Tovuz CAG members told that they provide information on problems to community council which later discussed with Community Mayor. In this case CAG acts as support unit to LG informing on recurrent community issues prioritized by non formal voting mechanisms. Observation of aforementioned opinions let us to conclude the following:

experimental group i.e. CAGs have mediated mechanism of community participation; CAG organizes meetings in sporadic manner and basically use social network to gather needed information; compared to control group (CVA CAGs and CBON) the involvement of community council is more intense in the experimental group; in both CVA and non CVA processes the majority of community issues are solved with WV interference and community contribution (labour force, assets, and funds) where possible. Only 18.5% of community issues identified in project communities (Stepanavan and Alaverdi ADPs) have been addressed with community own resources solely, while 38.2% has been addressed jointly with WV/other organizations and community, and 43.1% has been addressed by WV only16. The evaluation team could not find similar data from Tavush ADP for comparison.

The extent to which the project established sustainable capacities, processes

and systems Participation provides a way for local authorities to strengthen their communities, services and democracy. As the lead statutory body in their area, the representative body for their local community, it is in their interest to ensure that communities can participate in local level decision making process. Local authorities are key partners in the development of better participation. Many local authorities in CVA ADPs are taking on a leadership role, and are working in partnership with the community sector and other statutory bodies to strengthen participation in their locality. In particular, quantitative survey conducted among target community mayors show that: - the vast majority of mayors (75%) think that community participation significantly increases the quality of services; - 45% of village mayors consult with community residents (through participation in joint meetings) while setting community priorities, 35% consults with CAG members. Respected people of the community, education and health sector officials also considered as part of different CAGs. Table 8. Community priorities identification mechanisms used by community mayors (Stepanavan and Alaverdi)

16

For more details see the report on Evaluation Objective 2, p.19.

37

All the mayors are informed about CVA. All of them also stated that they could not address the raised issue due to limited community funds. All the mayors prove that both the quality and physical accessibility of health and education services have been improved which is not the case with municipal services. Though it should be mentioned that health and education services are mainly mentioned as crucial for community lives by community people while municipality activities rarely mentioned. Apparently this is the reason for community people not to focusing on municipality activities. One of the potential risks of promoting improved access to quality public services through application of CV&A approach is the fact that the main components of the approach are within the scope of responsibilities of community councils according to the national legislation17. Hence, it is recommended to strengthen the role of community councils by gradually involving them into different stages of the CV&A process.

Thus in order to provide sustainability of the project model it is quite reasonable to consider the following precondition: local authorities should have a duty to work with communities, community organisations and other statutory bodies to ensure that effective participation takes place in their locality. This set of duties would provide a mechanism to stimulate the development of participative practices in all local bodies, not just local authorities. Local authorities would be empowered to ensure that other service providers adopt participatory methods. The aim is to line up local processes so that citizens and community organisations can easily understand how to get involved, and how to influence the services that they care about. Analysing the possibilities of CVA model sustainability, the following issues identified as a result of evaluation need to be considered. There is so called dilemma in the demand for both participatory democracy and expertise in public management18. In the vast majority of cases citizen

17

According to the Republic of Armenia Law on Local Self Governance, each community has a community council

that has its roles and responsibilities set forth by the Law. Responsibilities of this elected body include but are not limited to a) ensuring citizen participation in the process of identification of community priorities that should inform community development plans; b) ensuring that selected representatives of community people take part in actual process of the development of community plans; c) conducting regular open meetings of the council, during which community people should participate and discuss the status of implementation of community development plans.
18

Anderson, J E., Public policymaking. Boston: Houghton Mifflin, 1990

38

participation should enrol people without necessary technical competence in order to keep so called no finger print on the process. But from the other hand technical competency is required in public management which makes it impossible to maximize both value preferences. Hence, people who are lacking in expertise or who lack confidence in their competence may exempt them from the participation. Evaluation results show that the lack of technical competency mainly refers to lack of knowledge about available funds in community budget and fundraising skills. Participation can be limited to the mere sanctioning of plans and policies and to the acquisition of respectability by the local authority in the community. This strategy indicates that citizens react to the proposals of service providers (including local authorities) more often than they propose their own concepts of future action. Another issue in the degree to which people may feel obliged to participate in public management is based on the variety of diversions for citizens which occupy their non-working time; barriers such as age and the illiteracy of some citizens; and the fact that some segments of the population may have little exposure to the media which could inform them of problems and possible solutions. In view of this, one should be realistic about the number of people who will be able to play an active role in the planning process. Unfulfilled expectations are another barrier to sustainable citizen participation. It can be predicted that people who feel that their participation will have little or no effect on the activities of local government will be reluctant to participate in planning. This attitude may also be the result of people's past experience, in which they have participated but have not had any noticeable effect on the course of events. All the time different people come, gather us, make promises and...so what? This is only useless talking and nothing more Extract from FG with Vardablur community members (the poor) The most obvious reason for this behaviour may be that, because of inadequate information, people often ask for things which cannot possibly be provided. It may also be that, because of inadequate information, people believe that their participation will achieve much more than it actually will. The main core of project sustainability is the establishment of CBONs in target communities. The CBONs are well-equipped and run by a competent staff. As all surveyed groups state this project is one of the major and important steps within the process of establishing self-sustainable community organizations. So far CBONs are functioning with the support of WV Armenia ADPs until they are ready to ensure their self-sufficiency. Both CBONs are actively functioning in the framework of fundraising activities, capacity building trainings. The CBONs jointly with the ADPs have developed a capacity building plans that is aimed at ensuring their further sustainability. The CB strategies are being actively implemented with involvement of the WV Armenia NO specialists and external experts as required. The strategies will be revised to ensure effectiveness thereof in terms of current capacities and gaps. Another core element in the sustainability system are trained and knowledgeable LGs. This assists them to understand the importance of cooperating with CBOs in addressing the community needs. Numerous joined activities implemented by LGs and CBOs created an atmosphere of trust and mutual accountability, which is preferable patter of interaction between the two bodies. All members of CBON agreed that as the result of the project the relations between LGs and community improved. All activities directed to increase knowledge on state entitlements on social services members of target communities are knowledgeable about the CBPM method and have an experience of analyzing and monitoring local public

39

service delivery processes. Community members are able to identify problems and address those problems in cooperation with CBON. Due to CBPM involvement and ownership of local communities of the local development activities increased. Data collected by CBON is stored in a joint database in order to identify issues for national level policy advocacy and/or problems demanding implementation of long term activities on local level.

Surveyed target groups states that community itself is the dominant stakeholder who needs to be responsible for continuation of citizen participation activities in decision making process. The problem is that so far almost all community residents involved in the survey perceive the whole process as one of WVA activities. Moreover, the major part of issues identified by community meetings were solved fully or partly through participation of WVA NO. To address this issue for the best sustainability of the community processes started WV Armenia has designed the next cycle of the project with consideration of the findings and recommendations presented in this report. More specifically the new project design envisages closer involvement of and collaboration with the community councils, who, as an institution, are the primary responsible for ensuring community participation in community level decision making through participatory planning, budgeting and monitoring thereof. The project is designed for three years and will be funded through WV UK PPA funds.

40

Conclusions and Recommendations Conclusions and Suggestions A) Project main achievements Improved quality of public services Significant increase of awareness on entitlements for public services among target community residents and service providers Increased level of social accountability Increased level of social solidarity and cohesion Dialogue between service providers and ordinary citizens. Structured and regular social participation Community action plans developed and implemented.

B) Project risks and limitations The absence of baseline evaluation significantly limited evaluation teams understanding of changes brought about by project initiative. Community councils have the mandate and similar functions in terms of ensuring the citizen participation in the identification of local priorities. There is a risk of creating a parallel structure by ADPs and CBOs. Low capacity among community representatives in identifying concrete steps for tackling local priorities. The CV&A database is less operational and more time consuming exercise. Some of ADPs TDFs have insufficient capacity in facilitating and conducting community discussions. Lack of awareness of available community budgets, which results in unrealistic action plans. Community councils and mayors are reluctant to conduct open discussion of the community budgets.

C) Recommendations

41

Provide ongoing capacity building training for CBOs and CBONs to enable strong capacity at local level (including their role in accessing and updating the knowledge on state entitlements on behalf of their communities) as well as ensure influence in higher level policy and decision making. Strengthen capacity building activities for community discussions facilitators. Provide educative trainings for community members before group discussions to for CVA process and goals better understanding. Continue supporting CBOs and CBONs at the ADP level add the technique of town-hall meeting into CV&A process in order to facilitate regular meetings between ordinary citizens and service providers/decision makers at both local and national levels. Conduct regular assessment of community residents knowledge about state entitlements to better target education activities.

Involve sectoral experts, as may be needed, in designing action plans and identifying the best possible solutions to the issues that are raised and prioritized by community residents through participating in community meetings and discussions. Strengthen activities to adjust community action plans with community development annual/4 year plans. The process of creating demanding citizens should go in parallel with creating responsible citizens Strengthen the role of community councils by gradually involving them into different stages of the CV&A process. Provide better connection with government planning cycle, such that the outcomes of the FGD, general meeting and action planning can inform government planning processes Elaborate structured process for regional and national meetings, i.e. issues raised, agreements made, issues pending, and actions to take Redesign the CV&A database making it more practical and flexible in terms of relevance to the ADP needs

42

Lessons learned from the Evaluation Process

Indicators need to be more clearly defined with involvement of respective Experts during the design, as well as at preparation for evaluation. As for data analysis process, it was quite hard because of huge secondary data received from FGDs. Hence, survey tools need to be more clearly defined. The quality of some of the evaluation findings collected through FGDs makes the further training of the ADP staff in FGD facilitation imperative for WV Armenia. It will be considered in the following evaluations as part of the preparatory work. The timeframe set for the evaluation report in the ToR proved to be slightly ambitious given the level of evaluation objectives, the scope of evaluation findings collected through the wide range of tools subject to analysis, as well as the number of evaluation team members to provide feedback and amendments to the drafts in terms of the time required for consolidation and addressing.

43

Appendices
Appendix I) FGD guide for community residents Objective 1. The Relevance of the project design to the existing needs

How would you describe the general situation in your community? What is the situation with social services delivery (education, health, etc)? What were communitys main issues and how they were solved over the past 3 years? How did community life changed over the last 3 years? How did community assist to those changes?

Objective 2. Project effectiveness

a.

b.
c. d. e. f. g. h.

Which structures/ services/ are more important for your/your family vital activity? Please assess the effectiveness of the following structures/services activities from the perspective of impact on community well-being /please score based on a 5-grade scale, where 1 means least effective, 5-most effective/ Regional governors office Village municipality School Kindergarten Medical services Church NGOs operating in the community Political parties Have your heart about community meetings/ gatherings/discussions organized/facilitated by World Vision? Have you participated in such meetings? If not, why? Who usually participate in mentioned community meetings/ gatherings/discussions/? How you learnt about those meetings? What made you participate? Why other community members participate (or not participate) in those meetings? What are those meetings general goal, main objectives? At what extend your ideas/opinions were reflected in activities/projects/ implemented as the results of those community meetings? What must be changed in the procedure of those meetings and what needs to be kept in the same way/format, place, time, frequency, participants, etc/

Objective 3. Integration of the project with the overall ADP design What do you think, how community meetings/ gatherings/discussions are linked with other activities/projects implemented in your community by World Vision?

Objective 4. Analyse the impact of the project model on citizen empowerment Are you aware of state entitlements over quality of education, health and public services? What entitlements/criteria do you know? How did you learn about those entitlements? What is the States role in providing those entitlements? How the knowledge about state entitlements does help you/community? How the issues related to social services quality were presented to authorities before organizing community meetings?

44

What major changes have taken place over the last 3 years from the perspective of raising community issues? What are the major achievements, results?

Objective 5. Analyse the sustainability of the project model Who is responsible/who is facilitating/ community meetings/ gatherings/discussions? How the process will go on without World Visions support? Would you like to participate in such meetings in the future? How the meetings must be organized in future?

Appendix II) FGD guide for CBON/CAG manager/staff Objective 1. The Relevance of the project design to the existing needs

Can you, please, describe when CBON/CAG was created? How it created and what are the purposes of this group? How long have you been the member of CBON/CAG? What are the main reasons of your participation in this group? What was the process of CBON/CAG members selection? (nominated, volunteered, previously/currently represented/s community in other structures) How you ensure equal representation of all members of community in CAG initiatives? (ASK ONLY CAG members) How your role has changed since the group began working? Did the composition of the CBON/CAG changed over the time? What capacities do you need to improve your activities within CBON/CAG? What mechanisms/channels existed before (CBO establishment) for community people to influence decisions regarding service provision in health/ education/ LG sectors? Objective 2. Project effectiveness In your perspective what are the objectives of CV&A. Can you name the steps of CV&A.? What are the actual results? (Negative/positive). How it influenced the quality of your services. How easy is to organize CV&A methodology in this area? What is the level of CV&A methodology efficiency (to what extend the inputs are correspondent with the outputs?) What is your contribution to the results? What was the contribution of other stakeholders (Others government, local state officials, World Vision, community members)

How did this project transform your relationships with these actors? How did this project transform your relationships with community? What elements of this process are proven to be effective and what needs to be improved?

As a CBON to what extend you think that your voice is heard by officials? Do you think that without WV the situation will remain the same or be different? What was the most significant change brought due to this project? What barriers of the project were identified and how they were addressed? Objective 3. Integration of the project with the overall ADP design How the CVA interlinked with other works supported by WV?

45

To what extend the information generated through the process (data base including) guide your actions and influence officials? How you prioritise issues raised by your CAG members? Can you, please say what kinds of projects/initiatives not connected with VW were implemented in your community by CBON? Objective 4. Analyse the impact of the project model on citizen empowerment What social groups of community take part in CVA processes (more men than women, young, old members of community... etc?) Was participation inclusive of all stakeholders groups and interests? Approximately how many people attended participation activities? Please, describe in more details how community members participation process did take place (Examples, cases) What types of issues are usually out of your influence to facilitate for solution? How do you report back to your members and communities? What is the added value of CBON in the process of CVA implementation? Objective 5. Analyse the sustainability of the project model Who is responsible for CV&A implementation?

In what ways do you think these activities could be continued without World Visions involvement? Do you think that this process will be continued in the future? How? What will be your role?

Appendix III) KII for ADP manager/staff Objective 1. The Relevance of the project design to the existing needs To what extent do you feel that the project is meeting community needs? Please describe in what way(s)? Are there any ways that it is not relevant? What assumptions do you think are necessary to enable the project objectives to be met? How participatory do you feel the project design process was? What opportunities existed for community participation? What opportunities existed for ADP staff participation? Do you know what the origin of this project and project objectives? Objective 2. Project effectiveness In your perspective what are the objectives of CVA? Can you name the steps of CVA? What are the actual results? Please provide specific examples? What are the enablers of results achieved? What are the barriers or constraints that you think still limit achievement? Are there any examples of improved quality of services? To what extent are changes as a result of service provider and/or LG actions? (i.e. high responsive) What have been the enablers of change, what have been the limitations? What outcomes of the project have surprised you the most? What is your contribution to the results? What was the contribution of other stakeholders? How do you feel your local level activities have informed WVA policy positions and influence activities at the national level? How do you capture and share the results / achievements with (1) community based stakeholders / (2) ADP / (3) NO / (4) SO? Do you think you are effective, are there ways that you can improve communication?

46

How do you think this project transforms the relationships of different actors (LG, service providers and citizens) in the community? Do you think the project has changed the way in which WV (ADP) relates to community and/or government? If so how? If it is the same in what ways? What should be avoided in implementation of the CVA approach? Do you have any reservations about the scaling up of CVA? If so, what are they? How equipped do you feel to facilitate this process?

Objective 3. Integration of the project with the overall ADP design Do you think this project helps to achieve the overall ADP program goal? If so, in what ways? Are there any ways in which there are contradictions between this approach and other operations of the ADP? How are the processes and objectives of CVA integrated within your overall ADP approach?

Objective 4. Analyse the impact of the project model on citizen empowerment What types of participation has been facilitated through the project? What forums / opportunities for participation are there / different levels (service facility / LG / Marz/ Did these opportunities exist before the project? Has the project contributed to increased influence of community members in how services are delivered and planning, management and budgeting decisions? In what ways, please provide examples? In what ways are the ADP designs, plans and budgets being influenced by community influence? How responsive do you feel service providers and local government are to the voices and influence of community? What enables their response, what are their constraints? Who participates in the CVA activities? Who doesnt participate? What special efforts, if any, are made by the project to encourage inclusion participation? What enablers for participation exist in the community for different groups? What are the challenges to participation for different groups? Who are the voiceless / most vulnerable in the community? What role have they had in the project activities? What value do the CAGs / CBONs have to supporting community participation? How representative are the actions of CAGs and CBONs to the expressed needs of communities? To what extent are citizens more aware of their entitlement to, and also their responsibility for primary education and health care? Did you ever participate or observe FGDs, community gatherings, marz level meetings? Give examples of such meetings or dialogues. Thinking about the FGD, community gatherings, marz level meetings, what went well during different dialogues? What factors contributed to making different processes work well? What areas needed improvement during such dialogues?

47

What has been your role in gatherings and dialogues? To what extent are you satisfied with your role during such dialogues/gatherings? How are issues concerning women, children and PWDs represented at gatherings and dialogues? What have been key outcomes from resolving such issues affecting these groups?

Objective 5. Analyse the sustainability of the project model Who is responsible for CVA implementation? We talked about changes above; do you think the results from the project will be sustained longer term? If so how, if not why not? Do you think this process will be continued in the future? How? What processes / structures for participation exist which will support participation beyond World Vision presence? What needs to be done for CVA activities to continue without World Vision involvement? To what extent do you think relationships between government and community can be continually strengthened beyond world vision presence? How equipped are CAGs and CBON to take on responsibility to leading processes long term? What strategies do you have to increase capacity of CAGs/ CBON to take on responsibility? What more needs to be done to better equip them to take on the role? To what extent is the local government supportive of the CVA processes and willing to sustain opportunity for community influence and the project objectives? Are there potential entry points for more linkage with local governance structures for citizen participation? Do you think this process should be continued in the future? Why, why not?

Appendix IV) KII guide for regional Officials Objective 1. The Relevance of the project design to the existing needs Do you see a role for citizen participation in governance and service delivery decision making? In what ways? Objective 2. Project effectiveness What do you understand to be CVAs objectives? To what extent do you feel the project has been successful in achieving these objectives? What in your opinion is the most significant change that has resulted from the initiative? Do you know of specific examples of priorities identified as a result of CVA processes?

48

Do you know if these priorities have been addressed? What opportunities and constraints does local and regional government face in addressing issues raised? From your knowledge of the initiative, what recommendations do you have to enhance the effectiveness of the process?

Objective 3. Integration of the project with the overall ADP design Can you tell us what else you know of WVs work in Armenia? Do you feel methodologies such as CVA are a valuable development approach for NGOs like WV? Why?

Objective 4. Analyse the impact of the project model on citizen empowerment Did channels for citizen input and influence with regard to governance and service delivery exist at local levels 5 years ago? What forms, now? Did channels for citizen input and influence with regard to governance and service delivery exist at regional levels 5 years ago? What forms and now? Who (of local communities) do you feel is least likely to participate in CVA-type processes? Are there ways this may be addressed?

Objective 5. Analyse the sustainability of the project model Who do you feel should lead CVA-type processes? Are there potential entry-points for the more systematic usage of such mechanisms for citizen voice at local and regional levels?

Appendix V)

KII guide for service providers Community name_____________________________ Interviewees N/S, position____________________ Interviewers N/S______________________________ Objective 1. The Relevance of the project design to the existing needs

What do you think, how much does civic participation assist to increasing the level of accessibility of social services? How have you participated in the elaboration of Policy influence through citizen empowerment project? Can you, please, tell how the process went on? Who else participated in the designing process? What mechanisms of the project especially facilitate community participation processes? How?

Objective 2. Project effectiveness Which are the project main objectives? Which are the project main steps? What where project significant achievements? How did the project affect the quality of community services? What was village municipality participation in the projects implementation as the result of community meetings/ gatherings? How did different stakeholders participate in different projects implementation as the result of community meetings/discussions /State, community, CBON, CAG./ What elements of CVA-type process should be kept and what needs to be changed / improved?

49

How did your relations with community/service providers changed as a result of the project?

Objective 3. Integration of the project with the overall ADP design

Do you feel that this approach is relevant to your community contexts? In what ways it is relevant? Are there any ways that it is not relevant? How is this approach linked to other activities/projects implemented by WV in your community?

Objective 4. Analyse the impact of the project model on citizen empowerment What types of participation has been facilitated through the project? Did these opportunities exist before the project? Has the project contributed to increased influence of community members in how services are delivered and planning, management and budgeting decisions? How responsive do you feel service providers and local government are to the voices and influence of community? What enables their response, what are their constraints? Who participates in the CVA activities? Who doesnt participate? What are the challenges to participation for different groups? Who are the voiceless / most vulnerable in the community? What role have they had in the project activities? Did you ever participate or observe FGDs, community gatherings, marz level meetings? Give examples of such meetings or dialogues What areas needed improvement during such dialogues? Objective 5. Analyse the sustainability of the project model Who is responsible for CVA implementation? How equipped are CAGs and CBON to take on responsibility to leading processes long term? What strategies do you have to increase capacity of CAGs/ CBON to take on responsibility? What more needs to be done to better equip them to take on the role? Are there potential entry points for more linkage with local governance structures for citizen participation? Do you think this process should be continued in the future? Why, why not?

Appendix VI) Questionnaire for community mayors ADP________________________ Community___________________ Community mayors N/S_________________ Interviewers N/S_______________________ a) What do you think, how much does civic participation assist to increasing the level of accessibility of social services

1. Totally 2. More assists

50

3. More not assists 4. Absolutely does not assist 5. D/K

b) Which of the following groups actively participate in the process of

raising community issues? /MARK AND PRIORITIZE 3 ANSWERS / 1. Community council 1_______________ 2. Representatives of education services 2_______________ 3. Representatives of health services 3_______________ 4. CAG members 5. CBON manager/staff 6. Church representatives 7. Community NGOs 8. Political parties 9. Village municipality 10. Community authorities /elders/ 11. Community members/citizens 12. Other______________________

c) Which of the following groups actively participate in solving 1. Community council

community issues? /MARK AND PRIORITIZE 3 ANSWERS / 1_______________ 2_______________ 3_______________

2. Representatives of education services 3. Representatives of health services 4. CAG members 5. CBON staff 6. Church representatives 7. Community NGOs 8. Political parties 9. Village municipality 10. Community authorities /elders/ 11. Community members/citizens 12. Other__________________

d) How do you, usually, make decisions on community priorities? 1. Consult with community council
2. Consult with CAG/CBON 3. Meet with education/health services representatives 4. Participate in community gatherings, meetings, discussions 5. Make decisions by your own 6. Other_____________________________________________

51

e) How often do you cooperate with Community active groups 1. Deciding every community priority
2. Regularly 3. Seldom 4. Never

f) How often do you cooperate with Community based organization

network? 1. Deciding every community priority 2. Regularly 3. Seldom 4. Never

g) What are the most urgent community issues? _______________________________________________________________ _______________________________________________________________ _______________________________________________________________ ________________________________________________________________ h) Are you aware of CVA tool? 1. Yes /Pass to k)/ 2. No /ASK i); j), k) finish the interview/
i) What projects have been implemented in your community within last 3 years? Who were participating in these projects/initiatives?

____________________________________________________________________________________________ ____________________________________________________________________________________________ ____________________________________________________________________________________________ _____________________________________________________________________________________________ How do you get aware of degree of communitys satisfaction from your services /SEVERAL ANSWERS ALLOWED/ 1. Meet with community members 2. Meet with community council 3. Meet with education/health services representatives 4. Analyze community members letters of complained 5. Other__________________________ j)

k) How would you describe accessibility of community services? 1. Excellent 52

2. Good 3. Satisfactory 4. Bad 99.D/K

l) Whats your opinion about CVA tool effectiveness? 1. Absolutely effective

2. More effective than not 3. More not effective than effective 4. Absolutely not effective 5. D/K

m) What are CVA main objectives? _________________________________________ _________________________________________ _________________________________________ n) What are the main steps of CVA process? _______________________________________________________________________________________________ _______________________________________________________________________________________________ _______________________________________________________________________________________________ _______________________________________________________________________________________________ __________ o) What are the positive and negative impacts of CVA
Positive 1. 2. 3. Negative 1. 2. 3.

p) What are the main results/achievements of CVA in your community? _______________________________________________________________________________________________ _______________________________________________________________________________________________ _______________________________________________________________________________________________ _______________________________________________________________________________________________ _____ q) How did different stakeholders participate in different projects implementation as the result of community meetings/discussions? /State, community, CBON, CAG. / _______________________________________________________________________________________________ _______________________________________________________________________________________________ _______________________________________________________________________________________________ _______________________________________________________________________________________________ _______________________________________________________________________________________________ _______________________________________________________________________________________________ _________________________________________________________________________________ r) How many of issues raised by community through CVA tool were addressed by village municipality? 1. All of them
2. Partly

53

3. None of them were addressed by village municipality s) How did CVA tool affect the accessibility and quality of the following
services? Education services quality 1. Definitely increased 2. More increased than not 3. More not increased 4. No significant changes 99. Dont/know Health services quality 1. Definitely increased 2. More increased than not 3. More not increased 4. No significant changes 99. Dont/know Education services accessibility 1. Definitely increased 2. More increased than not 3. More not increased 4. No significant changes 99. Dont/know Health services accessibility 1. Definitely increased 2. More increased than not 3. More not increased 4. No significant changes 99. Dont/know

Community services quality Community services accessibility 1. Definitely increased 1. Definitely increased 2. More increased than not 2. More increased than not 3. More nor increased 3. More nor increased 4. No significant changes 4. No significant changes 99. Dont/know 99. Dont/know t) Who should lead CVA-type process? 4. Village municipality 5. Community council 6. CAG 7. CBON 8. World Vision 9. Community members 10. Other____________________

u) How the process will go on without World Visions support _______________________________________________________________________________________________ _______________________________________________________________________________________________ _______________________________________________________________________________________________ _______________________________________

54