ACADEMIA Letters

Alternative to Impact Evaluation Methods in Developing

Countries: A Case Study of the Sahel Region of Africa
Keita Abdourahamane dit Baffa

The Sahel countries face chronic poverty and compounded disasters. They currently rank in
the bottom 20 countries within the Human Development Index (HDI) (UNDP, 2020). After
three decades of implementing projects and programs through economic aid , few tangible
changes have been achieved. Yet, it has been difficult to correlate these changes to any one
intervention. Hence, Impact Evaluation (IE) of project through an appropriate approach is
necessary in the context of the Sahel for many reasons. First, IE is a democratic requirement
that fosters the respect of citizens, beneficiaries, stakeholders, and donors. Second, IE will
lead to improvement of the relevance and effectiveness of projects and programs implemen-
tation through economic aid. It will inform the decisionmaker to continue, discontinue, scale
up or replicate the intervention. Third, IE is necessary to reorient and ensure sustainability
through the capitalization of knowledge and accountability of the actors. This article exam-
ines impact evaluation methods in Sahel using an analysis of the seminal literature from IE
Methods. It attempts to supplement existing knowledge on economic aid performance by de-
veloping a new conceptual framework of impact measurement more suitable to Sahel context.
The concept is called Reflective IE. This was achieved through a triangulation approach that
includes a literature review and a two-step interview process with primary stakeholders.

1. Introduction
Development practitioners and agencies have long sought to achieve impact with their work;
often highly and intrinsically motivated to create change (Hearn & Buffardi, 2016). The im-

Academia Letters, June 2021 ©2021 by the author — Open Access — Distributed under CC BY 4.0

Corresponding Author: Keita Abdourahamane dit Baffa, keitabaffa@gmail.com

Citation: Abdourahamane dit Baffa, K. (2021). Alternative to Impact Evaluation Methods in Developing
Countries: A Case Study of the Sahel Region of Africa. Academia Letters, Article 1216.
https://doi.org/10.20935/AL1216.
1
pact includes significant changes, expected and unexpected in the lives of individuals, groups,
and environment resulting in a causal link with the intervention. According to the Organiza-
tion for Economic Co-operation and Development (OECD) “impact is a positive and negative,
primary and secondary long-term effect produced by a developmental intervention, directly
or indirectly, intended or unintended” (2010, p. 24). The principal means for creating this
change in developing countries is the effective and efficient implementation of well-designed
programs and projects through economic aid. Developmental programs and policies based
on economic aids (bilateral and multilateral aid) are generally intended to produce positive
outcomes such as increased incomes, improved learning, reduced illness or malnutrition and
expand access to health services. Whether these changes could solely be attributed to eco-
nomic aids is an ongoing debate that many have attempted to resolve through IE of project
results. After more than 30 years of project implementation in Sahel countries, little changes
have been noticed in the living condition of the Sahel population. Sahel population still faced
poverty, food insecurity, compounded disaster, famine, malnutrition etc.. So, the questions of
performance of projects implemented through economic aids still remain. And one of the tool
that could measure the performance of project is evaluation. In general, two main approaches
for are used? Project/program evaluation are used:

1. Performance Evaluation (before and after evaluation using the same beneficiaries), and

2. IE (using experimental strategies with treatment and control group).

In general performance evaluation doesn’t present many challenges but IE methods have al-
ways a challenge. This research addresses the challenges that public and private organizations
face when using IE methods to measure the performance of projects/programs implemented
in Sahel financed through economic aid.

2. Research Purpose and Methodology

The purpose of this paper is to strengthen the existing knowledge by developing a new frame-
work that assists public and private organizations in evaluating the impact of projects/programs
in the context of developing countries especially in Sahel. This was achieved through a tri-
angulation approach that included a literature review, and a two-step interview process with
primary stakeholders based on phenomenological approach.

Academia Letters, June 2021 ©2021 by the author — Open Access — Distributed under CC BY 4.0

Corresponding Author: Keita Abdourahamane dit Baffa, keitabaffa@gmail.com

Citation: Abdourahamane dit Baffa, K. (2021). Alternative to Impact Evaluation Methods in Developing
Countries: A Case Study of the Sahel Region of Africa. Academia Letters, Article 1216.
https://doi.org/10.20935/AL1216.
2
3. Findings
A good IE method depends on its design and implementation. Findings from literature review
indicated that the primary difficulty private and‘ public agencies faced while measuring the
impact of projects through economic aid was the recurring problems associated with design-
ing of the IE. The difficulty was mostly due to the misconception of evaluation in relation to
the context of the intervention and the inadequacy of the IE approach to the intervention the-
ory of change. The review also showed that besides the design problems, emphasis must also
be given to the preparation of the IE methods. Project-based development and humanitarian
responses should be aligned to the context of implementation in the Sahel to bring about a
greater impact on the living conditions of the Sahel population. As previously mentioned,
the basis for any project planning in the Sahel is that disasters and shocks (drought, flooding,
famine, etc.) will happen at some stage of any initiative during planning, implementation and
after completion. It follows that projects should aim to reduce vulnerability, and reduce the
impact of those disasters, instead of having the more conventional development objectives.
Further, at some time impact evaluation using appropriate methods could be the best tool to
support Sahel countries in achieving their objectives related to improving the economic devel-
opment in Sahel. However, some concerns remain, especially when applying these methods
in the context of Sahel. For example, a major concern regarding the application of this method
is its cost-effectiveness. This cost-effectiveness is important as the “impact evaluation tech-
niques have different requirements in terms of sample size, data collection, complexity of
statistical analysis, and cost” (Hempel & Fiala, 2012). Even when we identify a best impact
evaluation method that will fit into a specific Sahel context, it may not be feasible most of the
time given the cost and the resources involved, especially when this is compared to the addi-
tional money that could be used to help more of the beneficiaries who are in need. Inputting
such an evaluative approach into practice, several challenges exist, including external validity,
ethical issues, partial or lack of compliance, attrition and spillovers that could severely affect
the quality of the impact evaluation in the Sahel context.
The semi-structured interviews focused initially on the participants’ personal knowledge,
understanding and perception of the IE methods. It was discovered that the personal thoughts
and perceptions about the IE methods were in harmony with the findings from the literature
review in some areas. For example, only a few advantages of the IE methods were identified
by the participants (13%), especially regarding the robustness and rigor of the method and
sometimes its capacity to demonstrate more evidence and attribution. With regard to the
advantages of IE, the majority of the participants (87%) stated that the design, management
and resource utilization were the most challenging factors in conducting IE. They referred

Academia Letters, June 2021 ©2021 by the author — Open Access — Distributed under CC BY 4.0

Corresponding Author: Keita Abdourahamane dit Baffa, keitabaffa@gmail.com

Citation: Abdourahamane dit Baffa, K. (2021). Alternative to Impact Evaluation Methods in Developing
Countries: A Case Study of the Sahel Region of Africa. Academia Letters, Article 1216.
https://doi.org/10.20935/AL1216.
3
particularly to the contextual issues, the cost of design and implementation and the specific
timeframe of implementation. There were also the ethical considerations, the time of the
design and implementation and the quality and expertise needed for the task. The complexity
of the method and the difficulty to attribute change to only one intervention were also cited.
In the Sahel context, these challenging factors were manifested by practical examples that
might question the application of IE methods, as seen in the following matters that were raised
from the different experiences of those actually practicing evaluation in the different Sahel
countries. Some of the salient points made are discussed below.

• Contextual Issues: Conducting IEs requires the implementation of a development project

without interruption. However, since 1990, political instability and insecurity in Sahel
countries have caused mass movements of the rural population which has affected the
smooth implementation of development projects and disrupted all efforts for impact-
ful development results. This contextual issues is completely out of the control of the
donor agencies implementing the projects in the Sahel.

• The cost of design and implementation: Based on the discussion with practitioners
of the IE methods, one person said, “the main complications start even before the IE
begins. What an expensive evaluation! and What is it useful for!”. From a few samples
of projects that went through IE, the average full cost (baseline, midline, endline) of
these methods was estimated at about 300,000 Euros (NGOs, 2018). Most practitioners
of IE believed that the exorbitant sums spent in such an approach could be better used
to fund other small and more resilient projects. This would mean using a much more
flexible, less complicated and affordable method to measure the impact of interventions
in the Sahel. In such a case, the remaining funds could be used to improve the living
conditions of the potential beneficiaries. This would also mean not depriving them of
an intervention from which they could truly benefit.

• Ethical Considerations: The ethical issue concerning the IE methods arose when some
potential recipients of the interventions were excluded from participation as the control
group during random control trial (RCT). This was indeed a real issue in Sahel countries
where almost all participants were potential beneficiaries and in real need. Moreover,
most of the projects were implemented in rural areas where the majority of potential
beneficiaries were extremely vulnerable and entitled to having access to the project pro-
visions. Thus, both the treatment and control groups should have had an equal chance
for getting treatment from the project interventions.

• Contamination Effect: In fact, in the Sahel context, especially in rural areas, the social

Academia Letters, June 2021 ©2021 by the author — Open Access — Distributed under CC BY 4.0

Corresponding Author: Keita Abdourahamane dit Baffa, keitabaffa@gmail.com

Citation: Abdourahamane dit Baffa, K. (2021). Alternative to Impact Evaluation Methods in Developing
Countries: A Case Study of the Sahel Region of Africa. Academia Letters, Article 1216.
https://doi.org/10.20935/AL1216.
4
 and cultural contexts and norms made it difficult for some beneficiaries to take advan-
tages of the benefit of an intervention without sharing this with others experiencing the
same need. A good example was the experience of Christian Aid (CAID) (2010) during
emergency food distribution to the most vulnerable in the Zinder and Maharadi regions
in Niger. After the departure of the program team, the village chief re-distributed the
food received among all the members of the village so that everyone was a beneficiary.
The ECHO Alliance (2015) project post-distribution monitoring in Niger was another
example During the focus group meetings with the beneficiaries, the women clearly
stated that cultural norms and customs would not allow them to be the only ones to
benefit from the cash and food when they knew that their neighbors did not have any-
thing to eat the night before. It seems clear that the social and cultural context was a
leading cause of contamination when implementing the project in the Sahel context.

• Attrition: Participants dropping out of the study was also a potential threat to the re-
liability of the IE methods used. In the context of the Sahel, it was almost impossible
to contain the attrition as long as migration was an integral part of the habits and cus-
toms of certain of the people, especially among the youths. According to Partnership
Program Arrangement’s (PPA) Participatory Vulnerable Capacity Assessment (PVCA),
during the dry season in Sahel, two out of every three young persons left the rural ar-
eas for the urban ones or migrated to other neighboring countries (CAID, 2010). In
summary, to answer the main research question mentioned above, the research com-
bined the findings from and qualitative data to identify the main characteristics of a
new framework that would be needed for IE. The subsequent section on Discussion of
the findings will detail these main characteristics.

4. Discussion
In light of above, it was clear that in the Sahel context, IE methods in development projects
faced a methodological difficulty, as opposed to the application of experimental procedures
in the field of medicine or other research domains. IE methods could not be applied in the
socio-economic and development context in the Sahel in the same manner as other sectors,
like medicine. In the Sahel’s socio-economic context, isolating the target groups from control
group might be very challenging. The application of IE methods is also faced with the almost
impossibility of having a control group identical to the experimental group in the development
and socio-economic context. It is often necessary to control several factors of dissimilarity
among the groups. Furthermore, The scope of change produced by anyone intervention can

Academia Letters, June 2021 ©2021 by the author — Open Access — Distributed under CC BY 4.0

Corresponding Author: Keita Abdourahamane dit Baffa, keitabaffa@gmail.com

Citation: Abdourahamane dit Baffa, K. (2021). Alternative to Impact Evaluation Methods in Developing
Countries: A Case Study of the Sahel Region of Africa. Academia Letters, Article 1216.
https://doi.org/10.20935/AL1216.
5
be very broad and attributing its impact on any given project is really difficult in the Sahel
context. Even with the most rigorous methods, in the social sciences the attribution of change
to a given project remains uncertain. The context of project implementation, especially in
the Sahel, has too many constraints and the IE methods do not eliminate all the uncertainties
about the attribution. It is difficult for the IE methods applied to control the many contributing
factors in development and for an individual to arrive at the conclusion about exactly which
changes can be attributed solely to a given project. However, for innovation within the project,
quasi-experimental designs could be used for learning and replication. Nevertheless, even
with this, it is better to integrate IE into the interventions by selecting only pathways from
the project theory of change that are not well-known in terms of project impact and not well
experimented by implementers. This becomes more of a learning agenda on a small scale to
be implemented and replicated for future interventions.
Finally, the study set out to answer the main research question posed which was: What are
the characteristics of a new conceptual framework that would better evaluates the impact of
projects funded through economic aid with respect to decreasing the poverty levels in the Sahel
region? To accomplish this, the research focused first on the effectiveness of the evaluation
methods used in the Sahel during the past ten years. Second, it identified the advantages and
challenges of these methods to arrive at the conclusion about the main characteristics of the
new framework for doing this which would require the followings:

• Design: There is the need for a more rigorous method of evaluation that can actually
demonstrate improvements in the living conditions of the beneficiaries. Such an ap-
proach should not present any ethical issues that prevent any participants from either
the treatment or control groups from benefitting equally from the intervention. Lastly,
the approach should not need a high level of expertise for implementation and it should
be easily adapted to suit different social and cultural contexts.

• Management: There should be a more flexible approach that does not need a specific
timeframe and can be implemented throughout the life of the project. In addition, there
should be a greater focus on the project’s internal monitoring system.

• Resources: The approach used should be affordable in terms of cost, even for smaller
projects which can create a balance between the methodology used and the achievement
of good quality results. Furthermore, the cost should be integrated within the regular
budget of the M & E system.

• Utilization: A robust evaluation method is needed to generate the kind of evidence that
provides for the acquisition of more accurate and reliable information that will lead to a

Academia Letters, June 2021 ©2021 by the author — Open Access — Distributed under CC BY 4.0

Corresponding Author: Keita Abdourahamane dit Baffa, keitabaffa@gmail.com

Citation: Abdourahamane dit Baffa, K. (2021). Alternative to Impact Evaluation Methods in Developing
Countries: A Case Study of the Sahel Region of Africa. Academia Letters, Article 1216.
https://doi.org/10.20935/AL1216.
6
 better understanding of the situation. This would then facilitate the proper management
and adaptation of these projects to meet the real needs of the recipients in Sahel

5. Contribution to Knowledge
The integration of the different findings led to the generation of a new framework of IE meth-
ods context called Reflective IE within the Sahel context, as illustrated in Figure 1 (attached)
Based on the characteristics of the new framework mentioned above, Reflective IE is de-
fined as a combination of performance evaluation and reflective monitoring which could be
the best alternative to IE methods for evaluating projects/programs in the Sahel. It should
further improve existing models (in particular performance evaluation) with more emphasis
on internal monitoring and this should be easily integrated into the normal project monitoring
and evaluation system. It will be easily imbedded into the regular system for monitoring and
evaluating the project. Reflective IE is composed of three steps, with 1 and 3 being a part of
the normal evaluation system, while steps 2 would be part of the routine monitoring system.
The combination of these three steps meets the requirement of the characteristics of the new
framework.

Academia Letters, June 2021 ©2021 by the author — Open Access — Distributed under CC BY 4.0

Corresponding Author: Keita Abdourahamane dit Baffa, keitabaffa@gmail.com

Citation: Abdourahamane dit Baffa, K. (2021). Alternative to Impact Evaluation Methods in Developing
Countries: A Case Study of the Sahel Region of Africa. Academia Letters, Article 1216.
https://doi.org/10.20935/AL1216.
7