Journal of Development Effectiveness

ISSN: 1943-9342 (Print) 1943-9407 (Online) Journal homepage: https://www.tandfonline.com/loi/rjde20

Improving well-being through mobile money: a

replication study in Niger

JP Meneses, ET Ventura, OA Elorreaga, C Huaroto, GG Aguilar & EP Beteta

To cite this article: JP Meneses, ET Ventura, OA Elorreaga, C Huaroto, GG Aguilar & EP Beteta
(2019): Improving well-being through mobile money: a replication study in Niger, Journal of
Development Effectiveness, DOI: 10.1080/19439342.2019.1679860

To link to this article: https://doi.org/10.1080/19439342.2019.1679860

View supplementary material

Published online: 05 Nov 2019.

Submit your article to this journal

Article views: 9

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=rjde20
JOURNAL OF DEVELOPMENT EFFECTIVENESS
https://doi.org/10.1080/19439342.2019.1679860

ARTICLE

Improving well-being through mobile money: a replication study

in Niger
a,b
JP Meneses , ET Venturac, OA Elorreagaa, C Huarotoa, GG Aguilara and EP Betetaa
a
GMINIS Research Group, Pontificia Universidad Católica del Perú, Lima, Peru; bAlberto Hurtado Medical School,
Universidad Peruana Cayetano Heredia, Lima, Peru; cSchool of Economics, Universidad Peruana de Ciencias
Aplicadas, Lima, Peru

ABSTRACT ARTICLE HISTORY

Around the world, there has been an exponential growth in cash-based Received 31 May 2019
programmes as part of social policies. Innovative payment mechanisms, Accepted 4 October 2019
such as mobile money, could have benefits for recipients. An experimental KEYWORDS
study suggests that the use of mobile money, Zap delivery, might improve Impact evaluation;
intra-household socio-economic dynamics on Niger. Our replication study replication study; mobile
evaluates the impact of Zap delivery in contrast to more traditional money; cash transfer
delivery mechanism. We examined original author´s findings and esti- program; nutrition
mated similar results about increasing household diet diversity without
decreasing their durable and nondurable assets. The heterogeneity eva-
luation by age groups suggests that the Zap delivery had a different
impact on older beneficiaries than younger ones. In addition, we evalu-
ated the robustness, which considers multiple imputation and Lee bounds
analysis; as well as a nutritional evaluation of children under 5 years using
anthropometric measures.

1. Introduction
Currently, there is an increased interest in cash-based programs. By 2013, these public policies had
been placed in at least 119 countries, including 37 in Africa. This context presents an unprecedented
opportunity to increase financial inclusion using a new payment channel that can decrease the
operating costs, improve the security and increase the outreach. Replacing cash transfers with other
payment mechanisms, such as mobile money, could have additional benefits and advantages for
recipients.
Our study independently assesses the findings in Aker et al. (2016) study, Payment mechanisms
and antipoverty programs: evidence from a mobile money cash transfer experiment in Niger.
The original research analysed an innovative mechanism of transference after severe droughts in
a humanitarian context. This study was a randomized controlled trial with a sample size of 96
villages1 that had been previously classified by the government of Niger as having produced less

CONTACT JP Meneses luis.meneses@upch.pe Alberto Hurtado Medical School Universidad Peruana Cayetano Heredia,
H. Delgado 430, SMP, Lima 31, Lima, Peru
Published Replication study: Meneses, JP, Ventura, E, Elorreaga, O, Huaroto, C, Aguilar, G, and Beteta, E, 2018. Mobile money and its
impact on improving living conditions in Niger: a replication study. 3ie Replication Paper 19. Washington, DC: International Initiative for
Impact Evaluation (3ie). Available at: http://doi.10.23846/RPS0019
Published Replication Data: Meneses, JP, Ventura, E, Elorreaga, O, Huaroto, C, Aguilar, G, and Beteta, E, 2018. Mobile money and its
impact on improving living conditions in Niger: a replication study. 3ie Replication Paper 19. Washington, DC: International Initiative for
Impact Evaluation (3ie). Available at https://doi.10.7910/DVN/3OZJ2Y
Supplemental data for this article can be accessed here
© 2019 Informa UK Limited, trading as Taylor & Francis Group
2 J. MENESES ET AL.

Figure 1. (a) Mean cost, (b) Mean cost (original paper).

than 50 per cent of their consumption needs during the 2009 harvest. The objective of the study was
to evaluate impact of three cash transference mechanisms:

● Cash: This mechanism provided the cash transfer manually, distributed in individual envelopes.
● Mobile: This mechanism included a manual cash transfer, a mobile phone with mobile money
transfer (m-transfer) enabled, training on how to use it and an account.
● Zap: This mechanism included an m-transfer-enabled, training on how to use it and an account;
the transfer was performed via the mobile phone and the beneficiaries had to take the mobile
phone to an m-transfer agent to obtain the transfer.

This study provides evidence that the Zap households bought more types of food items and increased
their diet diversity without decreasing their durable and nondurable assets. Furthermore, differences
were primarily due to the m-transfer intervention and not to the mobile phone.
The original study is particularly interesting for replication because its intervention design
allowed to isolate the potential effect of Zap respect to the other mechanisms of transference. In
addition, the data availability and original author`s cooperation were exceptional opportunities
that allowed us to reanalyze and make an independent inspection of their results. This circum-
stance is unusual and a minority of economic papers had been McCullough, McGeary, and Harrison
(2006, 2008), even when replicability is an ideal standard of good scientific practice (Mueller-
Langer et al. 2019).
In relation to replication studies, there is a general discussion about its relevance. According to
Brown, Cameron, and Wood (2014), internal replication research is essential because single studies
can strongly influence policy; and potential external replication studies (where the intervention is
conducted again in similar contexts) are difficult and rare. In this sense, there are initiatives interested
to show the availability and data access in journals for future replication studies. (Wood, Müller, and
Brown 2018). These practices will improve decisions to avoid scaling up ineffective interventions or
effective only in certain limited situations.
According to Klapper and Singer (2014), advancements in technology and electronic platform-
based business models have allowed many governments to increase the efficiency and scope of their
electronic-payments infrastructure. In this way, governments may benefit from making payments
more efficiently by lowering the cost of disbursing and receiving payments (CGAP 2011; Holloway,
Rouse, and Niazi 2017). Furthermore, electronic payments could increase incentives to save and
improve women’s economic participation and empowerment. In this sense, Aker et al. (2016) provide
new evidence about the benefits and opportunities of digital payment services such as mobile
money.
 JOURNAL OF DEVELOPMENT EFFECTIVENESS 3

Using electronic payments as delivery channels for transfers can further improve the efficiency of
government social programs. We reassessed the study independently from the results published by the
original authors in order to review and confirm the findings from the original article.

2. The pure replication

In pure replication (PR), we reconstructed the original article using cleaned data, kindly shared by the
original authors.2 . Hence, we estimated the two regression models found in the original publication:

Yiv ¼ β0 þ β1 Zapv þ β2 Mobilev þ β3 seedv þ θC þ εiv (1)

Yiv ¼ τ0 þ τ1 Zapv þ τ2 Cashv þ τ3 seedv þ θC þ μiv (2)

Yiv represents the outcomes of individual or household i in village v. Cashv is an indicator for villages
assigned to the cash group. Zapv is an indicator for villages assigned to the Zap group. Mobilev is an
indicator for villages assigned to the mobile group. The coefficients that captured the impact of the
mechanisms are β1 , β2 and τ1 .
Moreover, seedv is an indicator for the presence of a seed distribution programme at the village
level. Lastly, θC is geographic fixed effects at the commune level, which control potential differences
at commune level.3 Finally, εiv and μiv are the error terms.
The first regression allowed us to reproduce the Zap-cash coefficient (β1 ) and the mobile-cash
coefficient (β2 ), while the second regression described the Zap-mobile coefficient (τ 1 ). Following
Aker et al. (2016), we clustered the error term at the village level to account for the programme
design and correct for heteroscedasticity. The Pure Replication tables are presented in Appendix B.

2.1. Replication results

The results from the PR are presented and discussed in detail in Meneses et al. (2018). The topics
under evaluation were uses of cash transfer, impact on food security and nutritional status, timing of
cash transference expenses, mobile phone usage, intrahousehold decisions-making and mean cost
(in kilometres and hours, see Figure 1). Our PR findings are very similar to the original paper ones. The
original conclusions were confirmed with small changes on the number of observations considered
in the regression or the standard error (SE) value. The key findings are described as follows:

(i) Uses of cash transfer

The original results report 1,047 observations used for estimation on number of food and nonfood
items purchased; in contrast, our PR used among 1,047 and 1,035 observations for this purpose. In
addition, we found a difference of rounding in one of the SE for the variable transfer used to buy
staple grains (millet, sorghum). The original value of the SE is 0.01 for the mobile-cash coefficient,
whereas in the PR it is 0.00.

(ii) Impact on food security and nutritional status

For this component, the original authors considered a recognized indicator in the literature, the
household dietary diversity score (HDDS) (Kennedy, Ballard, and Dop 2011).4 Our PR used this
variable and confirmed the original results; Zap scores 0.51 higher than mobile (p < 0.01), and 0.28
higher than cash (p < 0.1).
The results are consistent with evidence about the impact of access to a new storage value, in this
case, m-transfer. In northeastern Burkina Faso, an area with low rainfall and a high propensity for
droughts, Gash and Gray (2016) showed that providing ways of storing value could significantly
4 J. MENESES ET AL.

improve households’ abilities to withstand shocks without having to reduce consumption or sell
livestock.
In relation to child nutritional status, we found a decreased level of statistical significance of the
variable diet diversity of children under five in relation to the original findings. The mobile-cash
coefficient estimated by our replication process does not have statistical significance, whereas it has
significance (p < 0.1) in the original study results.
Another difference was found in the standard deviation (SD) of household diet diversity score
mean for Cash group; we calculated 1.69, while the original authors calculated 1.70. Likewise, we
found another difference in the SE of coefficient Mobile-Cash for number of asset categories owned
(out of 11, excluding mobile phones): we estimated 0.10, whereas the original authors estimated
0.11. These differences are attributable to potential round-off errors based on software estimations
(McCullough and Vinod 1999).

(iii) Timing of cash transference expenses and mobile phone usage

The original study found statistically significant differences across the three treatment arms for the
variable obtained transfer the same day. In this case, Zap households were 36 and 39 percentage
points less likely to receive their cash on the same day it was available, in comparison to mobile and
cash households, respectively. Our PR found the same results.
Likewise, in relation to use of mobile phone, the original authors found that Zap and mobile
households were 33 and 15 percentage points more likely to use their mobile phones in comparison
to cash household, respectively (p < 0.01). Besides, Zap program recipients were 30 and 15 percen-
tage points more likely to make or receive calls than cash and mobile program recipients,
respectively.

(iv) Intrahousehold decision-making

Aker et al. (2016) evaluated three categories: decision-making regarding cash transfer, women’s
involvement in agriculture and clothing expenditures for Muslim festivals. The original analyses and
our PR obtained the same conclusions; there are statistically significant differences between the Zap
and cash groups in terms of visiting the market, selling grain for household and buying clothes for
children.5 Zap recipients were more likely to travel to weekly markets, were more involved in selling
grain and increased their expenditures on children’s clothing for festivals than were mobile or cash
households. According to Jack and Suri (2016), mobile money could contribute to the alleviation of
women’s poverty.

(v) Mean cost (in kilometres and hours)

Original authors presented evidence of the reduced costs to obtain the cash transfer. There are two
groups, because mobile and cash households received the cash transfer by the same mechanism.
The original study shows that Zap recipients incurred in lower costs in terms of kilometres travelled
and hours used to obtain the transfer. However, our PR identified some differences. In the original
article, the mean of distance (kilometres) to the cashpoint for Zap is around 1 while 0.8 in the PR.
Moreover, the mean of distance (hours) to cashpoint for cash/mobile is around 0.4 in the original
article and around 0.6 in the PR.
Our replication study presented some challenges. The original findings were obtained from two
documents shared by the original authors, the published version with the erratum and the corri-
gendum which are reported as original result tables in Appendix A. Furthermore, the authors
affirmed that per recipient cost was US$16.43 in cash and mobile villages and US$24.14 in Zap
villages. However, we could not identify any variable that allowed us to replicate the authors’ cost
calculations.
 JOURNAL OF DEVELOPMENT EFFECTIVENESS 5

3. Measurement and estimation analysis

The measurement and estimation analysis (MEA) can be implemented by redefining and recalculat-
ing the variables of interest, introducing additional control or interaction variables and using
alternative methodologies (Brown, Cameron, and Wood 2014).

3.1. Exploring heterogeneous impact

Aker et al. (2016) evaluated the impact of m-transfer as a cash transference mechanism for women in
vulnerable household. These beneficiaries had heterogeneous distribution of age and education,
features that could influence the impact of this intervention. In regarding, other studies explore the
heterogeneous impact of similar interventions based on head of household characteristics like age and
education level (Dammert 2008). Likewise, there is evidence of a higher probability of ICTs usage among
populations of 14–25 years old in rural areas in Peru (Guerrero and Ritter 2014). Furthermore, the mobile
money is more likely to be used by younger, wealthier and better educated populations, which would
suggest that these groups find more advantages in this financial tool (Mbiti and Weil 2011)
Based on this evidence, we proposed to analyze the heterogeneity of money transfer interven-
tions, according to the age and education of beneficiaries. However, we did not find variables to
define beneficiaries’ education level. Therefore, we could not explore the heterogeneous treatment
effects of education, even when other studies showed its impact on dietary diversity, nutritional
status (Oyekale and Oyekale 2009) and female empowerment (Iversen and Palmer-Jones 2014).
For our analysis, we hypothesized that younger program recipients would obtain more benefits
from m-transfer. It should be noted that the survey did not have a stratified design by age.
The beneficiaries6 were grouped based on shared behaviour according to age. For this purpose,
we used the classification developed by Newman and Newman (2017), with three groups.7

● Cohort 1: early and later adolescence (12–24 years old).

● Cohort 2: early adulthood (25–34 years old).
● Cohort 3: middle and later adulthood (35 years or more).

Our heterogeneity analysis included multiple interactions within the regression model. Likewise,
stratification was applied in order to isolate the effect of each age cohort. The regression model used
is composed of the following equations:

Yiv ¼ β0 þ β1 Zapv þ β2 Zapv :G2 þ β3 Zapv :G3

þ α1 Mobilev þ α2 Mobilev :G2 þ α3 Mobilev :G3 (3)
þ Xiv0 γ þ Xiv0 :Q2 γ2 þ . . . þ seedv þ seedv :G2 þ . . . þ θC þ θC :G2 þ . . . þ εiv

Yiv ¼ τ0 þ τ1 Zapv þ τ2 Zapv :G2 þ τ3 Zapv :G3

þ δ1 Cashv þ δ2 Cashv :G2 þ δ3 Cashv :G3 (4)
þ Xiv0 γ þ Xiv0 :Q2 γ2 þ . . . þ seedv þ seedv :G2 þ . . . þ θC þ θC :G2 þ . . . þ εiv

Yiv represents the outcomes of individual or household i in village v. Cashv is an indicator for villages
assigned to the cash group. Zapv is an indicator for villages assigned to the Zap group. Mobilev is an
indicator for villages assigned to the mobile group. The variables G2 and G3 describe the group to
which the beneficiaries belong and cohort 1 is the reference group to identify heterogeneous
impact.
For the Zap versus cash impact evaluation, our coefficients of interest are β2 and β3 , which
indicate the heterogeneous impact of cohorts 2 and 3 in relation to β1 (the impact coefficient for
cohort 1). Similarly, for mobile versus cash impact evaluation, the coefficients of heterogeneity are
α2 and α3 in relation to α1 (the impact coefficient for cohort 1). In the second equation, the impact
6 J. MENESES ET AL.

evaluation for Zap versus mobile is represented by coefficients τ2 and τ 3 in relation to τ1 (the impact
coefficient for cohort 1).
Finally, we applied Benjamini and Hochberg (1995) analysis to identify the false discovery rate
(FDR), which is the proportion of the rejected null hypothesis that was erroneously rejected. FDR
analysis is represented by q value. Coefficients with FDR lower than 10 per cent are presented with
q in the tables. The heterogeneity analysis tables are presented in Appendix C.
The tables present the coefficients of the interactions in the first three columns, with the
stratified results in the final three columns. The presentation of our results is based on the
recommendations suggested by Knol and VanderWeele (2012), in order to clarify the effect
modification by strata.
The randomized assignment by original authors suggests a similar age distribution across treat-
ment arms.8 Furthermore, we recognize that the original study did not have the intention to find
heterogeneous impact by age.9

3.1.1. Impact on uses of cash transfer

Several studies showed the relevance of age as an impact modifier. Dammert (2008) explored the
heterogeneous effects of conditional cash transfer by age of head of household as a continuous
variable. Also, Galiani and McEwan (2013) studied how children age modified the impact of
a conditional cash transfer program in outputs as school attendance and child labour. Therefore,
we will evaluate the heterogeneous impact between age groups in relation to the uses of cash
transfer.
The original author´s findings indicate that Zap households had more purchases of specific food
items than mobile households, except in millet and sorghum. However, our heterogeneous analysis
by age shows that the oldest Zap beneficiaries do not have a significant coefficient on corn, rice and
oil consumption, in comparison to oldest mobile beneficiaries. In addition, Cohort 3, in the Zap
versus mobile comparison, had a lower use of transfer to purchase grains such as corn and rice, with
a coefficient of −0.33 (p < 0.01) in relation to cohort 1. Between cohorts 3 and 1, purchase of oil
shows a coefficient of −0.18 (p < 0.1).
In our replication study, the Zap versus mobile interaction suggests an effect modification in
which the oldest cohort buys fewer food and nonfood items than the youngest group. An explana-
tion could be that this group uses the transfer to pay debt or for savings. According to Demirguc-
Kunt et al. (2015), in Niger 71 per cent of adults have borrowed money and 56 per cent have
borrowed from family or friends in the past 12 months.
Our heterogeneity analysis shows no differences between age cohorts in health, clothes and
school spending. These homogeneous results could be due to the elevated rates of births in young
women and the general presence of children under 15 years in households. According to the
Demographic and Health Surveys Niger 2012, 76 per cent of women aged 20–24 were married
before the age of 18. One in four girls is married before the age of 15 and the median age at marriage
increased from 15.1 years in 1992 to 15.5 years in 2006 and 15.7 years in 2012 (INS-Niger 2013).
Moreover, 89 per cent of households in Niger have children under 15 years (UN 2017).

3.1.2. Impact on food security, durable goods and nondurable goods

The original findings indicate that Zap households had dietary diversity scores higher than cash and
mobile households. The original results also suggest that Zap household members did not reduce
their contribution of public or private goods to the household. Our evaluation of heterogeneous
impact reaffirmed these findings.
Furthermore, in the comparison of Zap with mobile group, the original authors identified a higher
consumption of condiments and fruits, with a coefficient of 0.07 (p < 0.1) and 0.03 (p < 0.01)
respectively. Our heterogeneous analyses found that the impact on condiments is not significant
for cohort 1 and 3, while the impact on fruits is not significant for cohort 1 and 2. For cohort 2, the
Zap group increased the consumption of condiments in 14 percentage points in comparison to
 JOURNAL OF DEVELOPMENT EFFECTIVENESS 7

mobile group. Furthermore, in cohort 3, Zap households were 5 percentage points more likely to
consume fruits in comparison to mobile group. All differences described are statistically significant.
The heterogeneous impact on consumption of fruits could be explained by household income.
According to Kakwani and Subbarao (2005), the poverty level for households with elderly members is
much higher than the national average in several African countries.10 Fruit and vegetable consump-
tion rises with income and the income elasticity of demand for fruit and vegetables is more sensitive
for low-income households than high-income households (Ruel, Minot, and Smith 2005).11 This
suggests that the oldest group may benefit most by the income increase as a consequence of the use
of m-transfer.12
The benefits of Zap among later-adulthood populations could be helpful for designing future
interventions. A non-contributory social pension program could consider e-money as a delivery
mechanism. Salinas-Rodriguez et al. (2014) show that the impacts of the pension were significantly
higher among women and recipients with lowest income.
We found statistically significant differences in the number of asset categories owned by reci-
pients in cohort 3. For durable assets, cohort 3 had the greatest coefficient for Zap-cash comparison
(p < 0.1). Additionally, in cohort 3, Zap beneficiaries owned 0.28 more nondurable assets (flashlights,
petrol lamps and radios) than the mobile group members (p < 0.01).
According to Fernandez-Villaverde and Krueger (2011), during the first part of the life cycle,
households are forced to progressively accumulate durable goods as well as financial assets. In
Kenya, households with elderly heads showed higher level of durable goods ownership than
younger households, reflecting more years of potential accumulation (Zezza, de la Brière, and
Davis 2010).

3.2. Robustness analysis: multiple imputation and Lee bounds

We used multiple imputation (MI) to deal with missing values and the Lee bounds (LB) method for
attrition. The original study reported attrition of 7 per cent, balanced across treatment arms.
We also verified the attrition and missing data percentages, which are around 7 and 10 per cent
respectively. We did not find missing values for covariates at village level. In contrast, outcomes were
the unique variables where we found missing data.
We decided to use MI and LB as alternative methods to resolve potential missing values and
attrition of the original paper. The summary of the results are presented in Tables 1 and 2. We
present the complete MI results in Appendix E and LB results in Appendix F.

Table 1. Summary of results from multiple imputation robustness analysis.

After using MI, how is the statistical significance of
the results is affected?
Number of statisti-
cally significant Reduced
Original Study results in Aker et al. No longer
table number Table title (2016) Increased Unaltered Still significant significant
Table A4 Uses of cash transfer 12 0 7 4 1
Table A5* Food security and nutritional 16 1 15 0 0
status
Table A6 Leakage 1 0 0 0 1
Table A7 Location, knowledge and timing of 2 0 2 0 0
cash transfer
Table A8 Mobile phone ownership and 23 0 21 2 0
usage
Table A9 Intrahousehold decision making 6 0 6 0 0
Table A10 Alternative explanations 0 0 0 0 0
a
For Table 5, we do not include Panel B estimations, since there is not a baseline for children under 5.
8 J. MENESES ET AL.

Table 2. Summary of the results from Lee bounds robustness analysis.

Number of statistically signifi- After using LB, how many of
Table number Table title cant results in Aker et al. (2016) the results are still significant?
Table A4 Use of cash transfer 12 8
Table A5a Food security and nutritional status 16 12
Table A6 Leakage 1 1
Table A7 Location, knowledge and timing of 2 2
cash transfer
Table A8 Mobile phone ownership and usage 23 18
Table A9 Intrahousehold decision making 6 5
Table A10a Alternative explanations 0 0
a
Table 5 does not include Panel B estimations, since we do not have baseline to estimate attrition. Table 10 does not include
Panel A, since it is at the locality level and there is no attrition at that level.

3.2.1. Multiple imputation

MI is widely accepted as an accurate method for imputation (Enders 2010; Schafer and Graham 2002)
and has theoretical advantages with respect to other procedures. Lall (2016) and Arel-Bundock and
Pelc (2018) show the advantages of MI on political and epidemiological sciences.
The first step in the MI method is to define the predictive model. We considered the three
treatment assignments (Zap, mobile and cash), the seed treatment and the commune fixed effects as
predictors, which were the same covariates used by the original authors. Moreover, we estimated
robust SE using clusters at the village level, as did the original authors.
According to Enders (2010), it is necessary to include the covariates of the regression in the
imputation model in order to avoid a biased estimation. Moreover, it is particularly important to
include the treatment variables. If our MI model failed to have a strong predictive power, our
imputations would have a large variance, which would affect the significance of our point estimates.
If there is low or null variance in the imputation, then the MI results should be essentially the same as
the original study. We summarise these in Table 1 and Appendix E.

3.2.1.1. Multiple imputation results. In Tables E4-E10 (Appendix E), the coefficients tend to be of
lower magnitude and have higher SE, measured by p-values. In this regard, MI estimation produces
fewer significant coefficients. Even when the null hypothesis is rejected, the coefficients have a lower
statistical level of confidence. Changes in statistical significance of coefficients between MI and
original results are underlined in the tables.
We summarise the main results from the MI robustness analysis in Table 1. We define as a robust
result when the statistically significant coefficients from the original tables keep at least
a significance of 10 per cent.13 Most of the statistically significant original results from Aker et al.
(2016) are in Tables A4, A5, A6 and A8 (Appendix A).
The first step was to count the number of coefficients that were statistically significant in the
original study; we present these findings in column 3. Likewise, column 4 indicates how many of
these coefficients increased their significance in MI and column 5 indicates how many did not
change. Finally, columns 6 and 7 show the number of previously statistically significant coefficients
that reduce their p-values. Some retain their significance at 10 per cent and others lose it.14
We realised that MI results agree with the majority of the original authors’ conclusions.
Nevertheless, some coefficients presented changes. In Table E4, five Zap-cash coefficients lose
significance: transfer used to buy oil, food and nonfood items, other grains (corn, rice), condiments
and cowpeas; the last coefficient stops being statistically significant.
Table E5 presents changes that tend to strengthen the original author´s results. For instance, one
coefficient increased its significance in comparison to the original table: consumption of fats.
Also, in Table E6, the only statistically significant result in the original table for the alternative
explanation of leakage -program recipient received cash transfer- is no longer significant.
 JOURNAL OF DEVELOPMENT EFFECTIVENESS 9

According to Lall (2016), the pattern of missing values in public policy data sets is probably never
completely random. Furthermore, adopting MI can be expected to reduce bias and alter parameter
estimates in most studies, such that changes will be largest under three conditions: (1) the proportion
of missing data is high; (2) the data set contains a large number of variables that are strongly related to
missingness; and (3) hypotheses are tested on a heterogeneous sample in terms of missingness
correlates. For our case, we considered that the third condition could explain the different coefficients
estimated by MI, assuming a potential heterogeneous behaviour in the missing group.

3.2.2. Lee bounds analysis

We performed LB to analyse the robustness of the results from the reported attrition problem and
missing values. According to Horowitz and Manski (2000), bounds analysis is a widely accepted
methodology to test for attrition. Lee (2009) bounds methodology is one of the most used
(Glennerster and Takavarasha 2013; Molina and Macours 2017).15
A recent study by Molina and Macours (2017) notes the importance of dealing with attrition even
when it is balanced,16 finding that not controlling it could lead to an overestimation of the impacts.
LB methodology has two assumptions. First, the treatment should be randomly assigned, which is
satisfied by the original study design. Second, the effect of the attrition must be only in ‘one
direction’ (monotonicity assumption).
In LB, we had to deal with the statistical uncertainty in the estimation. Following the standard
procedure in LB analysis, we reported the combination of confidence intervals, at the 95 per cent,
from both bounds, and then the range obtained was evaluated if it included zero. If the range did not
contain zero, we concluded that the results are robust enough to deal with extreme bounds with Lee
(2009) methodology.

3.2.2.1. Lee bounds results. We present the results of the LB analysis in Appendix F. In each table,
we present the confidence intervals, with the lower and upper bound. The coefficients with a range
that does not include zero are presented with an asterisk. Tables F4-F10 show the range of the LB
estimation for the coefficients of interest.
Table 2 reports the summary of LB results. The second column indicates the number of coeffi-
cients that were statistically significant in Aker et al. (2016). The third column indicates the number of
robust coefficients in the LB estimation (those that do not contain zero in the estimated range).
We define robust in the LB when the statistically significant coefficients from the original Tables
A4-A10 keep at least with a significance p < 0.1. As Table 2 illustrates, 47 of 60 significant coefficients
keep their significant after this procedure. This suggests that original conclusions are maintained.
For example, in Table F4, LB confirms that Zap households, on a scale from 0 to 8, buy between
0.36 and 1.37 more types of food and nonfood items, compared to the Mobile group. However, the
Zap versus Cash coefficients related to use of the transfer to buy cowpeas, condiments or oil are not
robust.
In Table F5, Zap versus Cash effects related to HDDS (out of 12) or consumption of beans are not
robust to the LB procedure. Similarly, Table F8 shows that Zap versus Cash coefficients related to use
of communication with commercial contacts inside Niger and the probability that at least one
household member migrates are not robust.
In conclusion, the main coefficients of interest were robust for the LB test.

4. Theory of change analysis

The theory of change analysis (TCA) aims to provide a better understanding of the causal pathway
underlying the studied intervention, which is crucial to design public policies (Brown, Cameron, and
Wood 2014). In this sense, we focused on the impact of cash transfer mechanisms on nutritional
status of children under 5 years old.17
10 J. MENESES ET AL.

4.1. Nutrition evaluation

We develop a broader assessment about nutrition in children under 5 years to estimate the effect of
different transfer mechanisms. Changes in dietary diversity without a significant impact on malnu-
trition and the absence of stunting evaluation represented an opportunity for our study.
According to UNICEF strategies, nutritional status depends on three axes: (1) access to and
utilisation of adequate foods (including quality and safety); (2) the effectiveness of health services
and healthiness of the environment (including resources and the responsibilities of duty bearers);
and (3) the quality and level of maternal and child healthcare. Niger has a 42 per cent prevalence of
moderate and severe stunting (UNICEF 2016).
Initially, we used WHO malnutrition parameters for children under 5 years old to define wasting
and stunting. Moderate wasting is defined as having a weight-for-height Z-score falling below −2 to
−3 SD. Severe wasting is defined as weight-for-height Z-score falling below −3 SD. Moderate
stunting is defined as height-for-age Z-score falling below −2 to −3 SD. Severe stunting is defined
as height-for-age Z-score falling below −3 SD.
We created four new dichotomous variables for wasting and stunting evaluation: severe wasting,
moderate wasting, severe stunting and moderate stunting. In this sense, we used a regression model
to evaluate malnutrition status because it improved the assessment by analysing multiple covariates
in order to adjust the relation of interest. The control variables commune and seed were also
considered. We found it relevant to include such control variables to avoid bias in our findings.
Additionally, regressions are performed with the same structure of the regressions used in the
original study and PR for outcome estimation.
The results about wasting, its severity and the impact of cash transfer mechanisms are presented
in Table 3. In the original article, there is an evaluation of weight-for-height z-score as a continuous
variable. For severe wasting and severe and moderate wasting, a greater coefficient represented
a higher harmful condition. For this table, there is no statistically significant impact of the interven-
tions on wasting.

Table 3. Child wasting status.

Observations Mean Zap-cash Mobile-cash Zap-mobile
Variable cash cash (SD) Observations coeff. (SE) coeff. (SE) coeff. (SE)
B. Wasting
Weight-for-height z-score 212 −1.15 691 0.06 −0.03 0.09
(0.96) (0.12) (0.15) (0.13)
Severe wasting 212 0.01 691 −0.00 0.01 −0.01
(0.12) (0.01) (0.02) (0.02)
Severe and moderate wasting 212 0.17 691 −0.02 −0.00 −0.02
(0.38) (0.04) (0.06) (0.05)
SD, standard deviation; SE, standard error.
*p < 0.1; **p < 0.05; ***p < 0.01.

Table 4. Child stunting status.

Mean
Observations cash Zap-cash Mobile-cash Zap-Mobile
Variable cash (SD) Observations coeff. (SE) coeff. (SE) Coeff. (SE)
A. Stunting measurement
Height-for-age z-score 216 −1.42 702 −0.26 −0.06 −0.21
(1.51) (0.24) (0.18) (0.28)
Severe stunting 216 0.13 702 0.02 −0.03 0.05
(0.33) (0.04) (0.04) (0.04)
Severe and moderate stunting 216 0.32 702 0.07* −0.02 0.10
(0.47) (0.04) (0.06) (0.06)
SD, standard deviation; SE, standard error.
*p < 0.1; **p < 0.05; ***p < 0.01.
 JOURNAL OF DEVELOPMENT EFFECTIVENESS 11

Table 4 describes results about stunting and its severity, as well as the impact of cash transference
mechanisms, including a complementary analysis of the continuous variable height-for-age z-score
evaluation.
Regarding severe and moderate stunting, the Zap-cash coefficient is 0.07, which shows a small
and isolated positive relation between Zap intervention and severe and moderate stunting, with no
significant findings in the Zap-mobile coefficient.18 We considered that the absence of a nutritional
baseline evaluation is important to explain this finding, since we were not able to evaluate
a progression based on anthropometric measures. According to Bamberger (2010), the availability
of appropriate baseline data is always critical for performance evaluation.

4.2. Heterogeneity analysis by age

We performed a heterogeneity analysis to identify the age of children for whom this intervention had
a greater impact.19 Policymakers often find it important to understand how the benefits of social
programs might vary by individual or household characteristics, even if the average effect of the
program is not significant. This information could be relevant for focusing future public policies in
a specific age range.
We established two evaluation groups, sorted by age and grouped based on shared behavioural
patterns and impact of nutrition: 6–24 months (group 1) and 25–60 months (group 2). According to
Krebs, Lozoff, and Georgieff (2017), children under 24 months share specific opportunities and
vulnerabilities related to the fast central nervous system development and nutritional requirements,
as well as the impact of environment on their nutritional homoeostasis. Moreover, children under 6
months were excluded, based on their special characteristics, like exclusive breastfeeding, and the
duration of the original intervention.
Table H1 shows the coefficients of the interactions in the first three columns and the stratified
results in the final three columns.
Our analysis of heterogeneity did not find nutrition impact. We find significant impact between
Zap and cash for the reduction of severe wasting in group 2. This group (25–60 months), benefited
most from this intervention, with a statistically significant coefficient of −0.04. We also applied
Benjamini and Hochberg (1995) analysis for this coefficient; it showed FDR lower than 10 per cent.20
Although the significant coefficients suggest potential heterogeneous impact on the results, we
cannot establish a causal relationship, first because of the absence of baseline focused on children,
and second due to the limitations of power given by the number of observations.
Finally, we consider that the mediation analysis could be useful to amplify the causal analysis of
TCA section for future replication studies. According to Keele, Tingley, and Yamamoto (2015), the
mediation analysis decomposes the total treatment effect on the outcome into the indirect and
direct effects.

5. Conclusion
Our study replicated analysis and results of the paper, Payment mechanisms and antipoverty programs:
evidence from a mobile money cash transfer experiment in Niger by Aker et al. (2016). Through this
replication, we were able to verify the original results, using similar statistical methods as the original
authors. Using push-button replication, we replicate original author´s results using their code.
The PR section, our independent reproduction of findings, confirmed the comparability of our
results to the original authors’. We did not find any major differences in relation to the results shown
in the published paper and its complementary corrigendum. The minor observations we identified
do not affect the relevant results established by the original paper.
MEA included analysis of heterogeneity and robustness tests. The heterogeneity evaluation
suggests that the Zap intervention had a different impact on older beneficiaries than younger
12 J. MENESES ET AL.

ones. Robustness analysis considered MI and LB analysis, which confirmed that the original results
are robust to the evaluation of these methods.
Our nutritional assessment in the TCA described stunting and wasting status and severity in
children under 5 years. We were able to find significant impact between Zap and cash for the
reduction of wasting in group 2 (children 25–60 months old).
Finally, we thank the excellent disposition of the original authors to share their database, codes
and additional help while performing our replication study.

Notes
1. According to the National Institute of Statistics of Niger (2014), these 96 communities belong to the Department
of Tahoua, capital of Tahoua region, one of the country’s seven regions.
2. See https://sites.tufts.edu/jennyaker/research-2/ for the data sets.
3. The communes – Affala, Bambeye, Barmou, Kalfou, Takanamat and Tebaram – are representative of the
communes of the Department of Tahoua. In Appendix H, we present additional information on Niger’s
demographics.
4. The HDDS is the aggregation of 12 food groups: cereals, white tubers and roots, vegetables, fruits, meat, eggs,
fish and other seafood, legumes and seeds, milk and milk products, oil and fats, sweets, and spices (condiments
and beverages).
5. We identified two statistical parameters of the variable amount spent on children’s clothing for festivals that
had rounding differences from those estimated by the original authors.
6. The original survey was applied to women who received the intervention, but we found one male in the analysis.
We suppose this is due to the absence of a female receptor or a typing error.
7. We also evaluated quartile grouping (Appendix D). We did not find major differences in relation to our
behavioural grouping analysis.
8. In the original study, 96 villages were stratified by commune and villages and were randomly assigned to the
cash, mobile or Zap interventions (32 villages per group).
9. Although it was not a pre-specified analysis, we analysed heterogeneity by polygamy condition, but did not find
important differences.
10. In Malawi, Uganda and Zambia, the poverty gap ratio for households with elderly members is 6–20 percentage
points higher than the average ratio.
11. The fact that fruit and vegetables are an expensive source of energy is an important constraint for poor
households.
12. The original study suggests that reducing the transfer’s observability by m-transfer could also affect interhouse-
hold sharing, leaving more income available for the household (Jakiela and Ozier 2015).
13. In Table MI, the original results have 60 statistically significant coefficients, while MI suggests that two
coefficients lost their significance. This represents 3 per cent of all original significant coefficients from Tables
A4 to A10.
14. We refer to increases or reductions of significance if p-value increases or reduces at least 0.01.
15. Some recent and influential studies that use LB analysis to test for robustness are Hidrobo et al. (2014), Kremer,
Miguel, and Thornton (2009), Cunha (2014) and Drexler, Fischer, and Schoar (2014).
16. The original authors documented that there is no evidence of differential attrition across the experimental arms.
17. Children from 0 to 5 months were not included.
18. About programs with negative impact, in Nicaragua, children in coffee-producing households receiving condi-
tional cash transfers saw greater declines in height-for-age Z-score measure with a − 0.27 coefficient after
treatment (Dammert 2008). A program in Brazil found that 6 months after benefits began to be distributed,
beneficiary children were 0.13 z-scores lighter (weight-for-age) than excluded children (Morris et al. 2004).
Houngbe et al. (2017) found no evidence that unconditional cash transfers reduced the incidence of wasting.
19. Some replication studies included heterogeneity analysis as part of TCA. In this sense, Atanda (2019) replication
study evaluated the moderating factors arising from the heterogeneous implementation of the biometrically
authenticated payment system across districts and its influence.
20. According to Andrews and Buchinsky (2000), we estimated the coefficients using bootstrap. The statistical
significance of coefficient is maintained using 1498 repetitions.

Disclosure statement
No potential conflict of interest was reported by the authors.
 JOURNAL OF DEVELOPMENT EFFECTIVENESS 13

Funding
This study was funded by International Initiative for Impact Evaluation (3ie) Replication Programme on Financial
Services for the Poor and Bill and Melinda Gates Foundation .

Notes on contributors
Jean Pierre Meneses is a Physician and Master of Science at UPCH, and researcher at PUCP. As a Physician, his research
interests lie in neurology, neuroscience, nutrition and impact evaluation on vulnerable populations.
Edgar Ventura Neyra is Master in Economics, Professor at the UPC School of Economics and researcher at PUCP. His
primary field of research is social policy, and impact evaluation focuses on financial inclusion initiatives
Oliver A. Elorreaga is an Economist at PUCP and Master of Science in Epidemiological Research Sciences at UPCH. His
primary research interests include social policy, impact evaluation, as well as innovative interventions to aid margin-
alized populations.
Cesar Huaroto is a PhD student of Economics at Pontifical Catholic University of Chile. He works on economic
development, using applied econometrics to assess social policies on poverty and natural resources.
Giovanna Aguilar has a degree of Doctor of Economics from PUCP, and Master´s degree in Economics from PUC-Rio.
She is Full Professor and researcher at the PUCP´s Department of Economics. Her work focuses on microfinance,
development issues and the design of financial inclusion strategies.
Edmundo Beteta is Master of Arts of Economics at Georgetown University, Professor and researcher in Economics at
PUCP, whose research includes health economics, government and public policy, along with evaluation methods.

ORCID
JP Meneses http://orcid.org/0000-0001-7623-9606

References
Aker, J., R. Boumnijel, A. McClelland, and N. Tierney. 2016. “Payment Mechanisms and Antipoverty Programs: Evidence
from a Mobile Money Cash Transfer Experiment in Niger.” Economic Development and Cultural Change 65 (1): 1–37.
doi:10.1086/687578.
Andrews, D. W. K., and M. Buchinsky. 2000. “A Three-step Method for Choosing the Number of Bootstrap Repetitions.”
Econometrica 68: 213–251. doi:10.1111/1468-0262.00092.
Arel-Bundock, V., and K. J. Pelc. 2018. “When Can Multiple Imputation Improve Regression Estimates?” Political Analysis
26: 240–245. doi:10.1017/pan.2017.43.
Atanda, A. A. 2019. “Biometric Smartcards and Payment Disbursement: A Replication Study of Building State Capacity in
India.” Journal of Development Effectiveness 1–13. doi:10.1080/19439342.2019.1646782.
Bamberger, M. 2010. Reconstructing Baseline Data for Impact Evaluation and Results Measurement. World Bank.
Benjamini, Y., and Y. Hochberg. 1995. “Controlling the False Discovery Rate: A Practical and Powerful Approach to
Multiple Testing.” Journal of the Royal Statistical Society, Series B (methodological) 57 (1): 289–300. doi:10.1111/
rssb.1995.57.issue-1.
Brown, A. N., D. B. Cameron, and B. D. K. Wood, 2014. “Quality Evidence for Policymaking: I’ll Believe It When I See the
Replication.” 3ie Replication Paper 1. Washington, DC: International Initiative for Impact Evaluation (3ie).
CGAP. 2011. CGAP G2P Research Project: South Africa Report. Washington, DC: Consultative Group to Assist the Poor
(CGAP).
Cunha, J. 2014. “Testing Paternalism: Cash Versus In-Kind Transfers.” American Economic Journal: Applied Economics 6 (2):
195–230.
Dammert, A., 2008. “Heterogeneous Impacts of Conditional Cash Transfers: Evidence from Nicaragua.” IZA Discussion
Paper No. 3653. Institute for the Study of Labor (IZA).
Demirguc-Kunt, A., L. F. Klapper, D. Singer, and P. Van Oudheusden, 2015. “The Global Findex Database 2014: Measuring
Financial Inclusion around the World.” Policy Research Working Paper 7255. Washington, DC: World Bank.
Drexler, A., G. Fischer, and A. Schoar. 2014. “Keeping It Simple: Financial Literacy and Rules of Thumb.” American
Economic Journal: Applied Economics 6 (2): 1–31. doi:10.1257/app.6.4.1.
Enders, C., 2010. “Applied Missing Data Analysis.” Vienna Economics Papers 1302. New York: Guilford Press, University of
Vienna, Department of Economics.
14 J. MENESES ET AL.

Fernandez-Villaverde, J, and D Krueger. 2011. “Consumption and Saving over The Life Cycle: How Important are
Consumer Durables? Macroeconomic Dynamics.” 15 (5): 725–770.
Galiani, S., and P. J. McEwan. 2013. “The Heterogeneous Impact of Conditional Cash Transfers.” Journal of Public
Economics 103: 85–96. doi:10.1016/j.jpubeco.2013.04.004.
Gash, M., and B. Gray. 2016. The Role of Financial Services in Building Household Resilience. Washington, DC: Consultative
Group to Assist the Poor (CGAP).
Glennerster, R., and K. Takavarasha. 2013. Running Randomized Evaluations: A Practical Guide. Princeton: Princeton
University Press.
Guerrero, M., and P. Ritter. 2014. The effect of internet and cell phones on employment and agricultural production in
rural villages in Peru. Tesis de pregrado en Economía. Universidad de Piura. Facultad de Ciencias Económicas y
Empresariales. Programa Académico de Economía y Finanzas. Piura, Perú.
Hidrobo, M., J. Hoddinott, A. Peterman, A. Margolies, and V. Moreira. 2014. “Cash, Food, or Vouchers? Evidence from
a Randomized Experiment in Northern Ecuador.” Journal of Development Economics 107: 144–156. doi:10.1016/j.
jdeveco.2013.11.009.
Holloway, K., R. Rouse, and Z. Niazi. 2017. Women’s Economic Empowerment Through Financial Inclusion: A Review of
Existing Evidence and Remaining Knowledge Gaps (Rep.). New Haven, CT: Innovations for Poverty Action
Horowitz, J., and C. Manski. 2000. “Nonparametric Analysis of Randomized Experiments with Missing Covariate and
Outcome Data.” Journal of the American Statistical Association 95: 77–84. doi:10.1080/01621459.2000.10473902.
Houngbe, F., A. Tonguet-Papucci, C. Altare, M. Ait-Aissa, J. F. Huneau, L. Huybregts, and P. Kolsteren. 2017.
“Unconditional Cash Transfers Do Not Prevent Children’s Undernutrition in the Moderate Acute Malnutrition Out
(mam’out) Cluster-randomized Controlled Trial in Rural Burkina Faso.” The Journal of Nutrition 147 (7): 1410–1417.
doi:10.3945/jn.117.247858.
INS-Niger. 2013. INS-Niger and ICF International, 2013. Enquête Démographique et de Santé et à Indicateurs Multiples
du Niger 2012. Calverton, Maryland.
INS-Niger. 2014. Le Niger en Chiffres 2014. Available at: www.stat-niger.org/statistique/index.php/publication/publica
tions-de-l-ins/item/128-population
Iversen, V., and R. Palmer-Jones, 2014. “TV, Female Empowerment and Demographic Change in Rural India.” 3ie
Replication Paper 2. Washington, DC: International Initiative for Impact Evaluation (3ie).
Jack, W., and T. Suri. 2016. “The Long-run Poverty and Gender Impacts of Mobile Money.” Science 354 (6317): 1288–1292.
doi:10.1126/science.aah5309.
Jakiela, P., and O. Ozier. 2015. “Does Africa Need a Rotten Kin Theorem? Experimental Evidence from Village Economies.”
The Review of Economic Studies 83 (1): 231–268. doi:10.1093/restud/rdv033.
Kakwani, N., and K. Subbarao. 2005. Ageing and Poverty in Africa and the Role of Social Pensions. Washington, DC: World
Bank.
Keele, L., D. Tingley, and T. Yamamoto. 2015. “Identifying Mechanisms behind Policy Interventions via Causal Mediation
Analysis.” Journal of Policy Analysis and Management 34 (4): 937–963. doi:10.1002/pam.2015.34.issue-4.
Kennedy, G., T. Ballard, and M. C. Dop. 2011. Guidelines for Measuring Household and Individual Dietary Diversity. Food
and Agriculture Organization of the United Nations (FAO).
Klapper, L., and D. Singer. 2014. The Opportunities of Digitizing Payments. Washington, DC: World Bank.
Knol, M. J., and T. J. VanderWeele. 2012. “Recommendations for Presenting Analyses of Effect Modification and
Interaction.” International Journal of Epidemiology 41 (2): 514–520. doi:10.1093/ije/dyr218.
Krebs, N. F., B. Lozoff, and M. K. Georgieff. 2017. “Neurodevelopment: The Impact of Nutrition and Inflammation during
Infancy in Low-Resource Settings.” Pediatrics 139 (Suppl 1): S50–S58. doi:10.1542/peds.2016-2828G.
Kremer, M., E. Miguel, and R. Thornton. 2009. “Incentives to Learn.” The Review of Economics and Statistics 91 (3):
437–456. doi:10.1162/rest.91.3.437.
Lall, R. 2016. “How Multiple Imputation Makes a Difference.” Political Analysis 24 (4): 414–433. doi:10.1093/pan/mpw020.
Lee, D. 2009. “Training, Wages, and Sample Selection: Estimating Sharp Bounds on Treatment Effects.” Review of
Economic Studies 76: 1071–1102. doi:10.1111/roes.2009.76.issue-3.
Mbiti, I., and D. N. Weil, 2011. “Mobile Banking: The Impact of M-Pesa in Kenya.” Working Paper No. 17129. National
Bureau of Economic Research.
McCullough, B. D., and H. D. Vinod. 1999. “The Numerical Reliability of Econometric Software.” The Journal of Economic
Literature 37: 633–665. doi:10.1257/jel.37.2.633.
McCullough, B. D., K. A. McGeary, and T. D. Harrison. 2006. “Lessons from the JMCB Archive.” Journal of Money, Credit and
Banking 38 (4): 1093–1107. doi:10.1353/mcb.2006.0061.
McCullough, B. D., K. A. McGeary, and T. D. Harrison. 2008. “Do Economic Journal Archives Promote Replicable Research?
Can.” Journal of Economics 41 (4): 1406–1420.
Meneses, J. P., E. Ventura, O. A. Elorreaga, C. Huaroto, G. Aguilar, and E. Beteta, 2018. “Mobile Money and Its Impact on
Improving Living Conditions in Niger: A Replication Study.” 3ie Replication Paper 19. Washington, DC: International
Initiative for Impact Evaluation (3ie). doi:10.23846/RPS0019.
Molina-Millan, T, and K Macours. 2017. Attrition in Randomized Control Trials: Using Tracking Information to Correct
Bias, IZA Discussion Papers, No. 10711, Institute of Labor Economics (IZA), Bonn
 JOURNAL OF DEVELOPMENT EFFECTIVENESS 15

Morris, S. S., P. Olinto, R. Flores, E. A. Nilson, and A. C. Figueiro. 2004. “Conditional Cash Transfers are Associated with
a Small Reduction in the Rate of Weight Gain of Preschool Children in Northeast Brazil.” The Journal of Nutrition 134
(9): 2336–2341. doi:10.1093/jn/134.9.2336.
Mueller-Langer, F., B. Fecher, D. Harhoff, and G. G. Wagner. 2019. “Replication Studies in economics—How Many and
Which Papers are Chosen for Replication, and Why?” Research Policy 48 (1): 62–83. doi:10.1016/j.respol.2018.07.019.
Newman, B. M., and P. R. Newman. 2017. Development through Life: A Psychosocial Approach. Cengage Learning.
Oyekale, A. S., and T. O. Oyekale. 2009. “Do Mothers’ Educational Levels Matter in Child Malnutrition and Health
Outcomes in Gambia and Niger.” The Social Sciences 4: 118–127.
Ruel, M. T., N. Minot, and L. Smith. 2005. Patterns and Determinants of Fruit and Vegetable Consumption in sub-Saharan
Africa: A Multicountry Comparison. Geneva: World Health Organization (WHO).
Salinas-Rodríguez, A., M. D. P. Torres-Pereda, B. Manrique-Espinoza, K. Moreno-Tamayo, and M. M. Tellez-Rojo. 2014.
“Impact of the Non-contributory Social Pension Program 70 Y Mas on Older Adults’ Mental Well-being.” PloS One 9
(11): e113085. doi:10.1371/journal.pone.0113085.
Schafer, J., and J. Graham. 2002. “Missing Data: Our View of the State of the Art.” Psychological Methods 7 (2): 147–177.
doi:10.1037/1082-989X.7.2.147.
UNICEF. 2016. UNICEF Data: Monitoring the Situation of Children and Women. New York: United Nations Children’s Fund
(UNICEF). https://data.unicef.org/country/ner/
United Nations. 2017. “Population Division. Department of Economic and Social Affairs Division.” http://www.un.org/en/
development/desa/population/publications/database/index.shtml
WHO. 2006. WHO Child Growth Standards: Length/height-for-age, Weight-for-age, Weight-for-length, Weight-for-height
and Body Mass Index-for-age: Methods and Development. Geneva: World Health Organization (WHO).
Wood, B. D., R. Müller, and A. N. Brown. 2018. “Push Button Replication: Is Impact Evaluation Evidence for International
Development Verifiable?” PloS One 13 (12): e0209416. doi:10.1371/journal.pone.0209416.
Zezza, A., B. de la Brière, and B. Davis, 2010. “The Impact of Social Cash Transfers on Household Economic Decision
Making and Development in Eastern and Southern Africa.” Working paper presented at the African Economic
Conference, Tunis, October 27–29.