See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/374075521

A systematic review of the evaluation of agricultural policies: Using prisma

Article in Heliyon · September 2023

DOI: 10.1016/j.heliyon.2023.e20292

CITATIONS READS

0 48

4 authors, including:

Natalia Jaramillo Julián Andrés Castillo Grisales

National University of Colombia University of Antioquia
6 PUBLICATIONS 35 CITATIONS 3 PUBLICATIONS 0 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Natalia Jaramillo on 03 November 2023.

The user has requested enhancement of the downloaded file.

 Heliyon 9 (2023) e20292

Contents lists available at ScienceDirect

Heliyon
journal homepage: www.cell.com/heliyon

Review article

A systematic review of the evaluation of agricultural policies:

Using prisma
Lina M. Bastidas-Orrego a, f, *, Natalia Jaramillo b, e, 1, Julián A. Castillo-Grisales c, 1,
Yony F. Ceballos d, 1
a
Capital Contable Research Group, Corporación Universitaria Remington, Colombia
b
SUMAR Research Group, Fundación Universitaria María Cano, Facultad Ciencias Empresariales, Colombia
c
Ingeniería y Sociedad Research Group, Facultad de Ingenieria, Universidad de Antioquia, Medellín, Colombia
d
Ingeniería y Tecnologías de las Organizaciones y de la Sociedad (ITOS) Research Group, Facultad de Ingeneniería, Universidad de Antioquia,
Medellín, Colombia
e
Grupo de cerámicos y vítreos, Escuela de Física, Universidad Nacional de Colombia, Calle 59A 63-20, 050034, Medellín, Colombia
f
Mundo Organizacional, Facultad de ciencias Empresariales Research Group, Corporación Universitaria Remington, Colombia

A R T I C L E I N F O A B S T R A C T

Keywords: Background: The food security of each country depends on agricultural development, which is
Policy evaluation sensitive to the implementation of agricultural public policies. These must evolve as new rural­
Agricultural policies ities arise, with new phenomena, such as climate change, ecosystem services, changes in con­
Systematic review
sumer preferences, globalization, sustainability and ecological awareness. Hence, of ex-ante and
Variables of agricultural policy
ex-post evaluations of agricultural policies, are important because they provide timely informa­
Types of agricultural policies
tion to government entities. There are different methodologies for policy evaluation, which have
evolved over time.
Aims: This systematic review aims to identify manuscript that systematically review methodol­
ogies, policies and variables evaluated during the last 50 years to determine whether a policy has
been efficient. To assess the quality of the included manuscript and to describe the measures and
domains identified.
Methods: EBSCO, Dialnet, SciELO, Scopus, Science Direct, Dimensions and Web of Science were
searched. A total of 154 manuscript were identified, the review was finalized by reviewing the
title, and abstract and the review was finalized by reviewing the title, abstract and full text,
resolving disagreements. Of these 154 manuscripts, 37 met the criteria and were included in the
analysis. PRISMA checklists were used to evaluate the methodology.
Outcomes and results: It were found that there are few studies on the design of evaluation meth­
odologies for agricultural policies in the literature. Research shows that the latest policy evalu­
ation proposals present more complex methodologies involving tools such as machine learning
and agent-based modelling (ABM). On the other hand, the issue of sustainability as a policy is
seen in the agri-environmental policy evaluation.
Conclusions and implications: The evolution of agricultural policy methodologies can be observed
at the beginning with the use of quantitative methodologies, such as matrices, statistics and
econometrics. With the emergence of new variables, such as agri-environmental variables, citizen

* Corresponding author. Corporación Universitaria Remington, Carrera 51 # 49-59, Medellín, Antioquia, Colombia.
E-mail addresses: lina.bastidas@uniremington.edu.co (L.M. Bastidas-Orrego), nataliaisabeljaramillogomez@fumc.edu.co (N. Jaramillo), jandres.
castillo@udea.edu.co (J.A. Castillo-Grisales), yony.ceballos@udea.edu.co (Y.F. Ceballos).
1
These authors contribute equally to the study.

https://doi.org/10.1016/j.heliyon.2023.e20292
Received 7 December 2022; Received in revised form 31 July 2023; Accepted 18 September 2023
Available online 20 September 2023
2405-8440/© 2023 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license
(http://creativecommons.org/licenses/by-nc-nd/4.0/).
L.M. Bastidas-Orrego et al. Heliyon 9 (2023) e20292

participation and market opening, methodologies have become more comprehensive, combining
qualitative and quantitative variables. Methodologies were identified that evaluate robust agri­
cultural policies and others that focus on the evaluation of one or two policies. These studies are
important for research that focuses not only on the evaluation of agricultural policies but also on
their design and implementation processes.

1. Introduction

A public policy is a group of activities that governments implement to achieve economic and social objectives, seeking to improve
the quality of life of their group. Public policies are applied to different areas of the economy, among these are agricultural policies.
Public policies are change between cities or regions, since they depend on the characteristics of the territories and their needs [1].
Among public policies, there are some that focus on the promotion of economic sectors. The agricultural sector is one of the most
important sectors in the economic growth of a country. Given its importance, countries implement agricultural policies that seek to
impact the increase in agricultural production and in turn improve the quality of life of producers [2].
To begin this review, it is important to clarify public policies. Policies are associated with the institutional structures that char­
acterize a country’s political system [3], while public policies are actions and decisions taken by governments with the purpose of
achieving the common good and satisfying social needs [4].
From the public administration, the objectives set as a society are materialized and achieved through government interventions or
the participation of social actors. For its part, the policies arise from the objectives set before a need considered relevant to society [4].
From this perspective, public policy is the response to a need that manages to be built collectively and captures the attention of
government entities with the sole purpose of identifying actions that reflect a choice and political will around society’s benefits. Thus,
public policy plays a fundamental role in the construction of social reality since it does not represent an end by itself but rather might
reflect the differential power resources of social groups.
The evaluation of agricultural policies offers helpful information for the design and implementation of other policies related to
economic development, economic interests, prescriptions and requirements of international organizations, local environmental
conditions, and national institutions, among others, especially those that manifest themselves in international markets. Each country
presents different ruralities, which makes its agricultural sector work in a particular way; therefore, it must implement different types
of agricultural policies for each period of time, so each policy or group of policies implemented must have its own evaluation
methodology that lets us know its impacts on the sector and on society. Consistent with the above, works were found that evaluated
their policies their political reforms at certain times.
The environmental dimension of agricultural development has been progressively incorporated as a concept, giving rise to sus­
tainable agriculture. According to Ref. [5], the analysis of environmental policy in agriculture yields such diverse results that it is
impossible to perform an effective evaluation; for this reason, a Pareto inefficiency criterion is proposed as a tool to evaluate
agri-environmental public policies For this reason, it is interesting to analyze a combination of more robust methodologies that have
qualitative and quantitative tools that account for the complexity in decision-making when an agricultural policy is implemented.
Although public policies have improved in this century, the efforts are insufficient and the agricultural sector has not yet been
developed [6]. Agricultural policies seek sustainable productivity growth, environmental protection and adaptation to climate change.
This review is important because systematic reviews of evaluation methodologies of agricultural public policies focused on the use
of agent-based modeling have been performed. No methodologies were found in this work that could be applied in ex-ante or ex-post
evaluation of policy implementation specifically. It is also important for public policy-makers and researchers in the field to have a
compilation of the different evaluation methodologies, the typology of policies and the variables used in it. This work can serve as an
important source of information not only for policy design, but also for decision-makers on the continuation of an agricultural policy.
The objective of this review is to identify the methodologies, types of policies and possible variables that have been the subject of
evaluation for approximately 50 years. The results obtained will serve as support and guide for public policy-makers since it allows
information on the policies implemented and improvements in the design of new policies to be obtained. Therefore, this manuscript is
based on a systematic review using PRISMA (Preferred Reporting Items for Systematic Review and Meta-analysis) to infer and generate
evidence regarding the design and use of agricultural policy evaluation methodologies. However, PRISMA is not the only way to assess
the quality of the analysed papers.
The objectives are addressed by addressing the following research questions:

1. What methodologies were used to evaluate agricultural policies?

2. What kind of policies were evaluated?
3. What variables were analysed in the evaluation of agricultural policies?

1.1. Agricultural public policies

Agricultural public policies are classified into five large groups, which are aimed at: countries that do not have a macro agricultural
policy but a sectoral one, countries that have long-term protectionist policies focused on the sector, countries that have laws oriented

2
L.M. Bastidas-Orrego et al. Heliyon 9 (2023) e20292

towards agriculture, countries with governing laws for agriculture, which do not generate regulation and countries with macroeco­
nomic policies that have not managed to generate policies in the long term. Thus, to achieve sustainability in the agricultural sector, it
is important to assess the impacts of current and future policies that will ensure the agricultural future of a country [7].
Agricultural policies have the broad objective of improving the living conditions of the rural population, the agricultural economy
and food security, while seeking to reduce the environmental impact [7]. This implies that both the formulation and evaluation
processes must be rigorous and based on the economic, regional, cultural and political needs of the territory. This also suggests that the
policy evaluation process should be demanding and require the analysis of different evaluation methods or innovative techniques that
are adjusted to the different policy conditions of each territory or region, as suggested by Refs. [7–11].
One of the main challenges of agricultural public policies in the world is not only to seek the common benefit of people, but also of
nature and future generations [12]. Therefore, the design and evaluation of these policies is aimed at delivering sustainable food
production, biodiversity conservation, climate change mitigation, as well as improving the living conditions of the population [12,13].
Also, agricultural policies face even greater challenges, such as food quality, the population’s access to micronutrients or the avail­
ability of a wider food basket [14,15].
Finally, investments are needed in technology such as market information systems for producers, connectivity in rural areas and the
adoption of digital technology as a policy tool to help in the design, monitoring and evaluation of policies [16].

2. Method

2.1. Inclusion and exclusion criteria

The systematic review carried out included studies that describe the methodology for evaluating public agricultural policies related
to methodological descriptions of evaluation models that include variables related to public agricultural policies. Table 1 details the
inclusion and exclusion criteria used in this review to classify the search results. Inclusion and exclusion criteria were used to decide
whether a systematic review article should be included or excluded.

2.2. Type of study

Articles published in peer-reviewed journals were selected, as they are considered higher quality sources. It will focus on manu­
scripts that present reviews of evaluation models, and methodology for the evaluation of agricultural policies. Also, manuscripts where
the policy addressed agricultural or agri-environmental aspects were included. That are agricultural markets, agricultural production,
and quality of life of the farmer. Additionally, manuscripts dealing with environmental sustainability related to agriculture were
included. The type of document was articles or reviews. No language restrictions are applied.

2.3. Search procedures

This systematic review complies with the extension of the PRISMA statement for systematic reviews according to the Guidelines for
Review and Meta-analysis (PRISMA) [17]. Initial searches were performed up to June 30, 2022. The search included publications
between 1966 and the first semester of 2022. The search action will be carried out using EBSCO, Scopus, Dimension, Web of Science,
Science Direct, Dialnet, and SciELO (Table 2). There were no specifications for the publication date.
The keywords for the search in the databases were selected based on the need to identify the various forms of evaluation of

Table 1
Inclusion and exclusion criteria for manuscript.
Inclusion criteria

- Manuscripts that presented reviews of evaluation models, methodology for agricultural policy evaluation, evaluation were included.
- Manuscripts were included if the policy addressed agricultural or agri-environmental aspects. That is, agricultural markets, agricultural production, and quality of
life of the farmer. Manuscripts dealing with environmental sustainability related to agriculture were also included.
- Include types of agricultural policies.
- Published in peer-reviewed journals.
- Document types: articles.
- No language restrictions should be applied in worldwide.
An article was excluded if it met one or more of the following exclusion criteria:
- Analysis of impacts, excluding the evaluation model.
- Analysis of the historical evolution of an agricultural policy.
- Evaluation of agricultural public policy reforms.
- Document types: books, book chapters, abstracts, monographs, commentaries, dissertations and protocols.
- Manuscripts were excluded if the topic was not related to the objectives of the literature review (e.g., relationship between politics and policy).
- Other types of modelling that do not evaluate agricultural policies.
- Planning methodologies.
- Limited only to the analysis of variables or natural resources
- Evaluation of other public policies.
- Inclusion of ecosystem services, sustainability and sustainability in agricultural policies, without evaluating the policies.
- Duplicate documents.

3
L.M. Bastidas-Orrego et al. Heliyon 9 (2023) e20292

agricultural public policies according to the objectives of the study. The scope focuses on evaluation tools that allow the identification
of analysis variables, methodologies and types of policies associated with the agricultural sector. In this way, the evaluation of the
efficiency or effectiveness of a policy is ruled out because it is not consistent with the object of study. Consequently, alternative
keywords such as "land policy evaluation" OR "land policy evaluation" OR "land policy evaluation" were included and linked in the
search with the Boolean operator OR. The three main concepts were linked using the Boolean AND operator to ensure coverage of
relevant manuscript. The search was performed according to the following equation using Boolean search engines:
TITLE-ABS-KEY ("evaluation of agricultural policy" OR "evaluation of agrarian policy" OR "agricultural policy evaluation") AND
(LIMIT-TO (DOCTYPE, "ar") OR LIMIT-TO (DOCTYPE, "re")) AND (LIMIT-TO (SRCTYPE, "j")).

2.4. Search strategy

The first step of the review was to develop a search strategy to identify manuscript related to agricultural policy evaluation. The
selection criteria were used to classify the manuscript according to the topic of evaluation of agricultural policies. Manuscripts were
included if the policy addressed agricultural or agri-environmental aspects. That are agricultural markets, agricultural production, and
quality of life of the farmer. Manuscripts dealing with environmental sustainability related to agriculture were also included. Man­
uscripts were excluded if the topic was unrelated to the objectives of the literature review (e.g., the relationship between policy and
policy, evaluation of agricultural policy reforms).
The information for data extraction will include the year of publication, authors, title, year of publication, name of the journal,
Source title, abstract, language, risk factors or barriers that cause digital exclusion, and enabling factors that cause digital inclusion. An
Excel table was created to organize the data to ensure that all items meet the requirements. The search was completed on June 30,
2022, yielding 154 manuscripts and a table of updated systematic reviews that included database searches and the recording of reasons
for exclusion (Fig. 1). No automation tools were used. However, Mendeley’s reference management software was used to merge the
electronic search results and remove duplicates. Once duplicate files (35 manuscripts) were removed, 119 manuscripts remained for
review. Next, exclusion was made based on the document type criteria established in Table 1. This exclusion resulted in 87 manuscripts
to be analysed. Two researchers analysed the title and abstract, identifying 66 manuscripts. After this, the full text manuscript was
evaluated, the following exclusion ratios were applied (Table 3), leaving 39 manuscripts, of which it was not possible to find the full
text of 3 manuscripts. Finally, 37 manuscripts met the inclusion criteria. In case of disagreement in any of the criteria, consensus was
reached through discussion. The manuscript included were reviewed, analysed and documented several times.
From: Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated
guideline for reporting systematic reviews. BMJ 2021;372: n71. doi: 10.1136/ bmj.n71 [18].

2.5. Data collection process

After analyzing the 37 documents and identifying their objectives and results, it was defined that the research questions should
inquire about the methodologies, variables, and policies evaluated. The following research questions were formulated:
Question 1: What methodologies were used to evaluate agricultural policies? According to Ref. [19], there are three approaches to
policy evaluation. The first approach is oriented to meet the objectives, and the appropriate choice of policy instruments represents an
economic policy analysis method. A second approach focuses on government behavior based on the behavior and attitudes of voters,
political parties, and other groups in society. Moreover, lastly, the institutional approach refers to the environment in which policies
exist, highlighting the institutional framework, property rights, markets, contracts, and transaction costs.
On the other hand, policies are evaluated according to their process (design, implementation, and evaluation). According to
Ref. [20], the methodology applied in evaluating agricultural policies depends on the stage or cycle of public policies, regardless of the
type of policy. The evaluations are carried out ex-ante or before the implementation using qualitative methodologies, such as the
problem tree and theory of change, and quantitative methodologies, including cost-benefit analysis and computable general equi­
librium models. At the same time, the evaluation of policies in an ex-post stage allows for evaluating their impacts in different areas
(community, market, economy), in this context econometric impact assessment techniques and indicator matrices can be applied.
Question 2: What kinds of policies did you evaluate? For agricultural development, several policies have different objectives or
scopes. It is necessary to identify the types of policies as these show the interests of governments in the timeline. Agricultural policies
are related to rural development, both rurality and the agricultural sector in the world are diverse, and each country is a particular
case; therefore, various public policies focus on agriculture. Public policy analysts still use the four policy typologies defined by

Table 2
The table with the search coverage period - 1966 to 2022.
Database Coverage Language

Scopus 1974–2021 English

Science Direct 1977–2022 English
Web of Science 2016–2022 English
Dimension 1970–2022 English
EBSCO 1966–2022 English, Spanish, Arabic, Russian
Dialnet – –
SciELO – –

4
L.M. Bastidas-Orrego et al. Heliyon 9 (2023) e20292

Fig. 1. PRISMA 2020 flow diagram.

Table 3
Reasons for exclusion.
# Reason Reasons for exclusion Register

1 Market analysis or evaluation 7

2 Agricultural policy design 2
3 Food security assessment 1
4 Policy risk assessment 2
5 Natural resource management 4
6 Assessment tools 2
7 Economic impact of farming 1
8 Economic and environmental impacts of farming 1
9 Impacts on soil and water 1
10 Bird population model 1
11 Does not evaluate agricultural policies 3
12 Simulation of an agricultural system 2

5
L.M. Bastidas-Orrego et al. Heliyon 9 (2023) e20292

Ref. [21]. The typology is presented in Table 4.

In the study by the Economic Commission for Latin America and the Caribbean -ECLAC- [23], the authors present a classification of
agricultural policies as follows: land policies, large infrastructure policies, policies to strengthen the capacities of human resources, and
regulation policies. The authors also classify thematic or transversal policies as those whose main objective is not linked to agricultural
production. However, they impact agriculture in some way, such as those that seek to preserve the environment, agri-environmental,
security, and food sovereignty, fight against poverty, or promote territorial development.
However, the Food and Agriculture Organization of the United Nations (FAO) determines a taxonomy of agricultural policies,
which is defined by the requirements of the producer, resulting in three types of policies, detailed in the following table:
From previous classifications, the analysis of the agricultural policies evaluated in the 37 documents will be framed within the
taxonomy of agricultural policies of the FAO [24], defined in Table 5.
Question 3: What variables were analysed in the evaluation of agricultural policies? The variables are essential in policy evaluation
methodologies since they define the aspects considered. For the information analysis, the variables are classified as economic, envi­
ronmental, fiscal, and socioeconomic (Table 6). The economic variables have to do with those directly related to agricultural pro­
duction, the market for agricultural products, whether internal or external, the factors of production (land, capital, labor, technology),
and access to credit.
From the policy approach, the contributory variables, according to Ref. [25], are variables that can be manipulated in the political
process, while [26] affirm that these variables are finally the ones that are evaluated because, according to their character, their impact
is more significant.
The variables analysed change according to the type of methodology used, the stage in which the policy is found within the cycle,
and the type of Policy found (policies focused on the national or international market, land tenure, increased productivity, decreased in
the environmental impact, fiscal, monetary, among others). It also depends on the territorial scope given to the evaluation of the Policy
since there are policies at the level of regions, municipalities, countries, or agricultural products.

2.6. Outcomes and prioritization

The primary outcome was the identification of the types of methodologies used to evaluate agricultural policies. obtaining a
comparison of the initial and current evaluation methodologies, which allows for identifying a trend in the change of the evaluation
methodology. A secondary result was to identify the types of agricultural policies evaluated according to the FAO taxonomy. As a third
result, there are the types of variables evaluated in the identified methodologies.

2.7. Risk of bias individual studies

The investigators minimized the possible impact of information bias by ensuring that this work was an exploratory review of the
literature, including the most relevant databases and resources for locating relevant publications and by being careful about the
duplication of data.
The four experts (authors of this document) participated in the strategy developed to minimize selection bias. The first expert
carried out the first search, using the previously defined search equation, and subsequently carried out the exclusion process by type of
document and duplicate documents. The second expert again reviewed the results obtained in the first search and made a second
exclusion under the criteria associated with the topics already explained. The third expert analysed the aspects necessary to answer the
three research questions in the 37 documents analysed in depth. The fourth expert reviewed the results obtained by the third expert in
the 37 documents.

3. Results

3.1. Study selection

Table 7 shows the results of the search for the manuscripts in the databases. In Dialnet and SciELO, Latin American databases, no
manuscripts on this topic were found, despite having highly agricultural countries.

Table 4
Typology of policies according to Theodore J. Lowi.
Individual Environment

Direct Regulatory Policy: creation and control of norms and Redistributive Policy: refers to the redistribution of income from one social group to
laws. another, through financing or delivery of resources.
Indirect or Distributive Policy: They are related to the provision Constitutive Politics: They are the ones that define the rules of the game, that define the
distant of state services or facilities. political system

Source: Based on [22].

6
L.M. Bastidas-Orrego et al. Heliyon 9 (2023) e20292

Table 5
Taxonomy of agricultural policies according to FAO.
Taxonomy of agricultural policies Agricultural sector policies

Price policy, determined in part by macroeconomic policy. It is based on incentives, such as lower taxes. - Tax policies
- Pricing policy
- Foreign trade policy
- Exchange policy
Resource policies, must ensure access to resources to be able to produce (land, water, forests and fisheries). - Land tenure policy
- Water management policy
- Environmental Policy
- Agri-environmental policy
- Human resource policy
Access policies must ensure access to input and product markets. - Financial policy
- Support policy for inputs
- Policy of access to agricultural technology

Table 6
Typology of variables.
Variable Type Variables

Economic Agricultural production: Income, costs, expenses, profits, investment, agricultural and non-agricultural production, economic efficiency,
economic returns, budgets, and labor structure of the farm.
Market: prices, imports, exports, demand, supply, elasticity, agricultural GDP.
Factors of production and technological adoption
Environmental Environmental sustainability, ecosystem services, water balance, ecological indicators, biodiversity and water health.
Contributory Subsidies, taxes, transfers to support producers, costs of support policies, access to credit, and citizen participation in politics.
Socioeconomic People, consumer welfare, property rights, quality of life, socioeconomic variables.

3.2. Study characteristics

Table 8 shows the list of the 37 analysed manuscripts focused on the methodological design to evaluate agricultural policies, the
agricultural policies that have been evaluated and the variables involved in the evaluation by various evaluation methodologies.

3.3. Result of syntheses

The aim of this review was to identify articles that systematically review methodologies, policies and variables evaluated during the
last 50 years, in order to know if a policy has been efficient. To assess the quality of the included manuscript and to describe the
measures and domains identified.
The evaluation of agricultural policies is essential, on which work has been carried out from the methodological designs for
evaluating and measuring impacts on the economy and society; however, the research results have gained strength from 2012 onwards
the absence of studies before 2012 may be due to the lack of agricultural information. That is, there are deficiencies in agricultural
information systems, as stated by Ref. [45].
Table 9 shows the documents with the most references, the most cited are "The policy analysis matrix for agricultural development"
[31], and the second is "Representation of decision-making in European agricultural agent-based models" [52]. The first publication is
related to the design of an indicator matrix, which is currently used in some ex-post agricultural policy evaluations of the Organization
for Economic Cooperation and Development (OECD) and ECLAC; the matrix is an evaluation methodology widely used since its

Table 7
Database search results - (include jun-2022).
Database First Exclusion criteria Inclusion
search criteria
Document Type peer- Eliminating Quick Not possible to find Manuscript
reviewed journals duplicates review the full text evaluated

Scopus 26 6 2 2 0 3 12
Science 61 6 2 14 2 20 17
Direct
Web of 8 0 6 1 0 0 1
Science
Dimension 39 14 17 3 1 1 4
EBSCO 20 6 8 0 0 3 3
Dialnet 0 0 0 0 0 0 0
SciELO 0 0 0 0 0 0 0
TOTAL 154 32 35 20 3 27 37

7
 Table 8

L.M. Bastidas-Orrego et al.

List of reviewed papers.
Authors Title Source Country Territorial Methodology Typology Economic Environmental Contributing Socio- References
scope policy variables variables variables economic
variables

Byerlee D., Halter A.N. A Macro-Economic model Scopus Nigeria Economic Logistic Pricing x [27]
(1974) for agricultural sector sector Regression policy
analysis
Parikh KS (1977) An agricultural policy Science India Economic Other analysis Access x x x [28]
model for India* Direct sector policy
Baum K, Richardson J, A stochastic recursive Science United Farms Lineal Pricing x [29]
Schertz L (1984) interactive programming Direct States programming policy
model for farm firm policy model
analysis
Anderson R., Wilkinson Consumer demand for meat Scopus EUnited Economic Other analysis Pricing x x [30]
M. (1985) and the evaluation of States sector policy
agricultural policy
Monke E.A., Pearson S. The policy analysis matrix Scopus Not Economic Framework Pricing x [31]
R. (1989) for agricultural applicable sector policy
development
Roth MJ, Abbott PC, Evaluating agricultural Science Sub- Region Economic Pricing x [32]
Preckel PV (1990) price policy under dual Direct Saharan model policy
market regimes and Africa
institutional constraints
Michalek J. (1995) An Application of the Scopus Slovenia Region Economic Pricing x [33]
Policy Analysis Matrix for model policy
an Evaluation of
8

Agricultural Policies in the

Slovak Republic
Adesina AA, Djato K.K. Farm size, relative Science Côte Farms Descriptive Resources x [34]
(1996) efficiency and agrarian Direct d’Ivoire statistics policy
policy in Côte d’Ivoire:
profit function analysis of
rice farms
Rabinowicz E. (1996) Is Eastern Europe copying Scopus Europe Farms MAP Pricing [35]
Western agricultural policy
policy?
Ferto I. (1999) Restructuring of the Scopus Hungary Region MAP Pricing and x x [36]
Hungarian agri-food sector resources
policy
Semos AV (2001) Estimation of Producer’s EBSCO Greece Economic Analysis Access x x [37]
Assistance and Evaluation sector Matrix policy
of Agricultural Policy for
Olive Oil in Greece
Volk, Tinea (2005) Effects of agricultural Dimensions Slovenia Country Séameles Access and x x [38]
policy on the development pricing

Heliyon 9 (2023) e20292

of Slovenian agriculture policy
during the transition and
the process of accession to
the European Union
(continued on next page)
 Table 8 (continued )

L.M. Bastidas-Orrego et al.

Authors Title Source Country Territorial Methodology Typology Economic Environmental Contributing Socio- References
scope policy variables variables variables economic
variables

Oskam AJ, Meester G How useful is the PSE in Science Not Country Coupled model Pricing x [39]
(2006) determining agricultural Direct applicable policy
support?
Therond O, Methodology to translate Science Not Country Simulation Resources x x [40]
Belhouchette H, policy assessment problems Direct applicable policy
Janssen S, Louhichi into scenarios: the example
K, Ewert F, Bergez of the SEAMLESS
JE, Wery J, integrated framework
Heckelei T, Olsson
JA, Leenhardt D,
Van Ittersum M
(2009)
Viaggi D,Raggi M, Farm-household Science Europe Farms Quantitative Pricing x [41]
Gomez y Paloma S investment behavior and Direct method policy
(2011) the CAP decoupling:
Methodological issues in
assessing policy impacts
Mouysset L,Doyen L, Different policy scenarios Science France Farms Quantitative Resources x [42]
Jiguet F (2012) to promote various targets Direct method policy
of biodiversity
Luca L. (2012) Institutionalization of Scopus Rumania Country Structural Resource [19]
agricultural policy Equations policy,
evaluation in European access
9

Union policy and

pricing
policy
Luca L, Pelinescu E Methods to evaluate the EBSCO Rumania Country Equilibrium Resource [43]
(2012) agricultural policies and model policy,
their relevance for access
Romania policy and
pricing
policy
Akbar M,Jamil F Monetary and fiscal Science Pakistan Country Equilibrium Pricing x [44]
(2012) policies’ effect on Direct model policy
agricultural growth: GMM
estimation and simulation
analysis
Wang X,Shen Y (2014) The effect of China’s Science China Country Estimation Pricing x x [45]
agricultural tax abolition Direct model policy
on rural families’ incomes
and production
Kirchweger S, The dynamic effects of Science Austria Farms Estimation Not x x [46]

Heliyon 9 (2023) e20292

Kantelhardt J government-supported Direct model applicable
(2015) farm-investment activities
on structural change in
Austrian agriculture
Colen, L.; Gomez y Economic Experiments as a Dimensions Europe Country Linear Resource x [11]
Paloma, S.; Latacz- Tool for Agricultural Policy programming policy,
(continued on next page)
 Table 8 (continued )

L.M. Bastidas-Orrego et al.

Authors Title Source Country Territorial Methodology Typology Economic Environmental Contributing Socio- References
scope policy variables variables variables economic
variables

Lohmann, U.; Evaluation: Insights from access

Lefebvre, M.; the European CAP policy and
Préget, R.; Thoyer, pricing
S. (2016) policy
Dawson N, Martin A, Green Revolution in Sub- Science Rwanda Farms Simulation Access x x [47]
Sikor T (2016) Saharan Africa: Direct policy and
Implications of Imposed resources
Innovation for the policy
Wellbeing of Rural
Smallholders
Huber R, Rebecca S, Interaction effects of Science Switzerland Region Simulation Resources x x x [48]
François M, Hanna targeted agri- Direct policy
BS, Dirk S, Robert F environmental payments
(2017) on non-marketed goods and
services under climate
change in a mountain
region
Whittaker G, Färe R, Spatial targeting of agri- Science United Farms Surveys Resources x x [49]
Grosskopf S, environmental policy using Direct States policy
Barnhart B, Bostian bilevel evolutionary
M, Mueller- optimization
Warrant G, Griffith
10

S (2017)
Kremmydas D, A review of Agent Based Science Not Farms Econometric Not [50]
Athanasiadis IN, Modeling for agricultural Direct applicable model applicable
Rozakis S (2018) policy evaluation
Babu SC,Mavrotas G, Integrating environmental Science Nigeria Farms Econometric Resources x x x [51]
Prasai N (2018) considerations in the Direct model policy
agricultural policy process:
Evidence from Nigeria
Huber R., Bakker M., Representation of decision- Scopus Europe Farms Structural Not [52]
Balmann A., Berger making in European Equations applicable
T., Bithell M., agricultural agent-based
Brown C., Grêt- models
Regamey A., Xiong
H., Le Q.B., Mack
G., Meyfroidt P.,
Millington J.,
Müller B., Polhill J.
G., Sun Z., Seidl R.,
Troost C., Finger R.
(2018)

Heliyon 9 (2023) e20292

Kiryluk-Dryjska E, Reforms of the Common Science Europe Country Simulation Pricing x [53]
Baer-Nawrocka A Agricultural Policy of the Direct policy
(2019) EU: Expected results and
their social acceptance
(continued on next page)
 L.M. Bastidas-Orrego et al.
Table 8 (continued )
Authors Title Source Country Territorial Methodology Typology Economic Environmental Contributing Socio- References
scope policy variables variables variables economic
variables

Sy, Hieu Phan; Van, Review and Evaluation of Dimensions 51 Country Hydrological Pricing x x [6]
Thang Phan; Lam, Agricultural Policies in countries + model policy and
Tuan Nguyen; Thi Years 2015–2017 Vietnam access
Thanh, Xuan policy
Nguyen (2019)
Soulis K.X., Psomiadis A new model-based Scopus Greece Farms Machine Resources x [54]
E., Londra P., approach for the evaluation Learning policy
Skuras D. (2020) of the net contribution of
the european union rural
development program to
the reduction of water
abstractions in agriculture
Rashad, H. M. M. ; El- Agricultural policy EBSCO Egypt Country Machine Pricing x x [55]
Adamy, M. S.; evaluation of the most Learning policy and
Ragab, M. E.; El- important ceral crops, access
Maksoud, A. M. A. using the policy policy
(2020) anaysismatrix.
Fernández, VD; Mainstreaming evaluation Web Of Uruguay Farms Linear Resources x x x [20]
Aguirre, E; in public policies in Science programming policy and
Baraldo, J; Uruguay. Results and access
11

Hernández, E; lessons learned in policy

Laguna, H (2020) agriculture policies
Boysen-Urban K., Measuring the Trade Scopus Europe Country Systematic Access and x [56]
Brockmeier M., Restrictiveness of Domestic review pricing
Jensen H.G., Support using the EU policy
Boysen O. (2020) Common Agricultural
Policy as an Example
Mora-Herrera D.Y., A review of agent-based Scopus Not Not Systematic Not [57]
Huerta-Barrientos modeling for simulation of applicable applicable review applicable
A., Zúñiga-Escobar agricultural systems
O. (2021)
Lefebvre M., Barreiro- Can Economic Experiments Scopus Europe Country Other analysis Resources x x [58]
Hurlé J., Contribute to a More policy
Blanchflower C., Effective CAP?
Colen L., Kuhfuss
L., Rommel J.,
Šumrada T.,
Thomas F., Thoyer
S. (2021)

Heliyon 9 (2023) e20292

Stetter, Christian; Using Machine Learning to Dimensions Germany Farms Simulation Resources x x x [59]
Mennig, Philipp; Identify Heterogeneous policy
Sauer, Johannes Impacts of Agri-
(2022) Environment Schemes in
the EU: A Case Study
L.M. Bastidas-Orrego et al. Heliyon 9 (2023) e20292

Table 9
Most cited documents according to coupled data analysis in
VOSviewer [60].
Authors Citations

Monke E.A. (1989) [31] 214

Michalek J. (1995) [33] 5
Byerlee D. (1974) [27] 7
Ferto I. (1999) [36] 2
Huber R. (2018) [52] 68
Boysen-Urban K. (2020) [56] 3
Soulis Lx. (2020) [54] 5

Table 10
Methodologies used in the evaluation of agricultural policies.
Methodologies No. Manuscripts

Statistics Descriptive Statistics 12

Estimation model
Econometric Model
Balance model
Logistic regression
Framework Framework 2
Scenario method under the SEAMLESS-IF
Operations Research Linear Programming 7
Simulation
Matrix MAP matrix 3
Analysis Matrix
Quantitative Method Structural equations 5
Machine learning
Quantitative Method
Modelling Coupled model 2
hydrological model

publication. The second publication refers to a review of 20 agent-based models (SBA) that evaluate different aspects of the Common
Agricultural Policy (CAP) in the European Union. Each model addresses a different problem and heterogeneous decision-making
processes. They conclude that ABS in agriculture should be further developed, combining it with other modelling approaches that
seek to manage data since farmers’ social interactions, values, emotions, and learning are less and less modelled.
Of the 86 manuscript and reviews found, 43% (37) were focused on evaluating agricultural policies, which were found in several
languages, mainly in English. Of the 37 manuscripts reviewed, 7 analyses standard agricultural policies (CAP) in the European Union
(EU). Another 11 documents analyses specific CAP cases in different EU countries (France, Germany, Austria, Switzerland, Slovenia,
Greece, and Romania), adding 18 manuscript corresponding to 49% of the documents evaluating some CAP policies. According to the
official website of the European Community, the standard agricultural Policy (CAP) was created in 1962 and sought to support farmers
and guarantee food security in the European Union. Therefore, it is managed and financed with budget resources from the European
Union.

3.4. Q1 results. Methodologies used for the evaluation of agricultural policies

It was found that more than 90% of the evaluations on agricultural policies were in the ex-post stage. That is, after their application,
others were carried out ex-ante. Table 10 shows the evaluation methodologies used. 32% of the works used statistical tools, such as
descriptive statistics, estimation models, logistic regression, and econometric models, the latter being the most used since they focus on
evaluating the market and global policies. A representative use of simulation models is identified, with ABS being the most repre­
sentative, followed by linear programming.
According to the documents analysed between 1974 and 2000, the models’ application is based on linear programming, the general
equilibrium model, quantitative models, and the proposal and subsequent application of the Policy Analysis Matrix (PAM). As of 2011,
the trend moves to hybrid models, agent-based models, simulation, and experiments (discrete choice experiments, laboratory and field
experiments) without ceasing to use econometric models, which focus on micro and macroeconomic variables and serve to evaluate
market policies focused on the agricultural sector.
However, when an agricultural policy is evaluated, social variables that are difficult to quantify are linked, such as culture, self-
concepts, the producers’ values, and the peasants’ customs, among others. Some of these variables are left out of the most quanti­
tative types of evaluation, which is why there are combinations of models, as in the work of [41], which compare and discuss the role of
surveys and models in detailing the impact of agricultural policy reforms. The authors suggest using mixed methodologies to evaluate
the impacts of policies, especially those of the CAP since the work was done for the EU.
The problems are becoming more complex, meaning that the methodologies applied to policy evaluation are more sophisticated,

12
L.M. Bastidas-Orrego et al. Heliyon 9 (2023) e20292

and combining several methodologies is more common. For example [49], applied a hybrid optimization and simulation model to
analyze the fertiliser tax policy in an area of Oregon, United States. In this exercise, the evaluation of policies considered the
non-commercial valuation of ecosystem services. They found through the developed algorithm multiple different policy configura­
tions, which yielded similar results for the goals proposed by the agency.

3.5. ABS methodology - review manuscript

In total, three analysis manuscript were identified. However, it is highlighted that they are literature review manuscript in which
the importance of using agent-based modelling (ABS) is analysed as a widely used tool for evaluating agricultural policies. Despite
being works that do not have the evaluation of agricultural policies as their primary objective, they do offer important information
from the existing bibliography for the ex-ante and ex-post evaluation of agricultural policies [52]. analysed 20 European agricultural
ABM with a focus on the representation of the decision-making process. Depending on the approach, the models include different
elements at the farm level, which leads to ignoring values, social interactions, consideration of norms, and farmers’ decision-making
learning in SBAs in Europe. Therefore, they suggest that the new models improve the representation of farmers’ decision-making and a
better representation of the farming systems perspective.
Also [50], conducted a systematic review of the literature on peer-reviewed journal articles that aimed to use the AMB in evaluating
agricultural policies. They conclude that SBAs for agricultural policy evaluation can complement conventional (equilibrium-based)
agricultural models since farming systems can be easily modelled and include interactions between farms and market information. In
general, although it surpasses conventional modelling methods, this type of modelling has difficulties in being adopted by modelers
and has limitations in applying to large-scale evaluations.
[57] did work similar to that of [50]. In this work, the time window is between 2009 and 2019. It concludes that from 2009
onwards, the MBA is a technique that has been most used to model agricultural systems since it has proven to be a helpful tool for
integrating qualitative information. Quantitative and has been a framework of analysis for the society-nature relationship of agri­
cultural systems under an interdisciplinary and transdisciplinary approach at the micro and macro levels.

3.6. Q2 results. Evaluated policies

At least eight documents evaluated up to two and three agricultural policies, such as [11,20,38,43,53]. These authors conduct
evaluations of agricultural policies in general, focusing on evaluating the CAP in the EU. Policies were classified according to the FAO
taxonomy. It was identified that the price policies (Table 11) were the most evaluated during the entire period analysed. These policies
are related to taxes, market opening, protectionism for farmers, and price policies.
Agro-environmental policies are within resource policies, which are the second most evaluated. These policies have gained
strength, seeking that agricultural activity is responsible and balanced with the environment, presenting growing importance that
governments have given to environmental impacts and climate change as a response to the Kyoto Protocol and consequently to the
Objectives of the Millennium published by the UN. 21% of manuscripts are related to this type of policy. His study territory was in one
of the countries of the European Union, the United States, and one worked in Nigeria. A trend in the evaluation of agro-industrial
policies has been identified since 2017 with the work of [48].
However, for [48], agro-environmental measures are crucial to increasing support from governments that seek to conserve
biodiversity and provide ecosystem services from agricultural activities, and how these policies can influence the production of
non-marketed goods and services in marginal regions with agricultural vocation, which can generate a high probability of land
abandonment, which, in turn, has implications for territorial dynamics and impacts on rural communities.
The first work that attempts to evaluate this type of Policy are from Ref. [40], where they develop a method to adequately (and
transparently) implement scenarios for the range of models considered in an IAM (Integrated Assessment Model) framework such as
SEAMLESS-IF (Indicator Based).
Access policies are the least evaluated. However, they have a direct effect on farmers. The work done by the OECD [6] analysed the
changes in the agricultural policies of 51 countries associated with the organization between 2015 and 2017. They found that agri­
cultural support and aid have been declining, mainly in developed countries, and have had a slight rise in emerging countries. On the
other hand [47], shows how the support has benefited people with greater land ownership while the poorest have benefited less.

Table 11
Types of agricultural policies and number of countries where
they were evaluated.
Row Labels Country Account

Access Policy 13
Pricing Policy 20
Resource Policy 15
Grand Total 48

13
L.M. Bastidas-Orrego et al. Heliyon 9 (2023) e20292

Table 12
Evaluated variables.
Variable Typology Papers

Economics 24
Environmental 9
Contributive 13
Socio-economics 6
Grand Total 52

The economic variables are the most analysed in the

manuscript. They are related to the price policy,
which in turn is linked to economic and market var­
iables such as prices, elasticity, the value of produc­
tion, the value of expenses, input costs, the value of
imports and exports, and the labor market.

3.7. Q3 results. Variables used

The variables valued in the studies are directly related to the agricultural policies being evaluated. Thus, the economic variables are
mainly related to price policies and access policies, the environmental variables are related to resource and access policies, in this case,
land, and the contributory variables are mainly related to resource and access policies.
Table 12 lists the number of manuscripts that evaluated each of the variables. Generally, a policy evaluation methodology can
evaluate one or different types of variables.
The second most evaluated variable is the contributory one since these are related to direct support in money or transfers the
government grants to producers. These supports are related to credits, payments for ecosystem services, economic support for pur­
chasing inputs, payments for food production, or aid in general.
The agri-environmental variables began to be identified from 2009 onwards, and it is evident that there is a direct relationship
between support and transfers from the government, seeking to improve ecosystem services and achieve environmental sustainability
in agricultural production. This aspect is evaluated in the works of [20,40,48,51,58].
The work of [47] reflects on the limited assessment in Rwanda of impacts on basic needs, livelihoods, cultural identities, or the
vulnerability of rural households. These variables were not identified in the studies analysed in this literature review. In addition, it
was identified that social aspects such as citizen participation in policy design are not considered in the evaluations.
This review shows that economic and contributory variables have been decisive in the evaluation of agricultural policies. While
environmental variables began to emerge in 2009. On the other hand, socioeconomic variables are the least evaluated and those that
are directly related to the quality of life of farmers or producers. The evolution of the variables can be seen in Table 7.

4. Discussion

The aim of this review was to identify manuscript that systematically review methodologies, policies and variables evaluated
during the last 50 years to determine if a policy has been efficient. We analysed 37 manuscripts and the results of their research,
seeking to identify the types of methodologies used for their evaluation (for example, if they used simulation models based on agents,
matrices, surveys), the type of agricultural policy evaluated, and what variables were integrated into these methodologies. In addition
to this review, we found two reviews on agent-based modelling related to agricultural policies, limited only to an evaluation meth­
odology [40,47], which is very interesting because they can consider new relationships between actors in the rural environment
(peasants, government, suppliers, distributors), relationships that can be heterogeneous. However, these studies limit the identifi­
cation of other methodologies (qualitative, quantitative, mixed). On the other hand, no literature reviews were found that identify
possible agricultural policy evaluation methodologies.
It was found that agricultural policy evaluation models have evolved over time from using purely qualitative models such as
statistical models, operations research models and quantitative methods. It was identified that in the last five years, the analysed
studies show the use of mixed models (qualitative and quantitative), such as [48] who used an agent-based model, where they analysed
the impact of an agricultural policy on climate change [49]; designed hybrid optimization and simulation models to value nonmarket
policies and ecosystem services [51]; applied a Logit Regression Model for information analysis, taking into account people’s
perception of environmental concerns and agricultural policies in Nigeria [53]; presented a combination of a Partial Equilibrium
model and a movement theory model (game theory) to make an ex ante evaluation of CAP reforms in the European Union [20];
presented a methodological toolbox with qualitative and quantitative indicators to evaluate agricultural policies in Uruguay; and [59]
designed a model using Machine Learning to evaluate agricultural policies versus agri-environmental schemes in Germany, taking into
account farm heterogeneity.
The results of the analysed studies show that the trend in the design of methodologies to evaluate agricultural policies is focusing on
the combination of qualitative and quantitative models. Furthermore, the integration of farmers’ perceptions is also shown. It is
important to highlight in the review that farmers are the ones who ultimately receive government support through projects and public
policies, and they are also immersed in the new rural dynamics, which are widely studied from the theoretical perspective of the new
rurality.

14
L.M. Bastidas-Orrego et al. Heliyon 9 (2023) e20292

In the 37 manuscripts evaluated, it was identified that one or more agricultural policies were evaluated in combination, according
to FAO’s taxonomy of agricultural policies. Agri-environmental policies have been evaluated more frequently in this century. The
works of [42,48,49,51,54,59], evaluated agricultural policies associated with natural resource conservation or associated with
ecosystem service provision. Policies associated with economic transfers and policies affecting the agricultural market were the
policies with the highest number of evaluation models found.
There are few reviews related to the evaluation of agricultural policies, which is an important aspect in the rural development of
countries. Therefore, this review is an attempt to shed some light on this area. Information that may be useful for future research
focused on ex-ante or ex-post evaluation of agricultural policies around the world. In this way, the results of this work may help the
adoption of existing policy evaluation models or provide important information for the design of new evaluation models.
Even though agriculture is a very important sector for a country’s economic sustainability and food security, there is no significant
body of studies that evaluate of agricultural policies in the world. The low production of manuscript in the Latin American context
stands out, considering that they are highly agricultural. The OECD analyses the agricultural policies for this region and only for those
countries attached to it. However, each country in its public policy cycles has some policy evaluation methodology, either as a toolbox
or a simple matrix of indicators, as a validation process for administrative procedures.
On the other hand, we are cautious in the presentation of these results because incomplete descriptions of methodologies were
found or did not specify in depth the variables used in the evaluation. However, future studies may focus on more specific aspects, such
as the evaluation of agro-environmental policies or a type of project evaluation methodology. Another limitation is related to the
search criteria, since the manuscripts evaluated were limited to journal articles, published peer-reviewed papers and reviews. Were
excluded books and book chapters. In addition, this systematic review will not extend to a meta-analysis. Despite these limitations, the
proposed protocol adopts a well-established methodology based on the PRISMA method.

5. Conclusions

The results of this review made it possible to identify methodologies for evaluating public agricultural policies worldwide during a
period from 1972 to June 2022. It was found that there have been few systematic reviews related to methodologies for evaluating
agricultural policies; those found only focus on the use of agent-based models. In this work, 37 manuscripts were analysed, in which the
main methodologies used were statistics, with operational research tools and quantitative methods, highlighting that qualitative
methods have been barely used. The most evaluated type of policy is the price policy followed by resource policies (typology according
to FAO), coinciding with the economic variables as the most used variables in the evaluation methodologies found. The PRISMA
protocol was adopted to guide the systematic review, ensuring that the search processes had methodological rigor and transparency.
The most important findings of this review are the evolution and use of methodologies to evaluate agricultural policies and the
importance given to measuring the impacts of agri-environmental policies. On the other hand, this review highlights the few publi­
cations on this subject in Latin American countries, which have traditionally been agricultural and have multiple problems in the rural
sector.
Considering the impact of agricultural policies on the rural development of a country, this research is relevant because it provides
an important list of evaluation methodologies for these policies and their typology, generating relevant information for the design of
new methodologies for current policies. This manuscript will be of interest to those responsible for implementing agricultural policies
focused on rural development and policy makers because the impact of a policy must be measured ex-ante and ex-post of its imple­
mentation. Without the correct follow-up of a public policy, the public budget can be invested and not solve agricultural and rural
social problems.
Finally, this manuscript highlights the gaps in the use of agent-based models to evaluate how agricultural policies have impacted
farmers’ decisions, and it is perceived that this is an unexplored area of research that allows the design of new evaluation models based
on the joint use of qualitative and quantitative models. It is essential to highlight that these models must be evaluated jointly because
these policies are implemented in order to achieve sustainable rural development, and it must also be considered that the producers are
decision makers that could impact the design and implementation of policies.

Author contribution statement

Lina Bastidas-Orrego, Natalia Jaramillo: conceived and designed the experiments; analysed and interpreted the data, wrote the
paper.
Julian Castillo, Yony Ceballos: analysis tools or data, wrote and translate the paper.

Funding statement

This paper was funded by Corporación Universitaria Remington – 4000000352 – in agreement with Fundación Universitaria María
Cano, Institución Universitaria Digital de Antioquia y Universidad de Antioquia.

Data availability statement

The authors do not have permission to share data.

15
L.M. Bastidas-Orrego et al. Heliyon 9 (2023) e20292

Additional information

No additional information is available for this paper.

Declaration of competing interest

The authors declare no conflict of interest.

Acknowledgement

The authors wish to sincerely thank to the institutions of affiliations.

Appendix A. Supplementary data

Supplementary data to this article can be found online at https://doi.org/10.1016/j.heliyon.2023.e20292.

References

[1] B.G. Peters, Advanced Introduction to Public Policy, Edward Elgar Publishing, 2021.
[2] Yusuff Jelili Amuda, Evaluation of agricultural policies and programmes for sustainable future farming intensification in Nigeria, Int. J. Serv. Sci. Manag. Eng.
Technol. 13 (2022) 13, https://doi.org/10.4018/IJSSMET.316176.
[3] C. Knill, J. Tosun, Public Policy: A New Introduction, Bloomsbury Publishing, 2020.
[4] J. Wilson, Qué son y para qué sirven las políticas públicas? Iure 2 (2019).
[5] B. Czyżewski, Ł. Kryszak, Evolution of agri-environmental schemes worldwide. Comparing the agricultural policy of the EU, the US and the people’s Republic of
China, in: B. Czyżewski, Ł. Kryszak (Eds.), Sustain. Agric. Policies Hum. Well- Integr. Effic. Approach, Springer International Publishing, Cham, 2022,
pp. 175–193, https://doi.org/10.1007/978-3-031-09796-6_6.
[6] DrHP. Sy, Van DrTP, M.A.T.N. Lam, M.A.X.N. Thi Thanh, Review and evaluation of agricultural policies in years 2015-2017, J. Econ. Trade Mark Manag. 1
(2019) p70, https://doi.org/10.22158/jetmm.v1n2p70.
[7] J.M. Sánchez, J.P. Rodríguez, H.E. Espitia, Review of artificial intelligence applied in decision-making processes in agricultural public policy, Processes 8 (2020)
1374, https://doi.org/10.3390/pr8111374.
[8] A. Mergoni, K. De Witte, Policy evaluation and efficiency: a systematic literature review, Int. Trans. Oper. Res. 29 (2022) 1337–1359, https://doi.org/10.1111/
itor.13012.
[9] E. Kolesnik, L. Pavlova, T. Rybalova, Effectiveness evaluation of implementation of target programs for municipal education: improvement of approaches, Vieš
Polit Ir Adm 17 (2018) 568–585, https://doi.org/10.13165/VPA-18-17-4-06.
[10] K. Hendren, K. Newcomer, S.K. Pandey, M. Smith, N. Sumner, How qualitative research methods can be leveraged to strengthen mixed methods research in
public policy and public administration? Publ. Adm. Rev. 83 (2023) 468–485, https://doi.org/10.1111/puar.13528.
[11] L. Colen, S. Gomez y Paloma, U. Latacz-Lohmann, M. Lefebvre, R. Préget, S. Thoyer, Economic experiments as a tool for agricultural policy evaluation: insights
from the European CAP: economic experiments as a tool for agricultural policy evaluation, Can. J. Agric. Econ. Can. Agroeconomie 64 (2016) 667–694, https://
doi.org/10.1111/cjag.12107.
[12] G. Pe’er, A. Bonn, H. Bruelheide, P. Dieker, N. Eisenhauer, P.H. Feindt, et al., Action needed for the EU Common Agricultural Policy to address sustainability
challenges, People Nat. 2 (2020) 305–316, https://doi.org/10.1002/pan3.10080.
[13] F. Moreira, G. Pe’er, Agricultural policy can reduce wildfires, Science 359 (2018), https://doi.org/10.1126/science.aat1359, 1001–1001.
[14] P. Pingali, Agricultural policy and nutrition outcomes – getting beyond the preoccupation with staple grains, Food Secur. 7 (2015) 583–591, https://doi.org/
10.1007/s12571-015-0461-x.
[15] H. Eakin, S. Sweeney, A.M. Lerner, K. Appendini, H. Perales, D.G. Steigerwald, et al., Agricultural change and resilience: agricultural policy, climate trends and
market integration in the Mexican maize system, Anthropocene 23 (2018) 43–52, https://doi.org/10.1016/j.ancene.2018.08.002.
[16] M.-H. Ehlers, R. Huber, R. Finger, Agricultural policy in the era of digitalisation, Food Pol. 100 (2021), 102019, https://doi.org/10.1016/j.
foodpol.2020.102019.
[17] B. Hutton, F. Catalá-López, D. Moher, The PRISMA statement extension for systematic reviews incorporating network meta-analysis: PRISMA-NMA, Med.
Clínica Engl. Ed. 147 (2016) 262–266, https://doi.org/10.1016/j.medcle.2016.10.003.
[18] M.J. Page, J.E. McKenzie, P.M. Bossuyt, I. Boutron, T.C. Hoffmann, C.D. Mulrow, et al., The PRISMA 2020 statement: an updated guideline for reporting
systematic reviews, Int. J. Surg. 88 (2021), 105906, https://doi.org/10.1016/j.ijsu.2021.105906.
[19] L. Luca, Institutionalization of agricultural policy evaluation in European Union, Calitatea 13 (2012) 544–551.
[20] V.D. Fernández, E. Aguirre, J. Baraldo, E. Hernández, H. Laguna, Resultados y aprendizajes de la evaluación de políticas agropecuarias en Uruguay, Cuad. Claeh
39 (2020) 103–117, https://doi.org/10.29192/claeh.39.2.7.
[21] T.J. Lowi, Four systems of policy, politics, and choice, Publ. Adm. Rev. 32 (1972) 298–310, https://doi.org/10.2307/974990.
[22] A. Roth, Políticas Públicas. Formulación, Implementación Y Evaluación, Ediciones Aurora, 2004.
[23] E. Sabourin, M. Samper, O.S. Coordinadores, Políticas públicas y agriculturas familiares en América Latina y el Caribe: balance, desafíos y perspectivas, 2014.
[24] R.D. Norton, Agricultural Development Policy: Concepts and Experiences, Wiley, 2005.
[25] K.C. Land, D. MacRae, Review of policy indicators: links between social science and public debate, Duncan MacRae, Jr. Soc. Forces 65 (1986) 260–262, https://
doi.org/10.2307/2578948.
[26] M. González, E.M. Rincón, J. Sánchez, A. Oropeza, Modelos para el seguimiento y evaluación de programas de cáncer en Venezuela, Rev. Venez. Oncol. 21
(2009) 189–202.
[27] D. Byerlee, A.N. Halter, A macro-economic model for agricultural sector analysis, Am. J. Agric. Econ. 56 (1974) 520–533, https://doi.org/10.2307/1238604.
[28] K.S. Parikh, An agricultural policy model for India, IFAC Proc. 10 (1977) 533–543, https://doi.org/10.1016/S1474-6670(17)66505-4.
[29] K. Baum, J. Richardson, L. Schertz, A stochastic recursive interactive programming model for farm firm policy analysis, Comput. Oper. Res. 11 (1984) 199–222,
https://doi.org/10.1016/0305-0548(69)90010-0.
[30] R. Anderson, M. Wilkinson, Consumer demand for meat and the evaluation of agricultural policy, Empir. Econ. 10 (1985) 65–89, https://doi.org/10.1007/
BF01973542.
[31] E.A. Monke, S.R. Pearson, Review: the policy analysis matrix for agricultural development, Dev. South Afr. 7 (1990) 133–140, https://doi.org/10.1080/
03768359008439507.

16
L.M. Bastidas-Orrego et al. Heliyon 9 (2023) e20292

[32] M.J. Roth, P.C. Abbott, P.V. Preckel, Evaluating agricultural price policy under dual market regimes and institutional constraints, J. Dev. Econ. 34 (1990)
179–197, https://doi.org/10.1016/0304-3878(90)90081-L.
[33] J. Michalek, An application of the policy analysis matrix for an evaluation of agricultural policies in the Slovak republic, Oxf. Agrar. Stud. 23 (1995) 177–196,
https://doi.org/10.1080/13600819508424097.
[34] A. Adesina, Farm size, relative efficiency and agrarian policy in Cote d’Ivoire: profit function analysis of rice farms, Agric. Econ. 14 (1996) 93–102, https://doi.
org/10.1016/0169-5150(96)01181-4.
[35] E. Rabinowicz, Is eastern Europe copying western agricultural policy? Int. Polit. 33 (1996) 181–204.
[36] I. Fertő, Restructuring of the Hungarian agri-food sector, Acta Oecon. 50 (1999) 151–168.
[37] A. Semos, Estimation of producer’s assistance and evaluation of agricultural policy for olive oil in Greece, Agric. Econ. Rev. 2 (2001) 42–52, https://doi.org/
10.22004/ag.econ.26424.
[38] T. Volk, Effects of agricultural policy on the development of Slovenian agriculture during the transition and the process of accession to the European Union,
J. Agric. Sci. Belgrade 50 (2005) 75–88, https://doi.org/10.2298/JAS0501075V.
[39] A.J. Oskam, G. Meester, How useful is the PSE in determining agricultural support? Food Pol. 31 (2006) 123–141, https://doi.org/10.1016/j.
foodpol.2005.10.004.
[40] O. Therond, H. Belhouchette, S. Janssen, K. Louhichi, F. Ewert, J.-E. Bergez, et al., Methodology to translate policy assessment problems into scenarios: the
example of the SEAMLESS integrated framework, Environ. Sci. Pol. 12 (2009) 619–630, https://doi.org/10.1016/j.envsci.2009.01.013.
[41] D. Viaggi, M. Raggi, SG y Paloma, Farm-household investment behaviour and the CAP decoupling: methodological issues in assessing policy impacts, J. Pol.
Model. 33 (2011) 127–145, https://doi.org/10.1016/j.jpolmod.2010.10.004.
[42] L. Mouysset, L. Doyen, F. Jiguet, Different policy scenarios to promote various targets of biodiversity, Ecol. Indicat. 14 (2012) 209–221, https://doi.org/
10.1016/j.ecolind.2011.08.012.
[43] L. Luca, E. Pelinescu, Methods to evaluate the agricultural policies and their relevance for Romania, Lucr Științifice Univ Științe Agric Și Med Vet Banat
Timisoara Ser Manag Agric 14 (2012) 49–56.
[44] M. Akbar, F. Jamil, Monetary and fiscal policies’ effect on agricultural growth: GMM estimation and simulation analysis, Econ. Modell. 29 (2012) 1909–1920,
https://doi.org/10.1016/j.econmod.2012.06.001.
[45] X. Wang, Y. Shen, The effect of China’s agricultural tax abolition on rural families’ incomes and production, China Econ. Rev. 29 (2014) 185–199, https://doi.
org/10.1016/j.chieco.2014.04.010.
[46] S. Kirchweger, J. Kantelhardt, The dynamic effects of government-supported farm-investment activities on structural change in Austrian agriculture, Land Use
Pol. 48 (2015) 73–93, https://doi.org/10.1016/j.landusepol.2015.05.005.
[47] N. Dawson, A. Martin, T. Sikor, Green revolution in sub-saharan africa: implications of imposed innovation for the wellbeing of rural smallholders, World Dev.
78 (2016) 204–218, https://doi.org/10.1016/j.worlddev.2015.10.008.
[48] R. Huber, S. Rebecca, M. François, B.S. Hanna, S. Dirk, F. Robert, Interaction effects of targeted agri-environmental payments on non-marketed goods and
services under climate change in a mountain region, Land Use Pol. 66 (2017) 49–60, https://doi.org/10.1016/j.landusepol.2017.04.029.
[49] G. Whittaker, R. Färe, S. Grosskopf, B. Barnhart, M. Bostian, G. Mueller-Warrant, et al., Spatial targeting of agri-environmental policy using bilevel evolutionary
optimization, Omega 66 (2017) 15–27, https://doi.org/10.1016/j.omega.2016.01.007.
[50] D. Kremmydas, I.N. Athanasiadis, S. Rozakis, A review of Agent Based Modeling for agricultural policy evaluation, Agric. Syst. 164 (2018) 95–106, https://doi.
org/10.1016/j.agsy.2018.03.010.
[51] S.C. Babu, G. Mavrotas, N. Prasai, Integrating environmental considerations in the agricultural policy process: evidence from Nigeria, Environ. Dev. 25 (2018)
111–125, https://doi.org/10.1016/j.envdev.2018.01.001.
[52] R. Huber, M. Bakker, A. Balmann, T. Berger, M. Bithell, C. Brown, et al., Representation of decision-making in European agricultural agent-based models, Agric.
Syst. 167 (2018) 143–160, https://doi.org/10.1016/j.agsy.2018.09.007.
[53] E. Kiryluk-Dryjska, A. Baer-Nawrocka, Reforms of the common agricultural policy of the EU: expected results and their social acceptance, J. Pol. Model. 41
(2019) 607–622, https://doi.org/10.1016/j.jpolmod.2019.01.003.
[54] K.X. Soulis, E. Psomiadis, P. Londra, D. Skuras, A new model-based approach for the evaluation of the net contribution of the European union rural development
program to the reduction of water abstractions in agriculture, Sustainability 12 (2020) 7137, https://doi.org/10.3390/su12177137.
[55] H.M.M. Rashad, M.S. El-Adamy, M.E. Ragab, A.M.A. El-Maksoud, Agricultural policy evaluation of the most important cereal crops, using the policy analysis
matrix, Arab. Univ. J. Agric. Sci. 28 (2020) 445–457.
[56] K. Boysen-Urban, M. Brockmeier, H.G. Jensen, O. Boysen, Measuring the trade restrictiveness of domestic support using the EU common agricultural policy as
an example, J. Agric. Econ. 71 (2020) 27–49, https://doi.org/10.1111/1477-9552.12337.
[57] D.Y. Mora Herrera, A.H. Barrientos, O. Zuñiga Escobar, A review of agent-based modeling for simulation of agricultural systems, DYNA 88 (2021) 103–110,
https://doi.org/10.15446/dyna.v88n217.89133.
[58] M. Lefebvre, J. Barreiro-Hurlé, C. Blanchflower, L. Colen, L. Kuhfuss, J. Rommel, et al., Can economic experiments contribute to a more effective CAP?
EuroChoices 20 (2021) 42–49, https://doi.org/10.1111/1746-692X.12324.
[59] C. Stetter, P. Mennig, J. Sauer, Using machine learning to identify heterogeneous impacts of agri-environment schemes in the EU: a case study, Eur. Rev. Agric.
Econ. 49 (2022) 723–759, https://doi.org/10.1093/erae/jbab057.
[60] N.J. van Eck, L. Waltman, Software survey: VOSviewer, a computer program for bibliometric mapping, Scientometrics 84 (2010) 523–538, https://doi.org/
10.1007/s11192-009-0146-3.

17

View publication stats