Science of the Total Environment 682 (2019) 106–117

Contents lists available at ScienceDirect

Science of the Total Environment

journal homepage: www.elsevier.com/locate/scitotenv

Review

An efficient framework for conducting systematic literature reviews in

agricultural sciences
Thomas M. Koutsos ⁎, Georgios C. Menexes, Christos A. Dordas
School of Agriculture, Faculty of Agriculture, Forestry, and Natural Environment, Hellas, Aristotle University of Thessaloniki, 54124 Thessaloniki, Greece

H I G H L I G H T S G R A P H I C A L A B S T R A C T

• Steps on how to conduct an effective

systematic review
• Review and evaluate eligible articles
based on their strength of evidence
• Dissemination of systematic reviews
findings

a r t i c l e i n f o a b s t r a c t

Article history: Systematic review has generally been accepted as an effective, more complete, repeatable, and less biased type
Received 28 February 2019 literature review that can successfully lead to evidence-based conclusions. This study attempts to develop a
Received in revised form 19 April 2019 framework for systematic review with guidelines on how to conduct an effective systematic review for agricul-
Accepted 24 April 2019
tural research. Systematic reviews require more time and effort but they can be used to conduct a comprehensive
Available online 25 April 2019
literature review, identifying potentially eligible articles on primary agricultural research and answering certain
Editor: Damia Barcelo focused questions. A systematic review is also conducted as an example to examine whether systematic reviews
are used in agricultural sciences. It was found that in the last two decades about a third (N = 29 out of 89 or
Keywords: 32.5%) of the eligible studies, classified as reviews related to agricultural research, are available as free full-text
Agricultural research from publisher, while only eighteen of them were finally eligible to be included in this systematic review.
Systematic reviews © 2019 Elsevier B.V. All rights reserved.
Meta-analysis

Contents

1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
2. Comparison between narrative reviews, systematic reviews and meta-analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
3. Points of attention when conducting a systematic review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
4. Developing a systematic review framework for agricultural sciences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
4.1. Step 1: scoping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

⁎ Corresponding author.
E-mail addresses: tkoutsos@agro.auth.gr (T.M. Koutsos), gmenexes@agro.auth.gr (G.C. Menexes), chdordas@agro.auth.gr (C.A. Dordas).

https://doi.org/10.1016/j.scitotenv.2019.04.354
0048-9697/© 2019 Elsevier B.V. All rights reserved.
 T.M. Koutsos et al. / Science of the Total Environment 682 (2019) 106–117 107

4.2. Step 2: planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

4.3. Step 3: identification/search. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
4.4. Step 4: screening. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
4.5. Step 5: eligibility/assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
4.6. Step 6: presentation/interpretation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
5. Case study: “The systematic review used as literature review type in the agricultural research” . . . . . . . . . . . . . . . . . . . . . . . . . 110
5.1. Step 1: scoping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
5.2. Step 2: planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
5.2.1. Search strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
5.2.2. Eligibility criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
5.2.3. Strength of evidence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
5.3. Step 3: identification/search. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
5.4. Step 4: screening. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
5.5. Step 5: eligibility/assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
5.6. Step 6: presentation/interpretation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
6. Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
7. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
Appendix A. Supplementary material . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116

1. Introduction statistical evidence due to a small number of selected reports to be eval-

The need for a more objective and organized review of the existing
literature on a scientific topic has led to the gradual adoption and ac-
ceptance of the method of “Systematic Review” as an effective litera-
ture review method that can successfully lead to less biased
evidence-based conclusions (Egger et al., 1997; Olson et al., 2002;
Schünemann et al., 2017; The Campbell Collaboration, 2017; Munn
et al., 2018). The most commonly used and effective to an extent
method of reviewing the bibliography has been, for many years, the
narrative review (Levy and Ellis, 2006; Green et al., 2006), which, how-
ever, due to the potential subjectivity in the way that the references are
chosen, has been strongly criticized (Mann, 1994; Greenland, 1995;
Stroup and Thacker, 1995). Narrative reviews are based on the experi-
ence and subjectivity of the author, who is often an expert in this scien-
tific area. However, subjectivity and mainly the absence of a clear and
objective review method can bias the author's conclusions and lead
to several methodological flaws and misleading assumptions
(Mulrow, 1987; Petticrew, 2001; Cipriani and Geddes, 2003; Green
et al., 2006; Kowalczyk and Truluck, 2013).
On the contrary, systematic reviews and meta-analysis provide a rig-
orous, exhaustive and replicable method of reducing bias in selecting,
identifying and evaluating eligible articles (Cook et al., 1997; Cipriani
and Geddes, 2003; Sargeant et al., 2006; Higgins and Green, 2008;
Deeks et al., 2008; Cooper and Hedges, 2009; Schünemann et al.,
2017; Munn et al., 2018; Riley et al., 2019) from several scientific digital
sources compared with the subjective way of selecting relevant articles
in the traditional narrative type of review. Based on the defined review
protocol (Moher et al., 2010) the eligible studies are identified via struc-
tured query search and then are critically appraised and assessed to pro-
vide the most complete summary of current evidence relevant to the
focused scientific question that was set during the development of the
review protocol. As a result, through the process of a systematic review,
the investigation of the available literature is attempted in a compre-
hensible manner that can be repeated by an individual to check or ex-
tend the previous analysis (Mallett et al., 2012; Paré et al., 2015;
Schünemann et al., 2017). In this sense, previous review protocols can
be re-used and extended to include more recent advances to answer
new specific research questions. Organized search for scientific studies
using a quantitative comparison and statistical analysis can take the re-
view to a next level so-called “meta-analysis” (Antman et al., 1992;
Higgins et al., 2003; Higgins and Green, 2008; Schünemann et al.,
2017; Riley et al., 2019). The meta-analysis was first used by Karl Pear-
son in his attempt to address the problem of studies with reduced
uated (Glass, 1976).
Conducting a systematic review is necessary before starting a new
research, to study previous works and then briefly present the advances
made so far globally in this specific scientific research topic. In addition,
it is a very good test of whether something similar has been done in the
past and whether the potential results of a new research would provide
novel insights (Stroup and Thacker, 1995; Harwell, 1997; Kleijinen et al.,
1990). It should be noted that even if the systematic review process
identifies a very small number of references or even none from the se-
lected scientific sources, this information may be equally important
(Moher et al., 2010). While the systematic review is considered the pro-
cess of defining the best methodologically designed studies for a scien-
tific hypothesis based on selected criteria, the meta-analysis is the
statistical process of calculating an outcome based on the studies con-
sidered in the framework of the systematic review (Cipriani and
Geddes, 2003). Therefore, conduction a systematic review to answer a
scientific focused question in conjunction with the meta-analysis can
lead to more secure conclusions (Schünemann et al., 2017).
In agricultural research, several systematic reviews and meta-
analysis have been published in the last five years. In most cases, the
systematic review type was selected at first to obtain an interdisciplin-
ary and comprehensive overview of a topic and then answer a focused
question or propose an action-oriented conceptual framework for the
future agricultural research (Sargeant et al., 2006). Many agricultural
scientific sub-areas have already been examined using systematic re-
views, such as agricultural sustainable intensification and sustainable
development (Weltin et al., 2018; De Luca et al., 2017), adapting agri-
cultural systems to climate change, agricultural practices (Cooper
et al., 2016; Shaffril et al., 2019), pesticides (Schmolke et al., 2017; Liu
et al., 2019), heavy metal loss, agricultural emissions to the environ-
ment (Buckingham et al., 2014) or other environmental impacts de-
rived from agricultural activities (Sharma and Chaubey, 2017; Ouyang
et al., 2018) and lately medical-related topics (Auditeau et al., 2019;
Byeon et al., 2019; Seetaloo et al., 2019; Toth et al., 2019). The number
of systematic reviews and meta-analysis published during the last
years with topics related to agricultural or agro-environmental re-
search, shows that there is an increasing trend in using this type of liter-
ature review as an effective way of summarizing primary agricultural
research.
The main concern of this study is to provide a useful framework for
conducting systematic reviews in agricultural research. In addition, cur-
rent work presents an example of how to conduct a systematic review
based on the extent that the systematic review, as a literature review
108 T.M. Koutsos et al. / Science of the Total Environment 682 (2019) 106–117
type, has been adopted by agro-scientists. In the following sections, a
comparison between a traditional/narrative review and a systematic re-
view or meta-analysis is given and several points of attention when
performing a systematic review are highlighted. Next, a systematic re-
view framework for agricultural sciences is proposed and each step is
being described in detail.
Finally, based on the given framework of guidelines, a new system-
atic review is conducted as a case study entitled: “The adoption of the
systematic review used as a literature review type in the agricultural re-
search”. The topic of this systematic review was selected on purpose to
be broader for the following reasons: (1) to be attractive to a wider au-
dience; (2) to prove the efficiency of the given methodology regardless
the scientific topic being examined, and (3) to promote the adoption of
systematic review in agricultural research, because it is not used to the
extent that it should, according to the findings of the case study.
2. Comparison between narrative reviews, systematic reviews and
meta-analysis
There are at least 14 different review types and associated method-
ologies available to examine available literature (Grant and Booth,
2009): (1) critical review; (2) literature review (narrative); (3) map-
ping review/systematic map; (4) meta-analysis; (5) mixed studies re-
view/mixed methods review; (6) overview; (7) qualitative systematic
review/qualitative evidence synthesis; (8) rapid review; (9) scoping
review; (10) state-of-the-art; (11) systematic review; (12) systematic
search and review; (13) systematized review and (14) umbrella re-
view. The narrative review has been so far the most common and pop-
ular way of conducting a bibliographic review (Cipriani and Geddes,
2003). However, this type of review has the following disadvantages
(Fagard et al., 1996; Schünemann et al., 2017): (a) the selection of
the included studies and the choice of reference sequence used in the
text are biased and usually aim to support the approach of the new re-
search rather than be objective; (b) it is not clear or documented
whether the selection of the studies included is adequate to lead to
safe conclusions or whether the total number of the citations used is
sufficient for the review and (c) research based on the narrative type
of review using information from random scientific sources can lead
to methodological flaws and misleading conclusions (Cipriani and
Geddes, 2003).
On the other hand, Systematic Reviews provide a more rigorous and
transparent methodology to review existing literature (Van der Knaap
et al., 2008; The Campbell Collaboration, 2017). Contrary to narrative,
the systematic literature review type is a reliable and organized ap-
proach to investigate studies from several scientific sources and every
step during this process should be noted in detail (Schünemann et al.,
2017). However, Systematic Reviews may face some practical chal-
lenges such the need to have access to an extensive range of digital da-
tabases, which may be either expensive or difficult for non-academic
researchers, or the inevitable subjectivity due to the use of inclusion
and exclusion criteria used to screen eligible articles (Mallett et al.,
2012). In addition, due to the few eligible articles that are considered
and finally included at Systematic Reviews or due to their heterogeneity
the findings may be too broad or cannot provide clear recommenda-
tions (Duvendack et al., 2012; Hagen-Zanker et al., 2011).
Systematic reviews, compared with the traditional methods of
reviewing, can reduce the risk of getting inaccurate conclusions due to
subjective or incomplete selection of studies from random scientific
sources (Schünemann et al., 2017). However, conducting a systematic
review does not eliminate completely subjectivity problems, but at
least every step has been well documented and available to be checked
by anyone. The search strategy, inclusion criteria and any other limita-
tions, or any issue faced should be described in detail (Bailar, 1997;
Neinert, 1997). A basic comparison between narrative reviews, system-
atic reviews and meta-analysis is given at Table 1.
3. Points of attention when conducting a systematic review
For a successful systematic review, the most important point of at-
tention is to record all the procedures followed based on the established
review protocol (Moher et al., 2010; Schünemann et al., 2017; The
Campbell Collaboration, 2017). However, according to the authors'
opinion there are several individual attention points that should be
also considered when conducting a systematic review, such as:
• Systematic reviews should be conducted on a group basis involving at
least 2–3 researchers, specializing in the subject and having basic
knowledge of statistical analysis of data. This will reduce the subjec-
tive errors in the way the results of the individual studies are evalu-
ated and synthesized.
• Experts should always be consulted on individual scientific issues. Any
disagreement in the evaluation of the studies involved in the system-
atic review should be solved in consultation with a third party.
• Systematic reviews should not be over-optimistic in looking for the
required analytical studies. Usually, an initial quick search can give
an overview of the goal. In addition, the research focused question
should obviously be understandable and relatively simple.
• Every step followed during the systematic review takes time, but it is
necessary to be noted in detail. The search process should be clear so
that it can easily be repeated by another researcher. Defined search
terms (as combination of keywords and Boolean operators) and all
the inclusion and exclusion criteria set must be noted in detail. Keep-
ing record of all steps followed is very important as it may be neces-
sary to modify the processes at several steps.
• Consideration should be given to the topic of the systematic review
that is being planned to be carried out: What is the purpose of this re-
search? How well does it respond to the research case? Is there an ob-
jective procedure for extracting a summarized result? Are the criteria
used for including and excluding studies clear? Can the results of the
analysis lead to further research? It is likely that both the overall con-
clusions and the systematic review process will be criticized, so there
should be justification and any action taken in the analysis.
The summary of the results of the individual studies should be done
with caution. In some cases, the degree of heterogeneity of the individ-
ual studies or the nature of the data can lead the systematic review to
misleading conclusions (Higgins et al., 2003; Hardy and Thompson,
1998).
4. Developing a systematic review framework for agricultural
sciences
A systematic review aims to provide an exhaustive summary of cur-
rent literature to answer to a focused research question. One of its major
advantages is that the whole procedure is noted in detail and it can be
repeated for check or can be extended or changed depending on current
research needs. Several frameworks for conducting an effective system-
atic review have been proposed (Schünemann et al., 2017; The
Campbell Collaboration, 2017; Yannascoli et al., 2014; Wright et al.,
2007; Kitchenham, 2004; Khan et al., 2003; Zaza et al., 2000; Oxman
and Guyatt, 1993). Some publishers, such as Elsevier, provide also useful
instructions on how to perform systematic reviews or meta-analysis
(Kitchenham, 2004). The best known and probably the most used
framework is PRISMA statement (Moher et al., 2010), an evidence-
based aiming to help authors improve the reporting of systematic re-
views and meta-analyses (Moher et al., 2010).
Extending the basic steps provided by Prisma Flow Chart (Moher
et al., 2010), a new flexible and easy to follow framework is presented
for conducting a systematic review following these steps: (1) Scoping;
(2) Planning; (3) Identification; (4) Screening; (5) Eligibility/
 T.M. Koutsos et al. / Science of the Total Environment 682 (2019) 106–117 109

Table 1
Differences between narrative, systematic reviews and meta-analysis review.
(Adapted and modified from Cook et al., 1997; Grant and Booth, 2009; Higgins and Green, 2008, Schünemann et al., 2017).

Features Narrative review Systematic review Meta-analysis review

Short description Narrative review is a selective review
type usually prone to bias that broadly
covers a specific topic following no strict
search methods to locate and evaluate
relevant articles
Topic or Research Question • Often broad in scope
Data Sources and Search Strategy • Not usually specified,
• potentially biased and not provided
Selection criteria • Not usually specified, potentially
biased
Appraisal of included articles • Variable
Synthesis • Quantitate and qualitative or narrative
synthesis of the studies included
Inferences • Sometimes evidence-based
Classification • Classification may or not exist
Key references Webster and Watson, 2002; Green et al.,
2006; Levy and Ellis, 2006; Cronin et al.,
2008; Paré et al., 2015
Systematic Review is a type of review
based on search strategies to ensure that
the maximum extend of relevant articles
has been considered and synthesized
• Focused Research Question
• Explicit and comprehensive search
strategy with detailed list of data
sources
• Uniformly and consistently applied
inclusion and exclusion criteria
• Meticulous critical appraisal
• Quantitative summary is provided
when the data can be pooled
• Usually evidence-based
• Results are classified
• based on the strength of evidence
Higgins and Green, 2008; Liberati et al.,
2009; Kowalczyk and Truluck, 2013; Paré
et al., 2015; Schünemann et al., 2017,
Munn et al., 2018
Meta-analysis (usually after a Systematic
Review has been conducted) is the statistical
combination of the results from two or more
separate quantitative studies
• Focused Research Question
• Previously conducted Systematic Reviews
or quantitative studies
• Uniformly and consistently applied inclu-
sion and exclusion criteria
• Rigorous critical appraisal
• Quantitative summary that includes a
statistical synthesis
• Always evidence-based
• Results are presented
• based on the statistical analysis conducted
Higgins et al., 2003; Moher et al., 1999;
Higgins and Green, 2008; Borenstein et al.,
2009; Greenhalgh et al., 2011; Riley et al.,
2019

Fig. 1. Steps for performing a systematic review for agricultural sciences.

(Adopted and modified from Center for Health Communication and Participation, 2017.)
110 T.M. Koutsos et al. / Science of the Total Environment 682 (2019) 106–117
Assessment, and (6) Presentation (synopsis of findings, discussion and
presentation of the results) (Fig. 1).
4.1. Step 1: scoping
Scoping is very critical because it guides the review and it involves
three main sub-steps: (1) development of a review protocol (focused
research questions and study design); (2) identification of a few rele-
vant studies for a pilot review study; (3) search for previous systematic
reviews on current issue.
4.2. Step 2: planning
Planning includes the selection of the main keywords (and Boolean
operators) that will be used at the search string and how the search
will be performed, and it consists of the following sub-steps: (1) devel-
opment of the search strategy and building search queries using Bool-
ean operators; (2) identification of appropriate digital databases or
sources of eligible studies. The use of terms (keywords) along with the
right selection of Boolean operators (Fig. 2) is of vital importance and
it will determine the results of the search, as it might limit the number
of eligible articles based on the search strategy. This obviously can
lead to misleading results and unsubstantiated conclusions.
4.3. Step 3: identification/search
During this step the main search is performed (query execution)
based on the search strategy defined at the previous step. This step in-
cludes the following: (1) implementation of the pre-defined search
strategy; (2) examination of the resulted articles; (3) making changes
to the search strategy if needed; (4) performing additional searches;
(5) searching for additional sources for identifying articles, and (6) man-
ually selection of additional studies.
4.4. Step 4: screening
Screening involves the management of the resulted articles and con-
sists of the following sub-steps: (1) export of citations as the resulted
studies of the search queries; (2) import the exported citations into a ci-
tation manager (i.e. EndNote, Mendeley, Zotero); (3) remove of dupli-
cates; (4) update article information, and (5) thoroughly examination
of the selected articles. Another alternative to remove duplicates can
be the use of the advanced query editor tool available at several digital
scientific databases (i.e. Web of Science).
4.5. Step 5: eligibility/assessment
Several grading systems have been used for making judgements
about the strength of evidence of the eligible articles included in
Fig. 2. Terms and Boolean operators: (a) two terms should be present in the resulting records; (b) any of the two terms can be present; (3) second term will be excluded based on first
term; (4) any of the first or second term and any of third or fourth.
systematic reviews (Ferguson et al., 1998; Glasziou et al., 2001;
GRADE Working Group, 2004; Schünemann et al., 2017). The majority
of these systems suggest that the strength of evidence of the studies in-
cluded should be based on a hierarchy with evidence from systematic
reviews, meta-analysis and experiments at the top of the hierarchy
and evidence from observational studies or expert opinions at the bot-
tom of the hierarchy (Klein and Myers, 1999). According to GRADE
(GRADE Working Group, 2004) the strength of evidence of the studies
can have the following scheme (Dybå and Torgeir Dingsøyr, 2008):
(1) High: further research is very unlikely to change our confidence in
the estimate of effect; (2) Moderate: further research is likely to have
an important impact on our confidence in the estimate of effect and
may change the estimate; (3) Low: further research is very likely to
have an important impact on our confidence in the estimate of effect
and is likely to change the estimate and (4) Very Low: any estimate of
effect is very uncertain.
The following grading system is proposed for the assessment of eli-
gible studies related to agricultural research to address possible evi-
dence issues: (1) setting inclusion and exclusion criteria; (2) defining
the strategy for the strength of evidence; (3) assessing articles based
on their strength of evidence; (4) assessing the types of bias that may
exist (selection, performance, measurement or exclusion bias), and
(5) reading in depth the selected full-text of the articles. The proposed
grading system for the assessment of the agricultural research studies
is given at Table 2 (adapted and modified from Ferguson et al., 1998
and Glasziou et al., 2001).
4.6. Step 6: presentation/interpretation
This final step includes: (1) synopsis of the systematic review find-
ings; (2) study of the heterogeneity of the studies included (Hardy
and Thompson, 1998); (3) presentation of the results; (4) interpretation
of the findings; (5) discussion on the generalization of the conclusions;
(6) limitations of the systematic review, and (7) recommendations for
further research. A simple flow diagram is usually used to present the
different phases of a systematic review and the best known and widely
used is the PRISMA flowchart. The flow diagram of a systematic review
process as proposed by PRISMA (Moher et al., 2010) depicts the flow of
information through the different phases of a systematic review and it
maps out the number of articles identified, included or excluded, and
the reasons for these exclusions.
5. Case study: “The systematic review used as literature review type
in the agricultural research”
The case study conducted here is as an example of implementing the
guidelines provided by the proposed framework to demonstrate the
ease and efficacy of this methodology. In addition, this case study re-
sponds to a very interesting and critically-focused question as to
 T.M. Koutsos et al. / Science of the Total Environment 682 (2019) 106–117
Table 2
Grading the level of evidence of agricultural research studies.
Strength of evidence Type of agricultural research
Strong (I) Applied research, Adaptive/farm
level research, Strategic research,
Systematic reviews and
meta-analysis
Moderate (II) Case studies/Reviews
Modeling/Simulation
Some or Low (III) Opinion papers, conference
papers, workshops
Very low or Inconclusive (IV) Not applicable
whether systematic reviews and up to what extend have been adopted
to identify and evaluate articles in the agricultural sciences.
5.1. Step 1: scoping
(1) Development of a review protocol:
Focused question
“To what extend has the systematic review process been adopted
in reviewing studies with topics related to agricultural sciences?”
Study design
For conducting this systematic review, a comprehensive review
protocol was developed based on the following steps: (1) Scop-
ing; (2) Planning; (3) Identification; (4) Screening; (5) Eligibil-
ity/Assessment, and (6) Presentation (synopsis of findings,
discussion and presentation of the results). This review protocol
has been thoroughly discussed and approved by all authors.
(2) Identification of a few relevant studies for a pilot review
study:
Authors, after examining the digital sources available for
searching the eligible studies, they have reached a consensus
that the only source for searching will be the Web of Science
know as being the most complete digital source of scientific arti-
cles.
(3) Search for previous systematic reviews with the same or sim-
ilar topic:
No previous systematic reviews based on the focused questions
were found.
5.2. Step 2: planning
(1) Identification of appropriate digital databases or sources of
eligible studies:
The digital database chosen as the main source of the eligible
studies was ISI-Web of Science online database. An initial search
query was formed and used to investigate the total number of
studies that were classified as reviews (document type = “Re-
view”) and were also assigned to the Web Science categories
that are related to agricultural sciences. ISI-Web of Science Cate-
gories used: (1) Plant Sciences; (2) Agronomy; (3) Agriculture
Multidisciplinary; (4) Agricultural engineering; (5) Agricultural
Dairy Animal Science, and (6) Agricultural economics policy.
(2) Development of the search strategy and building the search
queries using Boolean operators
5.2.1. Search strategy
Finally, two main queries were developed using the advanced web-
based search tool of Web of Science to identify articles that reported the
use of a systematic review or meta-analysis. We extended our search to
locate studies with meta-analysis because in some cases the systematic
review although considered as a preliminary step it is not always men-
tioned at the abstract or title of the article. Additional search based on
111
Study design Study type Evidence
Experiments, Field trials, Systematic Experimental/ Substantiated: documented evidence
reviews Structured review
Case studies, Narrative reviews, Observational Partial substantiated: evidence under
Simulations conditions
Qualitative research, opinion papers, Descriptive Unsubstantiated: potential evidence
reports of expert committees. based on qualitative analysis or opinion
Not applicable Not applicable Not applicable
the given eligibility criteria was performed using Google Scholar to
identify other relevant studies. The eligibility of the selected studies
was confirmed by two independent reviewers as evaluators that were
kindly asked to offer their valuable opinion and any dispute was re-
solved during a consensus (Van der Windt et al., 2000).
5.2.2. Eligibility criteria
For the selection of articles, the following inclusion criteria were set:
(1) Studies that use systematic review or meta-analysis; (2) all studies
published in peer-reviewed English language journals; (3) document
type of studies set as ‘Review’. To exclude articles not related to agricul-
tural research (exclusion criteria) the query results were refined by
selecting the following Web of Science categories: (1) Agronomy;
(2) Plant Sciences; (3) Agricultural multidisciplinary; (4) Agricultural
engineering, and (5) Agricultural economics policy. The categories
used for the ‘meta-analysis’ were: (1) Agronomy; (2) Agricultural mul-
tidisciplinary, and (3) Agricultural economics policy.
As a result, the total number of articles related to “agriculture” with
document type “Review” at the initial screening was 5915 and after re-
fining using the specified agricultural related Web of Science categories
were 1726. Replacing the term “review” with the more specific terms
“systematic” and “meta-analysis” at the search queries, the total number
of studies referred to systematic reviews or meta-analysis was 280 and
198 respectively and after refining 46 and 43. Available as full-text arti-
cles (Open Access) were finally only 19 and 8, respectively, while two
more articles were added manually. The whole process of selecting the
eligible articles based on PRISMA is given at Fig. 3.
5.2.3. Strength of evidence
The main concern during the process of identifying and assessing the
eligible articles included in this systematic review was to find studies
with proven use of a typical review protocol of a systematic review or
documented reference to it.
For the assessment of the studies the articles included all articles
have been thoroughly examined and they were assigned a correspond-
ing level of strength of evidence based on the following grading evi-
dence system: (I) Substantiated: systematic reviews or meta-analysis
such as articles with structure of a typical systematic review or meta-
analysis following a review protocol; (II) Partially substantiated: narra-
tive reviews, reviews with no detailed review protocol used, studies re-
lated to or having a topic relevant to systematic reviews such as articles
that use the narrative literature review type or present a summary or
descriptive statistics of manually selected studies, and (III) unsubstanti-
ated: reviews with qualitative analysis or opinion papers or articles that
although have been reported as systematic reviews or meta-analysis
they do not follow the standards.
5.3. Step 3: identification/search
(1) Implementation of the pre-defined search strategy:
The queries set at the search strategy were executed and the
search history was saved using the ‘Save History’ feature of the
ISI-Web of Science website (*.wos file format).
112 T.M. Koutsos et al. / Science of the Total Environment 682 (2019) 106–117
Fig. 3. Systematic review flowchart based on PRISMA flowchart (Moher et al., 2010).
(2) Examination of the resulted articles:
All articles resulted from the search queries were thoroughly ex-
amined concerning the title, their abstract, the year of publica-
tion and the language used to confirm that the search results
were correct.
(3) Make changes to the search strategy if needed:
Initially, one of the terms used at the search queries was ‘agron-
omy’, which was considered to be more relevant to the agricul-
tural research. However, the search results were limited and
misleading and so the ‘agronomy’ term was finally replaced by
the term ‘agricultural’, which was more effective.
(4) Performing additional searches:
No additional searches were performed because the search strat-
egy, that was set, defined that the search should focus only on ISI-
Web of Science database with articles from peer-reviewed
Journals.
(5) Search for additional sources for identifying articles:
No additional sources were examined and the search was per-
formed only at the ISI-Web of Science database.
(6) Manually selection of additional studies.
No other review articles have been added manually.
5.5. Step 5: eligibility/assessment
5.4. Step 4: screening
(1) Export of citations as the resulted studies of the search queries:
The selected studies (16 systematic reviews and 8 meta-analysis
studies) were exported from IS-Web of Science using the Bibtex
format (*.bib file format) into two separate files.
(2) Import the exported citations into a citation manager:
A new Bibtex library was created using the JabRef tool and all cita-
tions from the two separate *.bib files were imported into this li-
brary.
(3) Remove of duplicates:
Using the feature ‘Quality/Find duplicates’ from the JabRef tool,
two studies were removed as duplicates. The resulted 22 studies
merged was saved as a new (*.bib) library file.
(4) Update article information:
This file with merged citations was imported into Zotero Citation
Manager. Update of article information was triggered to fill possi-
ble information gaps. All study characteristics were then exported
(*.usv file format) and then imported into an Excel file. Using the
Zotero feature ‘Create bibliography from the selected items’ avail-
able on the right mouse click on all selected entries, 22 citations
were finally copied to clipboard and then pasted into the manu-
script.
(5) Thorough examination of the selected articles:
Full-texts of the selected studies were downloaded and were ex-
amined thoroughly to confirm if they are indeed relevant to the
subject of literature review.
(1) Setting inclusion and exclusion criteria:
The inclusion criteria for the studies were the following: (1) stud-
ies should be systematic reviews or meta-analysis and (2) all
studies should be reviews related to agricultural research. The
exclusion criteria: (1) all articles should have been classified as
 T.M. Koutsos et al. / Science of the Total Environment 682 (2019) 106–117 113

reviews (ISI-Web of Science document type = Review) and
(2) all studies should have been published in peer-reviewed En-
glish language journals.
(2) Defining the strategy for the strength of evidence:
Adopting the proposed methodology, a four-level assessment
was developed for defining the strength of evidence of studies
included in the systematic review of this case study as follows:
(I) Strong; (II) Moderate; (III) Some or Low; (IV) Very low or In-
conclusive. The main concern was to examine the level of evi-
dence of the identified studies as follows: (I) Strong:
substantiated studies with topic related to systematic reviews
or meta-analysis, (II) Moderate: traditional or narrative reviews,
(III) Low: simple reviews with unsubstantiated conclusions and
(IV) Very low: reviews with inconclusive results.
(3) Assessing articles based on their strength of evidence:
Thirty studies (19 systematic reviews and 11 meta-analyses)
were examined thoroughly for their eligibility (topic, abstract
and content) and confirmed by two independent reviewers.
Three studies were removed as duplicates while other two
added manually. Several study characteristics were also exported
(year of publication, authors, journals, research areas, ISI-Web of
Science categories). Finally, twenty-nine studies in total were ex-
amined and assessed for their strength of evidence (Table 3).
As a result, out of the 29 selected studies that were examined,
eleven studies (N = 11 or 37.93%) were traditional reviews or
narrative reviews and they were classified as partially
substantiated with strength of evidence II and eighteen studies
(N = 18 or 62.07%) were systematic reviews or meta-analysis
studies and they were classified as substantiated with strength
of evidence I. Most of the studies identified as reviews with
topic related to the agricultural research have been published
within the last five years. Studies classified as systematic reviews
or meta-analysis (strength of evidence I) are slightly more in
number in comparison with the traditional or narrative studies
(strength of evidence I) (Fig. 5).
(4) Assessing the types of bias that may exist:
In this systematic review, only studies written in English were
identified based on the exclusion criteria, so any other non-
English written review article was excluded. Also, only reviews
related to agricultural research were examined based on the
search strategy and any other was excluded as well. It should be
noted also that some reviews remain unpublished because they
probably do not present the expected findings or conclusions,
while some other studies are under review and others may
never reach the publication stage for several reasons. Many au-
thors are reluctant to send their reviews for publication if the find-
ings are contrary to the already known, because they are afraid of
being rejected (Hobson, 2014). All the above cases may introduce
a bias in this systematic review that may alter the findings. Finally,
changes at the search strategy such as modification in terms lead
obviously to different search results having also different strength
of evidence.

Table 3
Assessment of the selected studies based on their strength of evidence.

# Citation Research Areasa Narrative Systematic Meta-analysis Evidence Strength

Review Review of
evidence

(1) Aravind et al., 2017 Agriculture * ++b II

(2) Banias et al., 2016 Agriculture; Chemistry; Food Science & Technology * +++ I
(3) Bertrand et al., 2015 Agriculture; Science & Technology - Other Topics * ++ II
(4) Chedraoui, 2017 Plant Sciences * ++ II
(5) Chen et al., 2018 Agriculture * +++ I
(6) Condon et al., 2013 Agriculture; Business & Economics; Food Science & Technology; Nutrition & * * +++ I
Dietetics; Public Administration
(7) Cooper et al., 2016 Agriculture; Science & Technology - Other Topics * * +++ I
(8) Dignac et al., 2017 Agronomy; Green & Sustainable Science & Technology * ++ II
(9) Felix et al., 2018 Agriculture; Science & Technology - Other Topics * * +++ I
(10) Garrett et al., 2017 Agriculture * * ++ II
(11) Gedamu, 2006 Agriculture * ++ II
(12) Holzkämper, 2017 Agriculture * +++ I
(13) Johnston et al., 2018 Agricultural Economics & Policy; Economics; Food Science & Technology; * +++ I
Nutrition & Dietetics; Planning & Development
(14) Lekberg and Koide, Plant Sciences * * +++ I
2005
(15) Massinon and Agriculture; Biotechnology & Applied Microbiology; Environmental Sciences & * ++ II
Lebeau, 2013 Ecology
(16) Medel et al., 2017 Plant Sciences * +++ I
(17) Mehmood et al., Agriculture; Science & Technology - Other Topics * ++ II
2017
(18) Menezes-Blackburn Agronomy; Plant Sciences; Soil Science * +++ I
et al., 2018
(19) Nayak et al., 2018 Agriculture; Plant Sciences * +++ I
(20) Pandey et al., 2016 Agriculture; Business & Economics; Food Science & Technology; Nutrition & * ++ II
Dietetics; Public Administration
(21) Pandey, 2017 Plant Sciences * ++ II
(22) Pathak et al., 2018 Agronomy; Plant Sciences * ++ II
(23) Robledo-Abad, 2017 Agriculture; Biotechnology & Applied Microbiology; Energy & Fuels * +++ I
(24) Schmidt et al., 2017 Agriculture; Forestry; Meteorology & Atmospheric Sciences * * +++ I
(25) Schut et al., 2014 Agriculture * +++ I
(26) Sharma and Agriculture * * +++ I
Chaubey, 2017
(27) Sibhatu et al., 2018 Agricultural Economics & Policy; Economics; Food Science & Technology; * +++ I
Nutrition & Dietetics; Planning & Development
(28) Uyttenbroeck et al., Agriculture; Biotechnology & Applied Microbiology; Environmental Sciences & * +++ I
2016 Ecology
(29) Weltin et al., 2018 Agriculture; Environmental Sciences & Ecology * +++ I
a
ISI-Web of Science research areas of the studies included.
b
Substantiated (+++); partially substantiated (++); unsubstantiated (+).
114 T.M. Koutsos et al. / Science of the Total Environment 682 (2019) 106–117

Fig. 4. Searches: (a) Initial search for type “review”; (b) Search for type “systematic review” and “meta-analysis”, First row: results refined based on Web of Science Categories; Second row:
refined entries based on their free full-text availability.

(5) Reading in depth the selected full-text of the articles:
The included twenty-nine articles in this systematic review were
read in depth and thoroughly examined.
5.6. Step 6: presentation/interpretation
(1) Synopsis of the systematic review findings:
Considering the total number of studies identified (N = 481: 280
+ 1 systematic reviews, 199 + 1 meta-analysis), most of them
are systematic reviews (N = 281 or 58.42%) and the rest are
meta-analysis (N = 200 or 41.58%). After filtering these studies
to only those strictly related to agricultural research the number
of studies for this systematic review was limited to N = 89 arti-
cles (Ν = 46 or 51.68% systematic reviews and N = 43 or
48.31% meta-analysis). Filtering again to those full text available
only N = 29 articles were found: 19 systematic reviews +11
meta-analysis +2 added manually – 3 removed as duplicates
(N = 20 or 68.97% systematic reviews, N = 9 or 31.03% meta-
analysis). This means that out of the total N = 481 articles (iden-
tified as systematic reviews and meta-analysis), related to the ag-
ricultural research, only a small percentage (N = 29 or 6%) of the
articles are available as full-text. Furthermore, out of the 29 stud-
ies in total, eighteen (N = 18 or 62.07%) of them were classified
as having strength of evidence I (systematic reviews and meta-
analysis) and eleven (N = 11 or 37.93%) as level II (narrative re-
views). The main conclusion is that only one-fifth of the system-
atic reviews and meta-analysis studies (N = 18 out of 89 or 20%),
related to agricultural research, are available as full-text manu-
scripts (Fig. 4). Concerning the review protocol used, in most of
the cases, it was formed and tailored to current research needs,
sometimes not so clear to be repeated and followed again. Fur-
thermore, none already known or previously successfully tested
review protocol was used in the eligible studies (except from
one study only with reference to PRISMA review protocol).
(2) Study of the heterogeneity of the studies included:
For advanced refine options it was used the ‘Results Analysis’ tool
of ISI-Web of Science, which provides a visualization of the

11 5 3 3
AGRONOMY PLANT SCIENCES FOOD SCIENCE GREEN SUSTAINABLE
TECHNOLOGY SCIENCE TECHNOLOGY

3
BIOTECHNOLOGY APPLIED
MICROBIOLOGY
2 2
AGRICULTURAL ECONOMICS NUTRITION
POLICY DIETETICS
5
AGRICULTURAL MULTIDISCIPLINARY

3
ENVIRONMENTAL SCIENCES 2
ECONOMICS

Fig. 5. Visualization tree-map of the studies included in the systematic review based on their ISI-Web of science categories.
 T.M. Koutsos et al. / Science of the Total Environment 682 (2019) 106–117 115

8
7

Number of articles
6
5
4
3
2
1
0
2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018
Year of publication

Systematic reviews or meta-analysis (level I) Narrative review (level II)

Fig. 6. Count of systematic reviews or meta-analysis with topics related to agricultural research.

resulted studies after selecting one of their characteristics, such
as the Tree-map of the studies based on their categories
(Fig. 5). As expected, most of the studies had been assigned to
‘Agronomy’ ISI-Web of Science category, while ten different cat-
egories of the studies in total were identified.
(3) Presentation of the results: (a)
Based on the year of publication of studies:
Most of the studies identified as reviews with topic related to ag-
ricultural sciences have been published during the last five years
(Fig. 6). Studies classified as systematic reviews or meta-analysis
(assessed as level I) were slightly more in comparison with the
traditional narrative reviews (assessed as level II), (b)
Based on the times of studies cited:
One the included studies classified as systematic review was
highly cited (146 citations) and almost in all cases studies classi-
fied as systematic reviews or meta-analysis were more cited than
the ones classified as simple reviews (Fig. 7), (c)
Based on the Open Access status of the studies (full-text avail-
ability):
Out of the 481 (379 derived from search and 2 added manually)
in total studies that were identified as systematic reviews or
meta-analysis, based on the search strategy, only 46 were finally
refined as systematic reviews and 43 as meta-analysis related to
agricultural research based on the ISI-Web of Science categories
(Fig. 8).
(4) Interpretation of the findings:
The number of reviews identified as systematic reviews or meta-
analysis was 379 (using the search term “agricultural”), while
after refining the results, the actual number of studies related
to agricultural sciences was reduced to 89 and finally only 18
studies were available as full-text manuscripts (Fig. 4). There
also is an increased trend in adopting the systematic literature re-
view type in the last five years (Figs. 6 and 7) that shows an
increased interest or even trust to the efficiency of this type of re-
view.
(5) Discussion on the generalization of the conclusions.
The search for this systematic review was performed using the
advanced search tool of the ISI-Web of Science website, querying
the Web of Science Core Collection which includes a very large
number of scientific articles peer-reviewed journals. The search
was executed based on the search strategy using the appropriate
search terms to identify eligible articles with no time-span. Au-
thors believe that the findings of this systematic review provide
a clear overview of the extend that the systematic literature re-
view type is adopted worldwide.
(6) Limitations of the systematic review:
In this systematic review, the identified studies were all written
in English. Also, the document type of all studies should be set
as ‘review’.
(7) Recommendations for further research:
Authors believe that further investigation is needed and focus on
the benefits of adopting meta-analysis in the agricultural re-
search. Meta-analysis can provide evidence for decision making
or propose action-oriented frameworks in several agricultural
scientific areas.
6. Discussion
Systematic reviews offer an effective way to examine current litera-
ture, since they are designed to answer one or more focused questions
with reduced bias in selecting articles in comparison with the tradi-
tional narrative type of review.
The case study here is an example that gives the opportunity to pres-
ent a framework with guidelines easy to use, thus it shows the extend of
adoption of this effective review type in agricultural research in the last

140
120
100
Cited

80
60
40
20
0
2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018
Year of publication

Systematic reviews or meta-analysis (level I) Narrative review (level II)

Fig. 7. Times cited of systematic reviews or meta-analysis in comparison with the narrative reviews with topics related to agricultural research.
116 T.M. Koutsos et al. / Science of the Total Environment 682 (2019) 106–117
500
450
400
199
Number of articles
350
300
250
200
150 280
100
43
50
46
0
Identified
Systematic reviews
Fig. 8. Presentation of systematic and meta-analysis studies based on their Open Access status.
years. Unfortunately, it was found that for the last two decades only a
third of the systematic reviews and meta-analysis studies in agricultural
sciences, (N = 18 out of 89 or 20%), classified as reviews with topic re-
lated to systematic reviews or meta-analysis are available as full-text
manuscripts, while only eighteen of them were eligible to be included
in this systematic review based on their strength of evidence.
The limited number of the available systematic reviews or meta-
analysis in agricultural research could be justified as follows: (1) the
methodology of performing a successful systematic review or meta-
analysis is not widely known to agro-scientists; (2) the heterogeneity
of the agricultural studies becomes an obstacle in the process of conduc-
ing a systematic review, on the grounds that they are studies under dif-
ferent conditions; (3) the biased traditional narrative type of review is
faster and more convenient way for performing a review for an individ-
ual scientist to lead to new a conclusion without spending time or effort
and (4) systematic reviews require time and detailed recording of all
the phases and procedures or steps followed, which may lead to longer
text length.
According to the authors' opinion the traditional narrative review
type can still be an effective and successful way of presenting current re-
search given the condition that the scientist conducting the review has
expertise in the topic or in the related scientific field. Otherwise, the bi-
ased selecting process of articles based on the narrative review type
may lead to false or misleading conclusions. Especially for the agricul-
tural research, the authors think that there are many cases where the
use of a systematic review or even meta-analysis can summarize pri-
mary agricultural research and provide robust conclusions based on
substantiated studies with assessed strength of evidence. For this rea-
son, authors encourage agro-scientists to adopt a framework with spe-
cific guidelines, easy to follow like the one proposed, aiming to
formulate carefully a review protocol and conduct efficiently a system-
atic review or meta-analysis to answer specific scientific questions and
summarize current literature.
7. Conclusions
Systematic review is a structured method for examining current lit-
erature that has been widely accepted and adopted as the evidence-
based way of summarizing primary research. The use of systematic re-
views in agriculture is recent but it is getting increased in the last few
years, maybe because of its efficacy to summarize knowledge about
the existing agricultural literature and indicate what should be
researched in order to understand better the agricultural systems and
processes.
Concerning the study case that has been examined as example, it can
be concluded that Systematic Reviews have been used for agricultural
sciences as a method of reviewing literature only during the last five
11
19
Refined Available
Meta-analysis
years. Based on the fact that findings for a specific scientific field derived
from different agricultural researches can be examined together to pro-
vide further information, one could assume that the use of systematic
reviews is a one-way road to combine several research studies to get a
final conclusion. Nevertheless, studies or experiments on the same sub-
ject performed under similar conditions can be used for systematic re-
views and following the review protocol an individual can proceed to
summarize the progress made so far in a specific scientific field. How-
ever, the lack of knowledge of how to conduct a successful systematic
review or the heterogeneity of the agricultural studies may be obstacles
in combining different studies to provide robust conclusions.
For this purpose and considering the given limitations, the use of a
guide for conducting a systematic review, such as the one proposed in
this study, is considered to be of great help for agro-scientists in their
agricultural or agro-environmental research.
Acknowledgements
We wish to thank Prof. A. Gatzianas, and Prof. I. Eleftherohorinos, for
their valuable comments on our manuscript.
Appendix A. Supplementary material
Supplementary data to this article can be found online at https://doi.
org/10.1016/j.scitotenv.2019.04.354.
References
Antman, E.M., Lau, J., Kupelnick, B., Mosteller, F., Chalmers, T.C., 1992. A comparison of re-
sults of meta-analyses of randomized control trials and recommendations of clinical
experts: treatments for myocardial infarction. JAMA 268, 240–248.
Auditeau, E., Chassagne, F., Bourdy, G., Bounlu, M., Jost, J., Luna, J., Ratsimbazafy, V., Preux,
P.M., Boumediene, F., 2019. J. Ethnopharmacol. 234, 119–153.
Bailar, G., 1997. The promise and problems of meta-analysis. N. Engl. J. Med. 337, 559
−560.
Borenstein, M., Hedges, L., Higgins, J., Rothstein, H., 2009. Introduction to Meta-analysis.
Hoboken. John Wiley & Sons Inc (2009).
Buckingham, S., Anthony, S., Bellamy, P.H., Cardenas, L.M., Higgins, S., McGeough, K., Topp,
C.F.E., 2014. Review and analysis of global agricultural N2O emissions relevant to the
UK. Sci. Total Environ. 487, 164–172.
Byeon, J.H., Kil, J.H., Ahn, Y.C., Son, C.G., 2019. Systematic review of published data on herb
induced liver injury. J. Ethnopharmacol. 233, 190–196.
Center for Health Communication and Participation, 2017. Exploring Systematic Reviews.
Available from. http://navigatingeffectivetreatments.org.au/exploring_systematic_re-
views.html (19 May 2017).
Cipriani, A., Geddes, J., 2003. Comparison of systematic and narrative reviews: the exam-
ple of the atypical antipsychotics. Epidemiol. Psichiatr. Soc. 12 (3), 146–153.
Cook, D., Mulrow, C., Haydens, R., 1997. Systematic reviews: synthesis of best evidence for
clinical decisions. Ann. Intern. Med. 126 (5), 376–380.
Cooper, H., Hedges, L.V., 2009. The handbook of research synthesis and meta-analysis. In:
Cooper, H., Hedges, L.V., Valentine, J.C. (Eds.), Research Synthesis as a Scientific Pro-
cess, 2nd ed. Russell Sage Foundation, New York, pp. 3–17.
 T.M. Koutsos et al. / Science of the Total Environment 682 (2019) 106–117
Cooper, J., Baranski, M., Stewart, G., Nobel-de Lange, M., Barberi, P., Fliessbach, A., ... Mader,
P., 2016. Shallow non-inversion tillage in organic farming maintains crop yields and
increases soil C stocks: a meta-analysis. Agron. Sustain. Dev. 36 (1). https://doi.org/
10.1007/s13593-016-0354-1.
Cronin, P., Ryan, F., Coughlan, M., 2008. Undertaking a literature review: a step-by-step
approach. Br. J. Nurs. 17 (1), 38–43.
De Luca, A.I., Iofrida, N., Leskinen, P., Stilitano, T., Falcone, G., Strano, A., Gulisano, G., 2017.
Life cycle tools combined with multi-criteria and participatory methods for agricul-
tural sustainability: insights from a systematic and critical review. Sci. Total Environ.
595, 352–370.
Deeks, J.J., Higgins, J.P.T., Altman, D.G., 2008. Cochrane handbook for systematic reviews of
interventions. In: Higgins, J.P.T., Green, S. (Eds.), Analysing data and undertaking
meta-analyses. John Wiley & Sons, Ltd, Hoboken, nj, pp. 243–296.
Duvendack, M., et al., 2012. Assessing ‘what works’ in international development: issues
and methods of risk of bias and meta-analysis in development interventions. Journal
of Development Effectiveness 4 (3), 456–471.
Dybå, T., Torgeir Dingsøyr, T., 2008. Strength of evidence in systematic reviews in soft-
ware engineering. Proceedings of the Second International Symposium on Empirical
Software Engineering and Measurement, ESEM 2008, October 9–10, 2008, Kaisers-
lautern, Germany https://doi.org/10.1145/1414004.1414034.
Egger, M., Zellweger-Zahnert, Schneider, M., Junker, C., Lengeler, C., Antes, G., 1997. Lan-
guage bias in randomised controlled trials published in English and German. Lancet
350, 326−329.
Fagard, R., Staessen, J., Thijs, L., 1996. Advantages and disadvantages of the meta-analysis
approach. Official Journal of the International Society of Hypertension 14 (2), S9–12
discussion S13. https://doi.org/10.1097/00004872-199609002-00004.
Ferguson, J., Dalton, C., McGettigan, P., Hill, S., 1998. Antibiotic resistance in animal enteric
bacteria and human disease – a review of the scientific literature. Joint Expert Tech-
nical Advisory Committee on Antibiotic Resistance (JETACAR). The University of
Newcastle, Australia.
Glass, G., 1976. Primary, secondary, and meta-analysis of research. Educ. Res. 5, 3−8.
Glasziou, P., Irwig, L., Bain, C., Colditz, G., 2001. Systematic Reviews in Health Care: A Prac-
tical Guide. University of Cambridge, United Kingdom.
GRADE Working Group, 2004. Grading quality of evidence and strength of recommenda-
tions. BMJ 328–1490.
Grant, M., Booth, A., 2009. A typology of reviews: an analysis of 14 review types and as-
sociated methodologies. Health Inf. Libr. J. 26, 91–108.
Green, B.N., Johnson, C.D., Adams, A., 2006. Writing narrative literature reviews for peer-
reviewed journals: secrets of the trade. J. Chiropr. Med. 5 (3), 101–117.
Greenhalgh, T., Wong, G., Westhorp, G., Pawson, R., 2011. Protocol–realist and meta-
narrative evidence synthesis: evolving standards (rameses). BMC Med. Res.
Methodol. 11, 115.
Greenland, S., 1995. Systematic Reviews. British Medical Journal Publishing Groups,
London.
Hagen-Zanker, J., McCord, A., Holmes, R., 2011. The impact of employment guarantees
schemes and cash transfers on the poor. ODI Systematic Review. Overseas Develop-
ment Institute, London.
Hardy, R., Thompson, S., 1998. Detecting and describing heterogeneity in meta-analysis.
Stat. Med. 17 (8), 841–856.
Harwell, M., 1997. An empirical study of Hedges' homogeneity test. Psychol. Methods 2,
219−231.
Higgins, J.P.T., Green, S., 2008. Cochrane Handbook for Systematic Reviews of Interven-
tions Version 5.0.1. The Cochrane Collaboration. 2008. Available at:. www.
cochrane-handbook.org.
Higgins, J., Thompson, S., Deeks, J., Altman, D., 2003. Measuring inconsistency in meta-
analyses. Br. Med. J. 327 (7414), 557–560.
Hobson, A., 2014. Why manuscripts submitted to an international peer reviewed journal
in education are rejected. Int. J. Mentor. Coach. Educ. 3 (2). https://doi.org/10.1108/
IJMCE-04-2014-0014.
Khan, K.S., Kunz, R., Kleijnen, J., Antes, G., 2003. Five steps to conducting a systematic re-
view. J. R. Soc. Med. 96 (3), 118–121.
Kitchenham, B., 2004. Procedures for performing Systematic Reviews, NICTA Technical
Report 0400011T.1. (ISSN:1353-7776). Empirical Software Engineering, National
ICT Australia Ltd, Australian Technology Park, Eversleigh, Australia.
Kleijinen, J., Riet, G., Knipschild, P., 1990. Vitamin B-6 in the treatment of the premen-
strual syndrome − a review. Br. J. Obstet. Gynaecol. 97, 847−852.
Klein, H.K., Myers, M.D., 1999. A set of principles for conducting and evaluating interpre-
tive field studies in information systems. MIS Q. 23 (1), 67–93.
Kowalczyk, N., Truluck, C., 2013. Literature reviews and systematic reviews: what is the
difference. Radiol. Technol. 85 (2), 219–222.
Levy, Y., Ellis, T.J., 2006. A systems approach to conduct an effective literature review in
support of information systems research. Inf. Sci. 9, 181–211.
Liberati A., Altman D. G., Tetzlaff J., Mulrow C., Gøtzsche P. C., Ioannidis J. P. A. et al. Moher
D., 2009. The prisma statement for reporting systematic reviews and meta-analyses
of studies that evaluate health care interventions: explanation and elaboration.
Ann. Intern. Med., 151(4), W-65.
Liu, T., Xu, S., Lu, S., Qin, P., Bi, B., Ding, H., Liu, Y., Guo, X., Liu, X., 2019. A review on re-
moval of organophosphorus pesticides in constructed wetland: performance, mech-
anism and influencing factors. Sci. Total Environ. 651 (Part 2), 2247–2268.
Mallett, R., Hagen-Zanker, J., Slater, R., Duvendack, M., 2012. The benefits and challenges of
using systematic reviews in international development research. Journal of Development
Effectiveness 4 (3), 445–455. https://doi.org/10.1080/19439342.2012.711342.
117
Mann, C., 1994. Can meta-analysis make policy? Science 266, 960−962.
Moher, D., Cook, D.J., Eastwood, S., Olkin, I., Rennie, D., Stroup, D.F., 1999. Improving the
quality of reports of meta-analyses of randomized controlled trials: the QUOROM
statement. Quality of reporting of meta-analyses. Lancet 354, 1896–1900.
Moher, D., Liberati, A., Tetzlaff, J., Altman, D.G., 2010. Preferred reporting items for system-
atic reviews and meta-analyses: the PRISMA statement. Int. J. Surg. 8 (5), 336–341.
Mulrow, C.D., 1987. The medical review article: state of the science. Ann. Intern. Med. 106,
485–488.
Munn, Z., Stern, C., Aromataris, E., Lockwood, C., Jordan, Z., 2018. What kind of systematic
review should I conduct? A proposed typology and guidance for systematic reviewers
in the medical and health sciences. BMC Med. Res. Methodol. 18, 5. https://doi.org/
10.1186/s12874-017-0468-4.
Neinert, C., 1997. Meta-analysis: science or religion? Contemp. Clin. Trials 10, 257S
−263S.
Olson, C., Rennie, D., Cook, D., Dickersin, K., Flanagin, A., Hogan, J., 2002. Publication bias in
editorial decision making. JAMA 287, 2825−2828.
Ouyang, W., Wang, Y., Lin, C., He, M., Hao, F., Liu, H., Zhu, W., 2018. Heavy metal loss from
agricultural watershed to aquatic system: a scientometrics review. Sci. Total Environ.
637-637, 208–220.
Oxman, A.D., Guyatt, G.H., 1993. The science of reviewing research. Ann. N. Y. Acad. Sci.
703, 125–133.
Paré, G., Trudel, M.-C., Jaana, M., Kitsiou, S., 2015. Synthesizing information systems
knowledge: a typology of literature reviews. Inf. Manag. 52 (2), 183–199.
Petticrew, M., 2001. Systematic reviews from astronomy to zoology: myths and miscon-
ceptions. BMJ 322, 98–101.
Riley, D., Moons, G., Snell, K., Ensor, J., Hooft, L., Altman, D., Hayden, J., Collins, G., Debray,
T., 2019. A guide to systematic review and meta-analysis of prognostic factor studies.
BMJ 364.
Sargeant, J.M., Rajic, A., Read, S., Ohlsson, A., 2006. The process of systematic review and
its application in agri-food public-health. Prev. Vet. Med 75, 141–151.
Schmolke, A., Kapo, K.E., Rueda-Cediel, P, Thorbek, P., Brain, R., Forbes, V., 2017, Develop-
ing population models: a systematic approach for pesticide risk assessment using
herbaceous plants as an example. Sci. Total Environ. 599-600, 1929–1938.
Schünemann, H.J., Oxman, A.D., Higgins, J.P., Vist, G.E., Glasziou, P., Akl, E., et al., on behalf
of the Cochrane GRADEing Methods Group and the Cochrane Statistical Methods
Group, 2017. Chapter 11: Completing ‘Summary of findings’ tables and grading the
confidence in or quality of the evidence. (updated June 2017). In: Higgins, J.P.,
Churchill, R., Chandler, J., Cumpston, M.S. (Eds.), Cochrane Handbook for Systematic
Reviews of Interventions. The Cochrane Collaboration Available from. www.train-
ing.cochrane.org/handbook, 5.2.0.
Seetaloo, A.D., Aumeeruddy, M.Z., Kannan, R.R.R., Mahomoodally, M.F., 2019. Potential of
traditionally consumed medicinal herbs, spices, and food plants to inhibit key diges-
tive enzymes geared towards diabetes mellitus management - a systematic review. S.
Afr. J. Bot. 120 (SI), 3–24.
Shaffril, H.A.M., Krauss, S.E., Samsuddin, S.F., 2019. A systematic review on Asian's
farmers' adaptation practices towards climate change. Sci. Total Environ. 644,
683–695.
Sharma, S., Chaubey, I., 2017. Surface and subsurface transport of nitrate loss from the se-
lected bioenergy crop fields: systematic review, analysis and future directions. Agri-
culture-BASEL 7 (3). https://doi.org/10.3390/agriculture7030027.
Stroup, D., Thacker, S., 1995. Quantitative methods in the review of epidemiologic litera-
ture. Epidemiol. Rev. 9, 1−30.
The Campbell Collaboration, 2017. Steps in proposing, preparing, submitting, and editing
of Campbell Collaboration systematic reviews revision - coordinating chairs.Available
from:. http://www.campbellcollaboration.org/ (17 May 2017).
Toth, B., Hegyi, P., Lantos, T., Szakacs, Z., Keremi, B., Varga, G., Tenk, J., Petervari, E., Balasko,
M., Rumbus, Z., Rakonczasy, Z., Balint, E.R., Kiss, T., Csupor, D., 2019. The efficacy of
saffron in the treatment of mild to moderate depression: a meta-analysis. Planta
Med. 85, 24–31.
Van der Knaap, L.M., et al., 2008. Combining Campbell standard and the realist evaluation
approach: the best of two worlds? Am. J. Eval. 29 (1), 48–57.
Van der Windt, D.A., Thomas, E., Pope, D., De Winter, A.F., Macfarlane, G.J., Bouter, L.B.,
Silman, A.J., 2000. Occupational risk factors for shoulder pain: a systematic review.
Occup. Environ. Med. 57, 433–442.
Webster, J., Watson, R.T., 2002. Analyzing the past to prepare for the future: writing a lit-
erature review. Manag. Inf. Syst. 26 (2), 11.
Weltin, M., Zasada, I., Piorr, A., Debolini, M., Geniaux, G., Moreno Perez, O., ... Schulp, C.J.E.,
2018. Conceptualising fields of action for sustainable intensification - a systematic lit-
erature review and application to regional case studies. Agric. Ecosyst. Environ. 257,
68–80. https://doi.org/10.1016/j.agee.2018.01.023.
Wright, R.W., Brand, R.A., Dunn, W., 2007. How to write a systematic review. Clin. Orthop.
Relat. Res. 455, 23–29.
Yannascoli, S., Schenker, M., Carey, J.L., Ahn, Jaimo, Baldwin, K., 2014. How to write a sys-
tematic review: a step-by-step guide, University of Pennsylvania. Am. Orthopt. J. 23,
64–69.
Zaza, S., Wright-De Aguero, L.K., Briss, P.A., 2000. Data collection instrument and proce-
dure for systematic reviews in the Guide to Community Preventive Services. Am.
J. Prev. Med. 18 (S1), 4474.