www.ietdl.

org
Published in IET Software
Received on 18th February 2013
Revised on 24th July 2013
Accepted on 23rd August 2013
doi: 10.1049/iet-sen.2013.0029

Special Issue on Empirical Studies in Software

Engineering

ISSN 1751-8806

Externalising tacit knowledge of the systematic

review process
Sandra Camargo Pinto Ferraz Fabbri1, Katia Romero Felizardo2, Fabiano Cutigi Ferrari1,
Elis Cristina Montoro Hernandes1, Fábio Roberto Octaviano3, Elisa Yumi Nakagawa2,
José Carlos Maldonado2
1
Computing Department, Federal University of São Carlos (UFSCar), São Carlos, SP, Brazil
2
Computer Systems Department, University of São Paulo (ICMC/USP), São Carlos, SP, Brazil
3
Computing Department, Federal Institute of São Paulo (IFSP), São Carlos, SP, Brazil
E-mail: fabiano@dc.ufscar.br

Abstract: Systematic Reviews (SRs) have recently intensified in Software Engineering. However, there is a lack of work that
makes explicit how the process to perform SR is in practice. The goal of this paper is externalising the process that reflects
how SRs are currently performed, transforming tacit knowledge into explicit knowledge. We describe the phases and
activities that compose the process and the relationship among them, and explore the iterative characteristic of this process,
focusing on intra- and inter-phase iterations that are necessary to conduct the process appropriately. To achieve the proposed
goal, we devised the process based on practical experience acquired for several years by research groups in Software
Engineering, which include graduate students and researchers who have applied SR. The process has been applied in several
SRs and seems to be effective in keeping the focus of the review at all phases. Moreover, the externalisation of the process
has been useful to help researchers improving the process execution quality. As the process reflects the practice and is
explained in details, it can be used as a guide to better understand the SR process and its details. This shall contribute to
improve all SR process phases, and hence the quality of SR results.

1 Introduction Perhaps one of the factors that can be promoting these

Systematic review (SR) (also known as systematic literature
review) was originally employed in the medical area and is
recognised as one of the key components to support the
evidence-based medicine. Inspired by the success in the
medical field, the SR has been addressed within a number
of other subjects, including economics, psychology, social
science and most health care disciplines [1, 2].
SRs, since their introduction in the field of software
engineering (SE), have gained substantial importance [3, 4],
and have been applied to various topics of interest [5–10].
The reasons why more and more SRs are conducted every
year can be attributed to the advantages of SRs, including
reduced likelihood of bias in results and the potential ability
to combine data for various quantitative studies such as
meta-analysis.
Despite its importance, the SR process is still largely
carried out manually, which is both challenging and
time-consuming. Based on literature, many potentially
problematic aspects of the process are identified.
Examples of such problems are: (i) formulation of
research questions [1]; (ii) conducting searches [1, 11–14];
(iii) primary study selection [1, 15]; and (iv) quality
assessment of primary studies [1, 16]. Therefore the
problems and the approaches adopted to tackle these
problems vary across studies.
problems is the lack of details in the description of the SR
process. The guidelines used in the SE area [17] were
created as an adaptation of several existing guidelines from
other disciplines, particularly medicine, and were described
at a relatively high level of details [18–20]. Although in
2007 a more comprehensive set of guidelines for
performing SR in SE became available [21], it remained the
same guidelines used by medical researchers (this was
reinforced by several discussions with researches from other
fields [20]).
Staples and Niazi [19] also mentioned that guidelines for
SR are well established in other disciplines. However, SE
researchers have yet to come up with a well understood
consensus about the conduction of SR. Several researchers
are working on improving the scientific support for SR in
SE [3, 15]. It is clear that concepts of evidence-based
practice initially developed in medicine should be adopted
in SE. However there are difficulties to use the medical
guidelines for performing SR in the SE context. A key
issue is to better understand the role of SR in SE [1].
Kitchenham et al. [2] highlight that adapting the SR
process to meet the needs of SE is an ongoing process.
Based on this scenario, the focus of this paper is to
externalise details describing how SR have been carried out
in practice. This is shown in Sections 3 and 4, with the aim
of transforming tacit knowledge into explicit knowledge.

298 IET Softw., 2013, Vol. 7, Iss. 6, pp. 298–307

& The Institution of Engineering and Technology 2013 doi: 10.1049/iet-sen.2013.0029

Tacit knowledge refers to information that is important to
adequately conduct SR, but is not explicit in the guidelines.
This tacit knowledge is often crucial for the interpretation
of the explicit knowledge. In this paper, we synthesised
tacit knowledge in a process, named I-SR, which highlights
the need for iterations which naturally occur in an attempt
to overcome the problems we are faced during the SR
process. We emphasise that the I-SR process results from
the experience obtained by these authors’ research groups in
the context of local research as well as international
collaborations. In Section 5, we also summarise the results
of SRs that have been performed in the last few years with
support of the process described in this paper, with special
attention to iterations and activities that were externalised in
our process. A description of related research and
conclusions and future work completes this paper (Sections
6 and 7).
2 Background
2.1 Basic concepts of systematic reviews
An SR is a ‘means of identifying, evaluating and interpreting
available research relevant to a particular research question, or
topic area, or phenomenon of interest’ [21]. Since its
introduction in 2004 in the SE field, SRs have been
increasingly used as a method for conducting SE-related
secondary studies [3, 4].
By analogy with medicine, Kitchenham [17] proposed the
use of SR as a methodology for performing unbiased
aggregation of empirical results in SE. Evidence-based SE
was first proposed as a means of advancing and improving
the discipline of (SE) [17]. SR provides a methodical and
structured process to support the conduction of literature
reviews [21].
Brereton et al. [1] state that the research question(s) is the
most critical element of an SR, because they are used to: (i)
construct search strings; (ii) determine the data that need to
be extracted from each primary study; and (iii) constrain
the synthesis process. Therefore the research questions are
the part of the protocol that should not be changed after the
protocol is reviewed and accepted [1].
2.2 Major challenges for performing systematic
reviews
Several authors have reported and discussed obstacles faced
by them – or by their research groups – while performing
SRs in the SE field. The major challenges are next
summarised.
2.2.1 Rigour of the search process: Zhang and
Muhammad [15] and Zhang et al. [22] highlight that the
rigour of the search process is one critical factor that
distinguishes SR from traditional, ad hoc literature reviews.
However, effective searching is far to be a trivial task.
Indeed, many SE researchers [1, 11–14, 16] have lately
reported difficulties to search relevant studies. Effective
searching strongly relies on the quality of the search string;
devising it is a time-consuming and error-prone activity
[22]. The performance of automated search itself is highly
dependent on the quality of the string, which implies a
continuous refinement process. However, defining a search
string may be a ‘double-edged sword’. As emphasised by
Boell and Cezec-Kecmanovic [23], on the one hand, the
more inclusive is the search string, the more irrelevant
IET Softw., 2013, Vol. 7, Iss. 6, pp. 298–307
doi: 10.1049/iet-sen.2013.0029
www.ietdl.org
documents one will retrieve; on the other hand, the more
precise it is, the more relevant documents will be missed.
Besides this, poorly written abstracts can complicate the
study selection process [1, 24–26].
2.2.2 Sequence against iteration: Currently
documented, largely adopted SR procedures [17, 18] do not
explicitly deal with the iterative nature of the SR process.
Nevertheless, previous findings reinforce the cyclic
characteristics of some phases or subsets of activities within
a given phase in the SR process. For example, in Zhang
and Muhammad’s [15] work, many interviewees mentioned
the study selection and data extraction are done in an
iterative way, specially when novices are involved in the
SR. This may even lead to a revision of the inclusion and
exclusion criteria and the data extraction form as well. In
another piece of work, Brereton et al. [1] concluded that
piloting the protocol is essential to find mistakes in the data
collection and in the synthesis procedures. Obviously,
piloting means that the search will be ‘officially’ executed
at some point in the future.
2.2.3 Lack of mature guidelines and checklists:
Staples and Niazi [19] mentioned that guidelines for SR are
well established in other disciplines, particularly in
medicine. However, SE researchers have yet to come up
with a well understood consensus about how to conduct
SR. There are several researchers that are working on
improving the scientific support for SR in SE [3]. It is clear
that concepts of evidence-based practice initially developed
in medicine should be adopted in SE. However, there are
difficulties to use the medical guidelines for performing SR
in SE context. A key issue is to better understand the role
of SR in SE [1]. Kitchenham et al. [2] highlight that
adapting the SR process to meet the needs of SE is an
ongoing process.
The next sections externalise the iterative process for
conducting SR, transforming tacit, experience-based
knowledge into explicit one. The process is described
together with a set of checklists to support approval
activities along the process. We also use examples extracted
from our SR to justify some design decisions made on the
process, in order to make the process explicitly iterative, as
well as to overcome the difficulties discussed in this section.
3 Process overview
The incremental SR process, hereafter called I-SR, is
compound of four phases, namely: (i) planning, (ii) initial
selection, (iii) final selection and extraction and (4)
synthesis. Its overall structure is depicted in Fig. 1. The
process layout is inspired in the evolutionary models for
software development. Such models aggregate iteration and
evolution in the development process, in which the
produced software artefacts evolve after each cycle of the
model. In I-SR, the artefacts evolve after executing each
phase similarly. In Fig. 1, phases are each represented as a
circle that comprises a set of activities that can be done
iteratively. The phases are briefly described in the sequence.
Their details are provided in Section 4.
The planning phase comprises the definition of the SR
protocol, a pilot search and the design of a preliminary
version of the data extraction form. It is the first phase of
the process and has the ‘define/revise protocol’ activity as
its starting point.
299
& The Institution of Engineering and Technology 2013
www.ietdl.org
Fig. 1 Process overview
The initial selection phase comprises activities related to
the pre-selection of primary studies. Basically, this phase
requires the customisation of the search string for each
target repository, running the searches, pre-selecting the
primary studies and checking the occurrence of new
candidate keywords to enhance the search string.
The final selection and extraction phase encompasses
activities related to the final selection of primary studies and
data extraction. As in initial selection, during this phase
some new candidate keywords may emerge, and hence must
be checked. Besides that, additional data extraction fields
may be necessary and thus must be considered in a revised
version of the data extraction form.
Finally, the synthesis phase addresses the analysis of the
extracted data, and reporting and publishing the results.
Filtering is required, for example, when it is necessary to
300
& The Institution of Engineering and Technology 2013
update the data sheets because of novel, up-to-date research
results that make previous ones obsolete.
It is important to highlight that packaging is performed in
parallel with the whole process execution. Furthermore, the
protocol permeates the whole SR lifecycle. Every designed
or produced artefact, every decision taken and every
procedure performed by the researchers are driven by the
protocol. Reporting all these items creates a package that
enables both process and results audit, as well as SR
replication, update and extension.
3.1 Supported execution scenarios
The I-SR process is devised to support SR in four different
scenarios: (i) original SR, (ii) SR replication, (iii) SR
extension and (iv) SR update (the last three are called
IET Softw., 2013, Vol. 7, Iss. 6, pp. 298–307
doi: 10.1049/iet-sen.2013.0029

‘re-execution’ scenarios. Depending on the current scenario,
some activities along the process must be executed in a
particular manner. Owing to this reason, most of the
activities within the planning phase emphasise the revision
of some artefact, result or procedure. Filtering-relates tasks,
which are later described in this section and in Section 4,
represent another example of customisable procedures based
on the current execution scenario.
To illustrate how the process support re-execution
scenarios, let us consider an update of an original SR.
During the second interaction (updating), the protocol is
revisited in order to identify required changes to make it
suitable for the update. For example, the exclusion criteria
require changes to restrict the search only for studies
published after the first SR. After running the searches, the
retrieved set of primary studies should be cleaned up to
avoid overlapping with the previous, original set of studies.
In the sequence, the studies are selected (i.e. classified as
selected or discarded [‘Selected’ means the study fulfils the
inclusion criteria, and ‘discarded’ means the opposite. In
the literature, the terms ‘included’ and ‘excluded’ are also
used.]) and all relevant information is extracted. Merging
(or filtering) tasks are also required to combine previous
and newly gathered data, as well as to avoid the presence of
obsolete evidence. Section 4 provides details on how to
treat the new studies independently, but still considering the
whole set of results, and how to aggregate the extracted
data. Related information can also be found elsewhere [27,
28].
3.2 Intra-phase and inter-phase iteration levels
The I-SR process is iterative in its essence. Two levels of
iteration can be found within and across the phases,
hereafter named ‘intra-phase’ iteration and ‘inter-phase’
iteration, respectively. In Fig. 1, intra-phase iterations are
represented by the arrows placed on the outermost ring of
each phase as well as by the directed edges placed within
the innermost circles. Inter-phase iterations, on the other
hand, are represented by the dashed, directed edges that go
from one phase to another.
Each directed edge – intra- or inter-phase – is labelled with
a guard-condition that must hold to activate the associated
transition. As an example, let us consider the ‘approve pilot
search’ in the planning phase. If the search results are
unsatisfactory (represented by the [not OK] label), the
researchers must re-execute the ‘revise keywords’ activity
and the subsequent ones, until the search results are
approved. Once the results are approved, be it in the first
iteration or not, the researchers can naturally move to the
next activity, which is ‘define/revise preliminary data
extraction’ form in this example.
3.3 Checklists and filters
Some activities along the I-SR process bring together
checklists to help the researcher assess the results obtained
after either partial or whole phase executions. For example,
the ‘approve selection’ and ‘approve extraction’ activities in
the final selection and extraction phase are both supported
by checklists. Note that a graphical mark is included in
each activity that is supported by a checklist (see the legend
of Fig. 1).
Some other activities in I-SR require data filtering. Their
goals depend on the ongoing review scenario (i.e. original,
replication, extension or update). For example, be a review
IET Softw., 2013, Vol. 7, Iss. 6, pp. 298–307
doi: 10.1049/iet-sen.2013.0029
www.ietdl.org
updating scenario in which new primary studies might be
pre-selected in the initial selection phase. Even though most
of the search engines enable the user to apply filters related
to publication date (e.g. published after a given year), such
filters are generally not fine-grained enough (e.g. published
after a particular month of the year) to avoid duplicates.
Therefore, after the researchers query all the target
repositories, they must filter the newly gathered results to,
in a further step, merge them with the previous round dataset.
4 Process in details
This section details each phase of the I-SR process,
emphasising the points where intra-phase and inter-phase
iterations can occur. Checkpoints and associated supporting
checklists are also described for each phase.
4.1 Planning phase
The objective of this phase is planning the SR and establish a
search string that is able to fetch appropriate studies. The plan
corresponds to a protocol, for which a template designed by
Kitchenham and Charters [21] has been used by the SE
community so far. Two protocol fields deserve special
attention since they can generate intra- or inter-phase
iterations: keywords and search string. Owing to this, they
are drawn as separate steps in Fig. 1. Intra-phase iteration is
assumed to occur because before going to next phase, the
search string must be improved until appropriate studies are
found. Inter-phase iterations triggering the re-execution of
activities of this phase can occur every time a study
selection is done. These iterations can be observed in Fig. 1
(see the connector labelled with ‘A’) and will be explained
in Sections 4.2 and 4.3.
The activities required for the planning phase are:
1. Define/revise protocol: all fields of the protocol must be
filled because they describe the intention of the SR, the way
the SR is carried out, and allow that the SR can be
replicated by other researchers.
2. Revise keywords: the keywords must be revised and
updated aiming at fetching appropriate studies. As they
must be improved until the researcher is satisfied with the
fetched studies, the revision occurs sometimes along the
process, particularly inside this phase, as an intra-phase
iteration. Besides, in every study selection activity (be it
initial or final), new candidate keywords can be spotted,
thus triggering an inter-phase iteration.
3. Revise search string: as the keywords are defined or
revised, they must be combined through logical operators,
generating a search string.
4. Run/revise pilot search: once the search string is defined, it
must be submitted to a selected search machine. This machine
is one of the set of sources (i.e. repositories) defined in the
protocol, and is used in this phase just to adjust the string.
5. Approve pilot search: the studies fetched from the search
executed in previous step must be analysed aiming at
verifying if they are in accordance to the objective of the
SR and its main question. As the researchers analyse the
results, they can realise that the search string can be
improved, triggering the intra-phase iteration, going back to
activity 2. This activity is assisted by the checklist shown in
Fig. 2.
6. Define/revise preliminary data extraction form: based on
the SR objective, the researcher is able to define an initial
301
& The Institution of Engineering and Technology 2013
www.ietdl.org
Fig. 2 Checklist for pilot search approval
list of data that must be extracted. This list is essential for
standardising the reading of the studies.
7. Approve protocol: it consists in checking if the protocol is
fully defined and aligned with the SR goals. If the protocol is
approved, researchers move to the next phase. Otherwise, a
new planning cycle starts and every activity is either
re-executed or revised to achieve protocol adequacy. The
checklist shown in Fig. 3 supports this activity.
4.2 Initial selection phase
This phase aims to fetch studies in all repositories specified in
the protocol. Besides, it includes a first (or initial) selection of
studies based on the reading of title and abstract (or even other
study excerpts, depending on the protocol definition). This
phase requires the application of the inclusion and
exclusion criteria defined in the protocol. During the
selection, filters are applied to discard retrieved documents
not of interest (e.g. studies that are retrieved from more
than one repository). Besides, if new keywords are
Fig. 3 Checklist for protocol approval
302
& The Institution of Engineering and Technology 2013
identified, they must be considered and will possibly trigger
an inter-phase iteration, causing a re-execution of some
steps of the planning phase.
The activities required for this phase are:
1. Adapt search string: since each search machine uses a
particular syntax for elaborating the search string, the string
defined in the planning phase must be adapted for each of
them. Note that some machines do not accept complex
strings (in terms of composition rules and numbers of
terms), and hence some categories of papers that can be
disregarded by the search string in a given machine cannot
be avoided in other machines. Dieste et al. [13] pinpoint
other important issues about search string construction
considering the most important scientific repositories in
computer science.
2. Query all repositories: after the string is adapted to each
search machine, it must be applied to every repository
aiming at retrieving relevant studies for the research question.
† ‘Filtering’: accomplishing this activity may require a
filter, which is applied while running any SR execution
scenario. For example, in a given scenario, search results
obtained from different repositories (e.g. Association for
Computing Machinery (ACM) Digital Library and IEEE
eXplore) may overlap and hence must be filtered to
remove duplicates; documents such as proceedings
foreword must also be discarded. Another example regards
the re-execution scenarios; in this case, studies that were
processed in a previous execution must be discarded.
3. Perform initial selection: all retrieved studies, from all
repositories, must be selected taking into account their titles
and abstracts (and possibly other excerpts). If a study can
be useful for answering the research question, it must be
selected; if not, it must be discarded. This decision is based
on the inclusion and exclusion criteria defined in the protocol.
† ‘Filtering’: accomplishing this activity may require a
filter, which may be applied while running any SR
execution scenario. For example, in an SR extension,
after the primary studies are selected, one (or some) of
them may be identified as a new, up-to-date version of
studies which were previously selected. In this case,
according to Kitchenham’s guidelines [17], only the
up-to-date study will populate the dataset. Note that such
situation is likely to occur in any execution scenario, be
it an original SR or a re-execution.
4. Check new keywords: as retrieved studies are read,
different keywords that were not considered in the protocol
Fig. 4 Checklist for initial selection approval
IET Softw., 2013, Vol. 7, Iss. 6, pp. 298–307
doi: 10.1049/iet-sen.2013.0029

may be identified. This can happen because of different
reasons: the researcher can be novice ‘in the research area’,
or the keyword can be a new term in that area. Every time
new keywords are identified, an inter-phase iteration is
triggered: the ‘revise keywords’ activity (planning phase)
must be executed, as well as the rest of the activities of that
phase. This aims to ensure that the search string and the
protocol are consistent with the updated keywords.
5. Approve initial selection: this activity aims to assess the
way the initial selection is performed and its results. If the
selection is approved researchers move to the next phase.
Otherwise, a new initial selection cycle starts to achieve
proper selection of studies. To assist this activity, we define
the checklist shown in Fig. 4.
4.3 Final selection and extraction phase
The objective of this phase is to read and select all studies
that are really of interest for the SR. Based on these studies,
data must be extracted aiming at supporting the synthesis of
the investigated theme. This phase is compound of six
activities and contains intra- and inter-phase iterations as
follows:
1. Perform final selection and pre-extraction: all studies must
be read in full and inclusion and exclusion criteria must be
applied again. Some times, studies that were selected in the
initial selection phase are discarded in this phase. In this
case, the researchers may note that the content is not
relevant for the SR objectives after the full reading.
† ‘Filtering’: similarly to the filter described for the
‘perform initial selection’ activity, this filter may be
applied while running any SR execution scenario. For
example, in a scenario that comprises either an original
SR or a re-execution, when the researchers fully read a
given primary study, they may find out that such study
either includes or updates the contributions and results
of previous studies published by the same authors. In
this case, the up-to-date study has kept the dataset and
the old versions must be removed therefore.
2. Check new keywords: as in the previous phase, new
keywords may be identified and the same inter-phase
iteration is triggered, deviating the execution to planning
phase again. Consequently, all the activities of that phase,
as well as the initial selection phase must be re-executed.
Note that the filters present in the process will support the
Fig. 5 Checklist for final selection approval
IET Softw., 2013, Vol. 7, Iss. 6, pp. 298–307
doi: 10.1049/iet-sen.2013.0029
www.ietdl.org
removal of duplicates (i.e. studies retrieved with the
previous version of the search string).
3. Check new data extraction fields and revise data
extraction form: as the studies are read, and the researcher
goes deeper into them, new information extraction fields
may arise, and the data extraction form must be updated. In
the ‘refine extraction’ activity, any remaining non-extracted
data will be handled.
4. Approve selection: as the SR must be done at least by two
researchers, this approval involves an agreement between
them. Hence, if they have different points of view with
respect to some studies or with respect to some new data
extraction fields, then an intra-phase iteration is triggered.
This generates a return to ‘perform final selection and
pre-extraction’ and all subsequent activities. The decision on
this iteration takes into account the checklist shown in Fig. 5.
5. Refine extraction: at this activity, the researchers are in
accordance with the selected studies and the data extraction
form. Therefore they must completely fill the form for each
selected study, and revise and (possibly) complete the
information extracted along the other activities.
6. Approve extraction: this approval aims to check whether
the extracted data are consistent across the full set of
selected studies and unbiased. If not, an intra-phase iteration
is triggered. To assist this activity, we define the checklist
shown in Fig. 6.
4.4 Synthesis phase
The objective of this phase is summarising the data collected
during the final selection and extraction phase, reporting the
process and the results in a clearly way, and publishing
them, providing the interested community with the gathered
evidence. This phase is compound of three activities as
follows:
1. Perform (meta) analysis: the data extracted in the previous
phase should be collected, analysed and summarised
following the procedures specified and approved in the
protocol. Kitchenham and Charters [21] mentioned that this
synthesis can be descriptive, but advices that quantitative
synthesis (i.e. meta-analysis) should be performed as always
as possible. Cruzes and Dyba [29] present guidelines on
how to conduct descriptive synthesis and Brereton et al. [1]
show guidelines of how to conduct meta-analysis in SE.
† Filtering: accomplishing this activity may also require a
filter, which is mainly intended to be applied during SR
updates and extensions. It comprises the revision of the
data on which (meta) analysis is performed. This is
required in order to keep only valid, non-obsolete
evidence in the data containers (e.g. tables, spreadsheets
and forms).
Fig. 6 Checklist for extraction approval
303
& The Institution of Engineering and Technology 2013
www.ietdl.org
2. Report results: the goal of the report is to make the
knowledge acquainted and the evidences found in the
primary studies explicit. During the report construction, the
researcher should ensure that the research questions are
being answered through clear results. Journals, conference
papers and technical reports are the most common formats
used to report the results.
3. Publish results: after preparing the report it should be
published to potentially interested parties. A way to select
adequate discussion forum for publishing the report is to
take into account the same forums considered by the SR
selected studies.
4.5 Packaging
Packing the SR means to group all necessary information for
replicating the review and making it auditable. Hence, all
artefact generated in all phases should compose such
package, as well as all procedures adopted for conducting
each activity. Any detail and any decision making should
be critically analysed for not becoming tacit knowledge
causing interference on the review replicability. Owing to
this, packaging is an action that must follow the whole
process as well as software quality assurance activities must
follow the software development. Particularly, some
protocol fields aim to guarantee that important procedures
(e.g. the initial selection procedure) are registered. Hence,
the quality of the protocol is very important since it is an
important item of the package.
The report that contains the results analysis is also part of
the package, since the transfer of knowledge acquired as a
result of the review is one of the goals of the review itself.
Therefore, aiming to externalise the importance of
packaging, it is an element that ‘involves’ all circles that
represent each phase of the process, as depicted in Fig. 1.
5 Evaluation
In this section, we summarise the results of systematic
literature reviews we have performed in the past few years.
For each SR herein described, we highlight the activities
described in the I-SR process that were fundamental for
achieving relevant results. When applicable, we also
contrast I-SR with other processes available in the literature.
5.1 Systematic review on testing of
aspect-oriented software
We have conducted a SR to characterise the state of the art in
testing of aspect-oriented (AO) software [30]. This SR,
hereafter called ‘AO testing SR’, has been updated four
times with support of the I-SR process described in this
paper. It had a main goal: identifying testing approaches
and fault types researchers and practitioners have been
investigating for AO software. Surveyed repositories are
IEEE eXplore, ACM Digital Library and others that are
relevant to the computer science area. The final selection
comprises 34 primary studies.
As previously mentioned, we have updated the SR four
times. For the updating rounds, the filters defined in the
I-SR process were fundamental to yield a precise dataset on
which data analysis has been performed. In the updating
rounds, a filter applied during the ‘query all repositories’
activity (initial selection phase) helped us to clean up the
search results. Even though a search filter – related to the
304
& The Institution of Engineering and Technology 2013
year of publication – was applied in the search engine,
the retrieved set of primary studies was overlapping the
previous set of selected studies. This happens because the
filters by publication date are generally not fine-grained
enough (for example, filters do not consider the month of
publication, thus the whole year is considered). Therefore,
before moving to the next activity, the filter was applied to
the results in order to remove duplicates.
Another example from the ‘AO testing SR’ regards updated
versions of primary studies. In an updating round, two
previously selected primary studies became obsolete
because of a more recent article published by the same
authors. Such up-to-date article revisited a set of testing
criteria proposed in previous conference papers by the
authors. By applying a filter in the ‘perform initial
selection’ activity (initial selection phase), the original
papers were removed from the dataset.
During the ‘perform (meta) analysis’ activity (synthesis
phase), we noted that research results published in a
primary study subsume previous results published by the
same authors. In these pieces of work, the authors
incrementally established a fault taxonomy for AO
programs. In the first SR round, in which the two first
primary studies were selected, the fault types were extracted
and used to compose a mapping among several fault
taxonomies. In the SR update round, the last publication led
to a revision of the map in order to include only the most
recent descriptions of fault types.
Note that applying filters as in the described examples have
saved time that would otherwise be wasted, and yielded a
more precise set of primary studies, since either some
studies were already present in the previous dataset or have
been updated by the original authors. Although Kitchenham
[17] states that this overlapping of primary studies must be
avoided in an SR, particular situations in which filters
should be applied are not documented in the existing
literature of SR process [17, 18, 28, 31, 32].
5.2 Systematic review on architectures for
embedded systems
A secondary study was conducted to help the comprehension
on how the research community has been investigating and
describing software architectures for embedded systems
Antonio et al. [33]. The basis string for running the
searches was (‘software architecture’ AND reference OR
model) AND (‘embedded system’ OR ‘embedded
software’). The surveyed repositories were ACM Digital
Library, IEEE eXplore and Scopus.
During the initial selection of primary studies, a new term
was identified – ‘real-time system’ – as frequently used in the
context of embedded systems. This led to a revision of the
search string, consisting of the addition of the new term,
and to the re-execution of the queries in all repositories.
The search performed with the revised string retrieved a
total of 573 primary studies. Applying inclusion and
exclusion criteria reduced the set to 104 items of interest.
Considering these 104 papers, 14 ones were retrieved
because of the additional term (i.e. real-time system).
Note that adding a new term allowed the authors to identify
a more comprehensive set of primary studies than the one
obtained with the original version of the string. The ‘check
new keywords’ (initial selection phase) of the I-SR process
addresses this addition of new key terms to the search
string. Similarly to the filtering activities illustrated in
IET Softw., 2013, Vol. 7, Iss. 6, pp. 298–307
doi: 10.1049/iet-sen.2013.0029

Section 5.1, this activity was not clearly defined in the
existing literature of SR process.
5.3 Systematic review on the use of visual data
mining (VDM) to support systematic reviews
The aim of this SR, hereafter called VDM-SR, was to
understand of the use of VDM techniques to support the SR
process, examining the types of techniques which may be
applied in each activity [34]. More specifically, the research
question was: ‘What evidence is there of VDM techniques
being applied to help in the SR process?’.
During the ‘perform final selection and pre-extraction’
activity (final selection and extraction phase in I-SR),
information necessary was collected using the data
extraction form, which was created and approved in the
planning phase. We collected information related to VDM
techniques and phases/activities of the SR process to which
VDM was applied. Although reading the studies and
collecting data, we found out that most of the papers were
not related to SE. Consequently, we decided to collect
additional information from the studies. In the ‘check new
data extraction fields and revise data extraction form’ (final
selection and extraction phase), we added a new field –
‘topic area’ – to the original data extraction form.
Consequently, the extraction was not approved and an
intra-phase iteration was necessary (from ‘approve
extraction’ to ‘refine extraction’, in the final selection and
extraction phase).
We reviewed 20 papers and our results indicate a scarcity
of research on the use of VDM to help with conducting
SRs in the SE domain. However, most of the studies (16 of
20) have been conducted in the field of medicine and they
revealed that the activities of data extraction and data
synthesis have more VDM support than other activities.
Although Kitchenham [17] mentions that the data
extraction is an iterative activity, she does not exemplify
the consequences of the insertion or absence of fields in
the data extraction form. The VDM-SR shows that the
information acquired with the new field topic area was
essential to conclude that there is a scarcity of research in
SE area and to signal an opportunity for future research.
5.4 Systematic review of experimental studies in
software inspection
The main goal of this study was to identify in literature which
software inspection processes were investigated through
experimental studies. We surveyed four repositories:
SCOPUS, IEEE eXplore, ACM Digital Library and Web of
Science. The final selection comprises 79 primary studies.
During the ‘perform final selection and pre-extraction’
activity (final selection and extraction phase in I-SR), we
collected necessary information to answer the research
questions using the data extraction form. In the planning
phase, we expected that the techniques mentioned in the
studies would be related to reading techniques, which are
commonly used to identify defects in artefacts. However,
during the pre-extraction activity, we noted that authors of
primary studies mention techniques to support other
inspection process phases or experimental process phases
(for instance: defects classification, defects estimation and
selection of inspectors).
We figured out this information should be collected to
answer the research question accordingly. Hence, a new
IET Softw., 2013, Vol. 7, Iss. 6, pp. 298–307
doi: 10.1049/iet-sen.2013.0029
www.ietdl.org
field was added to the data extraction form: ‘Which process
phase (or activity) is supported by the technique?’, related
to ‘check new data extraction fields and revise data
extraction form activity’. As mentioned before, although
Kitchenham [17] mentions that the data extraction is an
iterative activity, the activity flow designed for the I-SR
process allowed us to improve the data extraction form to
collect new pieces of information and to yield a more
complete data synthesis.
6 Related work
Kitchenham [17] proposed a three-phase process for SR in
SE. In the planning phase, the need for a new SR is
identified, so the researchers define the SR objectives and
create the protocol. Conducting the review phase is divided
into two stages: (i) the ‘selection execution’ which
encompasses the acquisition and analysis of primary studies
and (ii) ‘information extraction’, in which relevant
information is extracted from the included studies. Finally,
in reporting the review phase, the results of the primary
studies that meet the SR purpose are summarised; the
synthesis of results can be either qualitative or quantitative,
that latter based on statistical analyses. Unlike the
Kitchenham’s process, our I-SR process externalises the
intra- and inter-phase iterations that compose the SR process.
Kitchenham [17] also proposed guidelines to perform SR
in SE. In addition, Biolchini et al. [18] developed a
template protocol which facilitates SR planning and
execution in SE. The I-SR process introduces five
checkpoints which are not described in those previous
reports. One checkpoint was established to guarantee that
the search string is suitable. The pilot search must be
evaluated and if problems are found, the researchers must
return to revise keywords. Similarly, if problems regarding
the protocol, selection of primary studies, extraction forms,
extraction procedures are found, the I-SR processor and
some of its phases and/or activities may be re-executed.
In a previous work [28], we adapted the SR process
proposed by Biolchini et al. [18] to allow us to perform SR
updates. The adaptation concerned the update planning and
results filtering and merging. The filters proposed for the
I-SR process presented in this paper were inspired by
the filters proposed in our previous work. In spite of this,
the I-SR includes several other activities and iterations not
included in the mentioned processes [18, 28]. Examples are
the identification of new keywords, which implies a return
to the planning phase, the approval related activities and its
associated intra-phase iterations.
Molléri et al. [32] showed the SR process proposed by
Kitchenham [17] using the business process model notation.
The authors detailed the activities that compose the process,
the produced artefacts, participants involved and the
communication mechanisms. The process was formalised
through a group of elements, such as events, gateways and
communication flows. The representation created by Molléri
et al. can be used to guide the development of
computational tools to support the SR process. Similarly,
our work also represents the SR process. However, our
focus is not to represent the process in a formal notation,
but highlighting the iterative sequence flows of the
activities. Therefore we adopted arrows, which are useful
symbols to represent iterations along the process.
305
& The Institution of Engineering and Technology 2013
www.ietdl.org
7 Conclusions
SR is an important component to identify and aggregate
research evidence related to a specific research theme.
Despite their relevance, there is a lack of work that explains
in details the SR process. Owing to this fact, this paper
transformed tacit knowledge into explicit knowledge,
externalising the way that the SR process has been
conducted in practice.
Examples of factors externalised in the I-SR process,
described in this paper, were the iterations imposed by the
SR process, which are essential for maintaining the SR
quality. These iterations can occur at intra- and inter-phase
levels, and are triggered, in general, by unforeseen
situations during the planning phase. Examples are the
identification of new keywords for search strings or new
data extraction fields, which can be identified as the
researchers go deeply into the investigated theme (after the
initial selection of papers or data extraction activities).
Frequently, they impose updates in the protocol, which is
available during all tasks of each phase, as showed in the
I-SR process diagram (Fig. 1). Although iterative activities
were already mentioned by some authors [17, 18, 28, 31,
32], the I-SR process explicitly presents the moments when
these iterations happen and should be considered during the
execution of the SR process. Note that although the
inter-phase and intra-phase iterations have been observed in
practice in the studies described in Section 5, the process
should be applied by other researchers aiming to obtain
quantitative evidence of its adequacy.
We highlighted that besides the protocol, the packaging is
also present all the time, evincing that all information or
artefact generated during the review are important for
allowing review audit and replication. Particularly, there are
some protocol fields that must register essential procedures
for making the review replicable, thus not leaving important
information as tacit knowledge.
Note that systematic mappings (SMs) [31] follow a process
that is similar to an SR process. Therefore the process here
presented can be easily adapted for SM conduction. In the
context of our research groups, SM has been executed
through such process and some examples used in this paper
came from them.
As future work, we intend to apply a survey to identify SE
researchers’ experience in applying SR. This shall enable us
to enhance the quality of the I-SR process. Finally, we
consider that this paper can be used as a guide for novice
researchers as well as for researchers who intent to
understand the SR process deeply.
8 Acknowledgments
The authors thank the financial support received from
FAPESP (grant 2012/02524-4), INEP (Observatório da
Educação, project 3280) and CAPES, Brazil. The authors
also acknowledge the support granted by CNPq and
FAPESP, Brazil, to the INCT-SEC (National Institute of
Science and Technology – Critical Embedded Systems,
Brazil, grants 573963/2008-8 and 08/57870-9).
9 References
1 Brereton, P., Kitchenham, B., Budgen, D., Turner, M., Khalil, M.:
‘Lessons from applying the systematic literature review process within
the software engineering domain’, J. Syst. Softw., 2007, 80, (4),
pp. 571–583
306
& The Institution of Engineering and Technology 2013
2 Kitchenham, B., Brereton, P., Turner, M., et al.: ‘Refining the systematic
literature review process – two participant-observer case studies’, Empir.
Softw. Eng. (ESE), 2010, 15, (6), pp. 618–653
3 Zhang, H., Muhammad, A.: ‘An empirical investigation of systematic
reviews in software engineering’. Proc. Fifth Int. Symp. Empirical
Software Engineering and Measurement (ESEM), 2011, pp. 1–10
4 Petersen, K., Nauman, B.: ‘Identifying strategies for study selection in
systematic reviews and maps’. Proc. Fifth Int. Symp. Empirical
Software Engineering and Measurement (ESEM), 2011, pp. 1–10
5 Mendes, E.: ‘A systematic review of web engineering research’. Proc.
Fourth Int. Symp. Empirical Software Engineering (ISESE), 2005,
pp. 498–507
6 Hannay, J.E., Sjøberg, D.I.K., Dybå, T.: ‘A systematic review of theory
use in software engineering experiments’, IEEE Trans. Softw. Eng.,
2007, 33, (2), pp. 87–10
7 MacDonell, S.G., Shepperd, M.J.: ‘Comparing local and global software
effort estimation models – reflections on a systematic review’. Proc. First
Int. Symp. Empirical Software Engineering and Measurement (ESEM),
2007, pp. 401–409
8 Sulayman, M., Mendes, E.: ‘A systematic literature review of software
process improvement in small and medium web companies’.
Advances in Software Engineering, vol. 59 of Communications in
Computer and Information Science, 2009, pp. 1–8
9 Dieste, O., Juristo, N.: ‘Systematic review and aggregation of empirical
studies on elicitation techniques’, IEEE Trans. Softw. Eng., 2011, 37,
(2), pp. 283–304
10 Salleh, N., Mendes, E., Grundy, J.: ‘Empirical studies of pair
programming for CS/SE teaching in higher education: a systematic
literature review’, IEEE Trans. Softw. Eng., 2011, 37, (4), pp. 509–525
11 Dieste, O., Padua, A.: ‘Developing search strategies for detecting
relevant experiments for systematic reviews’. Proc. First Int. Symp.
Empirical Software Engineering and Measurement (ESEM), 2007,
pp. 215–224
12 Bailey, J., Zhang, C., Budgen, D., Turner, M., Charters, S.: ‘Search
engine overlaps: do they agree or disagree?’. Proc. Second Int.
Workshop on Realising Evidence-Based Software Engineering
(REBSE), 2007, pp. 1–6
13 Dieste, O., Griman, A., Juristo, N.: ‘Developing search strategies for
detecting relevant experiments’, Empir. Softw. Eng., 2009, 14, (5),
pp. 513–539
14 Kitchenham, B., Brereton, P., Turner, M., et al.: ‘The impact of limited
search procedures for systematic literature reviews – a
participant-observer case study’. Proc. Third Int. Symp. Empirical
Software Engineering and Measurement (ESEM), 2009, pp. 336–345
15 Zhang, H., Muhammad, A.: ‘Systematic reviews in software
engineering: an empirical investigation’, Inf. Softw. Technol., 2013,
55, (7), pp. 1341–1354
16 Riaz, M., Sulayman, N., Salleh, M., Mendes, E.: ‘Experiences
conducting systematic reviews from Novices’ perspective’. Proc. 14th
Int. Conf. Evaluation and Assessment in Software Engineering
(EASE), BCS-eWiC, 2010, pp. 1–10
17 Kitchenham, B.: ‘Procedures for performing systematic reviews, joint
technical report TR/SE-0401 (Keele) – 0400011 T.1 (NICTA)’.
Software Engineering Group – Department of Computer Science –
Keele University and Empirical Software Engineering – National ICT
Australia Ltd., 2004
18 Biolchini, J., Mian, P., Natali, A., Travassos, G.: ‘Systematic review in
software engineering: relevance and utility’. Technical report, PESC/-
COPPE/UFRJ, 2005
19 Staples, M., Niazi, M.: ‘Experiences using systematic review
guidelines’, J. Syst. Softw., 2007, 80, (1), pp. 1425–1437
20 Stapic, S., LÓpez, E.G., Cabot, A.G., Ortega, d.M., Strahonja, L.V.:
‘Performing systematic literature review in software engineering’.
Central European Conf. Information and Intelligent Systems (CECIIS’
12), 2012, pp. 441–447
21 Kitchenham, B., Charters, S.: ‘Guidelines for performing systematic
literature reviews in software engineering’. Technical report, EBSE
2007-001, Keele University and Durham University, UK, 2007
22 Zhang, H., Muhammad, P., Tell, A.B.: ‘Identifying relevant studies in
software engineering’, Inf. Softw. Technol., 2011, 53, (1), pp. 625–637
23 Boell, S., Cezec-Kecmanovic, D.: ‘Are systematic reviews better, less
biased and of higher quality?’. Proc. 19th European Conf. Information
Systems (ECIS), Helsinki, Finland, 2011
24 Dybå, T., Dingsøyr, T., Hanssen, G.: ‘Applying systematic reviews to
diverse study types: an experience report’. Proc. First Int. Symp.
Empirical Software Engineering and Measurement (ESEM), 2007,
pp. 225–234
25 Kitchenham, B., Brereton, O., Owen, S., Butcher, J., Jefferies, C.:
‘Length and readability of structured software engineering abstracts’,
IET Softw., 2008, 2, (1), pp. 37–45
IET Softw., 2013, Vol. 7, Iss. 6, pp. 298–307
doi: 10.1049/iet-sen.2013.0029

26 Dybå, T., Dingsøyr, T.: ‘Strength of evidence in systematic reviews
in software engineering’. Proc. Second ACM-IEEE Int. Symp.
Empirical Software Engineering and Measurement (ESEM), 2008,
pp. 178–187
27 Dieste, O., López, M., Ramos, F.: ‘Formalizing a systematic review
updating process’. Proc. Sixth Conf. Software Engineering Research,
Management and Applications (SERA), 2008, pp. 143–150
28 Ferrari, F., Maldonado, J.: ‘Experimenting with a multi-iteration
systematic review in software engineering’. Proc. Fifth Experimental
Software Engineering Latin America Workshop (ESELAW), Salvador,
Brazil, 2008, pp. 1–10
29 Cruzes, D., Dyba, T.: ‘Recommended steps for thematic synthesis in
software engineering’. Proc. Fifth Int. Symp. Empirical Software
Engineering and Measurement (ESEM), 2011, pp. 275–284
30 Ferrari, F.C.: ‘A contribution to the fault-based testing of aspect-oriented
software’. PhD thesis, Instituto de Ciências Matemáticas e de
IET Softw., 2013, Vol. 7, Iss. 6, pp. 298–307
doi: 10.1049/iet-sen.2013.0029
www.ietdl.org
ComputaÇão, Universidade de São Paulo (ICMC/USP), São Carlos/
SP – Brasil, 2010
31 Petersen, K., Feldt, R., Mujtaba, S., Mattsson, M.: ‘Systematic mapping
studies in software engineering’. Proc. 12th Int. Conf. Evaluation and
Assessment in Software Engineering (EASE), BCS-eWiC, University
of Bari, Italy, 2008, pp. 1–10
32 Molléri, J.S., Silva, L.E., Benitti, F.B.V.: ‘Proposal of an automated
approach to support the systematic review of literature process’. Proc.
25th Conf. Software Engineering & Knowledge Engineering (SEKE),
2013, pp. 488–493
33 Antonio, E.A., Ferrari, F.C., Fabbri, S.C.P.F.: ‘A systematic mapping of
architectures for embedded software’. Proc. Second Brazilian Conf.
Critical Embedded Systems (CBSEC), 2012, pp. 18–23
34 Felizardo, K., MacDonell, S., Mendes, E., Maldonado, J.: ‘A systematic
mapping on the use of visual data mining to support the conduct of
systematic literature reviews’, J. Softw. (JSW J.), 2011, 7, (2), pp. 450–461
307
& The Institution of Engineering and Technology 2013