Conducting Systematic

Literature Review
Lecture 3
Systematic literature review
• A systematic literature review (often referred to as a systematic
review) is a means of identifying, evaluating and interpreting all
available research relevant to a particular research question, or topic
area, or phenomenon of interest.
• Individual studies contributing to a systematic review are called
primary studies; a systematic review is a form of secondary study.
Reasons for Performing Systematic
Literature Reviews

• To summarise the existing evidence concerning a treatment or

technology
• e.g. to summarise the empirical evidence of the benefits and limitations of a
specific agile method.
• To identify any gaps in current research in order to suggest areas for
further investigation.
• To provide a framework/background in order to appropriately
position new research activities.
Features of Systematic Literature Reviews

• Clearly stated title and objectives

• Comprehensive strategy to search for relevant studies (unpublished and
published)
• Explicit and justified criteria for the inclusion or exclusion of any study
• Clear presentation of characteristics of each study included and an analysis
of methodological quality
• Comprehensive list of all studies excluded and justification for exclusion
Features of Systematic Literature Reviews

• Clear analysis of the results of the eligible studies

• statistical synthesis of data (meta-analysis) if appropriate and
possible;
• or qualitative synthesis

• Structured report of the review clearly stating the aims, describing

the methods and materials and reporting the results
Systematic Literature Review(SLR) Process
Variations of Reviews
• Systematic literature review (also referred to as a systematic review).
A form of secondary study that uses a well-defined methodology to
identify, analyse and interpret all available evidence related to a
specific research question in a way that is unbiased and (to a degree)
repeatable.
• Systematic mapping study (also referred to as a scoping study). A
broad review of primary studies in a specific topic area that aims to
identify what evidence is available on the topic.
The Review Process
• Planning the Review.

• Conducting the Review.

• Reporting the Review.

Planning the Review.

• Identification of the need for a review.

• Commissioning a review.
• Specifying the research question(s).
• Developing a review protocol.
• Evaluating the review protocol.
Conducting the review

• Identification of research.
• Selection of primary studies
• Study quality assessment.
• Data extraction and monitoring.
• Data synthesis.
Reporting the review

• Specifying dissemination mechanisms.

• Formatting the main report.
• Evaluating the report.
Need for SLR example real world
• Kitchenham et al. argued that accurate cost estimation is important
for the software industry; that accurate cost estimation models rely
on past project data; that many companies cannot collect enough
data to construct their own models. Thus, it is important to know
whether models developed from data repositories can be used to
predict costs in a specific company. They noted that a number of
studies have addressed that issue but have come to different
conclusions. They concluded that it is necessary to determine
whether, or under what conditions, models derived from data
repositories can support estimation in a specific company.
Determining the Research Question
• The structured question will determine the inclusion and exclusion
criteria:
• What is the population of interest?
• What are the interventions?
• What are the outcomes of interest?
• What study designs are appropriate?
Research question types in IT and computing.

• Assessing the effect of an Information technology.

• Taking stock of the state of the art in a problem domain.

• Investigating the success associated with an approach.

• Identifying the impact of technologies.

Review Protocol
• Background. The rationale for the survey.
• The research questions.
• Search Strategy.
• Selection Criteria and Procedures.
• Study quality assessment
• Data extraction strategy.
• Synthesis of the extracted data
• Project timetable
Search Strategy
• Formulating the search string.

• Generate terms linked with the RQs.

• list of synonyms, abbreviations, and alternative spellings

• Use Boolean operators(e.g. AND, OR, NOT) to construct the string.

Search Strategy
• Identify Libraries and databases

• Tailoring the search string for specific databases.

• Examples of digital libraries

• IEEExplore
• ACM Digital library:
• Google scholar (scholar.google.com)
• Citeseer library (citeseer.ist.psu.edu)
• Inspec (www.iee.org/Publish/INSPEC/)
• ScienceDirect (www.sciencedirect.com)
• Dbnlp
• Scopus
Study Selection

• Identify those primary studies that provide direct evidence about the
research question.

• Generate inclusion and exclusion criteria.

• Provide a selection ‘algorithm’.

Data Extraction
• Data to be extracted should be agreed upon a priori.
• Information will need to be collected relating to:
▪ Methodology
▪ Population( data type or problem area, Language for example)
▪ Approaches/Algorithms/Techniques being compared
▪ Outcomes evaluated
• Design a review-specific data extraction form.

• it should be performed by at least two independent readers.

Quality Assessment

• Ensure that retrieved studies follow scientific method when analyzing

and reporting data.

• What minimum quality criteria that they must contain?

Synthesis of Data

• Building up; putting together; making a whole out of the parts;

the combination of separate elements of thought into a whole;
reasoning from principles to a conclusion
• Narratively and/or statistically summarise/describe the data,
exploring similarities and differences between studies.
Writing and Communicating of the Review

• Articulate your findings by synthesizing and Abstracting.

• This can led to a new taxonomy framework sometimes.

• Clearly state the aims, describing the methods and materials and
reporting the results

• Review details and results are disseminated to relevant target

audiences using appropriate formats
Systematic Literature Review(SLR) Process