See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/223327265

Software engineering article types: An analysis of the literature. Journal of

Systems and Software, 81(10), 1694-1714

Article  in  Journal of Systems and Software · October 2008

DOI: 10.1016/j.jss.2007.11.723 · Source: dx.doi.org

CITATIONS READS

34 1,143

2 authors:

Michela Montesi Patricia Lago

Complutense University of Madrid Vrije Universiteit Amsterdam, Amsterdam, Netherlands
53 PUBLICATIONS   223 CITATIONS    272 PUBLICATIONS   4,172 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

HAPPYNESS: Emotion-aware sustainable service quality assurance View project

Wise/EU View project

All content following this page was uploaded by Michela Montesi on 05 April 2019.

The user has requested enhancement of the downloaded file.

This article appeared in a journal published by Elsevier. The attached
copy is furnished to the author for internal non-commercial research
and education use, including for instruction at the authors institution
and sharing with colleagues.
Other uses, including reproduction and distribution, or selling or
licensing copies, or posting to personal, institutional or third party
websites are prohibited.
In most cases authors are permitted to post their version of the
article (e.g. in Word or Tex form) to their personal website or
institutional repository. Authors requiring further information
regarding Elsevier’s archiving and manuscript policies are
encouraged to visit:
http://www.elsevier.com/copyright
 Author's personal copy

Available online at www.sciencedirect.com

The Journal of Systems and Software 81 (2008) 1694–1714

www.elsevier.com/locate/jss

Software engineering article types: An analysis of the literature

Michela Montesi a, Patricia Lago b,*
a
Department of Archives and Information Science, University of Amsterdam, Turfdraagsterpad 9, 1012XT Amsterdam, The Netherlands
b
Department of Computer Science, VU University Amsterdam, De Boelelaan 1081a, 1081 HV Amsterdam, The Netherlands

Received 9 February 2007; received in revised form 19 November 2007; accepted 25 November 2007
Available online 3 December 2007

Abstract

The software engineering (SE) community has recently recognized that the field lacks well-established research paradigms and clear
guidance on how to write good research reports. With no comprehensive guide to the different article types in the field, article writing and
reviewing heavily depends on the expertise and the understanding of the individual SE actors.
In this work, we classify and describe the article types published in SE with an emphasis on what is required for publication in journals
and conference proceedings. Theoretically, we consider article types as genres, because we assume that each type of article has a specific
function and a particular communicative purpose within the community, which the members of the community can recognize. We draw
on written sources available, i.e. the instructions to authors/reviewers of major SE journals, the calls for papers of major SE conferences,
and previous research published on the topic.
Despite the fragmentation and limitations of the sources studied, we are able to propose a classification of different SE article types.
Such classification helps in guiding the reader through the SE literature, and in making the researcher reflect on directions for
improvements.
Ó 2007 Elsevier Inc. All rights reserved.

Keywords: Article types; Article genres; Requirements for publication; Software engineering; Survey

1. Introduction the final stage of the research process and summarize it

In this study, we classify the types of articles or article
genres commonly published in software engineering (SE)
journals and conferences, such as surveys, case studies or
experience reports. The SE community has recently
stressed the need for clearer criteria of evaluation and more
precise guidance on how to write different types of articles
(Shaw, 2003; Wieringa et al., 2005). Our wish is to make a
contribution to this problem, and thus improve our under-
standing of what makes each type of SE research article a
good SE research article.
Defining the types of articles published within a scien-
tific community is not easy, since it implies dealing with
epistemological issues as well. Research reports represent
*
Corresponding author. Tel.: +31 20 5987745; fax: +31 20 5987728.
E-mail addresses: M.Montesi@uva.nl (M. Montesi), patricia@cs.vu.nl
(P. Lago).
as a whole. Typically, the methodology used to carry out
the research and the type or genre of the corresponding
article coincide, such as in the circumstance of case studies.
Therefore, when classifying the article genres published in a
field, one takes also into account the methodological pat-
terns of research in that field.
By using the expression ‘‘article genres”, we conceive of
research articles as standardized documents. Intended
readers have certain expectations about their form, con-
tent, and communicative purpose, in the same way as
film-goers have precise expectations for film genres such
as thrillers, westerns, or comedies. By considering research
articles as genres, we also want to stress that they play a
determined role in the communication among the members
of the scientific community. Kwasnik and Crowston (2005)
point out that, for a document to be defined as a genre,
‘‘social acceptance” is needed. In other words, research
articles can be considered genres because the scientific

0164-1212/$ - see front matter Ó 2007 Elsevier Inc. All rights reserved.
doi:10.1016/j.jss.2007.11.723
 Author's personal copy
M. Montesi, P. Lago / The Journal of Systems and Software 81 (2008) 1694–1714
community recognizes them as such, and assign them spe-
cific communicative purposes. ‘‘Article genres” and ‘‘types
of articles” are synonyms in this study, and so are the
words ‘‘article” and ‘‘paper”. With these premises, we are
now able to illustrate what a classification of article genres
can mean for SE.
1.1. Article genres reflect the patterns of research and
communication
Article genres published in a discipline reflect both the
nature of research and the patterns of communication
between different actors in that discipline. In a previous
study, Montesi and Mackenzie-Owen (2008) compared
the article genres published in SE, Biology, and Education
scientific research journals, with results showing two inter-
related aspects. Firstly, it was found that SE journals do
not attach to the methodological section the importance
that education and biology journals do. In most cases,
SE journals consider a three-section structure for major
articles (Introduction – Main Body – Conclusion), whereas
biology and education journals consider rather a four-sec-
tion structure that includes a method section as well. Some-
thing similar was observed by Posteguillo (1999). On an
epistemological level, the omission of any reference to the
methodological section seems to connect with the claims
of those who speak of a scant recourse to proper validation
and experimentation in the field (e.g. Fenton, 1993; Tichy
et al., 1995; Glass, 1995; Zelkowitz and Wallace, 1998).
Secondly, the cited study (Montesi and Mackenzie-Owen,
2008) revealed that in SE and education, the publication
of certain types of articles, such as the tutorial articles,
accounted for different patterns of communication than
in biology. In particular, it pointed to a more composite
audience, more likely to include practitioners besides
researchers, and to differentiate more evidently between
specialist and non-specialist readers.
If article genres carry in fact information on the nature
of research in a field and on the preferential audience they
address, we could make the following assumption: by con-
sidering the type of article, it is possible to assess the impact
of research in the community, including, with certain limi-
tations, its degree of applicability. In this hypothetical per-
spective, certain article genres can be regarded as more
applicable because they are more explicitly addressed to
practitioners and/or non-specialists (such as tutorial arti-
cles, for instance). By contrast, we could consider some
others less applicable because they are not based on a
sound methodology (such as those lacking a method sec-
tion, for example). This, of course, should first be proved.
However, some authors already claim that some genres,
such as case studies and the field studies, have more poten-
tial to connect theory and practice (Ardimento et al., 2003;
Segal et al., 2005). Being conducted in industry settings,
they imply an interaction between researchers and practi-
tioners which should make them more relevant for practice.
1695
Communication with other members of the community
who are not primarily involved in research is a subject of
debate in other disciplines as well, such as education or
library and information science (Haddow and Klobas,
2004). It is also especially burning in SE, as some consider
it crucial for the progress of the discipline (Glass et al.,
2002). Some SE researchers wonder whether it would be
possible to strengthen the impact of research in industry
settings, by increasing empirical research and experimenta-
tion (Tichy, 1998; Budgen and Kitchenham, 2005; Zelko-
witz et al., 2003). Research based on experimentation
would allow practitioners to find project solutions on a val-
idated basis of knowledge (Basili, 1996). Others propose
that technology adoption decisions by practitioners may
be supported or improved by Evidence-Based Software
Engineering (EBSE) (Dyba et al., 2005). In this perspective,
it would be the empirical or experimental type of article the
most likely to influence practice. But there is as yet no evi-
dence to support such claim. In fact, Punter (2003) found
that the practitioners using a specific SE portal were inter-
ested in various types of information, including but not
limited to experience reports and results of empirical stud-
ies about specific techniques, methods and tools. They also
valued current research directions, references to further
sources of information, or overviews of SE techniques,
methods and tools, with no specific preference for any of
these items.
1.2. Article genres play a unique role in each discipline
In our opinion and following Hjørland (1998) and
Hjørland (2002), article genres are shaped to conform to
the unique needs of each discipline. This can be argued
to detriment of those who often rely on comparisons with
older disciplines such as medicine, to contend that more
experimental and/or empirical research should be under-
taken (Pickard and Kitchenham, 1998; Kitchenham et al.,
2002; Dyba et al., 2005). This is not a unanimous position,
however. Some, for instance, claim that SE needs ‘‘intelli-
gent debate” more than experimentation (Fletcher, 1995).
Fletcher argues that computer science has never been con-
cerned with natural phenomena and empirical predictions
such as physics or medicine (Fletcher, 1995). Marcos
(2005) stresses this point as well. In her opinion, formal
and empirical sciences deal with the study of existing phe-
nomena and objects, whereas SE deals with the study of
methods and techniques necessary to create new objects.
Thus, SE research methods resemble software development
methods, and tend to be mostly of a qualitative and crea-
tive nature. Finally, Miller (2004) criticizes the often advo-
cated recourse to statistical significance testing in SE.
Statistical significance testing, in his opinion, would be
unsuitable for SE experiments and studies for several rea-
sons, and in particular for being intended for hard sciences.
With regard to this issue, it is significant that, even when
methodologies and relative article names are adopted from
other disciplines, the meaning is different in SE. As far as
 Author's personal copy
1696 M. Montesi, P. Lago / The Journal of Systems and Software 81 (2008) 1694–1714
we know, for example, a case study in SE is different from a
case study in medicine or law, as reported by Perry et al.
(2004), and it is probably different from a case study in
medicine or psychology. In short, research patterns and
methodologies are different from a discipline to another
because each discipline tackles different problems, aims at
different objectives, and deals with different objects. With-
out denying that more experimentation and/or empiricism
may increase the degree of applicability of research, we
would like however to focus on traditional patterns of com-
munication between researchers and practitioners in SE,
and on how different article genres reflect them.
1.3. Article genres help deciding how a good article should be
Besides the lack of rigor in conducting empirical
research, and the absence of the well-established research
paradigms that characterize other fields of science and
engineering (Tichy et al., 1995), some claim that software
engineers do not have clear guidance on how to write good
reports (Shaw, 2003). These three problems are indeed
interrelated as the research report reflects the different
stages of the research work. But, in addition, a fourth prac-
tical dimension may be involved. Reviewers and program
committee members are short of means for properly judg-
ing the work of other members of the community. For
example, they can qualify research as good or bad for very
different reasons, because there aren’t agreed-on criteria for
judging the quality of research reports (Wieringa et al.,
2005). The problem of assessing the quality of scientific evi-
dence affects practitioners also, even though they seem to
be more worried about the relevance for practice than
about the rigor of published research (Dyba et al., 2005).
Defining articles as article genres may contribute to this
problem as well because we assume that the requirements
of quality established for each type of article should com-
ply with the communicative purpose of the same articles.
In Shaw’s words, ‘‘It seems reasonable to assume that if
authors were more consciously aware of typical paper
types, they would find it easier to write papers that pre-
sented their results and supporting evidence clearly”
(Shaw, 2002). In this perspective, a first step to take would
be to answer questions such as these: What is a good
research report? How should it be? How can one judge
it? It is very difficult to give a satisfactory and definitive
answer to such questions without the consensus of a great
part of the community, which requires time and great
effort. A possible alternative is to summarize and organize
the written information currently available on the matter.
And this is precisely the aim of this study.
2. Methodology
The present description of the major types of research
papers published in SE is based exclusively on existing writ-
ten sources, and in particular on the following:
(1) Published, relevant papers, such as, for example,
Zelkowitz and Wallace (1998) or Sjøberg et al.
(2005). For the sake of clarity, we intentionally avoid
taking into account some studies published on the
topic. Glass et al. (2002), for instance, provide a clas-
sification of research papers in SE, based on several
criteria, among which topic, research approach, and
research method. Nonetheless, we have the impres-
sion, corroborated by Glass et al.’s results, that their
scheme of classification doesn’t suit SE research com-
pletely well, because it has been designed for all the
three computing disciplines: SE, Information Systems
and Computer Science.
(2) The SE journals included in the Journal Citation
Reports (2006) in May 2006. The JCR are elaborated
based on citation and article counts, and thus list the
most frequently cited and highest impact journals in a
field. This means that the journals reported in the
JCR are highly cited and used by the researchers of
a certain field. At the time we consulted it, the JCR
for ‘‘Computer Science (category) – Software Engi-
neering (sub-category)” listed 76 journal titles. In par-
ticular, we studied the corresponding ‘‘Information
to Authors”, which is intended to offer submitting
authors relevant guidance on the writing of proper
SE papers. At the time we consulted them, not all
journals provided information relevant to the present
study. Those that are cited here with their abbrevi-
ated names are reported in Appendix A. Some of
the titles reported in the JCR correspond, strictly
speaking, to magazines more than to journals. How-
ever, we use the term journal indifferently, as all are
included in this list. Similarly, some readers could
argue that certain titles do not strictly belong to the
field of SE but to other close fields. As a matter of
fact, some consider the JCR, which are elaborated
by the Institute for Scientific Information (ISI), inad-
equate for the computing disciplines. According to
Sidiropoulos and Manolopoulos (2005), for example,
the scientific areas in which each field is divided by
the ISI remain static over the years, and are unable
to follow the dramatic evolution of computer science.
Others point out that the JCR should take in confer-
ence publications, in order to reflect properly citation
and publication patterns in the computing disciplines
(Moed and Visser, 2007). With their limitations, the
JCR are all the same taken here as a point of refer-
ence, because they are elaborated on the basis of cita-
tion patterns within a certain field. Complementary
information is provided in Section 3.
When available, we also studied the corresponding
information for Peer Reviewers. Among the tools
available for reviewers, the Genre Submission Guide
of the journal IEEE Software (IEEE Software Guide)
stands out (IEEE, 2006). The Guide describes in detail
ten different manuscript types, providing guidance in
the following aspects: purpose, expected readers, sub-
 Author's personal copy
M. Montesi, P. Lago / The Journal of Systems and Software 81 (2008) 1694–1714
mitted word length, potential review criteria, and, in
most cases, also the standard outline. When journal
titles cited in this study are followed by the letter R
(Reviewers), then the information originates from
the corresponding information for reviewers.
(3) The instructions to authors of other relevant journals
not included in the list of the Journal Citation Reports.
These relevant journals have been pointed by a group
of three interviewed SE researchers. Examples are
ACM Computing Surveys, and the Knowledge Engi-
neering Review, for the description of special types
of papers such as surveys and tutorial articles, and
Journal on Software and System Modeling (SoSyM)
and Software Process: Improvement and Practice.
(4) The calls for papers of major SE conferences, as sug-
gested by the three interviewed SE researchers. Calls
for papers of 2005, 2006 and a few of 2007 were stud-
ied, depending on the information available over the
Internet. All the conferences whose calls for papers
were studied are reported in Appendix B. As for the
journals, not all conferences provide relevant infor-
mation for this study.
As commented above, article genres tend to coincide
with the methodology adopted in the research. This is
why we will mention sometimes the methodological
options available in SE. Different classifications of SE
methods are available, the one of the Software Engineering
Body of Knowledge (SWEBOK, (Abran et al., 2004)),
Zelkowitz and Wallace’s classification of empirical valida-
tion methods (Zelkowitz and Wallace, 1998), or Glass
et al.’s categories (Glass et al., 2002). Nevertheless, the
focus in this paper is on article genres, and questions
related to methodology will be addressed only if relevant
to this focus.
3. Article genres published in SE
The article genres reported here are described in the
order shown in Table 1. A section introducing the general
requirements for publication precedes the detailed discus-
sion of each article type. The space devoted to each type
depends on the information available and is consequently
biased in that sense. For example, long sections are devoted
to empirical and experimental research due to the abun-
dance of recent contributions to the topic. It is also possible
that some article genres are not mentioned at all because
not reported significantly in the written sources studied.
In order to help the reader follow the development of the
subject, a box with a few summarizing sentences precedes
each section. In the discussion and conclusion sections,
the reader will find a summary of the major points dis-
cussed for each article genre. We also provide a few refer-
ences to published papers, which we consider as good
examples of the genres commented on.
Our classification of article genres may be of help to var-
ious actors in SE, and in various ways:
1697
Table 1
Overview of the types of articles
III.A. Publishable papers: general
III.B. Extended versions of conference papers
III.C. Empirical research reports
III.C.(1) Observational studies
III.C.(2) Case studies
III.C.(3) Field studies
III.C.(4) Experimental papers
III.C.(5) Meta-analyses
III.D. Experience papers
III.E. Theoretical papers
III.F. Tutorial articles
III.G. Surveys
III.H. Short papers
III.I. Papers oriented towards practice
III.I.(1) Educational papers
III.I.(2) ‘‘How to”/ tool report papers
III.I.(3) Application papers
III.I. Opinion papers
 The readers and searchers can use Table 1 as a reading
guide to better determine how certain article types can
answer their expectations. This is especially true if they
come to Software Engineering as outsiders, e.g. if they
are working in multi- or interdisciplinary fields, or if
they are young researchers, approaching publications
for the first time. Moreover, our classification provides
an overview of the SE publication types, and guidance
about how to read and write them, and how to put their
reported type of research into context.
 Experienced researchers can get inspiration for further
reflection and further improvements. To this aim, the
summarizing box attached to each section about the
article types is named ‘‘reflection box”.
3.1. General requirements for publishable papers
Box 1 Reflection box on: general requirements for
publishable papers
– Regular papers may be shaped as different genres,
such as, for example, experimental papers, case
studies, or surveys.
– A typical SE paper has a three-section structure:
Introduction – Main Body – Conclusion.
– Appropriate style must be comprehensible.
– Relevance for practice, novelty, and originality are
the most important requirements for publication.
Papers published in journal main sections, usually called
‘‘regular papers” or sometimes also ‘‘original archival
papers”, may include different genres. IEEE Software, for
example, mentions ‘‘industry experience reports”, ‘‘case
studies”, and ‘‘tool reports”. Computer speaks also of ‘‘opin-
ion pieces” and ‘‘solutions to technical problems”. Other
journals include in this group of papers ‘‘tutorials”, ‘‘sur-
veys”, ‘‘review papers”, ‘‘theoretical papers”, ‘‘standards
discussions” (IEEE Micro), and ‘‘experimental papers”.
 Author's personal copy
1698 M. Montesi, P. Lago / The Journal of Systems and Software 81 (2008) 1694–1714
In the context of conferences, research reports are often
called ‘‘technical papers” or ‘‘technical solution papers”
(ESEC/FSE 2007, RE 2006, for example), but still the range
of article genres is fairly broad. At EASE 2006, for example,
technical papers may deal with ‘‘any aspect of product and
process evaluation and assessment, both qualitative and
quantitative, including: experiments (laboratory and field),
replication of empirical studies, case studies, surveys, obser-
vational studies, field studies, action research, evaluation
methodology, systematic reviews and meta-analysis”. On
the other hand, at ASE 2006 the only distinguishing feature
of technical papers is simply their describing innovative
research. Other conferences speak rather of full papers in
contrast with short papers (ISESE 2006 and CSMR 2006,
for example), or of scientific papers in contrast with experi-
ence papers (MoDELS 2006). Full papers are still widely
comprehensive in terms of topic and approach.
Regarding the layout of standard SE papers, findings of
a previous study (Montesi and Mackenzie-Owen, 2008),
revealed that the structure of SE regular papers may be
slightly simpler than in other fields. Following the instruc-
tions to authors of the SE journals reported in the JCR, a
paper in SE consists generally of only three clearly distin-
guishable parts: an introduction, a result section, some-
times called simply ‘‘main body” (IBM Syst J), and a
conclusion section. The method section is almost never
mentioned. This three-section structure is rarely described
further, although the headings that mark it are reflected
and confirmed in the elements of content to be reported
in abstracts. Sometimes, however, the instructions to
authors furnish indications on the content of the different
sections. The introduction, for instance, should tell why
the topic is important, and define a thesis or a problem.
In Computer, ‘‘Articles must have sufficient introductory
material; at least 15 percent of article length should be tuto-
rial [. . .]. A brief literature survey does not satisfy this
requirement”. In the conclusions, on the other hand,
authors outline future goals and lessons learned. Supple-
mentary material may complete the article, and can be pub-
lished exclusively online. In IEEE T Software Eng, for
example, supplementary material takes in proofs, code,
experimental data, short movies, appendices, animations
and audio files, and it is published on the Digital Library
with a pointer in the printed version of the paper.
Much emphasis is often placed on style and comprehen-
sibility. ‘‘Articles should be understandable to a broad
range of developers who require information they can
apply in their daily work. Writing should be down-to-
earth, practical, original, and comprehensible to all read-
ers, regardless of their specialty” (IEEE Software). To
improve comprehensibility, contributors should make use
of examples and visual material, such as figures, tables, dia-
grams, charts and photographs, define technical terms, and
avoid both jargon and academic language. For the same
reason, journals may recommend the use of appendixes
for detailed mathematical discussions, long intricate
proofs, or theorems (J Math Imaging Vis), as well as boxed
text or bullet points to better visualize the message of the
text (IEEE Software Guide).
The first requirement for a paper to be accepted and
published is to conform to the individual journal policy
about organization and style. In addition, authors should
also meet other requirements. The J ACM and other
ACM journals, for example, put forward three general cri-
teria for acceptance:
‘‘(1) the paper must be among the best papers of the year
in its area,
(2) the paper must be of interest to the broad
community,
(3) the presentation must be effective”.
A paper to be published in the ACM T Softw Eng Meth
should focus on ‘‘important research topics” and propose
‘‘reproducible, extensible, scalable” results, which are also
relevant to practice. Relevance to practice is particularly
stressed in connection with theoretical papers, but it is also
a desirable requirement for other sorts of papers in many
journals. It is crucial that papers indicate clearly what is
‘‘new and significant” of the work presented, and that they
go into as much details as possible, so that they can teach
something to the readers (Dr Dobbs J). Among the desirable
attributes of a good paper, two are especially stressed by
most journals, i.e. originality and novelty: ‘‘A paper may
describe original work, discuss a new technique or applica-
tion, or present a survey of recent work in a given field”
(IBM Syst J). Other attributes are: importance, significance
and relevance; correctness; technical soundness and preci-
sion; effective presentation, understandability and readabil-
ity. Authors should be able to address a broad audience,
and never forget about their potential readers while writing
an article (Commun ACM). ICSE 2006 and other confer-
ence guidelines put some emphasis on the ability to address
a software engineering audience with different levels and
areas of expertise, and on the quality of the presentation
(ASWEC 2007). Technical papers at ESEC/FSE 2007 must
clearly identify ‘‘their contribution to the state of the art
and/or their potential usefulness for industry”.
The instructions of IEE Proc-Softw, differently from most
journals, explain what the major attributes and requirements
of good papers should mean, shown in Table 2.
The importance of some of these requirements such as
comprehensibility and applicability is confirmed in
Anthony’s study (Anthony, 1999). Comparing SE intro-
ductions to the standard model of a research article in
other disciplines, he found that characterizing features of
SE introductions are, among others, a section with exam-
ples and definitions, and an evaluation of the research in
terms of application and novelty of the results.
Journals may also address peer reviewers, and provide
them with information in order to judge submitted articles.
Such information is usually very similar to that provided to
authors. The ACM T Softw Eng Meth -R asks reviewers to
check if acceptable papers contain: (a) motivation and clear
 Author's personal copy

M. Montesi, P. Lago / The Journal of Systems and Software 81 (2008) 1694–1714 1699

Table 2
Excerpt from the instructions of IEE Proc-Softw
Originality: is the work scientifically rigorous, accurate and novel? Does the work contain significant additional material to that already published?
Has its value been demonstrated?
Relevance: is the material appropriate to the scope of the journal to which it is submitted?
Motivation: does the problem considered have a sound motivation? Does the paper clearly demonstrate the scientific interest of the results?
Referencing: has reference been made to the most recent and most appropriate work? Is the present work set in the context of the previous work?
Clarity: is the English clear and well written? Poorly written English may obscure the scientific merit of your paper. Are the ideas expressed clearly
and concisely? Are the concepts understandable?
Length: unless previously agreed with the journals Managing Editor, all submissions must conform to the IEE Proceedings Length Policy
document

objectives, (b) demonstration of novelty and superiority
compared to related work, (c) evidence of effectiveness,
reproducibility, scalability, and practical relevance.
Reviewers should also ascertain if papers meet quality cri-
teria such as conciseness and technical precision, the con-
sistent use of terminology, and the definition of key
concepts. Focus and limitation of length must often be
evaluated together, as the ACM T Database Syst R sug-
gests: ‘‘TODS would like to discourage excessively long
papers (longer than 50 doubled-spaced pages including fig-
ures, references, etc.) and unnecessary digressions even in
shorter papers. This should help authors to focus on the
most important aspects of their submission [. . .]”. For the
rest, practically the same quality criteria as for authors
are brought up, such as clarity, readability and the ability
to target as broad an audience as possible.
Studying the abstracts of papers published at ICSE
2002, Shaw (2003) describes how accepted research papers
answered questions such as: What was your contribution?
What is your new result? How is your result different from
and better than prior work?, etc. According to Shaw, when
it comes to evaluating submitted papers, programme com-
mittees look for interesting, novel, and exciting results. In
addition, as SE knowledge grows incrementally, like in
other areas of science and engineering, programme com-
mittees are very interested in the authors’ interpretation
of prior work.
3.2. Extended versions of conference papers
Box 2 Reflection box on: extended versions of confer-
ence papers
– Extended versions of conference papers are a typi-
cal genre of SE.
– Extended versions have passed through a double
round of reviews.
– Many journals require that approximately one third
of the content be new.
– Publication on a journal allows discussing the
research beyond the time and space limitations set
by conferences.
A special type of articles are those based on one or
more papers previously presented at a workshop, a sym-
posium or a conference. For instance, van Vliet (2006)
is an extended version of van Vliet (2005). In the study
previously mentioned (Montesi and Mackenzie-Owen,
2008), comparing article genres in SE, education and
biology, it was found that extended versions of confer-
ence papers played a role only in SE. A distinguishing
feature of extended papers is the double round of
reviews they need to pass before journal publication
(i.e. conference and journal reviews). In a survey with
authors of extended versions and journal editors, Mon-
tesi and Mackenzie-Owen (accepted for publication)
found that the extension process may change depending
on the type of conference paper to extend (empirical,
theoretical, . . .).
Some journal instructions to authors regulate the pub-
lication of these articles. Among the other conditions, one
third of the content approximately must be new and ori-
ginal, and sometimes authors must submit both versions,
a description of the differences between the submitted
manuscript and the preliminary version, and reference
the previous version in the paper (SoSyM). Papers pub-
lished as such must be ‘‘important” (Theor Pract Log
Prog), ‘‘outstanding” (ACM T Database Syst), and ‘‘of
particularly high quality” (J Comput Sci Technol). New
material takes in explorations of further research, theo-
rems, proofs and/or implementation details (ACM T
Graphic).
If journals address reviewers the general tone doesn’t
change, though they offer some more clues on what to look
for. In the ACM T Database Syst -R, for instance, extended
versions of conference papers ‘‘should thoroughly consoli-
date the material, should extend it to be broader, and
should more carefully cover related research”. In addition,
the novel approach should be described more deeply, and
alternatives should also be considered more comprehen-
sively. In IEEE T Vis Comput Gr -R, while no new results
are required, key ideas, examples, and further elaboration
are expected. This confirms Shaw’s observation that in con-
ferences time and space limitations do not allow authors to
explore all the possible results of their research (Shaw,
2003).
 Author's personal copy
1700 M. Montesi, P. Lago / The Journal of Systems and Software 81 (2008) 1694–1714
3.3. Empirical research papers
Box 3 Reflection box on: empirical research papers
– Empirical research includes different research strat-
egies such as case studies, experimental research, or
qualitative research.
– Clear and comprehensive guidelines on how to
write a proper empirical research paper are not
yet available.
– The experimental context is considered important
for all types of empirical research.
– Empirical research is increasing in SE.
As defined by Basili et al. (1999), ‘‘an empirical study, in
a broad sense, is an act or operation for the purpose of dis-
covering something unknown or of testing a hypothesis,
involving an investigator gathering data and performing
analysis to determine what the data mean. This covers var-
ious forms of research strategies, including all forms of
experiments, qualitative studies, surveys, and archival
analyses”.
Kitchenham et al. (2002) provide some preliminary
guidelines for SE empirical research, which are based
mostly on published medical literature on the topic, being
the authors’ own experience as researchers and reviewers
the only direct connection with SE. These guidelines are
intended to cover all types of SE empirical research, such
as case studies, surveys, and formal experiments, whether
conducted in the field, in a laboratory or in a classroom.
In Kitchenham et al.’s opinion, clear guidance would help
both to assess the quality of published research and to
increase the likelihood that empirical study results can be
used in meta-analyses (see Section 3.3.5). Six basic topic
areas are treated in detail: (a) the experimental context of
the research, (b) the experimental design, (c) the conduct
of the experiment and the collection of data, (d) the analy-
sis, (e) the presentation of results, and (f) their interpreta-
tion. The description of the experimental context,
comprising industrial circumstances, research hypotheses,
and information on related research, is especially empha-
sized, for either observational studies or experiments. This
information should make the purpose of the research
understandable to all possible readers, whether researchers
or practitioners, besides helping to explain conflicting
research results.
Typically, journals do not comment in depth on this
type of research papers. The J Syst Software, for example,
encourages ‘‘theoretical and pragmatic developments with
emphasis on those that have been tested by government,
industry, or university empirical research”, but does not
specify the types of empirical research. In similar terms,
Empirical Softw Eng accepts different types of empirical
research (controlled experiments, field studies, qualitative,
etc.), giving preference to studies ‘‘that can be replicated
or expanded upon”, but doesn’t specify further. In the
IEEE Software Guide, by contrast, we find extensive, yet
not completely clear, information. Here a report of empir-
ical results resembles so much a case study that the two
types coincide in almost all respects. There are practically
no major differences, except perhaps a couple of possibly
significant divergences. The object of the research is the
‘‘project” in case studies, and the ‘‘participants” in empiri-
cal reports. In addition, case studies should report on tech-
niques, practices and tools, whereas empirical reports could
simply explain the practices. This blurred differentiation
between the two types may be the sign that historically
what has been understood as empirical research in SE are
case studies.
Comparing the 20 most cited SE articles of 1994 with
those of 1999, Wohlin (2005) found that a larger number
of empirical articles were included in the top list of 1999
as compared to that of 1994. By contrast, the topics prac-
tically had not changed. He mentions the ‘‘need for more
scientific approach to software engineering, including mea-
surement and empiricism”.
Good examples of this article genre are published in e.g.
Empirical Softw Eng.
3.3.1. Observational studies
Box 4 Reflection box on: observational studies
– Observational studies include all empirical research
strategies that impose little control over the devel-
opmental process. Item–Contextual factors, such
as the type of industry in which the products are
used, or the experience and skills of software staff,
affect the utility and generality of the results.
‘‘An observational method collects relevant data as a
project develops. There is relatively little control over
the development process other than through using the
new technology that is being studied” (Zelkowitz and
Wallace, 1998). In the classification of Zelkowitz and
Wallace (1998), observational studies include: field stud-
ies, case studies, project monitoring, and assertion. How-
ever, in contrast with Zelkowitz and Wallace, we regard
observational studies as different from case studies and
field studies, where observation may be (but is not neces-
sarily) undertaken. Assertion consists in the ‘‘developers
being both experimenters and subjects of study”. In the
sample of 612 SE papers they analyzed, assertion was
the most common validation method in the period
1985–1995. In a more recent paper, Zelkowitz et al.
(2003) classify assertion as an ‘‘informal method” of
validation.
According to Kitchenham et al. (2002), empirical studies
based on observing or measuring some aspect of software
 Author's personal copy
M. Montesi, P. Lago / The Journal of Systems and Software 81 (2008) 1694–1714
development in a particular company, should identify sev-
eral factors that might affect the generality and utility of
the conclusions. Among the others, they mention: (1) the
type of industry in which the products are used; (2) the
nature of the software development organization; (3) the
skills and experience of software staff; (4) the type of
software products used; and (5) the software processes
being used. For instance, Wu et al. (2003) present the
results of an observational-study of collaboration in soft-
ware design undertaken at a large software development
company.
Several observational studies can form the basis for
meta-analyses (Pickard and Kitchenham, 1998).
3.3.2. Case studies
Box 5 Reflection box on: case studies
– Case studies’ most distinguishing feature is their
descriptive character.
– It is difficult to distinguish case studies from experi-
ence reports or first-hand experience reports.
– The environment of the research is especially
important to evaluate case studies.
– They allow for different data collection procedures,
including controlled experiments.
– Their major advantage is that they reflect real world
situations.
– Some authors argue that, for this reason, they can
communicate better to practitioners.
A distinguishing feature of case studies is their descrip-
tive character. The Commun ACM, for example, comments
that: ‘‘Case studies should maintain an objective perspec-
tive on the systems they describe, and should be both ana-
lytical and descriptive”. Similarly, ICSE 2006 guidelines
insist on the descriptive nature of case studies, which
‘‘describe the application of one or more software engineer-
ing practice(s) in an industrial or organizational setting”,
and provide ‘‘a detailed description of how the practice
was applied and why”. Despite the distinctly descriptive
character of case studies, however, it is still difficult to dis-
tinguish them clearly from other reports, especially from
experience reports. In the IEEE Software Guide, for exam-
ple, we find that case studies resemble ‘‘industry experience
reports”. Here the few differences between the two genres
look formal rather than substantial (length, the nature of
references, etc.), because no epistemological reason justifies
them. Similarly, case studies and industry experience
reports share all but one of the review criteria, which seems
insufficient to fully explain their respective functions. In
industry experience reports, reviewers should establish if
the experience is likely to benefit the journal readers,
whereas in case studies they should see if implications of
1701
the results and future developments are addressed. In other
instructions to authors, case studies are difficult to distin-
guish from what is often called first-hand experience arti-
cles. However, they seem to entail a more objective
perspective, whereas the term itself first-hand experience
papers points to a more subjective report. Perry et al.
(2004) explanation comes as useful to clarify this issue:
Experience reports focus on a retrospective experience that
was illuminating of a problem, whereas case studies start
out with a research question, and answer that question
by systematically collecting data. Further, Perry et al.
(2004) defer the interested reader to exemplary research
case studies.
The IEEE Software Guide and several journal instruc-
tions, such as Computer, insist on the importance of the
environment or the context of case studies (‘‘type of appli-
cation domain, hardware and software platform, types of
people building and using the systems, etc.”). Only if the
environment is described in detail, can the readers tell
whether the findings obtained are relevant/applicable to
their situation. When it comes to describing a typical case
study outline, the IEEE Software Guide makes the ‘‘study
details” section change depending on whether a controlled
experiment or a non-controlled experiment was performed.
This clashes with Zelkowitz and Wallaces classification of
case studies as observational research reports, which as
such do not admit controlled experiments. However,
Kitchenham et al. (1995) and Wohlin et al. (2000), too,
assign case studies properties that one would regard as
more typical of quantitative research methods. For exam-
ple, the need for formulating previous hypotheses, or for
specifying certain design criteria such as external and inter-
nal validity. According to Wohlin et al. (2000) (p. 12), this
is possible because case studies afford diverse data collec-
tion procedures.
From a practical perspective, Zelkowitz and Wallace
(1998) stress that a case study’s major advantage consists
in the fact that it needs little additional costs to project
costs. Researchers can monitor certain attributes, such
as reliability and costs, in the development of a project
itself, which is going to happen regardless of the need to
collect experimental data. On the other hand, a weakness
of this method is that ‘‘each development is relatively
unique”, and comparing a development profile with others
is not always possible. Dyba et al. (2005), all the same,
point out that an advantage of their uniqueness is that
they reflect real world situations. As in medicine medical
treatments must be tested on real patients to obtain com-
pelling evidence, so can observational studies, case studies,
surveys and field experiments in SE ‘‘avoid the limited rel-
evance of small-scale, artificial software engineering
experiments”.
The empirical SE community often looks at case studies
with suspicion, because they lack objectivity and generaliz-
ability, and because replication is practically impossible.
Segal (2005), however, argues that the lack of objectivity
is a threat to any type of research that, in the circumstance
 Author's personal copy
1702 M. Montesi, P. Lago / The Journal of Systems and Software 81 (2008) 1694–1714
of case studies, can be minimized with recourse to different
sources of information. Furthermore, according to the
same author, an advantage of case studies is that they
may communicate more successfully to practitioners than
traditional experiments. In fact, there are indications, not
really supported by evidence, that case studies might be
more suitable than other genres for bridging the gap
between theory and practice, as mentioned in the introduc-
tion. Despite the low degree of external validity and gener-
alization, case studies would be sought by the industry
community particularly because carried out in real settings
and tailored to individual needs. Zelkowitz et al. (2003)
speak of a ‘‘self-interest of industry” which is concerned
with choosing the right methods for its own environment
more than with benefiting competitors through generaliza-
tion. Kitchenham et al. (1995), comparing case studies with
formal experiments, note that case studies are important
for industrial evaluation of software engineering methods
and tools because they can avoid the scale-up problems
of formal experiments.
The industry’s interest for this genre may explain why
case studies are sometimes assigned a persuasive function.
The instructions to authors of the J Visual Comp Animat,
for example, which is specialized in computer animation,
encourages ‘‘submission of case studies to demonstrate
to film makers what techniques have previously been
drawn on and their results”. As another example of indus-
trial interest, case studies presented at CAiSE 2006 pro-
vide the rationale for key decisions made at different
stages, including purchase and usage of information
systems.
3.3.3. Field studies
Box 6 Reflection box on: field studies
– Neither the function of field studies nor their
difference from case studies is clear in most
sources.
Field studies resemble case studies with a less intrusive
component (Zelkowitz and Wallace, 1998). In a case study,
the subjects themselves often collect the data, whereas in a
field study it is an outside group that monitors each subject
group. This approach is for instance well described by
Zhang et al. (2003), which report on a field study carried
out in an insurance company. However, some other times
the expression ‘‘in the field” seems to point to an involve-
ment of the reporting subjects, such as in ‘‘we seek to
include articles from practitioners working in the field
(including the user community)” (J Softw Maint Evol).
Even if field studies are often mentioned in the instructions
to authors of major journals or in conferences guidelines, it
is not clear what their function is. The difference with case
studies is not clear-cut either.
3.3.4. Experimental papers
Box 7 Reflection box on: experimental papers
– The terms ‘‘experiment” and ‘‘experimentation” are
often used improperly in SE.
– Many researchers complain about the lack of stan-
dardization regarding collection measures for
experimental research.
– Some also argue that current SE experimental
research is insufficiently representative of industrial
settings.
– It is particularly difficult to carry out true experi-
mentation in SE real settings.
Both Zelkowitz and Wallace (1998) and Sjøberg et al.
(2005) comment that the terms ‘‘experimentation” and
‘‘experiment” are often used incorrectly and/or inconsis-
tently in SE. Zelkowitz and Wallace (1998) observe that
what is called experimentation in SE hardly ever involves
a collection of data to confirm previously defined models
or theories. Sjøberg et al. (2005), on the other hand, note
that in SE the term ‘‘experiment” is often used as a syno-
nym for ‘‘empirical study”. Three definitions of experiment
are reported in Table 3.
Kitchenham et al. (2002), who define general guidelines
for empirical research, comment upon experimentation as
well. They stress one more time the importance of contex-
tual factors in designing formal experiments, which will
have to consider a rich and complex industrial setting. Fur-
ther recommendations are provided for the experimental
design. In particular, a proper experimental design should
describe the population under investigation, the rationale
and technique for sampling from that population, the pro-
cess for allocating and administering the treatments (or
‘‘intervention”), and the methods used to reduce bias and
determine sample size. Nonetheless, because data collection
measures are not standardized in SE, Kitchenham et al. con-
clude that it is difficult to know whether different studies are
really measuring the same thing and whether they are doing
that in the same way: ‘‘Still, much work remains in making
sure we mean the same things when we discuss and measure
variables in empirical studies”. The lack of standardization
of measurements in SE research is also stressed in Kitchen-
ham et al., 2001. Wohlin et al. (2000) argue that the informa-
tion furnished in experimental reports must be detailed
enough to allow other researchers to replicate the experi-
ment. They also describe in detail the different phases of
the experimental process and provide a standard outline
for reports of experiments (pp. 116–117). A good example
of experimental paper can be found in (Babar et al., 2006).
Sjøberg et al. (2005) focus specifically on controlled
experiments in SE. Their study relies on an analysis of
103 conference and journal articles published in SE major
journals and conferences between 1993 and 2002, and
 Author's personal copy

M. Montesi, P. Lago / The Journal of Systems and Software 81 (2008) 1694–1714 1703

Table 3
Definitions of experiment in SE
Zelkowitz and Wallace (1998):
‘‘A controlled method provides for multiple
instances of an observation in order to
provide for statistical validity of the
results”
Basili et al. (1999):
‘‘An experiment is a form of empirical study where
the researcher has control over some of the
conditions in which the study takes place and
control over the independent variables being
studied; an operation carried out under controlled
conditions in order to test a hypothesis against
observation”
Sjøberg et al. (2005):
‘‘Controlled experiment in software engineering
(operational definition): A randomized experiment
or a quasi-experiment in which individuals or teams
(the experimental units) conduct one or more
software engineering tasks for the sake of
comparing different populations, processes,
methods, techniques, language, or tools (the
treatments)”

reporting on controlled experiments. Their aim was to
measure to what extent the experiments were representative
of the industrial settings in terms of subjects, tasks, appli-
cations, and environment. They concluded by questioning
the fact that experimental research, the way it is now,
may be representative of the industrial settings, and by crit-
icizing several other aspects. The subjects taking part in the
experiments, for example, were in 87% of the cases stu-
dents, and in only 9% professionals; the applications, in
75% of the cases, were constructed for the purpose of the
experiment or constituted student projects, etc. They also
found that the reporting was often vague and unsystematic,
lacking generally consistent terminology. Not to mention
that both internal validity and external validity were
reported in a vague way, demanding for specific guidelines
for the community. Finally, they recommended that exper-
imental papers report rigorously on the following aspects:
‘‘the type and number of subjects, including the mortality
rate; context variables such as general software engineering
experience and experience specific to the task of the exper-
iments; how the subjects were recruited; the application
areas and type of tasks; the duration of the tasks; and inter-
nal and external validity of the experiments, including
being specific about the sample and target population of
the experiments”. Sjøberg et al. (2005) also identified a rela-
tion between experimentation and SE topics: most experi-
mental papers deal with ‘‘Software lifecycle/engineering”
and ‘‘Methods/Techniques”, using Glass et al. classifica-
tion scheme (Glass et al., 2002), and ‘‘Code inspections
and walkthroughs” and ‘‘Object-oriented design methods”,
using the IEEE Keyword Taxonomy (IEEE, 2002).
An explication of the poor representativeness of SE
experimental research may be the fact that true experimenta-
tion is often hard to carry out in SE. Laitenberger and Rom-
bach (2003) suggest that quasi-experimental designs would
fit better industrial settings than true experimental designs.
A major problem with true experimental designs in indus-
trial settings is that a random allocation of subjects is almost
impossible for schedule, availability, etc. Besides, the treat-
ment cannot be withheld. Quasi-experimental designs,
which do not require the allocation of subjects to be random,
may be a valid alternative, provided researchers indicate and
discuss explicitly all the possible threat to internal validity.
Zendler (2001) analyses briefly the history of SE experi-
mental research since the 1970s, highlighting that it has
increased considerably in the 1990s. Jedlitschka and Pfahl
(2005) summarize and complement the guidelines on the
conduction of experimental research in SE published since
1999. They insist on the fact that more research involving
different stakeholders must be carried out to achieve stan-
dardization. Tichy (2000) lists a series of dont”s for review-
ers of experimental research reports, inviting them to major
flexibility in judging the quality of such studies. He sug-
gests, among the other things, neither expecting perfection
nor decisive answers; not rejecting ‘‘obvious” results, and
not being casual about asking authors to redo their
experiment.
Authors are invited to submit experimental papers along
with other genres in the instructions to authors of many
journals, though they are not provided with any specific
information. The ACM T Softw Eng Meth only reminds
its contributors that results must be interpreted in term
of practice.
3.3.5. Meta-analyses
Box 8 Reflection box on: meta-analyses
– Meta-analyses allow to combine results of various
empirical studies.
– The lack of standardization in reports of empirical
research makes it difficult to apply meta-analyses in
SE.
Meta-analysis is a technique that allows combining the
results of various empirical studies. Its aim is ‘‘to provide
a quantitative and objective procedure for combining the
information from different studies” (Pickard and Kitchen-
ham, 1998). Such procedures should increase the confi-
dence of results that one obtains from only one study,
and, in general, make the process of reviewing the literature
more ‘‘scientific” (Djokic et al., 2001). For instance, Succi
et al. (2005) apply meta-analysis techniques on 200 projects
to study the distribution of selected object-oriented metrics.
According to Pickard and Kitchenham (1998), meta-
analyses follow three stages. First the analyst must decide
which studies to include. Individual studies should be of
the same nature (all case studies, or all cohort studies, etc.)
all exploring the same variables, for instance, or having
 Author's personal copy
1704 M. Montesi, P. Lago / The Journal of Systems and Software 81 (2008) 1694–1714
comparable measures. Then, in a second phase, the analyst
must estimate an effect size for each individual study in order
to make each of them comparable with the others. What
makes this stage difficult is that often raw data of individual
studies is missing, and the analyst cannot estimate the effect
size correctly. Finally, the results from individual studies are
combined. This technique is quite popular in medicine
where researchers tend to compare replicated experiments;
in SE, on the other hand, it would suit better observation
studies (Pickard and Kitchenham, 1998).
Miller (1999) is pessimistic about the applicability of
meta-analysis to SE research. According to him, in general,
the major obstacle to the introduction of meta-analyses in
SE would be the excessive variation within the several com-
ponents of the experiments in this field. Brooks (1997), on his
part, points out that there are not enough replications in the
SE literature, in order to perform meta-analysis routinely.
3.4. Experience report papers
Box 9 Reflection box on: experience report papers
– Experience reports are called by different names
(‘‘lessons learned reports”, ‘‘industry experience
reports”, etc.), and it is not clear if different names
imply variations.
– Experience reports appear to be connected with
application and practice.
– In some sources, practitioners are the expected
authors of experience reports.
– Soundness of the report and objectivity are impor-
tant publication requirements.
The papers classified under this heading have in com-
mon the notion of ‘‘experience”, remarked by some jour-
nals as first-person or first-hand experience. Other names
can also be used, such as ‘‘lesson learned reports”, ‘‘indus-
try experience reports”, ‘‘application papers”, etc., and it
remains unclear whether they are all to be considered as
the same type of experience reports. In the instructions to
authors, application, practice, and experience are interre-
lated, and especially the last two. With the name ‘‘applica-
tion papers”, however, different articles may be
understood, as explained further on. The IEEE Comput
Graph instructions on submissions to the ‘‘Applications
department” are a good example of how experience, appli-
cation and practice are in fact really close to each other:
‘‘Its existence is inspired by consistent feedback from the
readership reflecting a strong desire for more practical con-
tent, more real world applications, more practice and usage,
more implementation details, more experience-related
information, more practical application of research results,
and bridging the gap between theory and application”.
Despite their apparently less academic content, experi-
ence reports are widely sought for by some journals. They
may present ‘‘the results of using a CDA [Computer Aided
Design] in practice” (Comput Aided Design), describe ‘‘the
experience of applying functional programming to real
problems” (J Funct Program), or ‘‘how particular tech-
niques, practices, or tools were used in a practical setting”
(IEEE Software Guide). ICSE 2006 guidelines stress the
need for the report to provide ‘‘a critical review of experi-
ences during one or more phases of a software development
project”, and to describe all factors influencing the experi-
ence in a way that readers can understand its impact in
terms of success or failure. A typical example of ICSE
experience report is provided in (Best et al., 2007). In some
sources, practitioners are indicated as the most likely
authors of experience reports. According to Software Pract
Exper, for example, the aim of experience reports is ‘‘to
encourage practitioners to share their experiences with
design, implementation and evaluation of techniques and
tools for software and software systems”. It is in conference
guidelines, however, where we find a major emphasis on
practitioners as the most likely authors of experience
reports. At OOPSLA 2006, for example, ‘‘reports from
the field describing real-world experience” are called pre-
cisely ‘‘practitioner reports”. At METRICS 2005, one of
the authors should come from industry or government.
At WICSA 2007 ‘‘practitioners are asked to describe best
and worst experiences from the trenches, and researchers
are asked to describe areas where research progress is slo-
wed because of lack of access to real world problems”. This
preferential authorship could account for the connection of
experience, practice and application mentioned at the
beginning. While expected authors may often be practitio-
ners, expected readers, according to IEEE Software, are
software managers, maintainers, developers, applied soft-
ware researchers, and executives of companies interested
in investing in software technology.
The Empirical Softw Eng journal offers detailed indica-
tions on what and how to write in a proper ‘‘industrial
experience report”: ‘‘Describe the context from which the
results are reported [. . .]. Describe the software technolo-
gies [. . .]. Provide effectiveness and efficiency data leading
to the assessment of the technologies. Interpret the results,
explain their consequences, and draw conclusions. Discuss
the limitations of the results and conclusions”. Whatever
the topic they may deal with, practice and experience
papers do not need to make a novel contribution (J Funct
Program). The paper should give a well-informed, up to
date and accurate account of the application area by an
expert in that field. In fact, specific evaluation criteria
may apply. As opposed to regular research papers, they
need neither to discuss novel ideas nor to discuss related
published work in the literature (Empirical Softw Eng).
The J Funct Program also reminds that these papers are
difficult to be published in conferences due to their length.
This fact is confirmed by Kajiya (1993), who explains that
‘‘long papers” were introduced at SIGGRAPH to allow a
‘‘systems and application category papers”. Such papers
needed more space in order to ‘‘describe the experience that
the builders have had with the system”. A feature of
 Author's personal copy
M. Montesi, P. Lago / The Journal of Systems and Software 81 (2008) 1694–1714
‘‘industry experience reports” in the IEEE Software Guide
is a section with up to 10 brief bullet points summarizing
the ‘‘lessons learned”. RE 2006 guidelines also insist that
the focus of these contributions is on the ‘‘what” and the
‘‘lessons learned”, whereas explanations of the ‘‘why” are
better analyzed in other types of paper.
Some journals explain the criteria for acceptance of
experience reports. To be published in the ACM T Softw
Eng Meth, they ‘‘must provide thoughtful insights about
the development world or the application of a technology,
that result in the identification of new important challenges
for software engineering research”. In IEEE Comput Graph
those articles are not refereed. They are accepted if the con-
tribution is unique, and it gives some result such as, for
example, ‘‘how graphics was used to solve a previously
unsolved problem” or ‘‘why this solution worked, what
others were tried, and why this was better”. ICSE 2006
evaluation criteria for experience reports are the same as
for case studies. Soundness of the report, in this case,
involves describing the context of the experience (problem
domain, size of project, size of system, etc.), showing that
the experience is reported on the basis of observation and
analysis, and not of opinion, and measuring as accurately
as possible the important phenomena.
In the IEEE Software Guide, it remains undetermined
whether Industry experience reports imply a historical or
observational approach, i.e. whether the collection of data
takes place after the techniques, practices, and tools ana-
lyzed were implemented, or in the course of their imple-
mentation. Industry experience reports are considered
comparable to case studies, but the fact that they summa-
rize lessons learned and experience, however, points to a
historical nature. In Zelkowitz and Wallace (1998) experi-
ence reports are not mentioned as such. These authors
speak of ‘‘legacy data” and”lesson learned reports”. They
are both classified as historical methods, since both take
place when a project is completed, not depending on any
deadlines, but differ in the type of data they analyze. Leg-
acy data reports focus on quantitative data, including the
artifacts involved in the product, whereas lesson learned
reports reveal qualitative aspects of a project.
3.5. Theoretical papers
Box 10 Reflection box on: theoretical papers
– Theoretical papers are often of a methodological or
computational nature.
– Relevance for practice is the most important
requirement for publication.
Following the instructions to authors of the journals
analyzed, SE theoretical papers tend to be methodological,
but they also include computational papers, i.e. papers pre-
senting computational methods and algorithms. In most
cases, theoretical or methodological papers can be pub-
1705
lished only if they clearly show how the results contribute
to Software Engineering practice (ACM T Softw Eng
Meth), or when an understanding of the theory can lead
to better practical systems” (Software Pract Exper). In Sim-
ulat Pract Theory, for example, theoretical articles should
supply information on ‘‘potential benefits; any experience
gained; any innovation achieved through a specific
approach”. Then again, in Simul Model Pract Th, theoret-
ical articles must say: ‘‘Why the theory is relevant and how
it can be applied, what is the novelty of the approach and
what are the benefits and objectives of a new theory,
method or algorithm; what experience has been obtained
in applying the approach”. It is also necessary to include
applications and examples that demonstrate that any theo-
retical assumption is applicable to real world problems. In
Sci Comput Program, theoretical papers can deal with
philosophical and sociological issues on all aspects of com-
puter software production and usage, including ethics.
At CAiSE 2006, theoretical and conceptual papers are
classified as proper ‘‘research papers”, in contrast with
other types, such as case studies or experimental reports.
They should describe ‘‘the problem tackled, the state of
the art with respect to the problem, the solution suggested
and the potential – or even better, the evaluated – benefits
of the contribution”. Typical examples of SE theoretical
papers are published in the IEEE T Software Eng.
Relevance for practice as a condition for publication is
also stressed when journals address reviewers. Stricter crite-
ria than for other papers may be applied to evaluate theo-
retical papers, especially for what regards originality and
impact (ACM T Database Syst -R).
3.6. Tutorial articles
Box 11 Reflection box on: tutorial articles
– Tutorials are mainly educational, and introduce the
work done in a field in a highly comprehensible
manner.
– Two types of tutorial are considered in the IEEE
Software Guide: one focusing on applications and
the other on research activities.
– Tutorials address audiences with different back-
ground and levels of experience.
The educative inclination of many SE publications
materializes in a particular type of article, the tutorial.
Despite being published in most journals, tutorials are
rarely described. According to the ACM Comput Surv,
which publishes exclusively tutorials and surveys, a tutorial
is ‘‘a paper that organizes and introduces work in the
field”. A characterizing feature of it is the fact that basic
concepts of the field are emphasized and represented by
concrete examples. On its part, Presence-Teleop Virt sug-
gests that they ‘‘should develop to an appropriate level of
detail a significant topic that is of interest to, but not well
 Author's personal copy
1706 M. Montesi, P. Lago / The Journal of Systems and Software 81 (2008) 1694–1714
known to, a significant portion of the Presence readership”.
In Software Pract Exper, tutorials accomplish a second
function, that of documenting ‘‘well-timed techniques not
previously documented in computing literature”. Finally,
IEEE T Reliab highly regards tutorials that centre around
‘‘the chemistry & physics of failure”.
The IEEE Software Guide counts two types of tutorials,
the ‘‘research tutorial” and the ‘‘practice tutorial”. Each of
them develops one of the major aspects dealt with in a ‘‘sta-
tus report paper” (another genre described in the Guide).
Status reports discuss a given technology or a real-world
problem, providing both an explanation of its applications,
and a description of research trends and major barriers to
overcome. On the other hand, practice tutorials ‘‘Given a
specific technology (or real-world problem), explain how
the technology is applied (or how the real-world problem
is solved) today in industry”. And research tutorials ‘‘Given
a specific technology (or real-world problem), explain the
most promising research activities underway or proposed
that expand our knowledge of the technology (or of how
to solve the problem)”. Both types of tutorials and status
reports are allowed the same length (4300–5400 words)
and the same number of references and/or pointers to pri-
mary and secondary sources. Thus one expects a deeper
analysis and a major degree of specificity in tutorials. The
potential readership is practically the same in the three cases:
software developers, software managers, high-tech (corpo-
rate, for tutorials) executives, and applied software research-
ers. In all cases it is implied that readers are not experts in the
technology/real-world problem. Research tutorial and prac-
tice tutorial have a very similar outline, but the content of
the tutorial section differs correspondingly to their purpose.
In research tutorials, it deals with the major research activi-
ties that are expected to expand or refine the technology or
the problem, as well as their impact in practice. In practice
tutorials, the tutorial part provides taxonomy of applica-
tions, indicating in each case how the technology is applied
or how the problem is addressed.
Evaluation criteria of IEEE Comput Graph for tutorials
take in the following aspects, among the others: they must
fall within the scope of the journal, be interesting, well writ-
ten, and well organized, be understandable to most readers,
present background and historical material, current prob-
lems and issues, and assess applicability and limitations.
3.7. Surveys
Box 12 Reflection box on: surveys
– Surveys complement tutorials, but address a more
experienced audience.
– Surveys do not need to be novel, but should make a
contribution to the understanding of an area.
– They help researchers as well as practitioners to
keep up to date.
In most instructions to authors of SE journals, the name
‘‘survey” corresponds to what other disciplines, such as
education and biology, call ‘‘review” or ‘‘literature review”.
However, the terms are not exactly synonyms, because sur-
veys present some peculiarities. SE journals do not place
any special emphasis on the need for an exhaustive litera-
ture review, which is anyway required, but stress rather
other aspects. SE surveys include, for example, ‘‘material
that is difficult or impossible to find elsewhere” (Pres-
ence-Teleop Virt); they may focus ‘‘on emerging technolo-
gies or new aspects of existing technologies” (IEEE
Internet Comput), or summarize and organize recent
research results in a novel way (ACM T Math Software).
Effective presentation is highly expected (ACM T Math
Software). According to the ACM T Database Syst, two
types of surveys are possible: (a) those that summarize pre-
vious literature on a theoretical or systems research topic;
(b) and those that explain approaches implemented in com-
mercial systems. The former type presents a new way of
understanding how the papers in the field fit together,
whereas the latter reports the best industrial art, and is
acceptable even if it presents no new contribution. In a cer-
tain sense, for their educational and informative nature, SE
surveys are somehow similar to tutorials. The ACM Com-
puting Surveys explains that: ‘‘Basically a Computing Sur-
veys article, whether a survey or a tutorial, answers the
questions, ‘‘What is currently known about this area, and
what does it mean to researchers and practitioners?” It
should supply the basic knowledge to enable new research-
ers to enter the area, current researchers to continue devel-
opments, and practitioners to apply the results”. Such
definition retakes Goldberg (1982) and Wasserman (1984)
distinction between the two genres. According to these
authors, tutorials assume their audience to be inexpert
and emphasize basic concepts of the field providing con-
crete examples. By contrast, surveys assume their audience
to have a general knowledge of the field, and emphasize an
overall view of the literature.
Sometimes the variant more traditional designation of
‘‘review” is preferred in the journal instructions to authors,
as can other alternative names, such as ‘‘analyses” (Knowl
Eng R), or ‘‘state-of-the-art-report” (Comput Graph Form -
R). Then again, the term ‘‘survey” in empirical areas of SE
can refer to methods of research, in which participants are
requested to answer a questionnaire or give interviews
(Wohlin et al., 2000, pp. 10–12). This acceptation of the term
survey is closer to the understanding that the social sciences
have. In this way it is also understood by the instructions to
authors of the Commun ACM. Probably due to the ambigu-
ity of the term survey (meaning both literature review and
the questionnaire type of research), Zelkowitz and Wallace
(1998) refer to this research strategy as ‘‘literature search”.
Not many journals provide evaluation criteria for
reviewers who may have to evaluate surveys. The ACM
T Database Syst -R qualifies good surveys as focused, deep,
quite narrow, and educational for the journal readers. They
should make a contribution to the understanding of an
 Author's personal copy
M. Montesi, P. Lago / The Journal of Systems and Software 81 (2008) 1694–1714
important area or sub-area. When surveys summarize the
best industrial art, they can be accepted even if they give
‘‘no new contribution over what has been used in industry
for years”, as long as ‘‘the papers content is not to be found
in the published literature”. Surveys should also put recent
progress into a broader perspective and accurately assess
the limits of existing theories (IEEE T Vis Comput Gr -R).
Dyba et al. (2005) comment that surveys – that they call
‘‘systematic reviews” help keep up to date on the available
evidence regarding specific phenomena. In so doing, they
foster evidence-based SE research, and contribute to clos-
ing the gap between practice and research.
Budgen and Brereton (2006), on their part, discuss the
relevance of systematic reviews for the academic commu-
nity. Systematic literature reviews supply an objective sum-
mary of research evidence once a clear set of procedures is
established to review background material. Typical exam-
ples of SE surveys are those published on ACM Comput
Surv. Dyba et al. (2005) further argue that SE research is
bound to remain fragmented without ‘‘a research culture
that strongly advocates systematic reviews and replica-
tion”. To achieve this, the community needs to develop
and adopt specific guidelines for systematic reviews. The
best search methodologies should also be indicated in order
to reduce bias and guarantee as complete a coverage as
possible of the relevant information. Some guidelines are
available from the Inform Software Tech web site and pre-
sented by Kitchenham (2004), who synthesizes the infor-
mation reported in medical guidelines to perform
systematic reviews. More recently, Budgen et al. (2006)
have investigated the possibilities of aggregating evidence
in SE by looking at how this is done in other domains (like
Nursing & Midwifery and Criminology). In doing so, they
encountered domain-specific obstacles to perform system-
atic reviews. Potential authors of systematic reviews need
to deal with problems such as not standardized key-words
and abstracts of poor quality (Brereton et al., 2007).
Zelkowitz and Wallace (1998) remind that a major prob-
lem with this type of research is the fact that published lit-
erature tends to favor positive results to the detriment of
negative results with the resulting bias. Even with these lim-
itations, though, they consider surveys a validation method
that allows confirming existing hypotheses or enhancing
data collected in one project.
3.8. Short papers
Box 13 Reflection box on: short papers
– Short papers deal usually with practical or technical
topics.
– They usually undergo the same revision process as
full-length papers.
– Negative results or repetitions of previous results fit
well short papers.
1707
Different names are used to refer to this type of articles in
SE journal instructions to authors: Pearls, Short articles,
Short communications, Short contributions, Short papers,
Short submissions, etc. Short papers, like full-length papers,
allow for different methodological approaches and virtually
any kind of topic included in the scope of the journals,
although they tend to centre around some sort of practical
and/or technical application. In particular, short papers
may be ‘‘practitioner-oriented” (IEEE Software), describe
‘‘clever programming ideas” (J Funct Program) and ‘‘novel
and/or topical applications” (J Res Pract Inf Tech), or pres-
ent ‘‘problems on selected topics of computer science” (Algo-
rithmica). Sometimes they may also report ‘‘on rapidly
developing new topics” (Software Pract Exper). In the con-
text of conferences, short papers have an updating function.
At ICSE 2006 they are presented in an ‘‘informal and highly
interactive” section, which allows authors to get critical feed-
back that they can use for their final paper. Attendees, on
their part, keep abreast of emerging ideas and technologies
in the area. ICSE 2006 short papers, while dealing with the
same topics as those of the research track papers, cover work
with ‘‘solid technical basis” but not yet fully validated. At
QoSA 2006, short papers ‘‘can describe experience, ongoing
work, and new ideas”.
The maximum length admitted in general in SE journal
instructions to authors spans from 2000 to 5000 words. The
revision process is, as a general rule, as rigorous as for nor-
mal contributions. The publication process may be shorter
than for standard articles, but, in many cases, they are pro-
cessed as the other categories of papers. There are also
cases in which such articles may be reviewed only by the
editor in charge of the corresponding section (J Funct Pro-
gram). Reviewers must make sure that the paper improves
the state of the art ‘‘in some significant way” (ACM T
Database Syst R).
According to Tichy (2000), short papers would suit well
certain experimental studies, such as those reporting on
negative results or on repetitions with identical or very sim-
ilar results.
3.9. Papers oriented towards practice
3.9.1. Educational papers
Box 14 Reflection box on: educational papers
– Educational papers appear more commonly in con-
ferences than in journals.
Educational papers are mentioned both in conference
guidelines and in journal instructions, but they are
described mainly in the former case. At ICSE 2006, sound
research design and empirical analysis are strongly encour-
aged, whilst educational experience reports are accepted
provided they justify and explain the conclusions using
careful analysis and providing a thoughtful retrospective”.
 Author's personal copy
1708 M. Montesi, P. Lago / The Journal of Systems and Software 81 (2008) 1694–1714
Evaluation criteria are: ‘‘originality, importance of contri-
bution, soundness, empirical analysis, quality of presenta-
tion, and comparison to related works (as appropriate)”.
Typical good examples of educational journal papers are
published on the IEEE Transactions on Education, whereas
conference papers are to be found in the Proceedings of the
IEEE Conference on Software Engineering Education and
Training.
3.9.2. ‘‘How to”/ tool report papers
Box 15 Reflection box on: ‘‘How to”/ tool report
papers
– ‘‘How to” and tool articles explain how to perform
specific development tasks or on how to use specific
tools.
– The conditions of applicability/inapplicability of
the process or the tool at hand must be detailed.
‘‘How To” articles in the IEEE Software Guide ‘‘provide
readers with step-by-step instructions on how to perform a
specific software development task”. They should discuss
the distinction between tool-related and process-related
aspects, and in this sense they seem to be intended as com-
plementary to ‘‘Tool reports”. In the IEEE Software Guide
tool report articles ‘‘report the development of a new soft-
ware tool and its application”. However, ‘‘how to” and
tool report articles differ in the audience they address.
Whilst ‘‘how to” articles are intended for those ‘‘software
developers and/or managers who have never performed
the specific task, or who have performed it previously with
unacceptable results”, tool reports address ‘‘Software
developers, software managers, high-tech executives,
applied software researchers”, and assume that ‘‘readers
are not experts in the technology or class of tools being
addressed”. The word limit varies accordingly. ‘‘How to”
reports (2300–4300 words) have a word limit slightly under
the threshold fixed for major papers, such as tool reports
(4300–5400), among others. Structurally speaking, the
two reports are also different, as reported in Table 4.
Table 4
The structure of a ‘‘tool report” article and a ‘‘how to” paper according to the IEEE Software Guide
Introduction, explaining the problem the tool addresses;
The tool section, indicating who the intended users are, how and when it
would be used, and how it solves the problem;
State-of-the-art section, discussing alternative solutions to the problem,
and comparing the tool with similar ones;
How was the tool built, a ‘‘short” section (maximum 1500 words)
explaining the development process, the architecture, and the
algorithms; and
Summary and conclusion.
In reviewing ‘‘how to” reports, reviewers are asked to
check if examples are provided and if the conditions of
applicability/inapplicability are clear. Similarly, tool
reports should explain how the tool can be used in common
practice and for which applications it is most effective.
‘‘How to” or tool report papers, like for instance
(Arranga, 2000), are appearing less often in scientific pub-
lications. Instead, they are being replaced by online publi-
cations and web sites or blogs especially dedicated to
technologies and ‘‘how to” issues.
3.9.3. Application papers
Box 16 Reflection box on: application papers
– Application papers are similar and related to expe-
rience reports, tool reports and ‘‘how to” papers.
– They may focus on applications in different
domains.
– They usually stress problems and obstacles to
overcome.
As commented previously, different types of articles may
be expected under the name of ‘‘application papers”. In
some cases, an application paper may explain how to
implement a certain tool, method or technique. In some
others, it may focus on the application of software in spe-
cific environments and on solutions of specific problems.
Finally, it can be understood as an experience paper. We
don’t gain a deeper insight with the alternative name of
‘‘applied research paper” in use in the IEEE Software
Guide, whose purpose is to ‘‘Report to readers on the avail-
ability of some new research technology, technique, tool, or
language”. However, we can tentatively establish a differ-
entiation. Application or applied research papers are really
similar to tool and ‘‘how to” reports, but more concen-
trated on applications in specific domains. In addition, they
are also related to experience papers, with the difference
that experience reports focus on lessons learned, while
application reports on obstacles and problems to over-
come. As a matter of fact, in an applied research article,
according to the IEEE Software Guide, whilst the feasibility
Introduction, explaining the relationship of the task to software
development, who performs the task, when and how often;
How to section, providing step-by-step instructions and relative examples;
Tailoring the procedure section, describing the conditions under which the
above procedure is applicable, and how to tailor the procedure in
mismatched conditions
Summary and conclusion
 Author's personal copy
M. Montesi, P. Lago / The Journal of Systems and Software 81 (2008) 1694–1714
of the research has already been proved, it is expected that
more experimentation will confirm the effectiveness of the
results. Expected readers are: ‘‘Software developers and/
or managers who wish to use new technology experimen-
tally. Applied software researchers who want to learn what
others are doing”. Research results should be likely to be in
widespread commercial use within the next 5 years. The
article should address the classes of applications ideally sui-
ted to experiment, as well as further research endeavors.
The problematic nature of application papers is also
stressed by the ICSE 2006 guidelines, which publish appli-
cation papers within the achievement and challenges track.
Here papers address the critical issues pending in order
to permit major software engineering achievements, and
the technical challenges which remain in theory and prac-
tice. Achievements focus on SE contributions in different
application domains, whereas challenges are expected to
focus on major unsolved problems regarding currently
implemented systems, with the aim of raising interesting
and fundamental research questions. However, despite
their clear-cut purpose, such contributions do not adopt
necessarily a specific methodology, and achievement and
challenges contributions may appear as either case studies
or problem statements. On the other hand, ‘‘Problem state-
ments” as a type of paper appear in CAiSE 2006 too, with
no specific reference to application, but rather to ‘‘usage
and development”.
3.10. Opinion papers
Box 17 Reflection box on: opinion papers
– Opinion papers may cover general topics of interest
for the field, comment on previously published
papers, or explain a technical position.
– Requirements for publication include, among the
others: convincing arguments, brevity, and
interestingness.
Opinions may be given on a general topic of interest in
the field, or on a previously published paper – in this case,
often in the form of letters. In IEEE Comput Graph, an
opinion paper ‘‘[. . .] aims at describing and challenging
the status-quo of interactive computer graphics, and pre-
dicting its many opportunities by learning from the past,
questioning the present, and anticipating the future that
within a few decades evolved from the ideas and imagina-
tions of researchers and scientists to become a commodity
in all areas of our lives”. In the IEEE Software Guide, they
should encourage readers to think ‘‘out of the box”,
expand their perspectives, and raise controversy. Gener-
ally, opinion papers can be ‘‘provocative”, or ‘‘serious,
whimsical, or humorous” (Sigmod Rec). The general tone
is similar in conferences. ‘‘Vision papers” presented at
RE 2006 should be ‘‘revealing and thought-provoking”,
1709
whereas ‘‘position papers” at EWSA 2006 ‘‘present concise
arguments about a topic of software architecture research
or practice”.
Some journals distinguish further opinion articles,
sometimes depending on whether they give a technical
opinion or not. The Commun ACM, for example, separates
‘‘viewpoints” that are ‘‘carefully reasoned commentaries in
which all positions are substantiated by facts or principled
arguments”, from ‘‘technical opinions”, that are treated as
informal commentaries on technical matters. IEEE Comput
Graph publishes opinions both as viewpoint articles and as
‘‘Graphically Speaking” articles. In viewpoint articles, the
reader can find technical opinions, as well as the discussion
of challenges and limitations in todays methods, and of
potential areas of new research. These papers are evaluated
on the basis of the following criteria: uniqueness, brevity
(2–4 pages), high-quality figures, and the structured format
introduction actual visualizations–results. Different crite-
ria are considered for opinion papers of the ‘‘Graphically
Speaking” section. They must deal with some interesting
aspect of interactive computer graphics, should not be
obvious, should be short but provide a clear basis for the
viewpoint presented. A proper IEEE Software Guide opin-
ion essay should present the position, explain how it relates
to overall software development, and should provide evi-
dence and justification for the defended position, while
explaining opposite camps and common practices. IEEE
Software Guide potential reviewers should check if the posi-
tion is adequately supported with ‘‘conjecture, logic, intui-
tion, or experimental data”. The reporting of alternative
positions and ‘‘pointers to background information, posi-
tion justification, and opposing points of view” are also
required for this paper. Finally, reviewers should check
whether a sufficient number of readers are likely to disagree
with the position taken.
In our ‘opinion’ a good example of an opinion paper is
given by Booch (2007) as part of the series On Architecture.
4. Discussion
What we learned from this study is that the guidance
currently available to write and review SE papers is frag-
mented and still insufficient to help authors and reviewers
thoroughly in their tasks. In what concerns empirical and
experimental research, this is probably a consequence of
the methodological problems discussed in much SE litera-
ture (Fenton, 1993; Tichy et al., 1995; Glass, 1995; Zelko-
witz and Wallace, 1998), and commented on in the
introduction to this study. The not yet clear indications
about what to include in certain types of papers reflect
the absence of well-established procedures to carry out
research. Certainly, the members of the community have
implicit knowledge of what is not written on paper. For
example, if certain requisites of good papers, such as
importance, relevance, or significance may sound too gen-
eral to a standard reader, they may have, on the contrary, a
clear meaning for authors and reviewers, and especially for
 Author's personal copy

1710 M. Montesi, P. Lago / The Journal of Systems and Software 81 (2008) 1694–1714

the latter who, as a general rule, are experienced in writing
and judging papers. Still, it is true what stated by Wieringa
et al. (2005), that articles might be judged of a good or a
bad quality for different reasons. How to properly write
or judge papers depends on what is considered to be a
proper research methodology or research procedure, and,
as we have seen, the debate on this topic is in part still
open.
Nevertheless, despite the problems just mentioned, this
study has come up with a first classification of SE articles,
which is reported in Tables 5 and 6. It is undoubtedly an
improvable classification, but still it represents a necessary
basis for further developments. The commentaries included
under each article genre describe all of them in terms of
requisites for publication and communicative function,
sometimes pointing to unresolved issues such as an unclear
communicative purpose or possible terminological
problems.
Several researchers, especially those studying empirical
and experimental research, point out that improvements
need to be based on the consensus of at least a great part
of the community. Kitchenham et al. (2002) and
Jedlitschka and Pfahl (2005), for example, claim that it is
not yet clear which factors should be reported in the
research article regarding, for example, participants, con-
text or threats to validity. Nonetheless, consensus seems
necessary in other aspects as well. First of all, we did not
find a great agreement on the terminology used for each
article genre. The confusion existing on terms such as
‘‘experiment” or ‘‘empirical research” is a good example
of such problem. In this article, however, we have also seen
that it is not yet clear the difference between ‘‘case studies”
and ‘‘industry reports”, or between ‘‘experience reports”
and ‘‘application or applied articles”, to cite but a couple
of examples. It is very difficult to establish what a good
paper is, if we do not agree first on what we are speaking
about.
This terminological confusion is particularly undesirable
if it is definitely proved that certain article genres are more
appropriate than others to bridge the gap between theory

Table 5
Summary of the article genres described in the paper and their major aspects (part A)
Publishable papers: general
– Regular papers may be shaped as different genres, such as, for example, experimental papers, case studies, or surveys
– A typical SE paper has a three-section structure: Introduction – Main Body – Conclusion
– Appropriate style must be comprehensible
– Relevance for practice, novelty, and originality are the most important requirements for publication
Extended versions of conference papers
– Extended versions of conference papers are a typical genre of SE
– Extended versions have passed through a double round of reviews
– Many journals require that approximately one third of the content be new
– Publication on a journal allows discussing the research beyond the time and space limitations set by conferences
Empirical research reports
– Empirical research includes different research strategies such as case studies, experimental research, or qualitative research
– Clear and comprehensive guidelines on how to write a proper empirical research paper are not yet available
– The experimental context is considered important for all types of empirical research
– Empirical research is increasing in SE
Observational studies:
– Observational studies include all empirical research strategies that impose little control over the developmental process
– Contextual factors, such as the type of industry in which the products are used, or the experience and skills of software staff, affect the utility and
generality of the results
Case studies:
– Case studies’ most distinguishing feature is their descriptive character
– It is difficult to distinguish case studies from experience reports or first-hand experience reports
– The environment of the research is especially important to evaluate case studies
– They allow for different data collection procedures, including controlled experiments
– Their major advantage is that they reflect real world situations
– Some authors argue that, for this reason, they can communicate better to practitioners
Field studies:
– Neither the function of field studies nor their difference from case studies is clear in most sources
Experimental papers:
– The terms ‘‘experiment” and ‘‘experimentation” are often used improperly in SE
– Many researchers complain about the lack of standardization regarding collection measures for experimental research
– Some also argue that current SE experimental research is insufficiently representative of industrial settings
– It is particularly difficult to carry out true experimentation in SE real settings
Meta-analyses:
– Meta-analyses allow to combine results of various empirical studies
– The lack of standardization in reports of empirical research makes it difficult to apply meta-analyses in SE
 Author's personal copy

M. Montesi, P. Lago / The Journal of Systems and Software 81 (2008) 1694–1714 1711

Table 6
Summary of the article genres described in the paper and their major aspects (part B)
Experience papers
– Experience reports are called by different names (‘‘lessons learned reports”, ‘‘industry experience reports”, etc.), and it is not clear if different names
imply variations
– Experience reports appear to be connected with application and practice
– In some sources, practitioners are the expected authors of experience reports
– Soundness of the report and objectivity are important publication requirements
Theoretical papers
– Theoretical papers are often of a methodological or computational nature
– Relevance for practice is the most important requirement for publication
Tutorial articles
– Tutorials are mainly educational, and introduce the work done in a field in a highly comprehensible manner
– Two types of tutorial are considered in the IEEE Software Guide: one focusing on applications and the other on research activities
– Tutorials address audiences with different background and levels of experience
Surveys
– Surveys complement tutorials, but address a more experienced audience
– Surveys do not need to be novel, but should make a contribution to the understanding of an area
– They help researchers as well as practitioners to keep up to date
– They can have a role in bridging the gap between practice and research
Short papers
– Short papers deal usually with practical or technical topics
– They usually undergo the same revision process as full-length papers
– Negative results or repetitions of previous results fit well short papers
Papers oriented towards practice
Educational papers:
– Educational papers appear more commonly in conferences than in journals
‘‘How to”/ tool report papers:
– ‘‘How to” and tool articles explain how to perform specific development tasks or on how to use specific tools
– The conditions of applicability/inapplicability of the process or the tool at hand must be detailed
Application papers:
– Application papers are similar and related to experience reports, tool reports and ‘‘how to” papers
– They may focus on applications in different domains
– They usually stress problems and obstacles to overcome
Opinion papers
– Opinion papers may cover general topics of interest for the field, comment on previously published papers, or explain a technical position
– Requirements for publication include, among the others: convincing arguments, brevity, and interestingness

and practice. In this respect, distinguishing a preferential
audience for each type of paper, the IEEE Software Guide
confirms what we commented upon in the introduction,
that different types of papers are better suited for certain
readers. Further research should try to find out which types
of papers are best received by practitioners, if interaction
between research and practice remains a crucial problem
in the field. Practitioners would certainly benefit from
major terminological consistence. Regarding the article
genres more likely to be useful for SE practitioners,
researchers point out two genres: case studies (Ardimento
et al., 2003; Kitchenham et al., 1995; Segal, 2005) and sur-
veys, sometimes also called literature reviews (Dyba et al.,
2005). More vague indications exist on the utility for prac-
titioners of experience and tool reports, which however
have not yet been identified and described unequivocally,
or of tutorials. Evidence must be provided in all cases.
Finally, application papers are indeed a good example
of papers without a clear communicative function. Consid-
ering that they often deal with specific applications of soft-
ware in different domains, we could wonder if they could be
qualified as interdisciplinary as well. Glass et al. (2004) and
Vessey et al. (2005) studied the relationship of SE with
other disciplines in terms of ‘‘reference discipline”, i.e.
‘‘any discipline outside the SE field that SE researchers
have relied upon for theory and/or concepts”. They con-
cluded that hardly ever does SE rely on a reference disci-
pline, but this conclusion clashes with the way many SE
journals present themselves. Most of them insist on their
interdisciplinary and sometimes multidisciplinary nature.
Is the application type of paper the one more likely to deal
with interdisciplinary content? If so, how do writers handle
such content? No answer comes from the sources we have
studied.
5. Conclusion
In this article we have summarized the fragmented infor-
mation available on the different types of papers published
in SE journals and conferences. Our sources are the
instructions to authors of major publications in the field,
calls for papers of major conferences, as well as the relevant
 Author's personal copy
1712 M. Montesi, P. Lago / The Journal of Systems and Software 81 (2008) 1694–1714
literature published on the topic. What have we achieved as
a result?
Firstly, we can now count on a first classification of
research articles in SE, as summarized in Tables 5 and 6.
Each type of paper has been described in terms of desirable
requisites for publication and communicative function.
Relationships between different article genres have been
commented on, and a special emphasis has been placed
on unresolved issues, such as the terminological problems.
Secondly, we have learned that, without solving the
methodological issues currently matter for debate in SE,
the problem of properly writing and judging certain types
of SE papers remains open. Different authors have com-
plained about the lack of agreed-on criteria to write and
judge papers, as explained in other parts of this study.
However, how a good paper should be depends on well-
established patterns of research and methodologies, which
in some cases are currently under discussion in SE.
Finally, this study is fundamentally theoretical, as it
reports on what certain actors in the community regard
as desirable attributes for publication. Future research
can further refine article genres characteristics by studying
published articles in the framework of our classification.
Acknowledgements
The authors would like to thank Rafael Capilla and
Hans van Vliet for their comments and suggestions on ear-
lier versions of this paper. This work was partially sup-
ported by a grant from the Spanish Ministry of
Education and Culture.
Appendix A. List of journals
List of journal titles cited in the paper, in alphabetical
order with their corresponding acronyms. When available,
the acronym of the JCR has been used in the first place.
For the journals present in the JCR we also include the
Impact Factor in round brackets.
ACM Comput Surv – ACM Computing Surveys
ACM T Database Syst – ACM Trans. on Database Sys-
tems (1.833)
ACM T Graphic – ACM Trans. of Graphics (3.652)
ACM T Math Software – ACM Trans. on Mathemati-
cal Software (1.463)
ACM T Softw Eng Meth ACM Trans. on Software
Engineering and Methodology (3.000)
Algorithmica – Algorithmica, Springer (1.017)
Commun ACM – Communications of the ACM (1.797)
Comput Aided Design – Computer-aided Design (1.173)
Comput Graph Forum – Computer Graphics Forum
(0.972)
Computer – IEEE Computer (1.282)
Dr Dobbs J – Dr. Dobbs (0.050)
Empirical Softw Eng – Empirical Software Engineering
(0.966)
IBM Syst J – IBM Systems Journal (1.255)
IEE Proc-Softw – IEE Proceedings. Software (0.300)
IEEE Comput Graph – IEEE Computer Graphics and
Applications (1.152)
IEEE Internet Comput – IEEE Internet Computing
(2.304)
IEEE Micro – IEEE Micro (1.238)
IEEE Software – IEEE Software (1.333)
IEEE T Reliab – IEEE Trans. on Reliability (0.715)
IEEE T Software Eng – IEEE Trans. on Software Engi-
neering (1.967)
IEEE T Vis Comput Gr – IEEE Trans. on Visualization
and Computer Graphics (1.457)
Inform Software Tech – Information and Software
Technology (0.435)
J ACM – Journal of the ACM (2.197)
J Comput Sci Technol – Journal of Computer Science
and Technology (0.353)
J Funct Program – Journal of Functional Programming
(1.400)
J Math Imaging Vis – Journal of Mathematical Imaging
and Vision (2.197)
J Res Pract Inf Tech – Journal of Research and Practice
in Information Technology (0.205)
J Softw Maint Evol – Journal of Software Maintenance
and Evolution (0.457)
J Syst Software – Journal of Systems and Software
(0.744)
J Visual Comp Animat – Journal of Visualization and
Computer Animation (0.864)
Knowl Eng R – The Knowledge Engineering Review
Presence-Teleop Virt – Presence: Teleoperators & Vir-
tual Environments (0.628)
Sci Comput Program – Science of Computer Program-
ming (0.734)
Sigmod Rec – Sigmod Record (1.155)
Simul Model Pract Th – Simulation Modelling Practice
and Theory (0.361)
Simulat Pract Theory – Simulation Practice and Theory
Software Pract Exper – Software: Practice and Experi-
ence (0.406)
Software Process – Software Process: Improvement and
Practice
SoSyM – Journal on Software and System Modeling
Theor Pract Log Prog – Theory and Practice of Logic
Programming (1.372).
Appendix B. List of conferences
List of conferences (in alphabetical order with their cor-
responding acronyms) whose calls for papers have been
consulted.
ASE 2006 – IEEE/ACM Int. Conf. on Automated Soft-
ware Engineering
ASWEC 2007 – Australian Conference on Software
Engineering
 Author's personal copy
M. Montesi, P. Lago / The Journal of Systems and Software 81 (2008) 1694–1714
CAiSE 2006 – Conference on Advanced Information
Systems Engineering
CSMR 2006 – European Conference on Software Main-
tenance and Reengineering
EASE 2006 – Evaluation and Assessment in Software
Engineering
ESEC/FSE 2007 – European Software Engineering
Conference/ACM SIGSOFT Symposium on the Foun-
dations of Software Engineering
ESEM 2007 (METRICS + ISESE) – Empirical Soft-
ware Engineering and Measurement
EWSA 2006 – European Workshop on Software
Architecture
ICSE 2006 – Int. Conf. on Software Engineering
ICSM 2006 – Int. Conf. on Software Maintenance
ICSR 2006 – Int. Conf. on Software Reuse
ISESE 2006 – International Symposium on Empirical
Software Engineering
METRICS 2005 – IEEE International Software Metrics
Symposium
MoDELS 2006 – Int. Conf. on Model-Driven Engineer-
ing Languages and Systems
OOPSLA 2006 – Int. Conf. on Object-Oriented Pro-
gramming, Systems, Languages and Applications
QoSA 2006 – Int. Conf. on the Quality of Software
Architectures
RE 2006 – IEEE International Requirements Engineer-
ing Conference
SEKE 2007 – Int. Conf. on Software Engineering and
Knowledge Engineering
WICSA 2007 – Working IEEE/IFIP Conference on
Software Architecture
References
Abran, A., Moore, J.W., Bourque, P., Dupuis, R., Tripp, L.L., 2004.
Guide to the Software Engineering Body of Knowledge (SWEBOK).
IEEE.
Anthony, L., 1999. Writing research article introductions in software
engineering: how accurate is a standard model? IEEE Trans. Profess.
Commun. 42 (1), 36–46.
Ardimento, P., Baldassarre, M., Bianchi, A., Visaggio, G., 2003. Empirical
studies as a means for technology transfer. In: Proceedings of the
Second Workshop Series on Empirical Engineering, p. 8.
Arranga, E.C., 2000. Cobol tools: overview and taxonomy. IEEE Softw.
17 (2), 59–69.
Babar, M.A., Kitchenham, B., Maheshwari, P., 2006. The value of
architecturally significant information extracted from patterns for
architecture evaluation: a controlled experiment. In: Proceedings of the
Australian Software Engineering Conference (ASWEC). IEEE Com-
puter Society, Washington, DC, USA, pp. 379–390.
Basili, V.R., 1996. The role of experimentation in software engineering:
past, current, and future. In: Proceedings of the 18th International
Conference on Software Engineering (ICSE). IEEE Computer Society,
Washington, DC, USA, pp. 442–449.
Basili, V.R., Shull, F., Lanubile, F., 1999. Building knowledge through
families of experiments. IEEE Trans. Softw. Eng. 25 (4),
456–473.
Best, B., Jürjens, J., Nuseibeh, B., 2007. Model-based security engineering
of distributed information systems using UMLsec. In: Proceedings of
1713
the 29th International Conference on Software engineering (ICSE), pp.
581–590.
Booch, G., 2007. The irrelevance of architecture. IEEE Softw. 24 (3), 10–
11.
Brereton, P., Kitchenham, B.A., Budgen, D., Turner, M., Khalil, M.,
2007. Lessons from applying the systematic literature review process
within the software engineering domain. J. Syst. Softw. 80 (4), 571–
583.
Brooks, A., 1997. Meta analysis – a silver bullet-for meta-analysts.
Empirical Softw. Eng. 2 (4), 333–338.
Budgen, D., Brereton, P., 2006. Performing systematic literature reviews in
software engineering. In: ICSE, pp. 1051–1052.
Budgen, D., Kitchenham, B., 2005. Realising evidence-based software
engineering a report from the workshop held at ICSE 2005. SIGSOFT
Softw. Eng. Notes 30 (5), 1–5.
Budgen, D., Charters, S., Turner, M., Brereton, P., Kitchenham, B.,
Linkman, S., 2006. Investigating the applicability of the evidence-
based paradigm to software engineering. In: WISER’06: Proceedings
of the 2006 International Workshop on Interdisciplinary Software
Engineering Research. ACM Press, New York, NY, USA, pp. 7–14.
Djokic, S., Succi, G., Pedrycz, W., Mintchev, M.P., 2001. Meta analysis –
a method of combining empirical results and its application in object-
oriented software systems. In: Proceedings of the Seventh International
Conference on Object-Oriented Information Systems, pp. 103–112.
Dyba, T., Kitchenham, B.A., Jorgensen, M., 2005. Evidence-based
software engineering for practitioners. IEEE Softw. 22 (1), 58–65.
Fenton, N., 1993. How effective are software engineering methods? J. Syst.
Softw. 22 (2), 141–146.
Fletcher, P., 1995. The role of experiments in computer science. J. Syst.
Softw. 30 (1–2), 161–163.
Glass, R.L., 1995. Structure-based critique of contemporary computing
research. J. Syst. Softw. 28 (1), 3–7.
Glass, R.L., Vessey, I., Ramesh, V., 2002. Research in software engineer-
ing: an analysis of the literature. Information Softw. Technol. 44 (8),
491–506.
Glass, R.L., Ramesh, V., Vessey, I., 2004. An analysis of research in
computing disciplines. Commun. ACM 47 (6), 89–94.
Goldberg, A., 1982. Editorial policy. ACM Comput. Surveys 14 (2), 151–
157.
Haddow, G., Klobas, J., 2004. Communication of research to practice in
library and information science: closing the gap. Library Information
Sci. Res. 26 (1), 29–43.
Hjørland, B., 1998. Information retrieval, text composition, and seman-
tics. Knowledge Organization 25 (1/2), 16–31.
Hjørland, B., 2002. Domain analysis in information science. Eleven
approaches – traditional as well as innovative. J. Document. 58 (4),
422–462.
IEEE, 2002. IEEE keyword taxonomy. available at: http://www.com-
puter.Org/mc/keywords/software.htm.
IEEE, 2006. Genre submission guide. IEEE Software, available at http://
www.computer.org/portal/site/software.
Jedlitschka, A., Pfahl, D., 2005. Reporting guidelines for controlled
experiments in software engineering. In: Proceedings of the Fourth
International Symposium on Empirical Software Engineering (ISESE),
pp. 95–104.
Journal Citation Reports, 2006. Journal citation reports. Available
at http://portal.isiknowledge.com/portal.cgi?DestApp=JCR&Func=
Frame.
Kajiya, J., 1993. How to get your siggraph paper rejected. http://
old.siggraph.org/publications/instructions/rejected.html.
Kitchenham, B., 2004. Procedures for performing systematic reviews.
Tech. Rep. TR/SE-0401, Keele University, available at: http://
www.elsevier.com/framework_products/promis_misc/inf-systrev.pdf
(July).
Kitchenham, B., Pickard, L., Pfleeger, S.L., 1995. Case studies for method
and tool evaluation. IEEE Softw. 12 (4), 52–62.
Kitchenham, B.A., Hughes, R.T., Linkman, S.G., 2001. Modeling
software measurement data. IEEE Trans. Softw. Eng. 27 (9), 788–804.
 Author's personal copy
1714 M. Montesi, P. Lago / The Journal of Systems and Software 81 (2008) 1694–1714
Kitchenham, B.A., Pfleeger, S.L., Pickard, L.M., Jones, P.W., Hoaglin,
D.C., Emam, K.E., Rosenberg, J., 2002. Preliminary guidelines for
empirical research in software engineering. IEEE Trans. Softw. Eng.
28 (8), 721–734.
Kwasnik, B.H., Crowston, K., 2005. Introduction to the special issue:
Genres of digital documents. Information Technol. People 18 (2), 76–
88.
Laitenberger, O., Rombach, D., 2003. (Quasi-)experimental studies in
industrial settings. Lecture Notes Empirical Softw. Eng., 167–227.
Marcos, E., 2005. Software engineering research versus software develop-
ment. SIGSOFT Softw. Eng. Notes 30 (4), 1–7.
Miller, J., 1999. Can results from software engineering experiments be
safely combined? In: Proceedings of the Sixth International Sympo-
sium on Software Metrics (METRICS). IEEE Computer Society,
Washington, DC, USA, p. 152.
Miller, J., 2004. Statistical significance testing: a panacea for software
technology experiments? J. Syst. Softw. 73 (2), 183–192.
Moed, H., Visser, M., 2007. Developing bibliometric indicators of
research performance in computer science: an exploratory study.
Tech. Rep. CWTS Report 2001-01, Research Report to the Council for
Physical Sciences of the Netherlands Organization for Scientific
Research (NWO).
Montesi, M., Mackenzie-Owen, J., 2008. Research journal articles as
document genres: exploring their role in knowledge organization.
Journal of Documentation 64 (1).
Montesi, M., Mackenzie-Owen, J., accepted for publication. From
conference to journal publication. How conference papers in software
engineering are extended for publication in journals. Journal of the
American Society for Information Science and Technology.
Perry, D.E., Sim, S.E., Easterbrook, S.M., 2004. Case studies for software
engineers. In: ICSE’04: Proceedings of the 26th International Confer-
ence on Software Engineering. IEEE Computer Society, Washington,
DC, USA, pp. 736–738.
Pickard, L.M., Kitchenham, B., 1998. Combining empirical results in
software engineering. Information Softw. Technol. 40 (14), 811–821.
Posteguillo, S., 1999. The schematic structure of computer science research
articles. English Specific Purposes 18 (2).
Punter, T., 2003. What information do software engineering practitioners
need? In: Proceedings of Second Workshop on Empirical Studies in
Software Engineering. Rome, Italy.
Segal, J., 2005. When software engineers met research scientists: A case
study. Empirical Softw. Eng. 10 (4), 517–536.
Segal, J., Grinyer, A., Sharp, H., 2005. The type of evidence produced by
empirical software engineers. In: Proceedings of the 2005 Workshop on
Realising Evidence-based Software Engineering (REBSE). ACM
Press, New York, NY, USA, pp. 1–4.
Shaw, M., 2002. What makes good research in software engineering. Int.
J. Softw. Tools Technol. Transfer 4 (1), 1–7.
Shaw, M., 2003. Writing good software engineering research papers:
minitutorial. In: Proceedings of the 25th International Conference on
Software Engineering (ICSE). IEEE Computer Society, Washington,
DC, USA, pp. 726–736.
Sidiropoulos, A., Manolopoulos, Y., 2005. A new perspective to
automatically rank scientific conferences using digital libraries. Infor-
mation Process. Manage. 41 (2), 289–312.
Sjøberg, D.I.K., Hannay, J.E., Hansen, O., Kampenes, V.B., Karahasa-
novic, A., Liborg, N.-K., Rekdal, A.C., 2005. A survey of controlled
experiments in software engineering. IEEE Trans. Softw. Eng. 31 (9),
733–753.
View publication stats
Succi, G., Pedrycz, W., Djokic, S., Zuliani, P., Russo, B., 2005. An
Empirical Exploration of the Distributions of the Chidamber and
Kemerer Object-Oriented Metrics Suite. Empirical Softw. Eng. 10 (1),
81–104.
Tichy, W.F., 1998. Should computer scientists experiment more? IEEE
Computer 31 (5), 32–40.
Tichy, W.F., 2000. Hints for reviewing empirical work in software
engineering. Empirical Softw. Eng. 5 (4), 309–312.
Tichy, W.F., Lukowicz, P., Prechelt, L., Heinz, E.A., 1995. Experimental
evaluation in computer science: a quantitative study. J. Syst. Softw. 28
(1), 9–18.
van Vliet, H., 2005. Some myths of software engineering education. In:
ICSE’05: Proceedings of the 27th International Conference on
Software Engineering. ACM, New York, NY, USA, pp. 621–622.
van Vliet, H., 2006. Reflections on software engineering education. IEEE
Softw. 23 (3), 55–61.
Vessey, I., Ramesh, V., Glass, R., 2005. A unified classification system for
research in the computing disciplines. Information and Softw. Tech-
nol. 47 (4), 245–255.
Wasserman, A.I., 1984. Editorial policy. ACM Comput. Surveys 16 (2),
103–110.
Wieringa, R., Maiden, N., Mead, N., Rolland, C., 2005. Requirements
engineering paper classification and evaluation criteria: a proposal and
a discussion. Requir. Eng. 11 (1), 102–107.
Wohlin, C., 2005. An analysis of the most cited articles in software
engineering journals – 1999. Information Softw. Technol. 47 (15), 957–
964.
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M.C., Regnell, B., Wesslén,
A., 2000. Experimentation in Software Engineering: An Introduction.
Kluwer Academic Publishers, Norwell, MA, USA.
Wu, J., Graham, T., Smith, P.W., 2003. A study of collaboration in
software design. ISESE 00, 304.
Zelkowitz, M.V., Wallace, D.R., 1998. Experimental models for validating
technology. Computer 31 (5), 23–31.
Zelkowitz, M., Wallace, D., Binkley, D., 2003. Evaluation of new software
engineering technologies. In: Juristo, N. and Moreno, A.M., (Ed.),
Lecture Notes on Empirical Software Engineering, pp. 229–263.
Zendler, A., 2001. A preliminary software engineering theory as investi-
gated by published experiments. Empirical Softw. Eng. 6 (2), 161–180.
Zhang, X., Windsor, J., Pavur, R., 2003. Determinants of software
volatility: a field study. J. Softw. Maintenance 15 (3), 191–204.
Michela Montesi’s research area is scientific communication and the role
of information scientists in mediating communication between research-
ers, and research and practice. She obtained a PhD in Information Science
from the Complutense University of Madrid in 2004, and has spent the
last two years as a guest researcher at the University of Amsterdam.
Patricia Lago is Associate Professor in Software Engineering at the Vrije
Universiteit (VU University) in Amsterdam. Her research interests focus
on software and service architectures, software architecture knowledge
management and modeling. She co-authored various publications in
international journals, books and conference proceedings. She is co-
organizer of the International Workshop on SHAring and Reusing
architectural Knowledge (SHARK). She also acts as reviewer in several
international conferences and journals in the field. A full publication list is
available at www.cs.vu.nl/~patric.