Automatic Sentence Annotation for More Useful
Bug Report Summarization
Abstract—Bug reports have long been an important and useful
software artifact in software development projects with software
developers referring to them for various information needs. As
bug reports can become long due to a large number of comments
from various DevOps tools and conversations between developers,
users of bug reports may need to spend a lot of time reading them.
To address this problem, previous studies developed bug report
summarizers to extract comments from bug reports which were
deemed to be important. Comment importance was determined
based on human-created gold-standard summaries. We believe
creating such gold-standard summaries for evaluating bug report
summarization approaches is not a good practice for a number
of reasons. Consequently, we developed an automatic sentence
annotation method to identify content in bug report comments
which allows bug report users to customize a view for their
specific task-dependent information needs.
Index Terms—bug report summarization, text tagging, text
annotation, natural language processing
I. I NTRODUCTION
Software users and software developers file bug reports1
when they find unexpected behaviours or when they need
additional functionalities after a software product is released.
Those reports are stored in a bug tracking system such as
Bugzilla2 or Jira3 . Software project members, such as triagers,
developers, and project managers, refer to these reports for
completing their work. For example, a bug report triager reads
a bug report to decide the most suitable developer to fix a
bug, or compares two bug reports to determine whether they
are duplicates. A software developer reads through several bug
reports to understand the project history before fixing a new
bug. A project manager examines a collection of bug reports
to make decisions about the next product release. In all of
these examples, a bug report user has to read the bug report
description (i.e. the first comment) and a few or all of the
following comments. While some bug reports may be small
and concise, others may be large due to any combination
of long comment threads, long enumerated lists inside of
comments, or large stack traces or codes snippets.
A bug report can be viewed as a communication thread.
Once a reporter submits a bug description, others, including
other developers and the reporter themself, add comments. The
description contains important information, such as steps to
reproduce the bug, the expected software behaviour and the
actual behaviour of the software product. Comments contain
1 We use the colloquial term ”bug report” to refer to all change requests,
feature requests, or tasks created in a software project’s issue tracking system.
2 https://www.bugzilla.org verified 03/05/2020
3 https://www.atlassian.com/software/jira verified 03/05/2020
questions, clarifications, possible solutions, and attachments
such as patches or screenshots. As a bug tracking system
records most of this information as free-form text, we observed
different representations of the information in bug reports. For
example, the steps to reproduce a bug may be presented as an
enumerated list in one bug report and as a paragraph in another
report. Even the structured data such as code snippets, stack
traces and error messages can appear as or alongside free form
text in bug reports.
As bug reports are often long and contain a lot of textual
information, researchers and practitioners are interested in
creating bug report summaries [1]. The purpose of creating
summaries is to reduce the time spent on reading a bug
report for a particular software engineering task. For example,
Rastkar, Murphy, and Murray did a user study involving bug
report summaries for detecting duplicate bug reports [2], [3].
The findings of Lotufo, Malik, and Czarnecki’s user study
indicated that software developers found summaries useful for
a variety of tasks such as when looking for a solution or a
work-around for a bug, finding the status of an open bug,
and triage activities including searching for duplicate bugs,
determining bug priority, and closing bug reports [4].
Although several bug report summarization approaches have
been proposed [2]–[8], all of these studies focus on creating
a fixed-size summary. However, the restrictions imposed by
a fixed-size summary approach can reduce the quality of the
summary by missing important or useful information for a bug
report user [2]–[4]. Therefore, the summary of a bug report
should be flexible such that the user can include or exclude
information according to their information needs [1].
Previous bug report summarization efforts created con-
densed bug report summaries that didn’t provide for user
interaction [2], [3], [5]–[7]. Although Lotufo, Malik, and Czar-
necki provided two summary views: a condensed summary
and an interlaced summary where the interlaced summary
displayed the entire bug report by highlighting the sentences
extracted as the summary [4], their approach didn’t allow for
any user interaction with the summary. After evaluating the
results and comments of user studies conducted by Rastkar,
Murphy, and Murray and Lotufo, Malik, and Czarnecki, we
concluded that fixed summaries that miss information sought
by some bug report users is a significant flaw in these bug
report summarization approaches. Therefore, we introduce an
approach to automatically annotate the contents of a bug report
such that an interface can allow bug report users to find
and interact with the bug report to meet their task-specific
information needs.
 The contributions of this paper are as follows:
• We present a bug report schema describing its content
and structural dependencies.
• We present a sentence labelling schema to identify inten-
tions of comments on bug reports.
• We present a set of labelling modules to assign labels
to sentences in bug report comments according to the
labelling schema.
• We present a tool that allows bug report users to create a
customized summary for their specific information needs
according to the labels assigned to the sentences.
The rest of the paper is organized as follows. First, a
summary of previous bug report summarization research is
provided. Next, we give detailed arguments for why these
previous approaches may not be the best answer for solving
the bug report summarization problem. We then present our
approach for automatically annotating the sentences in bug
report comments and provide an example of how these an-
notations can be used to customize the view of a bug report
to meet specific bug report user’s information needs. Finally
we present an evaluation of our automatic bug report sentence
annotation process.
II. R ELATED W ORK
This section provides an overview of previous works re-
garding bug report summarization, visualisation of bug report
summaries, and tagging of the contents in bug report com-
ments.
A. Bug Report Summarization
Within the automatic bug report summarization literature,
there are three primary studies: Rastkar, Murphy, and Murray
[2], [3], Mani, Catherine, Sinha, and Dubey [5], and Lotufo,
Malik, and Czarnecki [4]. In short, it was found that the two
unsupervised learning models of Mani, Catherine, Sinha, and
Dubey and Lotufo, Malik, and Czarnecki performed better
than the supervised learning model proposed by Rastkar,
Murphy, and Murray.
The initial study of automatic bug report summarization
was conducted by Rastkar, Murphy, and Murray [2], [3].
This study provided a significant contribution in establishing
a corpus for automatic bug report summarization with 36
bug reports from four open source software projects. Three
human annotators were hired to create both abstractive4 and
extractive5 summaries for each bug report. Rastkar, Murphy,
and Murray applied a supervised learning model to create
extractive summaries by classifying a bug report sentence as
being in the summary or not. Their results indicated that,
even though the average precision was 57% for leave-one-
out cross-validation, the model performed poorly when trying
to summarize a bug report outside of the training corpus. This
was found to be because their supervised learning model was
highly sensitive to the training data.
4 An “abstractive summary” is created based on the semantics of the text.
5 An “extractive summary” is composed of selected sentences from the text.
The next study by Mani, Catherine, Sinha, and Dubey [5]
took an unsupervised learning approach to creating automatic
bug report summaries. This approach was taken as a result of
the observed shortcoming of Rastkar, Murphy, and Murray’s
supervised learning approach. Similar to Rastkar, Murphy, and
Murray, Mani, Catherine, Sinha, and Dubey created their own
bug report corpus with 19 bug reports and hired two annotators
to create both abstractive and extractive summaries. Also, they
used the bug report corpus created by Rastkar, Murphy, and
Murray to evaluate the quality of the generated extractive sum-
maries. Finally, their study introduced an automatic “noise”
identifier intended to differentiate between greeting sentences,
code snippets and stack traces. The authors found that the
extractive summaries improved when the noise was removed.
The final primary study of automatic bug report sum-
marization was conducted by Lotufo, Malik, and Czarnecki
[4]. Like Mani, Catherine, Sinha, and Dubey, the authors
focused on creating an unsupervised bug report summarizer.
In Lotufo, Malik, and Czarnecki’s approach, they considered
the similarity between a sentence and the bug report title, the
similarity between two sentences, and the use of a heuristic to
measure the agreement of two sentences. These characteristics
were used to compare the quality of the generated extractive
summaries and led to a more generalized approach for evalu-
ating bug report summaries.
As part of creating their corpus, the annotators of the
Rastkar, Murphy, and Murray corpus [2], [3] labelled the intent
of a sentence as a ‘problem’, ‘suggestion’, ‘fix’, ‘agreement’,
or ‘disagreement’. The label ‘meta’ was used for sentences
such as codes or URLs. Huai, Li, Wu, and Wang [7] im-
proved upon Rastkar, Murphy, and Murray’s summarizer by
incorporating sentence intentions into a logistic regression
classification model using the calculated probabilities of the
different intentions in Rastkar, Murphy, and Murray’s corpus
to modify the returned probabilities.
Jiang, Li, Ren, Xuan, and Jin [9] argued that feature
engineering is an important part of supervised learning mod-
els. The authors proposed to select features for bug report
summarizer using a crowd-sourced platform. The platform
broadcasts the task (i.e. create a bug report summary) and
asks volunteers to submit their responses. A response contains
summary sentences and reasons for choosing the sentences.
After analyzing the sentences and the reasons, the authors
created eleven features for bug report summarization. Three
of the eleven features were used in the Rastkar, Murphy,
and Murray’s bug report summarizer [2], [3] and one of
the features (bug description sentence) was used in Lotufo,
Malik, and Czarnecki’s unsupervised bug report summarizer
[4]. The evaluation indicated that the logistic regression with
crowd-sourced features out-performed the three bug report
summarizers created by Rastkar, Murphy, and Murray [2], [3],
Mani, Catherine, Sinha, and Dubey [5], and Lotufo, Malik, and
Czarnecki [4].
One of the most recent works on bug report summarization
proposed to apply deep neural networks for feature extrac-
tion. Liu, Yu, Li, Guo, Wang, and Mao [10] integrated an
auto-encoder network for feature extraction and applied it
to Rastkar, Murphy, and Murray’s corpus. The novel metric
called ‘believability’ measured the degree to which a sentence
is approved or disapproved within the conversation. Then the
sentence vectors were modified according to the corresponding
believability score. The performance evaluation showed that
the model out-performed the bug report summarizers created
by Rastkar, Murphy, and Murray and Mani, Catherine, Sinha,
and Dubey However, the model’s precision and Pyramid pre-
cision was lower than the unsupervised bug report summarizer
created by Lotufo, Malik, and Czarnecki.
B. Visualization of Bug Report Summaries
Lotufo, Malik, and Czarnecki addressed the problem of
bug report summary visualization by creating two different
views; one that only displays the summary sentences and
the other that has the entire bug report and highlights the
summary sentences. Their user study found that 56% preferred
the condensed summary view whereas 46% preferred the
interlaced view. Participants indicated that the interlaced view
was preferred as they did not trust the automatic summarizer
and that the sentence highlighting allowed such users to read
through the comments quickly. However, the interlaced view
was found to not be useful for very long bug reports. Yeasmin,
Roy, and Schneider [11] extended this work by using different
colours for the summary sentences so that a user can focus
his/her attention on the specific colour-coded sentences. We
use a similar multi-coloured approach in our task-relevant bug
report summarizer.
C. Tagging Bug Report Comments
As part of their work on providing an efficient interface for
bug report triaging, Bortis and van der Hoek commented that
information in bug report summaries should be interpretable
at-a-glance but still have the descriptive information readily
available [1]. In their work, tags were used to mark bug
reports with common characteristics. Similarly, in our work,
we developed tags to mark sentences with common attributes
such as codes, URLs, and quotes from other comments.
Mani, Catherine, Sinha, and Dubey [5] found that identify-
ing and removing noise when training the classifier improved
the performance of the classification model. They identified
syntax, stack traces, and error messages (labelled as code)
as well as greeting sentences (labelled as other) as noise.
Similar to Mani, Catherine, Sinha, and Dubey’s approach we
identify but do not emphasize the sentences that have noise
tags (e.g. URL, code, and off-topic). We provide an interface
that presents information to understand the bug report with the
flexibility of viewing other information if desired. Therefore,
unlike Mani, Catherine, Sinha, and Dubey’s approach which
completely removed noise sentences for the user, we allow
selective access to any information in the bug report.
As previously mentioned, Rastkar, Murphy, and Murray’s
corpus contains tags that describe the intent of a sentence. In
our work, we chose to refrain from using these tags because
of the inherent problems with human annotated information.
Instead, we propose a broad set of tags that can be extracted
based on keywords (or key phrases) and regular patterns such
as URLs or enumerated lists.
III. P RIOR A PPROACHES A IM AT THE W RONG TARGET
Although the previous works provided significant contribu-
tions for the summarization of bug reports, we believe that
they were aiming for the wrong target. In all of the prior
works, the target was to create an extractive summary that
was as close as possible to a ”gold standard” summary (GSS)
created by human annotators. However, based on observations
in prior work, specifically those by Rastkar, Murphy, and
Murray [2], [3] and our own work [removed for blind review],
we have concluded that such gold-standard summaries are
neither attainable in reality nor wanted in practice.
In the previous studies, the metrics used for evaluating
the summarization systems were those that are traditionally
used for a recommender system: precision, recall, and F-score.
Pyramid precision [12], a variation of the traditional precision
metric accounting for variation in the annotators, was also
used. The annotator-created extractive summaries were used
as the gold-standard when computing these metrics.
However, we found two issues with the gold-standard sum-
maries used for evaluation in these works. The first issue is
related to the word count restrictions placed on the automatic
summarization techniques, which were not applied (or at least
not enforced) on the gold-standard summaries. This resulted
in situations where the constraints placed on the automatic ap-
proach made it improbable, if not impossible, for the automatic
approach to achieve a high score in the evaluation.
The second issue is regarding the agreement, or rather the
disagreement, between the annotators. Both Rastkar, Murphy,
and Murray and Mani, Catherine, Sinha, and Dubey reported a
moderate level of agreement (kappa 0.4) between the human
annotators. This means that annotators themselves cannot
agree on what is a gold-standard summary. Although pyramid
precision attempts to address this issue, it has its own flaws.
Specifically, this metric depends on the number of annotators
(i.e. the number of tiers in the pyramid) and the reliability
of the annotators. Nenkova, Passonneau, and McKeown found
that at least five annotators are needed to have a stable pyramid
score for a summary [12] and the number of annotators was
below this threshold for all previous work.
Moreover, having five or more bug report summary an-
notators is not realistic for software engineering projects.
Assume that a summarization approach focuses on producing
bug report summaries for project managers. We would need
five or more project managers to reasonably establish the
gold-standard extractive summaries. Considering the resource
limitations of most software projects, it is unlikely that a
project would have five project managers for creating an
extractive summary corpus with the needed agreement level.
And project managers are only one type of bug report user.
This leads to a final problem with the goal of previous ap-
proaches; they did not consider the different information needs
that project members have for the summaries. In fact, it is
this difference in information needs between project members
that likely leads to annotator disagreement. For example, in
Rastkar, Murphy, and Murray’s user study, they found that
58% of users would like to have reproduction steps in the
summary when determining whether a bug report is a duplicate
or not. Similarly, bug reporting guidelines for projects (e.g.
[13]–[15] and studies on what are the important items for a bug
report [16] show similar disagreements about what constitutes
a ”good bug report”. Such disagreements about bug report
contents inevitably carry over into annotations for bug report
summaries. In other words, bug report summaries need to be
tailored to the information needs of the user.
Due to all of these points, we believe that creating a ”one-
size-fits-all” gold standard corpus is an unrealistic goal.
IV. AUTOMATED A NNOTATION FOR B UG R EPORT
S UMMARIZATION
This section presents our automated approach for annotating
sentences in a bug report to support task-specific views for
summarizing bug reports.
A. Bug Report Schema for Summarization
Bug reports contain both structured and unstructured data
[16]. Most of the comments appear as free-form text, a form
of unstructured data. Structured data in bug reports include
code snippets, stack traces, error reports, and attachments
[17]. After careful examination of the bug reports in Rastkar,
Murphy, and Murray’s corpus, which we considered to be a
representative sample, we developed a schema of a bug report
for summarization (Figure 1). According to this schema, a bug
report is composed of sentences. The content of some of these
sentences is of interest to a bug report user and we can use
features and keywords to identify the interesting sentences.
Note that the set of “interesting sentences” need not be the
same for every bug report user. Some of the sentences have
dependencies on other sentences in the bug report. We can
use these dependencies between sentences to apply techniques
such as clue words [18] and topic modelling.
Fig. 1. Bug report summarization schema.
The sentences in a bug report have various intentions. We
further refine the Content node of the bug report summariza-
tion schema with the four most common sentence intentions
that we observed in bug reports (Figure 2).
1) Description: The first group of sentences (i.e. comment)
is the Description. The description provides details about
the bug. Some bug descriptions are written as a paragraph
(i.e. unstructured) and others are well-structured. In all cases,
the three intentions of Steps to Reproduce, Actual
Outcome, and Expected Outcome are found here. In
Fig. 2. Bug report sentence intention schema.
a well-structured bug report description, these intentions are
easy to identify. We found that recently reported bug reports
were more likely to contain this structure. However, when ex-
amining the bug reports in the Rastkar, Murphy, and Murray’s
corpus, where some bug reports were reported as early as the
year 2000, automatically identifying these intentions was more
challenging as the descriptions were mostly of the unstructured
form. We created a label Des to annotate sentences in the first
comment.
Once a bug is reported, other bug report users (e.g. triagers,
developers, project managers) post comments. Upon careful
examination of the comments subsequent to the first comment
(i.e. the bug description), we found the following three inten-
tions.
2) Clarification: If a bug description or any other com-
ment is not clear enough, a bug report user requests further
information with a response appearing in a later comment.
We identify such groups of sentences as having an intention of
Clarification. As developers typically quote the original
sentence(s) from the previous comment(s) when they request
further information or when they respond to an information
request, this creates a communication structure similar to that
of email threads. We created the labels Org and Qt to annotate
original and quoted sentences respectively. The label CW was
created to annotate sentences that are related to the quoted
sentences.
3) Resolution: Sometimes when a bug reporter identifies a
problem, they also provide a solution. Alternatively, a solution
may be provided by another bug report user, typically the
developer who is responsible for resolving the report. We
refer to this intention as Resolution. To annotate sentences
related to bug resolution, we introduced a label Res.
4) Plan: A bug report user may introduce an idea to
solve an issue but not provide the actual solution as with
a Resolution comment. In this case, the comment is
considered to have a Plan intention.
a) Other labels: We mentioned five labels that are di-
rectly related to sentence intentions. Apart from those labels,
we created three additional labels to annotate sentences. The
labels OT, URL, and Code represent off-topic sentences,
URLs, and code snippets respectively. Such sentences appear
 TABLE I
D ESCRIPTION OF LABELS
Label Description
Des Bug report description including steps to reproduce, expected
outcome and actual outcome
CW Clue words
Org Original sentence if the sentences is quoted in later comment
Qt Quoted sentence from a previous comment
Topic Topic word is included
Res Resolution statements (contain fix, patch, attachment)
Plan Proposals for changes to design or implementation.
OT Off-topic sentence
URL Sentence contains a URL
Code Codes, stack traces, error messages and etc.
in different intentions in bug reports. Therefore, those three la-
bels are not directly related to any of the intentions introduced
above.
We created one more label, Topic, to annotate sentences
that participate in the ongoing conversation of the bug report.
This label is also not directly related to an intention.
B. Sentence Labels
We propose ten labels to identify the content found in bug
report sentences (Table I). The first seven labels (Des, CW,
Org, Qt, Topic, Res and Plan) are assigned to sentences
that are considered to contain useful information for bug report
users. The last three labels (OT, URL, and C) are assigned to
sentences considered to be unimportant or noise, which can
then be filtered out for the user.
1) Description [Des]: The bug report’s description contains
essential information for bug report users. For example, steps
to reproduce, observed results, and expected results are all
useful to triagers when determining duplicate bug reports or
to developers when creating a fix.
2) Clue Word [CW], Orginal [Org], and Quoted [Qt]:
Developers often quote sentences from other developers when
they need further information about that comment or when
they need to respond to that comment. For example, if a
developer posts a question, another developer will post the
answer by first quoting the question and then replying to it.
Or if a developer posts a way to solve a bug, another developer
will post a comment accepting the solution or requesting
more information by quoting the solution. Since these quotes
and responses create a chain-like communication between
developers, we use the labels Org for an original sentence
(i.e. start node) and Qt for when that sentence is quoted (i.e.
end node). The label CW is attached to a sentence if both the
response and the quote have words in common. CW denotes
the presence of “clue words” as proposed by Carenini, Ng,
and Zhou [18] for email summarization.
3) [Topic]: We observed that developers sometimes re-
spond to other comments without quoting the previous com-
ment(s). As these communication threads are not explicit, we
cannot directly identify such communication patterns in the
bug report. This can result in missing important information
and such sentences not receiving a label. The Topic label
addresses this problem by identifying sentences that participate
in the dominant topic of the bug report.
4) Resolution [Res]: Once an bug report is created, it goes
through different states, as indicated by the report’s status,
with the actual states differing between bug tracking systems
[19]. Within these different states, developers try to reproduce
the bug, provide screenshots or mock-ups of user interfaces,
create partial solutions, or create patches that may be accepted
as the solution by another developer. Such attachments, partial
patches and final solutions are of interest to bug report users
[16].
5) [Plan]: It is common for a discussion in a bug report to
revolve around the planning of how to fix a bug or implement
a feature. Such sentences are given the Plan label.
6) Off-Topic [OT]: During our inspection of bug reports,
we found comments which did not contribute to the on-
going conversation. Those comments were usually greetings
or appreciations towards another developer. The label OT is
assigned to such a sentence to reflect that it is an off-topic
comment in the bug report.
7) [URL]: URLs are often added to bug reports either
manually by bug report users or automatically by DevOps
tools. For example, we observed links to other bug reports
and to version control commits. Links to other bug reports
allow developers to investigate the content of these other bug
reports. This may help a developer in solving a current bug, or
a triager to create a “super bug” that gathers together a set of
reports that have similar behaviour. Links to version control
systems, such as Git and Subversion, allow developers
to inspect the changes made by a patch. Although URLs may
be considered as noise for summarization, a bug report user
could benefit from having these links accessible when a deep
investigation of all of the information is needed. Therefore,
we consider being able to identify URLs automatically and
allowing users to view them as important.
8) [Code]: Bug report comments can contain code snip-
pets, error messages, software configuration parameters, stack
traces and software version names and numbers. Although this
information may be relevant to a bug report, it may also not
be essential for understanding the bug report. Mani, Catherine,
Sinha, and Dubey identified such information as noise in their
study because the summarization models performed poorly
when noise was present. We assign the label Code to such
sentences so that the user can decide whether to show or hide
such information in the summary view.
C. Automatic Labelling of Bug Report Sentences
For a given bug report, each sentence is extracted from a
comment and passed to various modules for labelling. Figure
3 provides an overview of the process. The process proceeds
in the following manner.
First, a sentence is converted to lower case, stop words
and irrelevant symbols such as emojis are removed, and the
words are lemmatized. Digits or punctuation symbols are not
removed as these are necessary to identify URLs and code
snippets.
 Next, the text is passed to various modules for labelling.
The following is a detailed description of each of the labelling
modules. Note that the order of calling the URL Detector,
Code Detector, and Off-Topic Classifier is im-
portant because the Code Detector, and Off-Topic
Classifier depend on the output of the previously called
module. However, the rest of the modules are independent of
each other and they may be called in any order.
1) Description Labeller: The label Des is applied to a
sentence in the first bug report comment if it is determined
to either contain steps to reproduce the bug, observed be-
haviour or expected behaviour. When reporting an issue in
issue tracking systems such as Bugzilla, it is not required
to fill all or any textual details for steps to reproduce,
observed behaviour, and expected behaviour. When those
details are present explicitly, we used regular expressions
to capture the explicitly stated steps to reproduce, observed
behaviour and expected behaviour. The regular expressions
used to capture those statements are: step reproduce,
actual results, expect results.
If these regular expressions fail, another regular expression
is used for identifying any enumerated lists, with the assump-
tion that such a list is describing the steps for reproduction.
We used the regular expression ˆ\)?\d+(\.|\)) to capture
enumerated lists.
Finally, the sentences in the bug report description are
compared with the bug report title. If there is a common word
in both the sentence and the title we apply the Des label to
the sentence.
2) URL Detector.: All comments are passed into this
module to find sentences containing URLs. We use regular
expressions to identify URLs in comments and assign them
the label URL.
3) Code Detector.: By analysing Rastkar, Murphy, and
Murray’s corpus and identifying common patterns of noise
present in bug reports, we created regular expressions to cap-
ture those patterns. The Code Detector module identifies
program language syntax, error messages, stack traces, and
configuration parameters. The identified sentences are labelled
as Code. This module ignores the sentences identified by the
URL Detector.
4) Off-topic Classifier.: To determine the off-topic com-
ments, we follow Chowdhury and Hindle’s approach for
filtering off-topic comments from Python IRC channels [20].
We extracted comments from YouTube6 and Stack Overflow7
as positive (i.e. off-topic) and negative (i.e. on-topic) examples,
respectively, to train a classification model. The YouTube
dataset consists of comments from YouTube channels with
the keywords cooking, news and sports, as these were con-
sidered to likely contain a good representation of common
off-topic comments with respect to software development. For
the Stack Overflow dataset, we did not filter questions from
any programming language or technology types because the
6 https://developers.google.com/youtube/v3/getting-started
7 https://api.stackexchange.com/docs
bug report corpus contains data from four software projects.
We created a diverse dataset by extracting ∼3000 lines of
comments from each source.
During preprocessing, we converted all text to lowercase
and removed all punctuations symbols, extra white spaces,
and stop words. The comments from Stack Overflow were
manually filtered because we found some greeting comments.
We excluded those comments when training the models. Then
we lemmatized the words to remove the effect of variations of
the same word. We trained a Support Vector Machine (SVM)
classifier8 to categorize the bug report comments into off-topic
or on-topic comments.
This module ignores sentences previously labelled by the
URL Detector and Code Detector, and assigns the OT
label to sentences classified as “off-topic”.
5) Clue-word Detector: The entire bug report is passed into
this module to determine the clue-word relationships between
sentences. Identified sentences are considered when determin-
ing the Clarification intention. The Clue-word Detector
module uses the approach proposed by Carenini, Ng, and Zhou
[18] for email summarization and used by Rastkar, Murphy,
and Murrayin their bug report summarization approach. This
modules assigns the labels of CW, Org and Qt.
6) Topic Modeller: To determine the dominant topic of a
bug report, we apply a topic modelling approach from the
field of Natural Language Processing. Topic modelling is a
technique used to identify topics from a pool of documents
from different fields [21]. However, in our case, we are not
interested in identifying the different topics in a bug report.
Instead, we utilize the technique to capture the words of the
dominant topic in the bug report by extracting only one topic
from the topic model. Then we compare the sentences against
the set of topic words to find sentences that participate in the
dominant conversation of the bug report. We assign the label
Topic to these sentences.
In this study, we used Latent Dirichlet Allocation (LDA)
[21] to extract topic words. We created a corpus for each
bug report using a bag of word model and extracted ten topic
words. Since we were only interested in the most dominant
conversation, the number of topics was set to one.
7) Resolution Detector: All comments, except the first
one, are passed into this module to find sentences with the
Resolution intention. We use regular expressions to identify
attachments and comments that reflect the final solution or an
intermediate step towards the final solution. The label Res is
assigned to a sentence if that sentence has such keywords as
push, fix, patch, attach, and commit. Similar to clue
word matching, we do root word matching instead of exact
word matching.
D. Plan Detector
As we do not yet have a specific pattern or set of keywords
to identify Plan sentences, we did not implement such a
labelling module. This is due to the diversity of problems
8 A Naı̈ve Bayes classifier was also trained, but we found it produced too
many false positives.
 Fig. 3. Overview of the labelling and summarization process

described in bug reports and developers’ personal preference TABLE II

when composing comments. For example, if the proposed Q UALITY OF AUTOMATIC ANNOTATIONS
solution to a problem is small, such as changing a parameter Label Precision Recall F-Score
or adding one line to the source code, the comment could be Des 90.91% 48.34% 63.12%
a single sentence. On the other hand, a proposed solution may CW 65.33% 63.64% 64.47%
Org 92.31% 98.63% 95.36%
be complex and have multiple lines of code snippets and/or QT 92.68% 98.7% 95.6%
comments. Also, developers express their Plan intention in a RES 83.97% 77.06% 80.37%
variety of ways, such as enumerated lists and paragraphs. Con- OT (SVM) 58.23% 50% 53.8%
OT (Naı̈ve Bayes) 54.43% 50.59% 52.44%
sequently, we were unable to derive a general enough heuristic URL 86.84% 80.48% 83.54%
to assign the Plan label to comments with a sufficient level Code 63.81% 45.12% 52.86%
of accuracy. However, Chaparro et al.’s work [22] on discourse
patterns may provide a starting point for work in this direction.

V. R ESULTS AND D ISCUSSION
To develop and test our approach for automatically labelling
bug report sentences, we chose to use the bug report corpus
curated by Rastkar, Murphy, and Murray. This bug report
corpus is a collection of XML documents where sentences
are identified by a <Sentence> tag.
The choice to use Rastkar, Murphy, and Murray’s bug report
corpus was made for two reasons. First, the dataset is publicly
available9 and is the bug report corpus used in the previous
two studies of bug report summarization by Mani, Catherine,
Sinha, and Dubey [5] and Lotufo, Malik, and Czarnecki
[4]. Second, as the comment sentences are already identified
and verified, this removes errors occurring by a sub-optimal
sentence tokenisation approach on a raw bug report dataset.
A. Manual Labelling of Sentences
To evaluate the quality of our automatic sentence labelling
approach, four volunteers were asked to annotate by hand
9 https://github.com/HuaiBeibei/IBRS-Corpus verified 01/05/2020
the sentences in Rastkar, Murphy, and Murray’s corpus with
the labels from Section IV-B. One annotator examined and
labelled the entire corpus. The other volunteers were assigned
non-overlapping thirds of the corpus to label independently.
Once the labelling was completed, the first annotator met
individually with the other annotators to resolve any disagree-
ments. The agreed upon annotations were then compared with
the labels assigned by the approach. Note that our annotation
mechanism does not require annotators’ personal judgement
as we were searching for lexical characteristics or structural
characteristics in sentences. Therefore, annotating sentences to
indicate the intention(s) and annotating sentences to indicate if
it were a summary sentence are two different tasks. The latter
task involves personal judgement that introduces bias into
the summary and is why multiple annotators and agreement
rates are important when creating a bug report summary
corpora. We found that any disagreements in sentence labelling
were due to mistakes made during the annotation and/or
misinterpreted guidelines. After the resolution meetings, the
agreement rate was 100%.
B. Results of Automatic Labelling
Table II shows the precision, recall and F-score (Equations
1, 2 and 3) for each type of label assigned by our approach.
The remainder of this section provides a discussion for each
labelling module of these results.
# of labels correctly assigned
P recision = (1)
total # labels assigned
# of labels correctly assigned
Recall = (2)
total # labels assigned by annotators
2 × precision × recall
F − score = (3)
precision + recall
1) Description Labeller: The Description Labeller
achieved a 91% precision with a recall of 48%. After analyzing
the false negatives, we found that the module missed the root
form similarity of words such as rescale and scaling. For
example, in bug report GIMP #164995 the term scale was
in the title and the words rescaled and scaling appeared in the
bug report description. But the module lemmatizer failed to
find the root form of these words as scale, and the label Des
was not assigned.
Most of the bug reporters used plain text to describe ex-
pected and actual software behaviour in the description. How-
ever, some reporters included stack traces along with other
text. This resulted in the Code Detector identifying code
fragments and other program-related sentences that were part
of the description. However, recall that the Description
Labeller uses three different strategies to assign a Des
label: words in common with the title, enumerated lists, and
structured text. As a result, the Description Labeller
would assign the Des label to the same sentences as those iden-
tified by the Code Detector if the sentences appeared after
the identified steps to reproduce, actual behaviour, and ex-
pected behaviour. For example, lines 18-151 of the description
in Bugzilla #429126 is a stack trace. As the expected behaviour
is explained in lines 15-17, the Description Labeller
labelled every sentence afterwards (i.e. the stack trace) as
Des. In other words, the Description Labeller falsely
labelled 133 sentences in this bug report.
When investigating the reasons for a low recall rate
(48.34%), we found that the Description Labeller
failed to distinguish between the use of some words from
code, whereas a person would make this distinction. For
example, the title of Eclipse #260502 has the word class
inside brackets (i.e. ”class”) and the word class is repeated
in the first comment. To a person, it is obvious that the
reporter is describing class documentation, however to the
Description Labeller these sentences appeared to be
a piece of code and was not labelled.
2) Clue-word Detector: The Clue-word Detector
achieved a 65% precision and a 64% recall. We found that
the detector did well at capturing original (Org) and quoted
(QT) sentences. However, capturing clue words (CW) was not
as effective.
When investigating the clue words missed by the detector,
we found that certain words were not converted to the expected
root form by the lemmatizer. For example, localization was
not converted to the same root as localize. Although a person
can identify such words as clue words, the Clue-word
Detector failed to identify words having similar semantics.
3) Off-topic Classifier: The Off-topic Classifier
had a precision and recall of 58% and 50%, respectively.10
These results appear to be related to the positive data set
(i.e. from YouTube) mostly containing short greeting sentences
(e.g. “Thank you”, “Thanks”, and “Much appreciated”). Con-
sequently, we found that the classifier successfully detected
short greetings, but the classifier failed when the sentences
were long. For example, in Eclipse #224588 the sentence
“Again, I think it’s a brilliant idea Martin.” was not correctly
labelled as being off-topic.
4) Code Detector: Our Code Detector achieved preci-
sion and recall values of 64% and 45%, respectively. Regard-
ing these values, the regular expression patterns in the Code
Detector captured syntax embedded in regular English
sentences as opposed to proper code snippets. Such sentences
were not marked as code by the human annotators because
they observed that the programmers were discussing the code,
not providing code for context or as a solution.
When creating our labelling modules, we chose not
to remove symbols and stop words which are impor-
tant for detecting programming language syntax. This re-
sulted in some sentences that contain brackets, parenthe-
sis, and special symbols, which are common in program-
ming language syntax, being detected as code. For exam-
ple, Eclipse #223734 contains the sentence “According to
the JavaDoc in Platform.getResourceString(Bundle,String)...”,
which was labelled as code according to the regular expression
\w+[\.]?\w+\(((\w+,?\s?)?+)\) used in the mod-
ule. Another example is Eclipse #260502 which describes an
issue in a JavaDoc file. The reporter provided an example of
the order of defining a set of classes and interfaces. As these
comments contain the words class and interface we could
not write a regular expression pattern that identifies such a
comment as code, but also correctly excludes sentences that
contain the same words.
5) Resolution Detector: We found our Resolution
Detector to be effective with precision and recall rates
of 84% and 77%, respectively. Recall that the Resolution
Detector searched for keywords such as patch, push, com-
mit, attach, and fix. Most of the comments that contained such
keywords, in their various forms, were found by the human
annotators to describe either the solution or an intermediate
10 A classifier trained using a Naı̈ve Bayes algorithm was found to be
slightly worse with a precision and recall of 54% and 51%, respectively.
form of the final resolution for the bug report. However, certain
sentences were questions that inquired about a patch or a fix,
or a further explanation that contained one of the keywords.
For example, Eclipse #154119, contains the sentence “It would
be possible push parts of TextFieldNavigationHandler down to
Platform Text...”, which contains the keyword push but is not
a resolution statement.
6) URL Detector: The URL Detector was effective in
correctly identifying links, with a precison of 87% and a
recall of 80%. Although URL regular expressions are well-
known and developed, the reason that this detector did not
achieve as a higher score was due to URL patterns in stack
traces. We found that some stack traces contained the text
http://localhost which were labelled as URL by the
module. Such URLs were not considered useful to bug report
users by the annotators.
VI. T HREATS TO VALIDITY
One threat to the validity of our work is the generalizability
of our labelling process. Our regular expression patterns were
written by investigating the sentences in the Rastkar, Murphy,
and Murray’s corpus. This may mean that there are bug reports
in some software projects where there regular expressions
would fail. However, we believe this threat to be small, as
the corpus contains bug reports extracted from four different
projects where each project belongs to a different domain.
Therefore, we expect that our modules cover different code
styles, error reports, and stack traces in various programming
languages such as Java and C/C++.
Another threat to validity relates to the labelling of sen-
tences in Rastkar, Murphy, and Murray’s corpus by human
annotators. We used two annotators for each report and
resolved disagreements when they occurred. However, it is
possible that both annotators may have agreed on an incorrect
label. Similarly, it may seem that this approach would suffer
from the same annotator disagreement problem as expressed
in Section III. However, we consider that the annotations of
sentences from a fixed set of well-defined labels is much less
error-prone and subjective than the selection of sentences for
summarization.
VII. A TASK R ELEVANT B UG R EPORT S UMMARIZER
Having labelled the bug report sentences by their content
type or intention, an interface can be provided to the bug report
users which presents the labelling and allows the user to filter
the sentences according to their information needs. Figure 4
provides an example of the user interface for such a tool that
supports the creation of task-relevant bug report summaries.
The interface is modelled after Mozilla’s Bugzilla11 interface
to leverage familiarity with that or similar systems.
The tool provides access to all of the comments from
the bug report instead of displaying only the automatically
selected sentences as in a traditional bug report summarizer.
Similar to Bugzilla and other issue tracking systems, the user
can choose to expand or collapse individual comments at will.
11 https://bugzilla.mozilla.org/
At the top of the interface are checkboxes for the various
labels, allowing the user to show or hide sentences according
to the particular sentence label. On the right-hand side is
provided a list of the dominant topic words within the bug
report, allowing the user to filter the sentences by topic.
As mentioned, sentences are displayed with a label indi-
cating to which sentence category it belongs. The labels are
colour-coded to aid in understanding the relevance of the
sentence category. For example, the Resolution label is green
to indicate a high perceived relevance to a user, and the Off-
topic annotation is yellow to indicate that the sentence is likely
not relevant to the user. In Figure 4, the user has selected to
focus on sentences with the labels Des, CW, Qt, Org, Topic,
and Res. Sentences that are visible, but are not one of these
categories, are shown greyed out, such as “LOL, me too”
which has an OT label.
When the user selects a topic word that they are inter-
ested in, the interface highlights the word in the comments
to focus the user’s attention. In Figure 4 the user selected
preference in the topic word list, and the comments with
this word are highlighted (e.g. at the top of the screen).
VIII. C ONCLUSION AND F UTURE W ORK
In this study, we focused on creating customizable bug
report summaries based on our belief that fixed-length sum-
maries cannot support the different information needs of
diverse software development roles. Since different users are
interested in a variety of information, we developed a set
of labelling modules to identify the content or intention of
sentences in bug report comments. The empirical results indi-
cate that, in general, these labelling modules achieve sufficient
precision and recall scores for practical use. The code detector
and off-topic classifier received slightly over 50% F-score.
Even though those scores were not as high as the other sub-
modules, they provide a good first step in the direction of bug
report sentence labelling.
In the future, we plan to curate a better training set and
examine the use of NLP word embedding techniques to
improve the effectiveness of the Off-Topic Classifier module.
Also, we plan to examine how to improve the performance of
the Code Detector module.
Although we implemented a proof-of-concept task-relevant
bug report summarizer, it was designed to work with data
from Rastkar, Murphy, and Murray’s corpus. We plan to
implement a prototype that works with data from a ”live” bug
repository and conduct a user study to determine how useful
our automated sentence annotations are to software project
members. Also, the user study will involve project members
with different roles to better understand the importance of each
label to their tasks. Such a study will provide a baseline for
future approached towards task-relevant bug report summariz-
ers.
R EFERENCES
[1] G. Bortis and A. van der Hoek, “Porchlight: a tag-based approach
to bug triaging,” in 35th International Conference on Software
Engineering, ICSE ’13, San Francisco, CA, USA, May 18-26,
 Fig. 4. User interface of customizable bug report summary
2013, D. Notkin, B. H. C. Cheng, and K. Pohl, Eds. IEEE
Computer Society, 2013, pp. 342–351. [Online]. Available: https:
//doi.org/10.1109/ICSE.2013.6606580
[2] S. Rastkar, G. C. Murphy, and G. Murray, “Summarizing software
artifacts: A case study of bug reports,” in Proceedings of the
32nd ACM/IEEE International Conference on Software Engineering
- Volume 1, ser. ICSE ’10. New York, NY, USA: Association
for Computing Machinery, 2010, p. 505–514. [Online]. Available:
https://doi.org/10.1145/1806799.1806872
[3] S. Rastkar, G. C. Murphy, and G. Murray, “Automatic summarization of
bug reports,” IEEE Trans. Softw. Eng., vol. 40, no. 4, pp. 366–380, Apr.
2014. [Online]. Available: https://doi.org/10.1109/TSE.2013.2297712
[4] R. Lotufo, Z. Malik, and K. Czarnecki, “Modelling the ‘hurried’ bug
report reading process to summarize bug reports,” Empirical Softw.
Engg., vol. 20, no. 2, pp. 516–548, Apr. 2015. [Online]. Available:
http://dx.doi.org/10.1007/s10664-014-9311-2
[5] S. Mani, R. Catherine, V. S. Sinha, and A. Dubey, “Ausum: Approach
for unsupervised bug report summarization,” in Proceedings of the ACM
SIGSOFT 20th International Symposium on the Foundations of Software
Engineering, ser. FSE ’12. New York, NY, USA: ACM, 2012, pp. 11:1–
11:11. [Online]. Available: http://doi.acm.org/10.1145/2393596.2393607
[6] H. Jiang, N. Nazar, J. Zhang, T. Zhang, and Z. Ren, “PRST: A
pagerank-based summarization technique for summarizing bug reports
with duplicates,” International Journal of Software Engineering and
Knowledge Engineering, vol. 27, no. 6, pp. 869–896, 2017. [Online].
Available: https://doi.org/10.1142/S0218194017500322
[7] B. Huai, W. Li, Q. Wu, and M. Wang, “Mining intentions
to improve bug report summarization,” in The 30th International
Conference on Software Engineering and Knowledge Engineering,
Hotel Pullman, Redwood City, California, USA, July 1-3, 2018,
Ó. M. Pereira, Ed. KSI Research Inc. and Knowledge Systems
Institute Graduate School, 2018, pp. 320–319. [Online]. Available:
https://doi.org/10.18293/SEKE2018-096
[8] X. Li, H. Jiang, D. Liu, Z. Ren, and G. Li, “Unsupervised deep
bug report summarization,” in Proceedings of the 26th Conference on
Program Comprehension, ICPC 2018, Gothenburg, Sweden, May 27-28,
2018, F. Khomh, C. K. Roy, and J. Siegmund, Eds. ACM, 2018, pp.
144–155. [Online]. Available: https://doi.org/10.1145/3196321.3196326
[9] H. Jiang, X. Li, Z. Ren, J. Xuan, and Z. Jin, “Toward better
summarizing bug reports with crowdsourcing elicited attributes,” IEEE
Trans. Reliab., vol. 68, no. 1, pp. 2–22, 2019. [Online]. Available:
https://doi.org/10.1109/TR.2018.2873427
[10] H. Liu, Y. Yu, S. Li, Y. Guo, D. Wang, and X. Mao, “Bugsum:
Deep context understanding for bug report summarization,” in ICPC
’20: 28th International Conference on Program Comprehension, Seoul,
Republic of Korea, July 13-15, 2020. ACM, 2020, pp. 94–105.
[Online]. Available: https://doi.org/10.1145/3387904.3389272
[11] S. Yeasmin, C. K. Roy, and K. A. Schneider, “Interactive visualization
of bug reports using topic evolution and extractive summaries,” in
30th IEEE International Conference on Software Maintenance and
Evolution, Victoria, BC, Canada, September 29 - October 3, 2014.
IEEE Computer Society, 2014, pp. 421–425. [Online]. Available:
https://doi.org/10.1109/ICSME.2014.66
[12] A. Nenkova, R. Passonneau, and K. McKeown, “The pyramid method:
Incorporating human content selection variation in summarization
evaluation,” ACM Trans. Speech Lang. Process., vol. 4, no. 2, p.
4–es, May 2007. [Online]. Available: https://doi.org/10.1145/1233912.
1233913
[13] S. Tatham, “How to report bugs effectively,” https://www.chiark.
greenend.org.uk/∼sgtatham/bugs.html, accessed: 2020-02-03.
[14] “Apache tomcat bug reporting,” https://tomcat.apache.org/bugreport.
html#How to write a bug report, accessed: 2020-02-03.
[15] “Bug report writing guidelines - mozilla,” https://developer.mozilla.org/
en-US/docs/Mozilla/QA/Bug writing guidelines, accessed: 2020-02-03.
[16] N. Bettenburg, S. Just, A. Schröter, C. Weiss, R. Premraj, and
T. Zimmermann, “What makes a good bug report?” in Proceedings
of the 16th ACM SIGSOFT International Symposium on Foundations
of Software Engineering, ser. SIGSOFT ’08/FSE-16. New York,
NY, USA: Association for Computing Machinery, 2008, p. 308–318.
[Online]. Available: https://doi.org/10.1145/1453101.1453146
[17] N. Bettenburg, R. Premraj, T. Zimmermann, and S. Kim, “Extracting
structural information from bug reports,” in Proceedings of the 2008
International Working Conference on Mining Software Repositories,
MSR 2008 (Co-located with ICSE), Leipzig, Germany, May 10-
11, 2008, Proceedings, A. E. Hassan, M. Lanza, and M. W.
Godfrey, Eds. ACM, 2008, pp. 27–30. [Online]. Available: https:
//doi.org/10.1145/1370750.1370757
[18] G. Carenini, R. T. Ng, and X. Zhou, “Summarizing email conversations
with clue words,” in Proceedings of the 16th International Conference
on World Wide Web, ser. WWW ’07. New York, NY, USA:
 Association for Computing Machinery, 2007, p. 91–100. [Online].
Available: https://doi.org/10.1145/1242572.1242586
[19] J. Anvik, L. Hiew, and G. C. Murphy, “Coping with an open
bug repository,” in Proceedings of the 2005 OOPSLA workshop on
Eclipse Technology eXchange, ETX 2005, San Diego, California, USA,
October 16-17, 2005, M. D. Storey, M. G. Burke, L. Cheng, and
A. van der Hoek, Eds. ACM, 2005, pp. 35–39. [Online]. Available:
https://doi.org/10.1145/1117696.1117704
[20] S. A. Chowdhury and A. Hindle, “Mining stackoverflow to filter out
off-topic IRC discussion,” in 12th IEEE/ACM Working Conference
on Mining Software Repositories, MSR 2015, Florence, Italy, May
16-17, 2015, M. D. Penta, M. Pinzger, and R. Robbes, Eds.
IEEE Computer Society, 2015, pp. 422–425. [Online]. Available:
https://doi.org/10.1109/MSR.2015.54
[21] D. M. Blei, A. Y. Ng, and M. I. Jordan, “Latent dirichlet allocation,”
J. Mach. Learn. Res., vol. 3, pp. 993–1022, 2003. [Online]. Available:
http://jmlr.org/papers/v3/blei03a.html
[22] O. Chaparro, J. Lu, F. Zampetti, L. Moreno, M. D. Penta, A. Marcus,
G. Bavota, and V. Ng, “Detecting missing information in bug
descriptions,” in Proceedings of the 2017 11th Joint Meeting on
Foundations of Software Engineering, ESEC/FSE 2017, Paderborn,
Germany, September 4-8, 2017, E. Bodden, W. Schäfer, A. van
Deursen, and A. Zisman, Eds. ACM, 2017, pp. 396–407. [Online].
Available: https://doi.org/10.1145/3106237.3106285