Using Text Annotation Tool on Cyber Security
News – A Review
Mohamad Syahir Abdullah
School of Computing, Universiti Teknologi Malaysia,
Johor, Malaysia
msyahir38@utm.my
Mohd Aizaini Maarof
School of Computing, Universiti Teknologi Malaysia,
Johor, Malaysia
aizaini@utm.my
Abstract— Cyber-attack has become one of the main
concern in our everyday life and being reported
throughout online news website. As thousands of news
article existed, it is difficult to go through all the news
which lead to a slower analyzing process. Hence, a vital
text mining component known as Information
Extraction (IE) is needed in order to ease the
knowledge discovery process for the wide collection of
the cyber security news. To make IE process better and
easier, the usage of tool such as General Architecture
for Text Engineering (GATE) can help a lot especially
in creating annotated corpus. In this paper, we will
introduce and reviewing several annotation tools that
are freely available and also to discuss steps needed to
create an annotated corpus for the cyber security text
documents.
Keywords— Cyber-attacks, online news, Information
Extraction (IE), annotation tool, GATE
I. INTRODUCTION
Cyber-attack incidents have become a norm occurrence
in our cyber space. From time to time, more damages and
lost involving our cyber properties and information
technologies are inflicted [1]. As the number of incidents
increased, people need to start taking precaution steps to
avoid being a victim of the cyber-attacks. Despite that, the
fast growth of internet also resulting in a better information
dispersal through online news platform. Lot of information
especially on the cyber-attack incidents can be spread
instantly across the globe. However, as there are thousands
of news available online, it gives more difficulties for
people to access to a correct and precise information they
needed [2].
The difficulty is higher when dealing with different
kind of news format from the online website especially
involving the unstructured text such as news article [18].
News articles are usually written in lengthy and descriptive
manner. Although a lot of time will be consumed for a
lengthy document to be analyzed and processed, it can
provide us with opportunity to highlight or extract lots of
valuable information existed in the text through annotation
32
Authorized licensed use limited to: University of Exeter. Downloaded on June 23,2020 at 01:32:47 UTC from IEEE Xplore. Restrictions apply.
Anazida Zainal
School of Computing, Universiti Teknologi Malaysia,
Johor, Malaysia
anazida@utm.my
Mohamad Nizam Kassim
Cyber Security Responsive Division,
CyberSecurity Malaysia,
Seri Kembangan, Selangor, Malaysia
nizam@cybersecurity.my
process. The annotated text then can be used as a training
example in text analytics. However, as previously, a human
tend to work manually with the text, such as to annotate text
with a label which it has been proved to be a tedious, error
prone process and also consume a lot of time to be
completed [3].
Opportunely, the advancement of IE this past few years
has help to ease the text mining process. IE is important in
extracting information from the unstructured data or text.
IE help in conveying it into a more understandable human
language text by the support of natural language processing
(NLP) which are able to locate any instances related [4] to
the cyber security domain from the pile of documents
available. One of the important IE task is called Named
Entity Recognition (NER) which is usually used to locate
the targeted information that need to be extracted. NER is
also used for classifying the named entity into several
categories such as person, location, organization, date and
time. However, the performance of NER can easily be
affected with the absence of good training data.
To achieve or create a good dataset for training
purposes, it will start with data collection. Then, the
collected data should undergo a process called annotation.
Text annotation is a way of interaction with the text-
usually by marking or highlighting some part from the text.
It goes by underlining a certain piece of information that is
in our focus areas and is significant to convey the meaning
behind the text. As tedious it can be when manual annotate
is conducted [3], it surely help a lot in creating corpus for
the training purposes. Thus, to aid this process, the
introduction of annotation tools that is more precise, and
can be run automatically have help lots of researchers in
completing their works.
Some of the annotation tools are open-sourced. One of
the commonly used is the General Architecture for Text
Engineering (GATE). There are also another tools such as
brat rapid annotation tool (BRAT), WARP-Text and so on.
These tools are used quite often in text mining tasks
especially during data annotation. GATE for example
contains different kinds of plugin which related to NLP
components such as POS taggers, Named Entity
Recognition (NER) and so on. GATE also provide a complete text mining tasks. Figure 1 shows the annotation
friendly interface which let us to indicate part of the texts that can be made and the relationship between the entities
that represent our perceptions and ideas. In the meantime, are formed.
BRAT is an intuitive web-based tool that also supported by
BRAT has been associated in biomedical domain and
NLP. Meanwhile, BRAT can serve as an intuitive interface
works with various text format [13] [14] using both Python
that help the non-technical user to have a better annotation
and JavaScript which also based on a client-server
[3].
architecture as it provide services such as named entity
This paper consist of the following structures. In the annotation, dependency syntactic annotation and binary, n-
next section, we will discussing some related previous ary relation annotation.
works that has been made by others and several tools
(GATE, BRAT, WART-Text, etc.) used by them. Then, the
subsequent section discuss mainly on how to use GATE
tool that include the process from data collection to the
annotation process. Finally, the last section will contain the
conclusion of this paper and some of the future works that
can be done.

II. RELATED WORKS

Annotation tools can provide immense help to the

researchers and save lot of time and cost for annotating.
Using annotation tool can avoid us from any unintended
mistake compare to traditional human annotation that is
prone and vulnerable to errors and incompetency [5]. In
this section, several annotation tools that are popular
among the researchers such as GATE, BRAT and other
tools will be discussed.
A. GATE
This annotation tool is freely available and regarded as
an open-sourced language engineering infrastructure [6].
One of the many purposes of GATE is to annotate the text
document which is not only limited to English language.
GATE uses a UNICODE throughout and has been
practiced to annotate many other language [8] [9] such as
Indic language, where millions of corpus has been created
for the American National Corpus and EMILLE [7]. GATE
also specialized in extracting a specific knowledge from the
unstructured text obtained from the Web. [5] combined the
ontologies on targeted knowledge with GATE for the
extended extraction process on Semantic Web. In their
work, GATE help them to identify the part-of-speech
(subject, verb, object, etc.) of the documents and also
determine the NER such as person, date and location [5].
Furthermore, GATE supports variety of formats from the
input and output of the documents processed. It could
handle multiple formats [10] such as XML [11], HTML,
SGML, plain text and so on when loaded on, then
converted the input text into GATE document. Meanwhile,
GATE also provide automation process and decrease the
complexity behind the deployment where [12] create a
collaborative corpus annotation facilities on Web browser
by using the extension of the GATE.
Figure 1: Example of BRAT Interface [3]
C. OTHERS
Another annotation tool that is an open-sourced is
WARP-Text which is also a web-based. This tool designed
for annotating multiple layer of text and create relationship
among them at a different granularity levels [15] and
helpful in annotating atomic paraphrases task. One more
tool used in annotation process is WordFreak, created for
many kinds of text documents and enable to adapt to new
Java classes used to define a new task. [16] used this tool
for constructing an annotated corpus to support biomedical
information extraction.
III. ANNOTATION SCHEME
The process start with data collection from the cyber
security news website. Reliable news websites such as
Bleeping Computer, The Hacker News, ThreatPost,
Security Week and so on can be used for gathering the data
[18]. Another method is through the subscription into
certain news provider. Recorded Future.com through its
Cyber Daily section can provide this kind of service [17]
when subscribed. Notification email will be send daily that
contain sources of cyber security news.

B. BRAT
This tool is an intuitive web-based annotation tool
developed for annotating rich structured text that contain
various NLP tasks built in [3]. It provide a better interface
that can support even a non-technical user to navigate and
Figure 2: Part of the Email from Recorded Future.com

33

Authorized licensed use limited to: University of Exeter. Downloaded on June 23,2020 at 01:32:47 UTC from IEEE Xplore. Restrictions apply.
The links will redirect us to reliable news sources. This
method reduce the time to manually look for any particular
incident that happens on that day.
The next step is to convert the content of the webpages
into an easily readable and process able form. In this case,
we will convert into plain text. To do so, there are two
ways that can be done, first is using an open-sourced
online based tool called Textise {https://www.textise.net/}
where it provide service to removes everything else except
text that include images, videos, html tag and so on. The
text obtained through this method is then saved
accordingly into a .txt file. Next, we can use Python
application called Goose to extract the main body of the
news article, the title of the article, and also metadata on
any videos or images contain in the webpage. Goose will
just need to read the required news link, before extracting
the required text including to strip out any html tag and
unnecessary elements. However, note that some webpage
does contains multi layered or extra protection that will
disable both of the methods mentioned to scrap the text.
A. Loading the GATE Application
On the startup, GATE required you to load a new
resource as it usually in default contain no resource. By
clicking the Language/Processing Resource window, you
can either select ‘new’ to create a whole new session or
‘open’ to open past existing session. Next, load the text file
into GATE tool by clicking ‘GATE Document’ after
previous step. A new window will appear containing
parameter that can be selected. To load a file, click on
‘SourceURL’, a folder symbol on the right (refer circle area
in Figure 3) to choose the location of the file you need.
‘Open file’ dialog will be opened later so that you can
choose where your file reside and click ‘OK’ to load the
text file into GATE. Then, create a GATE Corpus by
following the same steps and select the loaded document.
Figure 3: 'SourceURL' Box for Loading the Text
B. Start Annotating
Select the text file that has been loaded. You can
directly annotate manually any piece of information you
want or use a built in GATE application to help your
annotation process. One of the commonly used is ANNIE
34
Authorized licensed use limited to: University of Exeter. Downloaded on June 23,2020 at 01:32:47 UTC from IEEE Xplore. Restrictions apply.
that consist several main processing resources such as
tokenizer, gazetteer, sentence splitter, POS tagger, NER
and so on. ANNIE can be loaded via ‘Load ANNIE’ icon
on top of the bar beside the folder icon. You can see the
ANNIE being loaded in the GATE panel under the
‘processing resources’ section. To run ANNIE with our
text, simply select ANNIE under the ‘Application’ section
and click it so that it will be shown on the main editor.
Before running the application, make sure to select the
intended Corpus that need to be annotated by selecting it
on the ‘Corpus’ area. Click ‘Run the Application’ on the
bottom of the editor. It will take some time to run the
application depending on the amount of your corpus.
View the documents in the corpus. You will have the
option to select the existing annotation loaded form
ANNIE. To add a new annotation, click ‘Annotation Sets’,
begin highlighting the word that need to be annotated.
Place the cursor on that word for a few second until a pop-
up window appear. Replace the ‘_NEW_’ with appropriate
tag to describe the annotation that has been made. Repeat
the process for any new annotation label or load an
annotation xml into the GATE.
C. Viewing and Saving the Annotation
Annotation that automatically done through ANNIE
application can be viewed by accessing the ‘Annotation
Sets’ at the top bar of the editor which will display the side
frame listing all available annotations such as Person, Date,
Location and so on. Clicking on any of these will display
the words related as shown in Figure 4 (‘Date’ is selected
and GATE will highlight any related words it detected).
Figure 4: Example when selecting one of the Annotation
Lastly, GATE work can be saved by two methods either
through Data Store (suitable for working with large
corpus) or by saving the annotated text as XML. For Data
Store, it need to be created first by selecting the ‘Data
Store’ and click ‘Create datastore’ to choose a new empty
folder. Any corpus or individual document can be saved in
the Data Store where it is easier to be retrieved when we
want to continue with our annotation. Next, to save into
XML form, simply select the document or corpus and
choose ’save as XML’ option and the data is saved in the
following format (figure 5 as viewed using xml reader) as
it contained all the previous tag made before. This data
later can be used directly as training dataset.
Figure 5: Example of XML format generated
IV. CHALLENGES AND LIMITATION
Even though the tool is able to help people annotating their
dataset faster and more manageable to comprehend, it still
hold some challenges and limitations to its capability.
First, learning to use the tool itself is quite difficult for
those who are new in the area, as there are low numbers of
manuals and tutorials available online. People need to
spend some time to read from the publication paper in
order to fully understand how to use the tool effectively.
Next, the learning curve also promptly increase when you
want to use different modules in your annotation. It will
needed you to ‘try and error’ and experimenting it by
yourself to see how the process actually works which will
cost you a lot of time. Finally, when dealing with large
dataset, it will still need to consume more time preparing
the annotation as most of the tools are still not fully
automatic. It is advised to have multiple people to work
together in the annotation process to achieve the end
product in a short period.
V. CONCLUSION
The availability and ease of use of open-sourced
annotation tools can help lot of works in text mining
especially in cyber security domain. As thousands of new
articles published every day, we certainly need a faster
way to process all the information. By using annotation
tool, Information Extraction tasks such as data annotation
and NER can be accomplished within a lesser amount of
time and provide better annotation with less error. Then,
the next step in text mining can be reinforced through a
good training data obtained from the annotation tool.
35
Authorized licensed use limited to: University of Exeter. Downloaded on June 23,2020 at 01:32:47 UTC from IEEE Xplore. Restrictions apply.
ACKNOWLEDGMENT
This research is funded by CyberSecurity Malaysia
under strategic collaboration with Cyber Threat
Intelligence Lab, School of Computing, UTM.
REFERENCES
[1] Zhao, Z., Ahn, G.J., Hu, H. and Mahi, D., 2012, September.
SocialImpact: systematic analysis of underground social
dynamics. In European Symposium on Research in Computer
Security (pp. 877-894). Springer, Berlin, Heidelberg.
[2] Shabat, H., Omar, N. and Rahem, K., 2014, December. Named
entity recognition in crime using machine learning approach.
In Asia Information Retrieval Symposium (pp. 280-288).
Springer, Cham..
[3] Stenetorp, P., Pyysalo, S., Topić, G., Ohta, T., Ananiadou, S.
and Tsujii, J.I., 2012, April. BRAT: a web-based tool for NLP-
assisted text annotation. In Proceedings of the Demonstrations
at the 13th Conference of the European Chapter of the
Association for Computational Linguistics (pp. 102-107).
Association for Computational Linguistics.
[4] Thompson, P., Iqbal, S.A., McNaught, J. and Ananiadou, S.,
2009. Construction of an annotated corpus to support
biomedical information extraction. BMC bioinformatics,
10(1), p.349.
[5] Alani, H., Kim, S., Millard, D.E., Weal, M.J., Hall, W., Lewis,
P.H. and Shadbolt, N.R., 2003. Automatic ontology-based
knowledge extraction from web documents. IEEE Intelligent
Systems, 18(1), pp.14-21.
[6] Cunningham H, Maynard D, Bontcheva K, Tablan V 2002.
GATE: A framework and graphical development environment
for robust NLP tools and applications. In Proceedings of the
40th Anniversary Meeting of the Association for
Computational Linguistics (ACL’02). Philadelphia, US.
[7] Baker P, Hardie A, McEnery A, Cunningham H, Gaizauskas
R. 2002. EMILLE, A 67-Million Word Corpus of Indic
Languages: Data Collection, Mark-up and Harmonisation. In
Proceedings of 3rd Language Resources and Evaluation
Conference (LREC'2002), pages 819--825.
[8] Gamback B, Olsson F 2000. Experiences of Language
Engineering Algorithm Reuse. In Proceedings of the Second
International Conference on Language Resources and
Evaluation (LREC), pages 155--160, Athens, Greece.
[9] Pastra K, Maynard D, Hamza O, Cunningham H, Wilks Y
2002. How feasible is the reuse of grammars for Named Entity
Recognition? In Proceedings of 3rd Language Resources and
Evaluation Conference (LREC’02), Gran Canaria, Spain..
[10] Bird S, Liberman M 1999. A Formal Framework for Linguistic
Annotation. Technical Report MS-CIS-99-01, Department of
Computer and Information Science, University of
Pennsylvania. http://xxx.lanl.gov/\-abs/cs.CL/9903003.
[11] Ide N, Bonhomme P, Romary L 2000. XCES: An XML-based
Standard for Linguistic Corpora. In Proceedings of the Second
International Language Resources and Evaluation Conference
(LREC), pages 825--830, Athens, Greece.
[12] Cunningham, H., Tablan, V., Bontcheva, K. and Dimitrov, M.,
2003. Language engineering tools for collaborative corpus
 annotation. In Proceedings of Corpus Linguistics (Vol. 2003).
[13] T. Ohta, S. Pyysalo, J. Tsujii, and S. Ananiadou. 2012.
Opendomain anatomical entity mention detection. In
Proceedings of DSSD.
[14] M. Neves, A. Damaschun, A. Kurtz, and U. Leser. 2012.
Annotating and evaluating text for stem cell research. In
Proceedings of BioTxtM..
[15] V. Kovatchev, M. Ant`onia Mart´ı, and Maria Salam´o. 2018.
Etpc - a paraphrase identification corpus annotated with
extended paraphrase typology and negation. In Proceedings of
LREC-2018.
36
Authorized licensed use limited to: University of Exeter. Downloaded on June 23,2020 at 01:32:47 UTC from IEEE Xplore. Restrictions apply.
[16] Thompson, P., Iqbal, S.A., McNaught, J. and Ananiadou, S.,
2009. Construction of an annotated corpus to support
biomedical information extraction. BMC bioinformatics,
10(1), p.349.
[17] Kallus, N., 2014, April. Predicting crowd behavior with big
public data. In Proceedings of the 23rd International
Conference on World Wide Web (pp. 625-630). ACM.
[18] Abdullah, M.S., Zainal, A., Maarof, M.A. and Kassim, M.N.,
2018, November. Cyber-Attack Features for Detecting Cyber
Threat Incidents from Online News. In 2018 Cyber Resilience
Conference (CRC) (pp. 1-4). IEEE..