DEVELOPING AUTOMATED AMHARIC HATE SPEECH POSTS

DETECTION MODEL FROM FACEBOOK USING DEEP

LEARNING

A MASTER’S THESIS

BY

SURAFEL GETACHEW

DEPARTMENT OF SOFTWARE ENGINEERING

ADDIS ABABA SCIENCE AND TECHNOLOGY UNIVERSITY

NOVEMBER 2020
DEVELOPING AUTOMATED AMHARIC HATE SPEECH POSTS
DETECTION MODEL FROM FACEBOOK USING DEEP
LEARNING

By

SURAFEL GETACHEW

A Thesis Submitted as a Partial Fulfillment for the Degree of Master of

Software Engineering
To

DEPARTMENT OF SOFTWARE ENGINEERING

ADDIS ABABA SCIENCE AND TECHNOLOGY UNIVERSITY

NOVEMBER 2020
Certificate

ii
Declaration

iii
 Abstract

During the last few years, social activities over the internet especially on social media
platforms increased drastically, but unfortunately, social networks have also become the
place for hate speech proliferation by which most people’s social lives are disturbed
because of hate speech posts and conflicts triggered by those posts. Studies confirm that
online hate speech has different offline consequences. Even though there are a lot of
researches on automated hate speech detection most of them are for other language and
there is a scarcity of labeled data to apply automated analysis and detection methods on
Amharic dataset. Therefore the research on automatic detection of hate speech posts
attracted our attention.

As a solution to those problems, this research aimed to prepare a labeled huge Amharic
dataset by collecting posts and comments from selected Facebook pages of activists that
participated actively. Those Facebook data sets are labeled manually as hate and free based
on the guidelines and pre-processed by applying data cleaning and normalization
techniques. In this research, the recurrent neural network models for automated hate speech
posts detection from Amharic posts on Facebook are developed by using Long Short Term
Memory (LSTM) and Gated Recurrent Unit (GRU) with word uni-grams for feature
extraction and word2vec to represent each unique word by vector representation. The
experiment was conducted on those two models by using 80% of the data set for training
and 10% for validation to train the model and to select the best hyper-parameters
combination for automated hate speech posts detection. The remaining 10% of the dataset
was used for testing the model after training.

As a result Parameter combination 12 of LSTM based RNN achieves an accuracy of 97.9%

to detect posts as hate speech or free. Which is assured by testing the models using models
performance test and inference on user-generated data.

Keywords:- Hate speech, Amharic posts, RNN, LSTM, GRU, Hyper-parameters, Deep
learning.

iv
 Acknowledgments

First and foremost I would like to praise the almighty God without the need of any reason
and St. Maryam (Dingel Mariam ) for the strength she gives to me from the beginning to
the end of this research work and in my whole life too. I gratefully would like to give my
special thanks to my advisor Dr. Kulla Kekeba for his brilliant guidance, constructive
suggestion, and encouragement during this research. Secondly, I want to acknowledge my
friends for their great help in the data collection process as it was so tidy process because
of the block of previously available automated data collection tools.

My special thanks also go to Mr. Getachew Tesfaye for providing expertise and insights
into hate speech laws and helping me to draft hate speech annotation Guidelines. Also, I
would like to thanks Mr. Aschalew Getachew, Abdu Mohammed, Kirubel Getachew,
Yohannes Alemu, and Ms. Zufan Dessalegn for their major role in the dataset annotation
process as annotator and for anyone who participated or contributed to this work. Next, I
would like to thank Mr. Tesfahun Gebre and Mr. Ahmed Ali for their help on linguistic
parts and valuable comments and suggestions on this study.

Last but not least, I am also grateful to my family, especially my mother Tsehaynesh Ali,
and my father Getachew Tesfaye with my brothers Kirubel Getachew and Samuel
Getachew for their love and endless support and encouragement throughout my academic
life. Using this opportunity, I would like to express my deepest gratitude to my current
fiancé and future wife Melaknesh Dessalegn. Thank you for your love, care, and moral
encouragement you have given me throughout my study.

v
 Table of Contents
Certificate ........................................................................................................................... ii
Declaration ........................................................................................................................ iii
Abstract .............................................................................................................................. iv
Acknowledgments................................................................................................................v
Table of Contents ............................................................................................................... vi
Abbreviations and Acronyms ............................................................................................ ix
List of Tables ..................................................................................................................... xi
List of Figures ................................................................................................................... xii
CHAPTER 1 INTRODUCTION .........................................................................................1
2.1 Background ...........................................................................................................1
2.2 Statement of the problem ......................................................................................3
2.3 Research Questions ...............................................................................................5
2.4 Objective ...............................................................................................................5
2.4.1 General objective ...........................................................................................5
2.4.2 Specific objectives .........................................................................................5
2.5 Motivation .............................................................................................................6
2.6 Significance of the study .......................................................................................7
2.7 Scope and limitation ..............................................................................................8
2.7.1 Scope ..............................................................................................................8
2.7.2 Limitation of the research ..............................................................................8
2.8 Thesis organization ...............................................................................................9
CHAPTER 2 LITERATURE REVIEW ...........................................................................11
3.1 Feature of social media posts ..............................................................................11
3.2 Hate speech .........................................................................................................11
3.3 Hate speech in Ethiopia on social media .............................................................14
3.4 Natural language processing and Amharic language ..........................................15
3.5 Hate speech detection techniques ........................................................................17
3.5.1 Feature extraction.........................................................................................18
3.6 Machine Learning for Hate Speech Detection ....................................................21

vi
3.6.1 SVM (Support Vector Machine) ..................................................................22
3.6.2 Naïve Bayes .................................................................................................22
3.6.3 Random Forest .............................................................................................23
3.6.4 Logistic Regression ......................................................................................23
3.7 Deep Learning for Hate Speech Detection ..........................................................23
3.7.1 Deep Learning Algorithms for Hate Speech Detection ...............................24
3.8 Summary of Related works .................................................................................27
CHAPTER 3 METHODOLOGY .....................................................................................32
4.1 Introduction .........................................................................................................32
4.2 Data Set Preparation ............................................................................................34
4.2.1 Data collection .............................................................................................34
4.2.2 Data Set Labeling .........................................................................................37
4.2.3 Labeling Guideline.......................................................................................37
4.3 Data pre-processing .............................................................................................38
4.3.1 Tokenization ................................................................................................38
4.3.2 Normalization ..............................................................................................39
4.4 Feature extraction ................................................................................................40
4.5 Evaluation............................................................................................................41
4.5.1 Inter annotator agreement ............................................................................41
4.5.2 Model performance evaluation metrics .......................................................42
CHAPTER 4 EXPERIMENT AND IMPLEMENTATION.............................................44
5.1 Implementation Environment ..............................................................................44
5.2 Deployment Environments..................................................................................46
5.3 Implementation of data preprocessing ................................................................46
5.3.1 Implementation of Data Cleaning ................................................................47
5.3.2 Implementation of Tokenization ..................................................................48
5.3.3 Implementation of normalization .................................................................48
5.3.4 Implementation of feature representation ....................................................49
5.4 Data exploration ..................................................................................................51
5.5 Data visualization ................................................................................................51

vii
5.5.1 Distribution of class .....................................................................................52
5.5.2 Word n-grams ..............................................................................................52
5.5.3 Word clouds of the dataset ...........................................................................53
5.6 Neural network model implementation ...............................................................54
5.6.1 Experimental setup.......................................................................................54
5.6.2 Training ........................................................................................................56
5.6.3 Testing..........................................................................................................58
CHAPTER 5 RESULT AND DISCUSSION ...................................................................61
6.1 Results .................................................................................................................61
6.1.1 Dataset annotation result ..............................................................................61
6.1.2 Model evaluation result................................................................................62
6.2 Discussion ...........................................................................................................67
CHAPTER 6 CONCLUSION AND RECOMMENDATIONS .......................................70
7.1 Conclusion...........................................................................................................70
7.2 Contribution ........................................................................................................71
7.3 Recommendation .................................................................................................71
References .........................................................................................................................72
Appendixes .......................................................................................................................79
9.1 Amharic hate speech data set labeling guideline ................................................79
9.2 The calculation for inter-annotator agreement ....................................................81

viii
 Abbreviations and Acronyms

AI Artificial Intelligence
API Application Programming Interface
BOW Bag of Words
BSD Berkeley Software Distribution
CBOW Continuous Bag of Words
CNN Convolutional Neural Networks
CNTK Microsoft Cognitive Toolkit
CPU Central Processing Unit
DBOW Distributed Bag of Word
DNN Deep Neural Network
EU European Union
GAATS Graph-based Automatic Amharic Text Summarization System
GloVE Global vector
GPU Graphics Processing Unit
GRU Gated Recurrent Unit
HDFS Hadoop File System
HTML HyperText Markup Language
IR Information Retrieval
KDT Knowledge Discovery from Text
KNN K-Nearest Neighbors
LR Logistic Regression
LSTM Long Short-Term Memory
LVQ Learning Vector Quantization
ML Machine Learning
NB Naïve Bayes
NER Named Entity Recognition
NLP Natural Language Processing
NLTK Natural Language Toolkit
ix
OCR Optical Character Recognition
OSN Online Social Networks
PDF Portable Document Format
POS Part of Speech
PV-DM Paragraph Vector-Distributed Memory
RF Random Forest
RMSProp Root Mean Square Propagation
RNN Recurrent Neural Network
SGD Stochastic Gradient Descent
SRM Structural Risk Minimization
SVM Support Vector Machine
TF-IDF Term Frequency–Inverse Document Frequency

x
 List of Tables
Table 2.1 Summaries of related works ............................................................................. 29
Table 3.1: Categories and description of public Facebook pages ..................................... 35
Table 3.2 Selected Facebook pages and number of post and comments filtered out ....... 36
Table 3.3 Amharic characters normalization .................................................................... 39
Table 3.4 Illustrations of actual and predicted class ......................................................... 42
Table 4.1 Description of implementation environments................................................... 44
Table 4.2 The values of selected hyper-parameters .......................................................... 56
Table 5.1 Annotation result of common datasets.............................................................. 61
Table 5.2 Evaluation metrics using RNN-LSTM and parameters combination ............... 64
Table 5.3 Evaluation metrics using RNN-GRU and parameters combination ................. 65

xi
 List of Figures
Figure 1.1: Social media usage growth in Ethiopia [2]. ..................................................... 2
Figure 3.1 Research flow .................................................................................................. 33
Figure 3.2 Sample of the data set before applying pre-processing ................................... 38
Figure 3.3 Sample of the dataset after applying pre-processing ....................................... 39
Figure 3.4 Method of building hate speech texts detection model ................................... 41
Figure 4.1 Python code to load the post and labels........................................................... 47
Figure 4.2 Data cleaning sample code .............................................................................. 47
Figure 4.3 Sample code to remove emoji ......................................................................... 47
Figure 4.4 Sample code of tokenization ........................................................................... 48
Figure 4.5 Sample code of normalization implementation ............................................... 49
Figure 4.6 Sample code of vector representation of words .............................................. 50
Figure 4.7 First tokenized word vector representation ..................................................... 50
Figure 4.8 Dataset sizes used in papers [25] ..................................................................... 51
Figure 4.9 Class distribution of labels .............................................................................. 52
Figure 4.11 Most frequent unigram word ......................................................................... 53
Figure 4.12 Word cloud of data set after pre-processing .................................................. 54
Figure 4.13 Sample code for selected hyper-parameters values ....................................... 55
Figure 4.14 Sample code to split data into training, validation, and test set .................... 57
Figure 4.15 Sample of the training code ........................................................................... 58
Figure 4.16 Sample code of testing models performance ................................................. 59
Figure 4.17 Sample code of user inference testing ........................................................... 60
Figure 5.1Parameter combination comparison for LSTM ................................................ 63
Figure 5.2 Training and validation loss and accuracy ...................................................... 65
Figure 5.3 Test accuracy of parameters combination for RNN-GRU .............................. 67

xii
 CHAPTER 1 INTRODUCTION

2.1 Background

There are now 5.15 billion unique mobile phone users in the world, and there are 3.96
billion social media users in the world totally among them 21.14 million users are from
Ethiopia, which shows the rapid development of social networking websites and their
usability by users from all over the world to easily expressing an opinion and
communicating each other. Nowadays it is easy to spread individuals’ ideas within the
population using different languages through which the idea can reach the desired web
technology users. It results in a huge amount of user-generated data available for an
enormous online audience. Those opportunities are widely used to express hateful
statements to large groups or specific individuals with malicious intention [1].

Facebook is a widely used social media site by most of the community members according
to stat counter global status data from September 2019 till September 2020 from the overall
60.15% of social media usage is held by Facebook [2] as shown in Figure 1.1. Facebook
users in Ethiopia use the Amharic language to trigger deadly ethnic clashes among a
population of 107 million through ugly Facebook content, which is a language supported
by Facebook [3]. The Amharic language is a widely spoken language which is the working
language of Ethiopia. It has 275 characters mainly consonant-vowel pair and it is the
second-largest Semitic language. While using Facebook Ethiopian users mostly use Geez
scripts to share more essential posts and to communicate through comments and replies,
even though Amharic texts can be written by using the Latin alphabets [4].

1
 Social Media users statistics in Ethiopia from Sep 2019-2020
with specified percentage in each month of the year
90
Number of users in percentage
80

70

60

50

40

30

20

10

0
Instagra VKontak
Facebook Pinterest YouTube Twitter Tumblr LinkedIn reddit Other
m te
2019-09 77.6 9.74 5.57 5.96 0.62 0.15 0.12 0.03 0.22 0
2019-10 74.64 9.69 5.42 7.58 2.09 0.14 0.16 0.02 0.25 0.02
2019-11 74.4 10.64 6.32 6.24 1.91 0.11 0.16 0.1 0.1 0
2019-12 57.34 17.15 8.97 12.7 3.08 0.16 0.41 0.05 0.14 0
2020-01 56.94 15.79 9.29 13.74 3.69 0.27 0.12 0.07 0.06 0.04
2020-02 60.76 15.4 8.87 9.56 4.84 0.28 0.14 0.06 0.07 0.02
2020-03 58.39 14.96 10.66 9.6 5.86 0.17 0.2 0.05 0.08 0.02
2020-04 52.89 13.92 13.06 12.69 6.64 0.22 0.18 0.27 0.11 0.01
2020-05 53.99 19.11 17.73 7.04 1.23 0.27 0.22 0.29 0.09 0.02
2020-06 57.44 15.02 16.32 9.34 0.86 0.44 0.22 0.31 0.03 0.03
2020-07 23.19 30.91 33.33 10.2 0.45 0.41 0.54 0.76 0.2 0.02
2020-08 57.8 15.02 18.06 7.62 0.57 0.16 0.24 0.33 0.17 0.03
2020-09 56.69 15.19 18.5 7.86 0.84 0.21 0.24 0.22 0.22 0.03

Social Media Platforms

2019-09 2019-10 2019-11 2019-12 2020-01 2020-02 2020-03
2020-04 2020-05 2020-06 2020-07 2020-08 2020-09

Figure 1.1: Social media usage growth in Ethiopia [2].

The explosive growth in hate speech and its erosion to democracy, justice, peacebuilding,
and public trust has increased the demand for automated hate speech detection and
technological intervention because hate speech posts are more influential in Ethiopian
communities as individuals become more sensitive to something which they think is mine

2
due to that they follow posts about those things and participate through comments and
share. Those posts shared by different users deliberately or unknowingly create a bad
feeling on some side and which became a cause for conflicts, that is why the government
of Ethiopia mostly blocking social media sites while rebellion arises to minimize their
effect [5]. So that, having organized knowledge about those hate speech posts to take some
measures by the government need the application of automated hate speech posts detection
based on the Amharic language because of the huge amount and unstructured data used in
social media is difficult to analyze manually to solve those type of problems. Therefore,
we will develop the model which can detect hate speech posts by applying deep learning
techniques and algorithm.

2.2 Statement of the problem

In the case of Ethiopia, most peoples used social media especially Facebook to share a lot
of information and it became influential social media in the country, because of its high
usability by the community. Hate speech posts on Facebook has become a major problem
for both social media users and the government by triggering the users for conflict through
hateful speech which bring hatred among social groups to initiate for violent act based on
their ethnicity, political attitude, religion and other social assets [5] [6]. When the
government wants to know the situations that happened because of hateful posts and the
reaction of the society for hate speech posts related to the real-time situation using posts
and comment to forward a decision based on the situation is difficult to do so manually
because of the huge amount of social media data.

The Amharic language is the widely used language by most social media users in Ethiopia
and it is used to post essential texts about business, social, political, or any other activities
followed by other languages English and afaan oromifa [7]. Detecting hate speech texts
from Amharic posts are difficult because of a unique set of punctuation marks and digits
of Amharic text and its rich verb morphology, compound words, abbreviations, spelling
variation, and unstructured nature of social media posts [8] to overcome these difficulties

3
and to provide a prediction based on those texts after analytics is done on them the
development of automated hate speech detection model using deep learning is needed.

A lot of researches on sentiment analysis and hate speech are being undertaken for English,
other languages [9], and also for Amharic. But to the best of the researchers' knowledge,
Zwedie and Wang developed a model to detect Amharic text hate speech using machine
learning techniques using medium-sized data set and achieved an accuracy of 79.8% by
which they classify the speeches [5], and also they identify vulnerable community based
on their ethnic identity using their hate speech detection technique to ban contents
discussed already identified community by identifying the most hatred ethnic group among
Amhara, Tigre and Oromo [9] , and also Yonas studied about hate speech detection on
Amharic posts on Facebook using machine learning algorithms by collecting 5000 datasets
from posts of activists and most followed pages to achieve an accuracy of 79% to classify
the speech as hate or offensive [10].

The research gaps are identified based on the studies conducted on Amharic hate speech
detection is that the researchers use small-sized data set collected from Facebook, they used
machine learning algorithms with low accuracy to classify hate speech as hate and
offensive but hate speech is always changing and new trends emerge. To stay ahead of
these trends, we try to identify potentially inflammatory speeches that haven’t been
reviewed for possible removal from Facebook by improving the test accuracy of the model.

Therefore, the contribution of this thesis will be studying both hate and offensive speeches
of Amharic posts on Facebook to detect hate speech posts by studying their features with
more datasets. The main aim is to develop hate speech posts detection model for Amharic
posts on Facebook using deep learning by improving the accuracy of previously developed
models of Amharic hate speech detection using machine learning through deep study of
neural networks hyper-parameters and their effect on Amharic Facebook posts analysis and
preparation of Amharic labeled dataset for researchers.

4
2.3 Research Questions

We have 3 basic research questions to be answered by this thesis, those are:-

1. How can deep learning be used to detect hate speech text from Amharic datasets?

2. Which deep learning algorithm could be used to detect hate speech posts from

Amharic Facebook posts?

3. Which hyper-parameters combination of the neural network could have the best

performance?

2.4 Objective
2.4.1 General objective

The general objective of the thesis is to develop a model to detect hate speech posts by
analyzing the data from Amharic Facebook posts and the comments of users using deep
learning approaches.

2.4.2 Specific objectives

The specific objectives of the thesis include:-

 To analyze the nature of Amharic social media posts.

 To prepare the labeled Amharic Facebook dataset.
 To develop a model to detect hate speech posts.
 To find the best appropriate hyper-parameters combination of the model for
detecting hate speech posts.
 To train the model using the training data set.
 To test the model using the testing data set.
 To evaluate the performance of developed models.

5
2.5 Motivation

Hate speech has become a popular topic in recent years, which is reflected by devastatingly
growing political attention in addition to increased media coverage of this problem, which
provides a motive to focus on automatic detection of hate speech posts.

The main reasons are European union commission directives, which have been conducting
different initiatives to decrease hate speeches and accusation of Twitter and Facebook by
European regulator for not being good enough at removing hate speech from their platform
and the need for an automated technique to classify text as hate speech programmatically
to make its detection easier and faster [11]. Another reason is that there is a lack of available
data for hate speech detection even though it is a very important task.

In our country, most social media users use Facebook because of its easy accessibility on
the mobile phone in a cost-effective manner and its ease of use to share information,
exchange massage, and other useful features. Even though most of the users used to chat
and grab useful information, there are a lot of groups, individual, political parties, and
others who used it to share their idea and try to get some advantage from conflicting
different social groups through conflict triggering and hateful posts which can develop
hatred situation among social groups.

Such type of situations occurred repeatedly in our country and forced the government to
block the internet totally which can highly affect the income of the telecommunication
from the internet and the information inquiry of users, which is all because of social media
influence on the social life of the community and government by spreading conflict and
rebellion triggering posts. For this reason, in Ethiopia Hate speech and Disinformation
prevention and suppression proclamation were stated to prevent deliberate dissemination
of hate speech by law, which gives us a motive by stating a duty for social media service
providers to prevent the dissemination of disinformation and hate speech through its
platform and to act within 24 hours to remove takeout of circulation of disinformation or
hate speech upon receiving notification about such communication or post [12]. And also

6
Facebook as a company seeks input from experts and civil society organizations
everywhere as part of the process of removing inflammatory speeches from their platform
by planning to prevent potential offline harm that may be related to content on Facebook
[13].

Therefore to provide notification about hate speech posts to conduct essential measures by
social media service provider Facebook an automated Amharic hate speech posts detection
model is going to be developed by using deep learning neural networks.

2.6 Significance of the study

In Ethiopia, there is a lot of social media user especially on Facebook because of easy
accessibility for users and there are a lot of posts from different users and those posts pose
influence on the community either positively or negatively. One of the negative impacts of
social media is triggering conflict and create serious disagreement and quarrel among the
social group which affects both the societies and the government.

The significance of the research will be:-

 To detect hate speech posts and the comments of the community from Amharic
posts on Facebook.
 To guide the ministry of peace by helping them to easily detect hate speech
posts on Facebook to report for the company.
 This model will help the social media platform especially Facebook by
improving their Amharic hate speech posts text detection system.
 To help the user to be protected from hate speech and conflict triggering posts
on Facebook by improving hate speech detection systems developed by other
researchers previously.

7
2.7 Scope and limitation
2.7.1 Scope

Hate speech posts detection is based on Amharic text analytics which is one of the complex
and recent research disciplines and its difficulty is because of the lack of publicly available
Amharic language processing tools and datasets. Considering this and other factors the
scope of the research will be limited to:-

 Amharic text posts from Facebook by preparing a data set.

 Attention is given to the most common Amharic words which express the
hate speech opinion.
 Detection and classification of Amharic texts from Facebook as hate speech
or free speech.
 Designing a deep learning model to detect posts from other posts that can
help to provide a prediction of posts.

2.7.2 Limitation of the research

The study of this research is going to be limited with different limitation because of lack
of studies of Amharic social media texts analytics especially using deep learning
techniques and algorithm due to the reason than the researchers can’t get resources for the
applicability of neural network algorithms for Amharic texts processing, also lack public
datasets for Amharic posts and comments from Facebook and the need of huge volume of
data set for training neural network algorithms combined with the denial of accesses of
tools which were previously available to scrape the data from Facebook poses some
limitation to this thesis. Based on those limitations this thesis is limited to:-

 Geez, script texts only not including Amharic posts using Latin letters and English
posts.
 Posts from Facebook only, not other social media or websites.
 Normal texts only not poetry expression.

8
2.8 Thesis organization

The research paper organized under six chapters

Chapter One discussed above, and it includes the introduction of the study, the motivation,
and the statement of problems, the research questions that would be answered in the
proposed solutions, the objective of the study, the methodology, the scope and limitation,
and the application results of the study.

Chapter Two presents the literature review and related works on automated hate speech
detection, the definition of hate and offensive speeches and their implication for conflict
triggering texts, features of social media posts and comments in Ethiopia, Methodologies
previously used for hate and offensive speeches detection worldwide and specifically for
Amharic language, machine learning algorithms and techniques and their efficiency to
detect hate speeches and related works.

Chapter Three discusses the research methodologies that have been used in this thesis
work, methods used for data collection, data preprocessing, data post-processing, and data
analysis, methods used to develop a deep learning model to detect hate speech texts and
neural network algorithms, and evaluation metrics of the model.

Chapter Four discusses the experiment and implementation of hate speech texts detection
model by applying deep learning techniques and neural network approach using the
PyTorch framework of Facebook with feature extraction, models training, and testing
techniques, data processing and analysis techniques using the model and finally,
implementation and experimentation of proposed solution including data set description,
working environment analysis and implementation and evaluation of model by comparing
with other machine-learning-based models, discusses the result of dataset preparation and
models implementation related with different features.

Chapter Five presents the discussion of the result of the experiment on data preprocessing
and also the evaluation of the model based on evaluation metrics.
9
Chapter Six presents a conclusion, contribution, recommendation, and identifies future
works for further research direction on this research topic.

10
 CHAPTER 2 LITERATURE REVIEW

Extraction of relevant information and knowledge pattern from a huge amount of text data
requires the application of natural language processing and text mining which is a sub-part
of data mining. To bring out useful information from plentiful text there are a lot of studies
on natural language processing. But, those studies already have been conducted in a few
languages and there are several techniques and features from those studies which can be
applicable for Amharic language processing. In this chapter, the literature reviews are
conducted to get general information about the feature of social media posts, the writing
system of Amharic language on and off Facebook, feature representation of hate and
offensive posts, the machine learning techniques, and neural networks approaches to use
for hate speech posts detection.

3.1 Feature of social media posts

Applying traditional methods to the pattern or information detection from social media
texts poses unique challenges because of the unstructured nature of social media texts with
its abundancy, time sensitivity, and short length characteristics [14] [15]. Social media texts
have real-time nature by which users post something related to the real-time situation when
the event is happening so that relating the real-time event with the text poses a different
challenge than traditional text mining [16]. Even though Twitter restricts the length of texts
a user can post no longer than 140 characters Facebook allow 63,206 characters per post
but the need of users to post short texts which can implicitly transmit huge information but
cannot provide sufficient context information through traditional text mining technique
[17].

3.2 Hate speech

In hate and offensive speech detection, there is no single definition that can agree with
everybody, due to this it became a topic to hotly debate by experts. Hate speech is
understood to be bias-motivated, hostile, and malicious language targeted at a person or

11
group because of their genuine or discern natural characteristics [18]. There are many
examples of the term ‘hate’ will be used to describe speech that constitutes a slur, toxic
language, or an instance of verbal abuse against a wide range of targets, in a way that does
not distinguish ‘hate speech’ from other types of speech such as offense or insult [19].
There is also Legal definitions of hate speech from governments of different countries, by
which governments are increasingly defining hate speech in their criminal codes in an
attempt to directly regulate harmful rhetoric both on and offline [20]. The hate speech
definitions vary from perspectives of different sources as described below:-

 The international committee of the UN defines hate speech as “a form of other-

directed speech which revokes the basic principles of human equality and dignity
and wants to degrade the supporting base of groups and individuals in the
estimation of society” [21].
 The European Court of Human Rights defines hate speeches as "a speech that
covers all types of expression which roll out, ignite, advocate or justify xenophobia,
anti-Semitism, racial hatred or other types of hatred based on impatience with
annoyance, including parochialism, demonstrated through aggressive
ethnocentrism and nationalism, hostility and discrimination against migrants,
minorities, and individuals from the origin of emigrants” [22].
 Ethiopian Hate Speech and Disinformation Prevention and Suppression
Proclamation Page 12339 under Proclamation No. 1185 /2020 “Hate speech is the
speech that intentionally promotes discrimination, hatred, or attack against a
discernable group of identity or person, based on race, ethnicity, gender, religion or
disability” [12].
 Facebook defines hate speech as “Objectionable content that directly attacks the
people based on their protected characteristics such as religious affiliation, sexual
orientation, caste, sex, race, ethnicity, national origin, gender, gender identity, and
serious disease or disability”. Generally, Facebook defines an attack as
dehumanizing or violent speech, calls for exclusion statements of inferiority or
segregation [23].

12
The definition of hate speech on Facebook and Ethiopian law is the most relevant definition
for our research because the scope of our research data set is limited to Facebook and the
model is built for Facebook Amharic hate speech detection. But, when we summarize the
definition of hate speech from different works on hate speech detection hate speeches have
the following features:-

1. Hate speech has a specific target based on specific characteristics of groups like
ethnic origin, political view, or religious view.
2. Hate speech incites violence or hatred toward a specific community and uses
languages that attack or diminish those groups.

In addition to hate speech, there is language or expression that offends and negatively
characterizes an individual or group of people. This kind of speech causes someone to feel
upset, annoyed, insulating, angry, hurt, and disgusting [10].

The difference between hate speech and offensive language is often based upon understated
linguistic distinctions [24]. Offensive speech occurs when there is a low degree of hate
speech characterization occurs. The speech may contain a target but do not directly incite
violence, attack, or diminish based on the target group characters because people often use
offensive language to make a point in the debate on heated conversation and condemn
another violent act performed by other people. Any speech that contains a sarcastic,
mocking, and offensive that offend can be considered offensive if it conducts using
language contains high negative, abusive, dirty words phrases in both oral and text.

Generally, we have identified both hate speech and some offensive speeches as the conflict
triggering texts based on the level of their offensiveness because of their nature of annoying
some specific group by inciting violence or offending them for the reason that Facebook
has allowed some offensive and humor posts [25].

13
3.3 Hate speech in Ethiopia on social media

The Internet is among one of the best innovations of the human being by which people
from different race, religion, and nationality has brought together to interact with each
other. The most common social media platforms like Facebook and Twitter connect
billions of peoples from different areas and points of view to share their ideas and opinions
instantly. But there are several hostile consequences like online harassment, trolling, cyber-
bullying, and hate speech [26]. A social media nature enables peoples to create and share
their opinion to participate online. These online platforms are often developed but used
incorrectly to spread content that promotes attacks or diminishes, that incites violence or
hate against groups or individual. Because of the anonymity and mobility of social media
platform features for security and privacy, enable their users to hide their true identity
behind the screen, and express or spread hateful contents.

Hate speech in Ethiopia is growing with the increase of social media users in the country.
The widespread hate speech using social media is an open secret. However, there is a law
that indirectly used for hate-related issues, which is anti-terrorism law, the law prohibits
“the use of any telecommunication network or apparatus to disseminate any terrorizing
message” or “obscene message”, this includes the usage of any social media platform and
other forms of commutation platform to disseminate terrorizing message [5]. However, the
law has been used to restrict messages or speech criticizes government policy and officials.
This law has got backlash from the national and international organization and academic
community because of the applicability of the law contradict freedom of speech of human
right law [27]. Currently, law-maker, government officials, and politicians in Ethiopia have
been aware of hate speech on social media and the lawmakers drafting a new hate speech
and fake news law on the march 23 of 2020 by the proclamation cited as “Hate Speech and
Disinformation Suppression and Prevention Proclamation No.1185 /2020” [12].

14
3.4 Natural language processing and Amharic language

Natural language refers to the language spoken or written by human beings, as opposed to
Artificial languages, computer understandable language, mathematical, or logical
languages [28].

In the 1950s natural language processing began as a combination of linguistics and artificial
intelligence. To index and search huge volumes of text efficiently NLP employs highly
scalable statistics-based techniques that help to distinguish it from text information
retrieval (IR) [29]. It is so important because it helps us to build hate speech detection
models by processing the linguistic data sets by takes chunks of information as input in
form of voice or text or both to manipulate them as per the algorithm of machine learning
inside the computer. Thus the input speech or text is going to be transformed into the output
of an NLP.

NLP is a collection of algorithms and methods used to extract meaning and grammatical
structure from input natural language to perform useful tasks like the generation of natural
language that helps to builds output depends on the rules of the target language, which is
spoken by peoples from a specific region and the specific task. NLP is useful in tutoring
systems, duplicate detection, computer-supported instruction, and database interface fields
as it provides a pathway for increased interactivity and productivity [30].

As stated in the first paragraph of this section natural language processing is a language
spoken or written by the human being and Amharic is a language from the Semitic family
that is written from left-to-right by using unique geez script. It is one of the sub-Saharan
countries Ethiopian’s working language. Amharic lacks capitalization and in total has 275
alphabets mainly consonant-vowel pairs and it has its writing script known as Fidel “ፊደል“
having 34 base characters and other six characters represented for each base characters
formed from the combinations of the consonant of base characters and vowel. It is the
Federal Government of Ethiopia's’ official working language and spoken in the Amhara

15
Region completely. It has generally about 21 million speakers (including second-language
speakers) and is the most widely spoken Ethiopian Semitic language [31].

Alphabets of Amharic can be written from left to right side in a tabular format of seven
columns where the beginning column denotes the character serving as the base and the
others represent the characters that are derived from their vocal sounds. For example, the
following characters show consonants and vowels modified from the base character like መ
modified as ሙ, ሚ, ማ, ሜ, ም, ሞ [32].

The Amharic language is one of under-resourced language which doesn't have more
language processing tool and techniques like other languages have which could help
computational linguistic researchers to go more detail and develop useful higher-level
Internet or computer-based applications and models but, there are researches on Amharic
language text mining. Text mining preprocessing tasks are essential in the Amharic
language to remove stop words or words which are less relevant. To achieve this there is
an aggregated method of frequency and entropy measures of words to identify Amharic
stopwords automatically. There are different proposed approaches researchers use to
generate and remove stop words from natural language documents of different languages
those are the supervised approach, dictionary-based approach, an automated algorithm
based on the frequency of words, deterministic finite automata entropy measure [33].

The Amharic writing system is more challenging than English to detect the intention of the
text and to characterize, categorize or classify it because of factors like the redundancy of
characters which has a similar sound like ሀ or ሐ people use such characters for semantically
same words interchangeably, writing mechanism of compound words using hyphen or
space and sometimes they may merge but they have different lexicon when they put
separately, spelling variation of same words which can be written using different letters
but pronounced as the same. Inconsistency showed in abbreviating Amharic words and
there are many ways of writing words especially loan words, words that are taken from
other languages. For example, the word computer can be written as ኮምፒዩተር or ኮምፒውተር
[34] [35].

16
There are around ten punctuation marks in the Amharic language, but only a few of them
are used in a computer system. Also, most of them are sentence separator marks.
Punctuation mark such as ፡ (Hulet neteb) which could act as (separator of word or space),
። (Arat Neteb) which could serve as (full stop (period)), ፣ (Netela Serez) which is an
Amharic synonym for (comma), and ፤ (Dereb Serez)/ (semicolon) [10].

For the Amharic language, there are text classification, summarization, and prediction
techniques and approaches by different scholars the first approach is the neural network
approach for automatic text classification by applying a learning vector quantization
algorithm which is the supervised version of the Kohonen neural network and used to
classify Amharic texts after preprocessing is accomplished on the text through
tokenization, stop word and number removal, stemming, and dimension reduction by using
the classifier which is constructed using LVQ-neural network algorithm LVQ network has
competitive and linear layer by which the competitive layer learns to classify input vectors
from subclass and the linear layer transform the competitive layers subclass into target
class defined by the user [36].

3.5 Hate speech detection techniques

Hate speech detection problem has been studied by a different researcher, and they used
different techniques to detect hate speech propagation on social media and other online
web platforms. Just as there is no clear agreement on the definition of hate speech, there is
no consensus concerning the most effective methods to detect on social media platforms.

Most automated hate speech detection approaches used binary class classification to
classify the speech as hate or free speech by labeling posts and comments using binary
class and classify them using algorithms of machine learning like SVM, naïve Bayes, and
others [5] [37] [38]. Also, there is another type of approach called the multi-class approach
which labels posts and comments in multiple classes as hate, offensive, and clean and a lot
of multi-class classification approaches classify hate speech as racism, sexism, religion,
anti- Semitism, nationality, and politics, but most of those studies are using the English

17
language [39] [24] [40]. Though some researchers have developed for languages other than
English for the like of Amharic [5], Arabic [41] [42], Indonesian [43] [44], Italian [45],
Dutch [46], German [47] uses machine learning algorithms and techniques to detect hate
speeches.

3.5.1 Feature extraction

Feature extraction consists of a collection of the derived values (features) from the input
data (text in this specific scenario) and producing distinctive properties which are
informative and non-redundant, to improve the learning and classification task of the deep
learning algorithms. Upon their extraction, there is usually a subset of features that will
contain more relevant information [48].

For text classification features are extracted by selecting the subset of terms occurring in
the dataset used for the training set, which is called feature extraction. Feature selection
helps for two main purposes. The First purpose is to perform training more efficiently by
applying a classifier to decreasing the size of the effective vocabulary. Second, feature
selection often helps to increases detection accuracy by eliminating features, which are
noisy and less relevant words. The feature extraction of text techniques that are commonly
used for hate-speech detection are weighted word, word embedding techniques, and text
mining feature extraction methods. The majority of the research papers adopt strategies
already known in text mining to the specific problem of automatic detection of hate speech
[42] [19] [49]. Also, few research papers use specific hate speech detection features used
to tackle the problem of hate speech automatic detection.

N-gram is a sequence of N items and letters, words, or phonemes by which some specific
N item pairs certainly occur much more frequently than other pairs. N-grams are a kind of
multi-order Markov model, by which the probability of a particular Nth position item
depends on the previous N-1 items, this can be computed from the dataset [31]. This feature
extraction method allows improving hate speech detection performance than BOW because
it incorporates the context of each word instead of using words with characters or syllables.

18
But for abusive language detection character N-gram features is more predictive than token
N-gram features [50]. For automatic hate speech detection N-grams are one of the most
used techniques in most researches which uses either deep learning or machine learning
techniques [51] [52] [44].

Word embedding is a technique to learn features by mapping each word or phrase from
the vocabulary dictionary to the N-dimensional vector of real numbers. It is another way
to improve the performance of hate speech detection based on deep neural networks by
considering the semantics of documents. It is widely used in deep learning models to
processing natural languages automatically, based on the representation of words in a large
corpus by projecting a set of words of size X into a vector space of dimension Y such that
Y < X, to facilitate the study of the meaning words and to improve the learning performance
of the model [5] [51] [53]. Word2Vec, doc2vec, fastText, and Global Vectors (GloVe) are
currently among the most accurate and usable word embedding methods that can convert
words into meaningful vectors. These methods have achieved a lot of attention in text
categorization because of their abilities to capture the syntactic and semantic relations
among words. The use of word embedding for text classification tasks has become a
standard approach. Deep learning-based text classification gives a vectored value to a word
using word embedding or statistical methods. The different aspects of words will be
encoded by the dimensions of generated vectors.

Word2vec can learn the representation of each word by using a real number vector with
its semantic features because it is a network of artificial neurons of two layers, thus layers
allow grouping similar words into a single vector [54]. Vector representation of words is
used to represent words with similar meanings by similar vector representation. The
Word2Vec model learns high-quality word embedding and is widely used in various NLP
tasks to the text preprocessing phase by transforms a text into a row of numbers. The idea
of word2vec representation can be extended to documents by teaching the machine for
sentences or documents instead of teaching feature representations for words using
word2vec. The word2vec model has been implemented in either a continuous bag of words
and continuous skip-gram or similar distributed models of word or vector representations
19
[55]. Deep neural networks like Bidirectional Encoder Representations from BERT
Transformers and its successors by which deep networks generate contextual
representations of words for extended periods on large corpora using the attention
mechanisms after it has been trained [56].

Patihullah uses word2vec features extraction to represent the vector value of each word to
create a model of word distribution in their Indonesian hate speech detection model
development process by using a vocabulary builder to collect all unique words from the
corpus and build vocabulary, context builder to find out the relationship between the
appearance of one word with other words around it by using the concept of the context
window or also called a sliding window and neural network to conduct training so that each
word can be represented by a vector [57]. And also Sebastian and Riehle use word2vec for
feature extraction by transformed each word that appeared at least two times in the training
into its vector representation and determined the mean vector of all word vectors which is
used as inputs for the features to automatic analytics of German texts to detect hate speech
[58].

Word Vector representation has an extension named Doc2vec, which represents the whole
of a document in a single digital vector to easily identify the similarity between the
documents. It can learn the representation of documents through two models, namely
Distributed Memory which is equivalent to a continuous bag of words, and Distributed Bag
of Words, which is equivalent to Skip-Gram in Word2vec. By using doc2vec the
documents will be represented numerically without consideration of the length documents.
But it is not like words because documents do not arrange in the logical structure like
words. A word vector is generated for each word, and a document vector is generated for
each document [59].

The GloVe refers to Global Vectors which is an unsupervised model that provides
geometrical encoding by utilizing co-occurrences of words. Through the use of it, the word
embedding for words can be obtained from the dataset [60]. It uses the distributional
statistics of words to create word vectors, particularly how frequently words co-occur

20
within a large text corpus. By using the created word vector glove model can be used to
measure semantic similarity between word pairs by calculating their Euclidian or cosine
distance. In glove model, the pre-trained word vector representations come in different
version and dimension of word vector, like glove.840B.300d which consist 840 billion
word vectors with 300 dimensions there are different dimensions (25, 50, 100, 200, 300)
that will capture different granularities of meaning. By summing the GloVe vectors, per
dimension, for all unique terms in a post we can obtain the resulting features which are
known as GloVe features [61].

3.6 Machine Learning for Hate Speech Detection

The ability to learn from past observations and the increase in data volume, variety, and
velocity leads to automation in text processing techniques which leads to machine learning
which is a subpart of an Artificial Intelligence (AI) that helps systems to automatically
learn and improve based on the training dataset without being explicitly programmed to do
a specific task. In machine learning approaches of text processing LR, SVM, DT, random
forest tree, rule induction, K-nearest neighbor, K-means and the Hebbian algorithms are
the common machine learning algorithms. Whereas the newly emerged and highly applied
approaches, the deep learning approaches also include CNN, DNN, and the RNN are the
basic and highly adapted machine learning techniques of text processing for hate speech
detection.

A lot of researches on hate speech are being undertaken for English, other languages [62]
, and also for Amharic, Zwedie and Wang developed a model to detect Amharic text hate
speech using Naïve Bayes and random forest machine learning techniques with word2vec
and TF-IDF feature extraction methods using medium-sized data set and achieved an
accuracy of 79.8% by which they classify the speeches as hate or not [5] , and also Yonas
[10] studied about hate speech detection on Amharic posts on Facebook using three
machine learning algorithms which are SVM, Naïve Bayes, and random forest by
collecting 5000 datasets from posts of activists and most followed pages and word unigram,

21
bigram, n-gram, TF-IDF and word2vec feature extraction using 5 fold cross-validation to
achieve an accuracy of 79% to classify the speech as hate or offensive.

Machine learning approaches can be categorized into supervised, unsupervised, and semi-
supervised approaches. But, the most common approach to be used in hate speech detection
is the supervised method by which machine learning algorithms will apply what they have
been learned using the training dataset to the new data to predict the desired output. This
approach is domain-dependent because it depends on manual annotation of a huge volume
of text for instant most of the research work reviewed for this research uses a classifier
algorithm such as support vector machine (SVM), Random Forest (RF), Naïve Bayes (NB),
Logistic regression (LR) and Deep learning method. Most of the research studies for
automatic hate speech detection uses more than two classification algorithms for
computational comparison reason and used to suggest which algorithm has high
performance and accuracy for their proposed detection model [24] [41] [63] [64].

3.6.1 SVM (Support Vector Machine)

SVM is an impressive approach for classifying high-dimensional data with the use of the
Structural Risk Minimization (SRM) principle. It has been conveyed as a discriminative
classifier which is further accurate than most former classification prototypes. It has high
accuracy to handle imbalanced class but it needs a large data set and higher training time
to perform well with noisy datasets [10]. In 2017, SVM’s held the best results for text
classification tasks, but in 2018 deep learning took over, especially in hate speech detection
as described here [65].

3.6.2 Naïve Bayes

It is a simple classification technique depends on the Bayesian rule with the dimensionality
of the inputs is high. In this Bayes rule applied to documents and classes. This model gives
class labels to problem instances, represented as vectors of feature values.it has high
performance for small dataset even though the multi-class prediction is needed [42] [10].

22
By iterating through the training data set, the Naive Bayes classifier finds out the number
of occurrences of each of the words, which are bigram, and checks if the test data has words
with the same feature as the training data. After the training set is preprocessed by applying
preprocessing techniques, the bigram feature vectors were extracted from every tweet to
yielded better results for the hate tweets reviews [66].

3.6.3 Random Forest

The random forest algorithm is a data classification algorithm which is versatile and can
work on high dimensional datasets, with a large number of attributes by avoiding
overfitting it can provide reliable feature importance estimate and many other incidental
pieces of information about the dataset apart from the class tags, in addition to that it is
loosely coupled, and can be easily parallelized but, it requires more computational
resources to modeling and prediction [10] [67].

3.6.4 Logistic Regression

Logistic regression is the algorithm that can be regularized to avoid overfitting to predict
the probability of events by using a logit function. This logit function is a sigmoid function
that mapped continuous variables to discrete values, and the outputs have an excellent
probabilistic interpretation but it is weak to handle high dimensional data and it is
dependent on transformations for non-linear features [10] [68].

3.7 Deep Learning for Hate Speech Detection

Deep learning is part of a broader family of machine learning methods based on neural
network classifiers which are a network of units, where the input units show terms and
output units speak for the category. Each unit receives a set of inputs. There are two types
of separators according to classes that are single linear perception and multiple perceptions.
In single-layer classes are not separated. Multiple layers are used to find non-linear
classification boundaries. Generally, deep learning is a machine learning mechanism that
helps computers to learn and to do what humans naturally could do. Like machine learning
23
algorithms they can be learned through supervised, unsupervised, or semi-supervised ways.
Nowadays, Deep Learning models show excellence in analytics to be done on the text and
hate speech detection tasks specifically because of its dependency on the neural network
classifiers with deep knowledge. Its attempt to learn in a real sense to identify patterns in
the provided text and tries to emulate the event in layers of neurons. The performance of
deep learning models depends upon the best choice of neural network algorithm and its
hyper-parameters as well as techniques to represent features. Some research works like
[51], [69], [70]propose a deep learning neural networks based hate-speech text detection
using Convolutional Neural Networks, Recurrent Neural Network, and Long short-term
memory respectively, and also, Zwedie and Wang identify vulnerable community based
on their ethnic identity using their hate speech detection technique to ban contents
discussed already identified community by identifying the most hatred ethnic group among
Amhara, Tigre and Oromo by utilizing classical and neural network machine learning
libraries in python by preparing the word embedding from posts and comments extracted
from selected Facebook pages for word2vec feature extraction by 300 embedding
dimensions and using RNN neural network with 128 hidden layers, output dimension size
of 50, 100 epochs of training and RMSProp optimizer [9].

3.7.1 Deep Learning Algorithms for Hate Speech Detection

Deep learning emerges as a promising field of machine learning by teaching machines with
neural network architecture and there are several new neural network algorithms emerge
every few months. In the section below we have reviewed those algorithms.

3.7.1.1 Deep Neural Network (DNN)

It is one of a deep learning neural network from a supervised learning model of machine
learning which has a power that comes from the ability of DNN to find data representations
that are useful for classification and they are widely explored to handle NLP tasks [70].
DNN is an artificial neural network with a more hidden layer between the input and output
layers. It is designed to learn by using a multi-connection of layers. The input consists of

24
the connection of the input feature space with the first hidden layer of the DNN and every
single layer only receives the connection from the previous and provides connections only
to the next layer in a hidden part. The layer for input is implemented via TF-IDF, word
embedding, or some other feature extraction method. The layer for output is equal to only
one for binary classification or the number of classes for multi-class classification. In
multiclass DNNs, each learning model is generated or the numbers of nodes in each layer
as well as the number of layers are randomly assigned. For multi-class classification, the
layer of output should use softmax [71] [72].

3.7.1.2 Recurrent Neural Network (RNN)

RNN is a standard algorithm mainly used for sequential data that is more widely used and
complex and it has an extraordinary feature that can map a node as an input to many nodes
as output. According to Corazza, Michele et al. RNNs are a useful tool for hate speech
detection because of their ability to remember the sequence of inputs by considering their
order differs from the feed-forward model. They implement the model by combining both
sequences of word embedding and social media features using Keras by fed sequence of
word embedding to a recurrent layer whose output is concatenated with social media
features [73]. Mossie and Wang use the RNN variants such as LSTM, Bidirectional-LSTM,
Gated Recurrent Unit (GRU), and Bidirectional GRU models will be tested and evaluated
for social media dark side content detection using transfer learning emphasis on hate and
conflict and these variants perform well in both short and long texts compared with CNN
[70] [74].

LSTM is a special type of recurrent neural network that addresses problems by conserving
long term dependency more expertly in comparison to the basic recurrent neural network.
To overcome the problem of vanishing gradient LSTM is particularly useful. LSTM uses
multiple gates to carefully regulate the amount of information that flows into the state of
each node. By using multiple levels of abstraction LSTM can compute the representation
of a document from the representation of words in the document. To represent each input
word to train the LSTM the low-dimensional, continuous, and real-valued vectors are used
25
[45]. Generally, LSTM can retain the knowledge of earlier states to be trained for tasks that
require large memory or state awareness of tasks due to this reason it can overcome the
limitation of RNN because it consists block of memory cells state by which signal flows
by regulating input, forget and output gates to control what is stored, read and written on
cell. LSTM is used by Google, Apple, and Amazon [75].

Gated Recurrent Unit (GRU) is another variant of recurrent neural networks that contains
two gates which are update gate and forget gate. Those gates do not possess any internal
memory, and non-linearity is not applied. GRU is now popular in the community who is
working with recurrent networks. The main reason for the popularity is the computation
cost and simplicity of the model. The researchers used GRU to detect hate speech in the
Indonesian language by using it to solve the vanishing gradient problem which comes with
the standard recurrent neural network [76]. Generally, GRU combines the forget gate and
input gate into one update gate and has an additional reset gate and it is increasingly popular
for NLP [77].

3.7.1.3 Convolutional Neural Network (CNN)

CNN is a normal multilayer neural network that is identified as a potential solution for hate
speech detection problem in social media datasets by using the word2vec pre-trained
vectors as the main tools for vector representation words and character n-grams. Most
researchers employed pre-trained CNN vectors to predict the classes to achieving a better
F-score than pre-trained word2vec and character-n-grams [51]

CNN uses pooling to minimize the complexity regarding computational resources by

reducing the size of the output from the first layer to the next layer in the neural networks
by conserving important features of different pooling techniques which could help to
reduce outputs. Although CNN is built for the task of image processing it has also been
effectively used for hate speech detection [75]. Generally, it is an essential method for
string, text, and consecutive data classification.

26
3.8 Summary of Related works

This section presents a comprehensive review of basic related works to the area of
automatic hate speech detection using social media datasets for Amharic language and
other languages to clearly understand the general technique, method, and results of existing
studies.

For detection of hate speech on Amharic dataset, Mossie and Wang perform a preliminary
study on it, They create a dataset of 1821 posts and comments collected from Facebook
and 4299 instances of keyword and phrase extracted for posts and comments, then classify
the speech as “hate” and “not hate” using machine learning classifier algorithms NB model
with achieved 73.02% and 79.83% and Random forest achieved 63.55 and 65.34%
accuracy respectively for both word2vec and TF-IDF feature extraction methods [5]. In
their second study, they propose an approach for the detection of hate speech to identify
minority groups vulnerable to hatred on social media specifically Facebook by using the
distributed processing framework of spark to collect and pre-process posts automatically
by crawled selected Facebook pages [9]. After preparing the dataset they extract features
using word embedding mechanisms like Word2Vec and word n-grams to use it for a variety
of RNN which is GRU specifically. Finally, they clustered hate words using Word2Vec
and can predict potentially hatred targeted ethnic groups. In their experimental results,
feature extraction using Word2Vec word embedding performs better result with GRU
achieves the best result to successfully identify the Tigre ethnic group as the highly
vulnerable community in terms of hatred compared with Amhara and Oromo.

Yonas studied hate speech detection on Amharic posts on Facebook using three machine
learning algorithms which are SVM, Naïve Bayes, and random forest by collecting 5000
datasets from posts of activists and most followed pages and word unigram, bigram, n-
gram, TF-IDF, and word2vec feature extraction using 5 fold cross-validation to achieve an
accuracy of 79% to classify the speech as hate or offensive [10].

27
Likewise, Del Vigna et al. study an Italian online hate campaign on social network sites.
They collect the data from Facebook comments that appeared on an Italian public page and
labeled the dataset as no hate, weak hate, and strong hate, and then by merging weak and
strong hate as hate they form the second dataset. The author's design and implement two
classifiers algorithm for the Italian language, one traditional machine learning algorithm
named SVM and the other is deep learning RNN named LSTM algorithm. By conducting
two different experiments with both datasets that at least 70% of the annotator agreed on
the class of the data. SVM and LSTM achieved an F-score of 80% and 79% for binary
classification and 64% and 60% for ternary classification, respectively [45].

Ibrohim et al. studied automated hate speech detection for the Indonesian language on
social media. By collecting tweets and create a binary class dataset comprising hate speech
and non-hate speech to classify it using the BOW model, word n-gram, character n-gram,
and negative sentiment feature extraction methods with Naïve Bayes, SVM, BLR, and
RFDT models. Then achieved 93.5% F-measure the best performance with the
combination word n-gram with RFDT than other combined models [44].

Gambäck et al. [51] present a deep learning-based hate speech classification system for
twitter using a dataset prepared by Benikova et al. [69] that has four class categories
sexism, racism, both (sexism and racism), and not hate speech. Using four features
embedding like word2vector, character n-grams, Random vector, and combined character
n-grams with word vectors. These four features embedding with deep learning CNN, the
models Tested by using 10-fold cross-validation, their best performing model uses
word2vec embedding with higher precision and recall, with a 78.3% F-score.

28
 Table 3.1 Summaries of related works

S.N Authors and Year Dataset size Number of class Feature Algorithms Accuracy
extraction
1. Mossie and Wang 6120 posts Binary class Word2vec Random 79.83%
(2017) [5] and comments classification and TF-IDF Forest and and
(Hate and no Naïve 65.34%
hate) Bayes
2. Mossie and Wang 5876 posts Binary class Word2vec GBT, RF, 92.56%
(2019) [9] and comments classification with TF-IDF, RNN-
TF-IDF LSTM,
alone, and RNN-GRU
word N-
grams
3. Yonas (2019) [10] 27162 posts Binary class N-grams, TF- SVM, NB, 72.54%,
and comments classification IDF, and RF 70.78% ,
word2vec and
75.39%
4. Djuric et al. Binary class Paragraph2v Logistic 80%
(2015) [54] classification c & BOW regression
with TF &
TFIDF
5. Fauzi et al. (2018) 705 tweets Binary class BOW with NB, KNN, 83.2%,
[43]
classification TF-IDF SVM and 76.8%,
RF 81.3%,

and

77.6%

29
6. Kiema Kiilu et al. 45645 tweets Binary class sentiment NB 67.47%
(2018) [78]
classification analysis & N-
Gram
7. Watanabe et al. More than Binary class unigrams J48graft, 87.4% for
(2018) [39] SVM, and
14,000 tweets classification with binary
RF
(hateful and sentimental, and
offensive), and semantic, and 78.4% for
ternary class pattern ternary
classification features
(hateful,
offensive, and
clean)
8. Alfin et al. 1,100 tweets Binary class bag-of-word NB, SVM, 90.2%,
(2017) [37] Bayesian
classification (BOW), 86.5%,
LR (BLR),
word n-gram and 91.5%,
(RFDT)
and character and
n-gram 93.5%
9. Davidson et al. 24,802 tweet Multiclass bigram, Logistic 90%
(2017) regression
classification unigram, and
trigram
features with
TF-IDF
10. Tulkens et al. ( 5759 Multiclass word2vec, SVM 46% F
2016) [46]
comments classification Dictionary- score
based

30
11. Gambäck and 6,655 tweets Multi-class word2vec, CNN 78.3% F-
Kumar (2017)
classification as ( Random score
[51]
racism, sexism vector,
both and non- character n-
hate) grams, and
word2vec+ch
aracter n-
grams

12. Badjatiya et 16,000 tweet Multi-class Random CNN, 93% F

al. (2017) [69] LTSM, and
classification as ( Embedding score
Fast Text
racist, sexist, and, Glove
both and neither) Embeddings

31
 CHAPTER 3 METHODOLOGY

In this chapter, the methodology used in the research to collect, pre-process, and build the
data set will be discussed. The methods used for training the neural network model and
techniques used to answer the research questions to achieve the thesis objective will be
presented. Finally, the methods for feature engineering and the evaluation metrics to
evaluate the result of the experiment will be discussed.

4.1 Introduction

To develop hate speech posts detection models using a deep learning approach the literature
reviews are conducted on different aspects to find the research gaps and formulate research
objectives. And the data set is collected, preprocessed, and prepared for training, validation,
and testing. By using the prepared dataset the model is trained, validated, and tested by
using the deep learning approach as depicts in Figure 3.1.

32
 Conducting Literature
Defining Research Problem Find research gaps
Review

Preparing the Data Set

Formulate Research
Objectives
Select Facebook Collect Posts
pages and posts and Comments

Applying
Annotate the
Preprocessing
data
Techniques

Build the model

Conduct the Experiment

Evaluate the model

Writing Thesis Report

Figure 4.1 Research flow

33
4.2 Data Set Preparation

In this study, to detect hate speech texts from Amharic posts and comments on Facebook
a new dataset is built because of the lack of published Amharic Facebook dataset. Even
though there is publicly available dataset those are for the English language [39] [26], so
that to achieve the goal of building the data set the following tasks are performed.

1. Mostly followed pages of activists and news pages are selected because, over the
previous years, speeches by activists in Ethiopia have spread rapidly across social
media [74].
2. The posts and comments were collected from those pages manually because
Facebook blocks data collection tools like netvizz.
3. Label the datasets collected from those pages as hate speech and free speech based
on the guideline in Appendix 9.1.

4.2.1 Data collection

Mostly followed Facebook pages of activists, news pages, and broadcasting pages of
channels are selected because It's common for the social network communities to
commonly posting on political and religious issues, Then the data collected from their posts
and the comments of their followers [5] [9].

To collect the post and comments, first of all, publicly accessible Facebook pages, and
activists which do not use a Facebook page but have more followers and posted on their
timeline selected.

The justification we have selected Facebook as a data source is that Facebook has become
the most essential social network in the world and scholars also showed how, in countries
with limited Internet penetration like Ethiopia, Facebook has become almost a synonym
for the Internet [9] and it is most widely used for many parties in dispatching the unreserved
depravity [45].

34
 Table 3.1 shows that the categories of Facebook pages and activists we have screened out
to collect the dataset from them based on the following metrics:-

 Facebook pages that have more than 50,000 like based on the study of Yonas, could
help us to get more active public pages [10].
 Activists without public page but will have posts with more than 300 comments per
post because it shows they have a lot of active participants for their posts.
 The pages and activists have to use the Amharic language as the main language for
their posts.

Table 4.1: Categories and description of public Facebook pages

No Categories Description

1. Journalist and blogger pages They write content for the blog and deliver news
consistently.
2. Religious Activists pages They actively participate in religious affairs and could
have more followers based on their religion.

3. Political Activists page They actively participate in political affairs and they
could deliver posts consistently based on their view and
could have more followers from different backgrounds.

4. News media and broadcasting Delivering the news to the public

pages
5. Government office pages A group of people who holds governmental power and
deliver information for the public.

6. Political party pages A group of peoples with a common political view.

The study conducted based on the data collected from different activists’ pages as shown
in table 3.2 most of them have more than 100,000 followers and more than 300 comments

35
 per post. Posts that have more than 300 comments are selected from March 2018 to April
2020.

Table 4.2 Selected Facebook pages and number of post and comments filtered out

Categories No Page Name Number of Number of

followers Post and
comments
filtered out
Political activist 1. Getachew Shiferaw 167,043 1249
Political activist 2. Muluken Tesfaw 153,586 1138
Political activist 3. Abel Wabella 125,131 897
Religious activist 4. Zemedkun Bekele 574, 174 1143
Political activist 5. Daniel Berhane(ዳንኤል ብርሃነ) 187,763 1347

Political activist 6. Miky Amhara 115,768 1266

Political activist 7. Dereje Gerefa Tullu 105,876 982
Journalist and 8. Getu Temesgen - (ጌጡ ተመስገን) 878,249 867
blogger
Religious activist 9. Aman Mezmur (ሱፊ) 107,038 589

Political activist 10. Jawar Mohammed 1,901,842 1247

Political activist 11. Natnael Mekonnen 320,968 1623
Political activist 12. Taye Dendea Aredo 225,010 1170
Political activist 13. Taye Bogale Arega (Ilma Dasu Odaa) 158,245 1150
Religious activist 14. Tadele tibebu - ታደለ ጥበቡ 199,015 896

Journalist and 15. Meskerem Abera 54, 960 1263

activist
Political activist 16. Seyoum Teshome 203,540 1346
Political activist 17. Henoke Yeshetlla 169,126 1300
Political activist 18. Ayalew Menber 99,491 836
Political activist 19. Alula Solomon (Al-Solomon) 138,190 987
36
Political activist 20. Dereje Habtewold 119,129 769
Political activist 21. Habtamu Ayalew Teshome 77,832 518
Political activist 22. Abebe Gellaw 145,219 1174
EFDRE prime 23. Abiy Ahmed Ali 2,088,989 1873
minister
Public media 24. Ethiopian Broadcasting Corporation 1,611,281 769
Public media 25. waltainfo.com 376,573 564
Religious media 26. Bilal Media BST 706,526 1389
Political party 27. የአማራ ብሔራዊ ንቅናቄ National 201,180 976

Movement of Amhara
Political activist 28. Tamagne Beyene 446,433 965
Political activist 29. Yared Shumete - ያሬድ ሹመቴ 215,078 875

Political activist 31. Ahmedin Jebel official - አህመዲን ጀበል 603,799 784

Total number of collected post and comments 31,952

4.2.2 Data Set Labeling

Data set labeling is a process of classifying the data on two different labels which are hate
and free by combining both hate and offensive speeches to be labeled as a hate tag and free
speech labeled as a free tag. All of the unique posts are labeled by ten annotators from
different backgrounds of culture, religion, and ethnic group. Among those ten annotators,
five of them are given the same number of posts to measure the inter-annotator agreement
among them, and the labeling task is performed based on the guideline given by the
researcher.

4.2.3 Labeling Guideline

To help the annotator to label posts and comments collected from Facebook guidelines
have prepared based on the definition from UN general definition of hate speech and
offensive language [21] and Ethiopian hate speech law [12] by using the annotation style

37
other researchers used [10] to annotate their data. The instructions used for labeling
mentioned in Appendix 9.1.

Figure 4.2 Sample of the data set before applying pre-processing

4.3 Data pre-processing

After preparing our data set we are going to perform data preprocessing before using it on
our deep learning model by using Amharic text preprocessing methods to clean up
collected posts and comments. First of all, we cleaned our data set by removing irrelevant
characters, punctuation marks like ārati net‟ibi‟ (፡፡), net‟ela šerezi‟ (፣), ‟diribi šerezi‟ (፤)
and question mark (?), emojis, blank values, and URL. After that, we apply the following
2 preprocessing techniques.

4.3.1 Tokenization

It is a process of chopping the data set into a piece of tokens by using white space and a
new line as a boundary maker using the NLTK tool.

38
4.3.2 Normalization

It is a component, which is used to convert Amharic letters with the same pronunciation
and different writing representation to a common word with common representation. In
Amharic language characters like ሀ, ኀ, ሐ, ኃ, ኻ, ሓ and ሃ, ሰ and ሠ, ጸ and ፀ, አ, ኣ, ዐ and ዓ
represent the same consonants with the same pronunciation and it can be used
interchangeably without any meaning differences, therefore, we normalize those type of
characters by converting into a single character as shown in Table 3.3 [32].

Table 4.3 Amharic characters normalization

Sound characters Normalized to

ሀ, ኀ, ሐ, ኃ, ኻ, ሓ, ሃ ሀ
ሰ, ሠ ሰ
ጸ, ፀ ፀ
አ, ኣ, ዐ, ዓ አ

Figure 4.3 Sample of the dataset after applying pre-processing

39
4.4 Feature extraction

To build the most efficient model for hate speech texts detection using deep learning neural
network approaches the word embedding feature extraction have used as one of the
methodologies to detect hate speech texts because by using the Word2Vec word
embedding feature extraction method with RNN-GRU [9] Zwdie and Wang can achieve
the best performance for vulnerable community identification. As specified in section 3.5.1
of the literature review there are different word embedding mechanisms to represent the
text by vector representation. To do this task 104,320 unique words extracted from 30,000
datasets to pass into an embedding layer to have a more efficient representation for our
input data with one-hot encoded vectors. By using an embedding as input to the network
learns a different embedding table on its own for learning semantic representations. After
input words are passed to an embedding layer, the new embedding will be passed to LSTM
cells and The LSTM cells will add recurrent connections to the network and give the ability
to the system to include more information about the sequence of words in the data set. After
all of this, the LSTM outputs will go to a sigmoid output layer. We're using a sigmoid
function because hate speech and free becomes 1 and 0, respectively, and a sigmoid will
output predicted values between 0 and 1. Finally, the hate speech text detection model
looks like:

40
Figure 4.4 Method of building hate speech texts detection model

4.5 Evaluation

In this section, the evaluation metrics to measure inter-annotator agreement, and to evaluate
the models' overall performance are going to be discussed.

4.5.1 Inter annotator agreement

After successful completion of dataset preparation and preprocessing, some sections of the
cleaned dataset provided for five different annotators from different ethnic, religious, and
political points of view background based on the annotation convention specified by [79].
Cohens Kappa will be used for calculating the agreement level among annotator by

41
calculating it using the Kappa value interpretation standard of [10] the inter-annotation
agreement is interpreted as:-

 Poor if kappa <0.20

 Fair if 0.20 <= kappa<= 0.40
 Moderate if 0.40<= kappa<=0.60
 Good if 0.60 <= kappa <=0.80
 Very good if 0.80 <= kappa <=1.00 and 1.00 is perfect agreement.

4.5.2 Model performance evaluation metrics

To evaluate the performance of our neural network-based deep learning model we have
designed a method of evaluation metrics that enable us to measure its performance by using
specified parameters as measuring criteria. The hate speech text detection model detects
the text at hate speech or free speech for this model we use accuracy, precision, recall, and
F-measure as evaluation metrics based on previous researchers study [32] [80] [10].

Table 4.4 Illustrations of actual and predicted class

Actual hate speech Actual free

Predicted hate speech True hate speech False hate speech
Predicted free False free True free

We represent true hate speech as TH, true free as TF, false hate speech as FH, and false
free as FF.

4.5.2.1 Accuracy (A)

It is measured as a ratio between the number of correctly classified hate speech and free
texts by the total number of texts.

𝑇𝐻 + 𝑇𝐹
𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 =
TH + TF + FH + FF
42
4.5.2.2 Precision (P)

It measures the exactness or correctness of the model. The high precision shows that the
model achieves better correctness on hate speech text detection.it is calculated by dividing
true hate speech by the sum of true hate speech and false hate speech.

𝑇𝐻
𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 =
TH + FH

4.5.2.3 Recall (R)

It is the measure of the completeness of the classifier by which the high recall implies the
model is better and does not miss correct hate speech classes. it is calculated by dividing
true hate speech by the sum of false free and truly free.

𝑇𝐻
𝑅𝑒𝑐𝑎𝑙𝑙 =
TH + FF

4.5.2.4 F-measure (F)

It is a performance measure of test accuracy that combines Recall and precision into a
single measure of performance, which is calculated by dividing the product of Precision
and Recall by their average.

𝑅𝑒𝑐𝑎𝑙𝑙 ∗ 𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛
𝑅𝑒𝑐𝑎𝑙𝑙 = 2 ∗
Recall + precision

43
 CHAPTER 4 EXPERIMENT AND IMPLEMENTATION

In this chapter, the experiment conducts on the dataset presented by implementing the
neural network model built for hate speech posts detection. It elaborates on the
development and implementation environment, software libraries, and tools used for data
visualization and data analysis. And then implements the data preprocessing techniques,
mentioned in the methodology section and will explore their result. The experiment will
be performed on the model by dividing the data set into training, validation, and test data
set by setting up an experimental setup. Finally, the model will be tested and evaluated
based on the evaluation metrics mentioned in Chapter Three.

5.1 Implementation Environment

To perform data preparation, data preprocessing and data analysis using the neural network
model different tools and libraries are used to support our model implementation. In this
research, the python programming language and PyTorch framework are used for deep
learning algorithms implementation with different data processing tools because python is
the choice of developers, researchers, and data scientists. Table 4.1 shows that the list of
implementation environments used in the thesis.

Table 5.1 Description of implementation environments

Implementation tools
Tools Version Description
Anaconda navigator 1.9.12 It is a graphical user interface (GUI) of
desktop that is included in Anaconda®
distribution to allow us to manage conda
packages easily, launch applications,
environments, and channels without
using command-line commands.
44
Jupiter notebook 6.0.3 An open-source web application that
allows us to create and share documents
that contain visualizations, live code,
equations, and narrative text. Uses
include numerical simulation, data
visualization, statistical modeling, data
cleaning, and transformation.
Python 3.7.0 A general-purpose programming
language that is suitable for deep
learning algorithms implementation.
Notepad++ 7.8.6 It is an editor of source code and normal
texts to use it on Microsoft Windows. It
supports tabbed editing and we use it for
data preparation and to manage the data
annotation process of annotators.
Pytorch 1.4.0 It is a library for an optimized tensor of
deep learning using CPU and GPU
which is an open-source library of
machine learning used for developing
and training the neural network-based
deep learning models. We use it to train
our RNN based hate speech text
detection model.
NumPy 1.18.1 Array processing for number, strings,
and objects. We use it to handling our
data set features for training and testing
of the model.
Matplotlib 3.1.3 Publication quality figures in python.
We use it for data visualization.

45
 NLTK 3.4.5 It is a python program that helps to work
with Natural language data. We use it
for data pre-processing tasks.
cuDNN 7.6 It is a deep neural networks library of
GPU-accelerated primitives. It provides
highly tuned implementations for
standard routines such as forward and
backward convolution, pooling,
normalization, and activation layers.
And we use it to train our neural
network by tuning the performance of
processing.
Pandas 1.0.3 High performance, easy to use data
structures, and data analysis tools. We
use it for data reading, manipulation,
writing, and handling the dataset.

5.2 Deployment Environments

The tools used for implementation were installed on a personal computer HP Pavilion
Gaming Laptop 15-cx0xxx equipped with Intel® Core™ i7-8750H CPU @2.20GHz,
2208Mhz, 6 Core(s), 12 Logical Processor (s), 16 Gigabyte of physical memory, 18.8
Gigabyte of total virtual memory, and 1000 Gigabyte hard disk storage capacities. The
operating system is Windows 10 Enterprise, 64 bits.

5.3 Implementation of data preprocessing

The data preprocessing used as one of the methodologies of the research as specified in
section 3.2 of Chapter Three and data preprocessing implemented in this section by
importing required packages and load the data set to perform data preprocessing steps. The
data set loaded for preprocessing is shown in the figure below.

46
 Figure 5.1 Python code to load the post and labels

5.3.1 Implementation of Data Cleaning

To the implementation of data cleaning, the irrelevant characters and symbols including
Amharic language punctuation marks, special character, and emojis are removed. The
codes are shown below to realize the implementation of data cleaning by import the RE
package to remove Amharic characters and emojis in the dataset.

Figure 5.2 Data cleaning sample code

Figure 5.3 Sample code to remove emoji

47
5.3.2 Implementation of Tokenization

To tokenize our data set newline character used to split each line and a space delimiter used
to get a token of words by using the NLTK tool.

Figure 5.4 Sample code of tokenization

5.3.3 Implementation of normalization

The dataset is normalized using the RE package and implemented using the Amharic
language data normalization process. The following python code helps to represent
common words with different representations with a common representation. The code
specified in figure 4.5 shows the sample code for the implementation of normalization.

48
Figure 5.5 Sample code of normalization implementation

5.3.4 Implementation of feature representation

To extract features and represent them by the number word2vec feature representation is
used, by which each tokenized words are represented by the vector and to implement it by
building a dictionary that maps words to integers. The code shown below illustrates the
implementation of the vector representation of tokenized words.

49
Figure 5.6 Sample code of vector representation of words

After the representation of tokenized words using a vector representation system, the
contents of the first line data, tokenized reviews are represented as a vector shown in the
figure below.

Figure 5.7 First tokenized word vector representation

50
5.4 Data exploration

In the process of data set preparation, after additional preprocessing steps, 30,000 posts
and comments filter out and labeled as hate speech and free class which is more than the
data set size used in most papers as shown in the figure below [81].

Dataset size used in papers

18
16
14
Number of Papers

12
10
8
Number of papers
6
4
2
0
0-1000 1000-10,000 10,000-100,000 More than
100,000
Data Set Size

Figure 5.8 Dataset sizes used in papers [25]

5.5 Data visualization

In this section, the dataset will be visualized by using different visualization parameters to
measure the distribution of the data set among specified labels and whether it is balanced
or imbalanced. And also assess the neural network word embedding vector and measure
the quality of the word vector using different features by performing data visualization
experiments to find out how well the vector captures the linguistic relationship between
words.

51
5.5.1 Distribution of class

The experiment performed on the data set to measure the distribution of the dataset in 2
classes called hate speech and free to prove that whether the dataset is balanced or not and
to find out the most frequent unigram words. The data set has 2 labels which are hate speech
and free by which 15949 of 30,000 are hate speech and others are free, which means
53.16% are hate speech text and 46.84% are free as shown in the figure below.

Class Distribution of Dataset

16500
16000
15500
Data size

15000
14500
14000
13500
13000
Class

Hate Free

Figure 5.9 Class distribution of labels

Generally, the dataset is nearly balanced because there is no huge gap in percentage among
2 labels.

5.5.2 Word n-grams

By this section of the data visualization, the most frequent unigram words are detected by
implementing the python codes.

52
 15 MOST FREQUENT UNIGRAM WORDS
አማራ ኢትዮጵያ ጃዋር ፈጣሪ ወያኔ ጀግና የኦሮሞ ትግራይ
ህወሀት አብን ሌባ ጠላት ጥላቻ ባንዳ ምድረ

1370
NUMBER OF WORDS IN THE DATASET

703

458

410

405

373

336

314

309

307

259

213

204

168

165
15 MOST FREQUENT WORDS

Figure 5.10 Most frequent unigram word

5.5.3 Word clouds of the dataset

Word cloud is data visualization for text to communicate important information at a glance.
The more frequent a word appears in a text, the bigger and bolder it appears in the word
cloud. The words are arranged artistically, thereby it helps the most popular words to
resonate with the targeted audience in an engaging and fun way.

53
Figure 5.11 Word cloud of data set after pre-processing

Based on the result of visualization the word like አማራ', 'ኢትዮጵያ', 'ጃዋር', 'ፈጣሪ', 'ወያኔ', 'ጀግና',
'የኦሮሞ', 'ትግራይ', 'ህወሀት’, ‘አብን', ህዝብ', 'ሰው', መንግስት', 'ሀገር', እግዚአብሄር', 'ቀን', 'ሀሳብ', 'ችግር', 'ጉዳይ'
are the most influence maker words in the dataset to affect the model training and testing
because most our tokens rotate around words in a cloud of the dataset. The code shown
below illustrates the implementation of generating a word cloud of the dataset.

5.6 Neural network model implementation

This research depends on deep learning neural network approach to implement a hate
speech text detection model based on the dataset from Amharic posts on Facebook. To
perform the neural network model implementation as mentioned in previous sections the
data preprocessing and data visualization implements using python tools.

5.6.1 Experimental setup

To implement a recurrent neural network that performs hate speech text detection models
in PyTorch which is a framework for Python based on Torch, which is a library for deep

54
learning. To produce results different parameter combinations were performed for both
LSTM and GRU. The datasets split into training, validation, and test set by using the split
fraction of 80:10:10 and different batch size, learning rate, embedding dimension,
optimizer, number of an epoch, and dropout as shown in Figure 4.13 Because the successful
implementation of neural network algorithms strongly depends on the parameter settings
used. The value of each parameter is used to produce good results. Therefore, the random
search technique is used to tune the hyper-parameters by choosing some parameters.

Figure 5.12 Sample code for selected hyper-parameters values

The following hyper-parameter values are selected for LSTM-RNN and GRU-RNN
models to experiment by interchanging their values as shown in Table 4.2.

55
Table 5.2 The values of selected hyper-parameters

Parameter Selected hyper-parameters and values

combinations Batch size Learning Optimizer Epoch Dropout
rate
PC1 32 0.1 Adam 20 0.1
PC2 64 0.01 Adam 50 0.25
PC3 128 0.001 Adam 100 0.5
PC4 32 0.1 Adamx 20 0.1
PC5 64 0.01 Adamx 50 0.25
PC6 128 0.001 Adamx 100 0.5
PC7 32 0.1 SGD 20 0.1
PC8 64 0.01 SGD 50 0.25
PC9 128 0.001 SGD 100 0.5
PC10 32 0.1 RMSProp 20 0.1
PC11 64 0.01 RMSProp 50 0.25
PC12 128 0.001 RMSProp 100 0.5
PC13 32 0.1 Adadelta 20 0.1
PC14 64 0.01 Adadelta 50 0.25
PC15 128 0.001 Adadelta 100 0.5

5.6.2 Training

After splitting our dataset into training, test, and validation by split ratio of 80:10:10, which
means from the 30,000 data set 27,000 is used for training and validation, and the remaining
3000 is used for testing. And then 27,000 data set is split by 90:10 split fraction to use
24,000 for training and 3000 for validation as shown in Figure below.

56
Figure 5.13 Sample code to split data into training, validation, and test set

To train the recurrent neural network-based model using GPU an LSTM and GRU take in
an input size, batch size, the number of the hidden node, learning rate, and optimizers and
activation functions to minimize loss and improve the accuracy of the model. There are
weights initialized between the input layer and the hidden layer. LSTM and GRU loop
through the training dataset to perform a forward and backward pass to update the weights
based on the learning rate.

57
Figure 5.14 Sample of the training code

5.6.3 Testing

After data preprocessing and training of the LSTM and GRU model using a clean dataset,
it is tested by using the unseen test dataset, which is 10% of the complete data set (3000)
in number.

58
5.6.3.1 Test model performance

It is a performance test of model using a testing dataset which is 10% of the dataset by
which we'll see how the trained model performs on all of the defined test data. In this case,
the model Attempts to predict the labels for the given testing datasets and uses the test
labels to calculate the accuracy of those predictions. To evaluate the performance of the
model the precision, recall, accuracy, and F-score are calculated.

Figure 5.15 Sample code of testing models performance

59
5.6.3.2 Inference on user-generated data

The second way to test the model is an inference on user-generated data by which the model
provided the text that consists of the post at a time without a label and sees what the trained
model predicts. By this time the model will accept new input text data from the user and
predicting an output label, this is called inference. While using this method the data
preprocessing and feature representation steps applied to the data set will be applied for
this single post to make it suitable for testing and to match the model. For this type of
testing, a predict function that takes in a test dataset with models training hyper-parameter
passed to preprocess and predict as hate or free speech.

Figure 5.16 Sample code of user inference testing

60
 CHAPTER 5 RESULT AND DISCUSSION

In this chapter, the result of the experiment was done in the previous chapters discussed.
To present the result of the data annotation process and the experiment which was done on
the dataset using RNN based deep learning models and evaluate them using evaluation
metrics to elaborate implication of the result on this research area.

6.1 Results
6.1.1 Dataset annotation result

By selecting 1000 posts and comments using a simple random sampling technique from all
of 30,000 labeled posts and comments to give the same chance for each data to be selected
for annotation. To provide them for 5 annotators to measure their agreement while
annotation. Each of the annotators had 1000 similar posts to label them as hate speech or
free. Then the Fleiss’s kappa is going to be calculated because we have 5 annotators and
Cohen’s kappa is not applicable for more than 3 annotators.

Table 6.1 Annotation result of common datasets

Annotator 1 Annotator 2 Annotator 3 Annotator 4 Annotator 5

Hate speech 585 580 562 554 576
Free 415 420 438 446 424
Total 1000 1000 1000 1000 1000

The inter-annotator agreement for 1000 posts among 5 annotators has resulted in Fleiss’s
kappa of 0.664 as we mentioned in appendixes 9.2 which is a very good level of agreement
and it is a substantial result because we have much more annotator and more data set than
usual which could affect the inter-annotator agreement among annotators.

61
6.1.2 Model evaluation result

The RNN based hate speech text detection model is built and the experimental setup is
prepared by setting up different parameters to perform the model's evaluation by changing
parameters like the number of an epoch, batch size, learning rate, dropout, and optimizers
(Adam, Adamx, SGD, RMSProp, and Adadelta). By changing those parameters testing
accuracy is measured. Finally, the model performance on each hyper-parameters
combination evaluated using the evaluation metrics called Testing Accuracy, Precision,
Recall, and F1-score as we stated in our methodology section.

6.1.2.1 Result of RNN-LSTM

In the first experiment of the model evaluation, the Adam optimizer was used in Parameter
combination 1 the model can achieve 77.6% testing accuracy. Then the Adam optimizer is
used with parameter combination 2 which is different from parameter combination 1 and
it can achieve test accuracy of 84.6%, and in parameter combination 3, it provides test
accuracy of 88.8%. In those first three experiments, the optimizer is similar but the batch
size and number of epochs are increased, and then the accuracy also increased. Based on
this evaluation using their test accuracy, RMSProp outperforms Adam, Adamx, SGD, and
Adadelta as the best optimizer.

Using PC 5 the Adamx optimizer is used to achieve an accuracy of 86.4%, and then another
experiment is performed by changing the dropout that specified by other researchers as it
has a high impact on the performance of the model by changing it from 0.25 to 0.5 with
changing other parameters and the model can achieve the test accuracy of 92.3% in PC6.
This implies that the dropout has a maximum effect on these parameter combinations.

The parameter combination 7 achieves an accuracy of 83.1%, and then another experiment
is performed by changing the SGD optimizer momentum parameter to 0.9 and the model
achieves 83.5% accuracy. This shows that a change in the momentum of this optimizer has
a low impact on performance.

62
By performing another experiment to monitor the effect of the number of epochs and
learning rate, while the number of epoch and learning rate increased the test accuracy also
increased as shown in the bar chart, 5 times with different parameters combination when
the number of epochs and learning rate increased test accuracy also increase.

After experimenting with the LSTM based RNN model by 15 different parameters
combination different test accuracies achieved as the bar chart in the figure below
demonstrates. The x-axis represents individual parameters combination and the y-axis
represents the result of test accuracy.

TEST ACCURACY OF ALL PARAMETERS

COMBINATION FOR RNN -LSTM
PC1 PC2 PC3 PC4 PC5 PC6 PC7 PC8
PC9 PC10 PC11 PC12 PC13 PC14 PC15
TEST ACCURACY IN PERCENTAGE

97.9
92.3

91.3

91.1

93
88.8

87.6
86.4

85.3
84.6

88
83.1

82.2
77.6

81

15 HYPERPARAMETERS COMBINATION

Figure 6.1 Parameter combination comparison for LSTM

As shown in the table below the evaluation metrics in section 4.5.2 are used to evaluate the

LSTM model by combining different parameters.

63
Table 6.2 Evaluation metrics using RNN-LSTM and parameters combination

Parameters Accuracy Precision Recall F-score

Combination

PC1 77.6% 0.8 0.78 0.79

PC2 84.6% 0.86 0.85 0.85

PC3 88.8% 0.9 0.89 0.89

PC4 81% 0.83 0.81 0.82

PC5 86.4% 0.88 0.86 0.87

PC6 94.7% 0.95 0.95 0.95

PC7 83.1% 0.85 0.83 0.84

PC8 85.3% 0.87 0.85 0.86

PC9 87.6% 0.89 0.88 0.88

PC10 82.2% 0.84 0.82 0.82

PC11 91.3% 0.92 0.91 0.92

PC12 97.9% 0.98 0.98 0.98

PC13 88.0% 0.89 0.88 0.89

PC14 91.1% 0.92 0.91 0.92

PC15 93% 0.94 0.93 0.93

The training and validation accuracy and loss graph of our best model is shown in the

Figure below, by which we can deduce that our model is not overfitting and the model's

performance for the validation dataset become stable starting from epoch around 80 and

can achieve its best performance at 100 epochs.

64
Figure 6.2 Training and validation loss and accuracy

6.1.2.2 Result of RNN-GRU

For the second experiment, another GRU model was built by using a gated recurrent unit

as a base and applied previously described 15 parameters combinations on our algorithm

and evaluated it based on the evaluation metrics, by which parameter combination 12

perform 88% for our data set which is better than all other parameter combinations, but it

is much smaller than the best result achieved by LSTM which is 97.9% test accuracy

achieved by parameter combinations 12.

Table 6.3 Evaluation metrics using RNN-GRU and parameters combination

65
Parameters Accuracy Precision Recall F-score

Combination

PC1 69.8% 0.72 0.7 0.71

PC2 75.3% 0.78 0.75 0.76

PC3 83% 0.85 0.83 0.84

PC4 63.9% 0.67 0.64 0.65

PC5 69% 0.72 0.69 0.7

PC6 83% 0.85 0.83 0.84

PC7 70.9% 0.74 0.71 0.72

PC8 77.5% 0.79 0.76 0.77

PC9 87% 0.88 0.87 0.88

PC10 73% 0.66 0.63 0.64

PC11 80.5% 0.83 0.81 0.81

PC12 88% 0.89 0.88 0.89

PC13 66.1% 0.69 0.66 0.68

PC14 77.1% 0.79 0.77 0.78

PC15 85% 0.87 0.85 0.86

66
 TEST ACCURACY OF ALL PARAMETERS
COMBINATION FOR RNN -GRU
PC1 PC2 PC3 PC4 PC5 PC6 PC7 PC8
PC9 PC10 PC11 PC12 PC13 PC14 PC15
TEST ACCURACY IN PERCENTAGE

88
87

85
80.2
83
83

77.5

77.1
75.3

70.9
69.8

73
69
63.9

56.1
15 HYPERPARAMETERS COMBINATION

Figure 6.3 Test accuracy of parameters combination for RNN-GRU

6.2 Discussion

The first research question for this thesis was: “How can deep learning be used to detect
hate speech texts from Amharic datasets?” Based on the result of the experiment, the
recurrent neural network showed that this type of neural network could be used for text
analysis of Amharic datasets. The result over the models covering two classes hates speech
and free showed some very good results. Over the Amharic post and comments datasets,
the fifteen parameter combinations have been evaluated using test accuracy, precision,
recall, and F-score. Based on the result of the experiment Parameter combination 12 of the
LSTM model is the best combination that has the highest percentage on the test dataset
among those 15 combinations of the model, by which the model takes RMSProp optimizer,
0.001 learning rate, 128 batch size and dropout value of 0.5 with 100 epochs to achieve
97.9%, which could answer the research question 2 and 3, that says “Which deep learning

67
algorithm could be used to detect hate speech posts from Amharic Facebook posts? ” and
“Which hyper-parameters could affect the performance of the neural network?”. Since the
LSTM based RNN models show a good result on both validation data and test data, this
indicates that the neural network has learned from the training data and can take the
knowledge with them when new data has to be analyzed as we test it using inference on
user-generated data.

The evaluation of the data annotation process on our Amharic dataset is compared with
other researcher’s jobs while our datasets are different. Initially, the inter annotation
agreement for the binary class dataset by which both researchers Mossie and Wang [5] and
Yonas [10] has 0.64 and 0.54 for binary and ternary class respectively, but the dataset used
for this experiment has achieved 0.664 Fleiss kappa of the inter-annotator agreement for
binary class classification by annotating it using 5 different annotators with 1000 common
data set for each of them. This indicates that the number class and the number of data set
to be annotated affects the inter-annotator agreement and the annotation process to not have
more than this agreement [5] [10].

Word2vec feature representation is used to represent the dataset by vectors and to make it
suitable for the model by building a dictionary that maps words to an integer. In the case
of feature extraction, we did not use other feature representation methods like TF-IDF, and
glove embedding because most of the researchers feature representation result evaluation
shows that word2vec feature representation outperforms other methods. According to
Mossie and Wang in three of their works [9], [74] [5] word2vec embedding outperforms
pre-trained GloVe, LDA2Vec, and TF-IDF for feature models and in the work of Yonas
[10] also word2vec has the best performance for feature extraction than TF-IDF. By using
the experimental result of researchers on feature extraction and representation as evidence
we use word2vec feature representation and we make our experiment by using the
parameter called embedding dimension which is used for vector representation of word by
which embedding dimension 400 is selected.

68
In the process of hate speech detection, there are different feature representation
mechanisms and machine learning and deep learning algorithms used to implement the
model. We are using a single feature representation mechanism called word2vec
representation with LSTM and GRU by changing the combination of other selected hyper-
parameters. After experimenting, the result of our best parameter combinations is going to
be compared with other models' best performer algorithms. By using machine learning
algorithms with word2vec feature representation Yonas [10] achieve 75.39%, 70.78%, and
72.54% accuracy respectively for RF, NB, and SVM, and Mossie and Wang [5] uses NB
and RF with w2vec to achieve 79% and 65% accuracy respectively. By using deep learning
for vulnerable community identification they can achieve accuracy of 92% by using RNN-
GRU with word2vec [9] and by their latest work published on April 20-24 of 2020 75%,
86.4%, 91.2%, and 92.5% accuracy achieved by using Bi-GRU, Bi-LSTM, LSTM, and
GRU respectively for social media dark side content detection, even though it is not
recommended to compare models with the different dataset and experimental setup because
there are a lot of factors which can pose influence on their performance. But, our model
achieves an accuracy of 97.9% for our Facebook Amharic dataset.

Finally, the LSTM based RNN model for hate speech text detection from Amharic posts
can detect hate speech texts by training with the dataset from Facebook, which can solve
the problem that caused by hate speech posts from Facebook by detecting and reporting
them to Facebook by collaborating with the ministry of peace.

Generally, this model faced different difficulties in data collection and preparation because
of the lack of Amharic Facebook data and the inability of collecting the dataset
automatically using previously available data collection tools because Facebook blocked
them for privacy reasons, this makes our data preparation stage tidy.

69
 CHAPTER 6 CONCLUSION AND RECOMMENDATIONS

In this chapter, the conclusion derived from the proposed hate speech text detection model
using deep learning and the findings of LSTM and GRU models for Amharic data analysis
will be presented, and also we will describe our contributions in this research work and the
recommendations for other researchers.

7.1 Conclusion

Nowadays, in social media, there are a huge amount of data exchanged among users which,
poses a lot of influence on users life positively or negatively, among those huge amounts
of data hate speech texts on Facebook takes the major role to affect users life negatively by
initiating them for conflict based on their race, religion, ethnic group or any other attributes
which could differentiate individuals.

In this research, a solution for Amharic hate speech texts on Facebook is developed by
using a deep learning approach by which, a model that can detect hate speech texts using
recurrent neural networks is built. To successfully implement the model we start with
defining hate speech text which is the text consist of both hate and offensive language
based on their formal definition and we explore existing techniques for hate and offensive
speech detection. And methods to collect, prepare, preprocess, and process the dataset
using the neural network by using different parameters combination to compare the result
of each parameter change.

In this study, the dataset is collected and labeled manually because of strict social media
rule which prevents data collection tools, by which 30,000 data set is collected and
annotated to the binary class of hate speech and free speech. Based on this data set the
LSTM and GRU model is trained and tested by splitting the data set into a training,
validation, and test set using the split ratio of 80:10:10. The experiment performed using
this dataset with different parameters on GRU and LSTM based RNN model by feature
70
representation of word2vec resulted in better test accuracy of 97.9% by RNN-LSTM,
which is better than other machine learning models for hate speech detection.

Finally, we find out that using deep learning neural network models for Amharic text data
analysis it is possible to detect hate speech posts from Facebook, and LSTM results in
better efficiency than GRU based on our dataset. And the change of neural network hyper-
parameters makes the change in the accuracy of the deep learning model.

7.2 Contribution

This research has the following contributions:

1. Hate speech posts detection model: by this research automated hate speech post

and comments detection model is built by using the recurrent neural network called

LSTM.

2. Amharic hate speech data set: to the researches to be conducted on Amharic hate

speech detection, the dataset is prepared and published1.

7.3 Recommendation

We developed a hate-speech text detection model using a recurrent neural network for
detecting the speech as hate speech or free but we recommend:-

 To classify hate speech texts into multiple classes of their base.

 To apply this hate speech text detection model on image data sets by using image
processing and NLP with deep learning neural network algorithms.

1
Getachew, Surafel (2020), “Amharic Facebook Dataset for Hate Speech detection”, Mendeley Data, V1,
doi: 10.17632/ymtmxx385m.1 To access this dataset click

71
 References

[1] "Data reportal," [Online]. Available: https://datareportal.com/global-digital-

overview#:~:text=The%20number%20of%20mobile%20phone,in%20the%20past%2012%20months..
[Accessed 12 Augest 2002].
[2] "Social Media Stat counter," Apr 2019-2020. [Online]. Available: https://gs.statcounter.com/social-
media-stats. [Accessed 18 May 2020].
[3] Maggie Fick, Paresh Dave, "Facebook's flood of languages leave it struggling to monitor content,"
Reuters, 2019.
[4] Hudson, Grover, "Linguistic analysis of the 1994 Ethiopian census, Northeast African Studies," 1999.
[5] Zewdie Mossie, Jenq-Haur Wang, "Social network hate speech detection for the Amharic language".
[6] Iginio Gagliardone, Alisha Patel and Matti Pohjonen, "Mapping and Analyzing Hate Speech online:
Opportunites and Challenges for Ethiopia," 2014.
[7] Kalo, Ameyu Etana, "Social Media and Journalism, Journalists and Media Outlets’ Use of Social Media
Networks in Ethiopia," 2015.
[8] Lars Asker1, Atelach Alemu Argaw1, Björn Gambäck2 and Magnus Sahlgren, "Applying Machine
Learning to Amharic Text Classification," 2017.
[9] Zewdie Mossie, Jenq-Haur Wang, "Vulnerable community identification using hate speech detection
on social media," ScienceDirect, 2019.
[10] Yonas Kenenisa, "Hate Speech Detection for the Amharic Language on Social Media," 2019.
[11] B. a. R. M. a. C. G. a. C. B. a. K. N. a. W. M. Ross, "Measuring the reliability of hate speech annotations:
The case of the European refugee crisis," arXiv preprint ar X iv:1701.08118, 2017.
[12] FEDERAL NEGARIT GAZETTE OF THE FEDERAL DEMOCRATIC REPUBLIC OF ETHIOPIA,
"Hate Speech and Disinformation Prevention and Suppression Proclamation," pp. 12339-12345, 23
March 2020.

72
[13] S. Chakrabarti, Facebook, 21 May 2020. [Online]. Available:
https://about.fb.com/news/2019/06/social-media-and-conflict/. [Accessed 24 Augest 2020].
[14] Fattane zrrinkalam, Ebrahim bagheri, "Event identification on social network," vol. I, 2016.
[15] H. X.hu, "Text analytics on social media, in mining text data," Springer US, 2012.
[16] F.Atefeh, W.Kherich, "A survey of techniques for event detection on Twitter," Comput. Intell, vol. 31.
[17] X.Yan, J.Guo, Y.Lan, J.Xu, X.chneg, "A probabilistic model for bursty topic discovery in microblogs,"
in AAAI Conference on artificial intelligence, 2015.
[18] Cohen-Almagor, Raphael, "Fighting hate and bigotry on the Internet," Policy \& Internet, vol. 3, pp. 1-
-26, 2011.
[19] P. Fortuna and S. Nunes, "A Survey on Automatic Detection of Hate Speech in Text," ACM, vol. 51,
pp. 1-30, 2018.
[20] Haraszti, Miklos, and others, " Hate Speech and the Coming Death of the International Standard before
It Was Born (Complaints of a Watchdog," The content and context of hate speech rethinking regulation
and responses, 2012.
[21] U. Nations and I. Introduction, "I.8. International Convention on the Elimination of All Forms of Racial
Discrimination (Excerpts)," 2013.
[22] "Recommendation No. R (97) 20 of the Committee of Ministers to the Member States on ‘Hate Speech,"
p. 106–108, 1997.
[23] Objectionable Content:Hate speech, "Facebook," [Online]. Available:
https://www.facebook.com/communitystandards/objectionable_content. [Accessed 16 May 2020].
[24] T. Davidson, D. Warmsley, M. Macy, and I. Weber, "Automated Hate Speech Detection and the
Problem of Offensive Language," arXiv Prepr, 2017.
[25] P. a. N. S. Fortuna, "A survey on automatic detection of hate speech in text," ACM Computing Surveys
(CSUR), pp. 1--30, 2018.
[26] B. Mathew, R. Dutt, P. Goyal, and A. Mukherjee, "Spread of hate speech in online social media," 2018.
[27] Gagliardone, Iginio and Pohjonen, Matti and Beyene, Zenebe and Zerai, Abdissa and Aynekulu,
Gerawork and Bekalu, Mesfin and Bright, Jonathan and Moges, Mulatu and Seifu, Michael and

73
 Stremlau, Nicole and others, "Mechachal: Online debates and elections in Ethiopia-from hate speech to
engage in social media," Available at SSRN 2831369, 2016.
[28] Arbi, Mohamed Zied, "Natural Language Processing," arbinatural.
[29] Nadkarni, Prakash M, and Ohno-Machado, Lucila and Chapman, Wendy W, "Natural language
processing: an introduction," Journal of the American Medical Informatics Association, pp. 544--551,
2011.
[30] Reshamwala, Alpa and Mishra, Dhirendra and Pawar, Prajakta, "Review on natural language
processing," IRACST Engineering Science and Technology: An International Journal (ESTIJ), vol. 3,
pp. 113--116, 2013.
[31] "Wikipedia," [Online]. Available: https://en.wikipedia.org/wiki/Amharic. [Accessed 16 05 2020].
[32] Mebrahtu Tadesse G/Medhin, Trilingual Sentiment Analysis on Social Media, Addis Ababa, Ethiopia,
2018.
[33] Miretie, Sileshi Girmaw, and Khedkar, Vijayshri, "Automatic Generation of Stopwords in the Amharic
Text," International Journal of Computer Applications, vol. 975, p. 8887, 2018.
[34] Nobata, Chikashi, and Tetreault, Joel and Thomas, Achint and Mehdad, Yashar and Chang, Yi,
"Abusive language detection in online user content," Proceedings of the 25th international conference
on world wide web, pp. 145--153, 2016.
[35] T. Tilahun, "OPINION MINING FROM AMHARIC BLOG," Addis Ababa, 2013.
[36] Kelemework, Worku, "Automatic Amharic text news classification: Aneural networks approach,"
Ethiopian Journal of Science and Technology, vol. 6, pp. 127-137, 2013.
[37] Alfina, Ika and Mulia, Rio and Fanany, Mohamad Ivan and Ekanata, Yudo, "Hate speech detection in
the Indonesian language: A dataset and preliminary study," 2017 International Conference on Advanced
Computer Science and Information Systems (ICACSIS), pp. 233--238, 2017.
[38] Kiilu, Kelvin Kiema and Okeyo, George and Rimiru, Richard and Ogada, Kennedy, "Using Naive
Bayes algorithm in the detection of hate tweets," International Journal of Scientific and Research
Publications, vol. 8, 2018.

74
[39] Watanabe, Hajime, and Bouazizi, Mondher and Ohtsuki, Tomoaki, "Hate speech on Twitter: A
pragmatic approach to collect hateful and offensive expressions and perform hate speech detection,"
IEEE Access, pp. 13825--13835, 2018.
[40] Raufi, Bujar, and Xhaferri, Ildi, "Application of Machine Learning Techniques for Hate Speech
Detection in Mobile Applications," 2018 International Conference on Information Technologies
(InfoTech), pp. 1--4, 2018.
[41] Albadi, Nuha, and Kurdi, Maram and Mishra, Shivakant, "Are they our brothers? Analysis and detection
of religious hate speech in the Arabic Twittersphere," 2018 IEEE/ACM International Conference on
Advances in Social Networks Analysis and Mining (ASONAM), pp. 69--76, 2018.
[42] Al-Hassan, Areej, and Al-Dossari, Hmood, "Detection of hate speech in social networks: a survey on
the multilingual corpus," in 6th International Conference on Computer Science and Information
Technology, 2019.
[43] Fauzi, M Ali and Yuniarti, Anny, "Ensemble method for Indonesian twitter hate speech detection,"
Indonesian Journal of Electrical Engineering and Computer Science, vol. 11, pp. 294--299, 2018.
[44] Ibrohim, Muhammad Okky, and Budi, Indra, "A dataset and preliminaries study for abusive language
detection in Indonesian social media," Procedia Computer Science, vol. 135, pp. 222--229, 2018.
[45] Del Vigna12, Fabio and Cimino23, Andrea and Dell’Orletta, Felice and Petrocchi, Marinella and
Tesconi, Maurizio, "Hate me, hate me not: Hate speech detection on Facebook," in Proceedings of the
First Italian Conference on Cybersecurity (ITASEC17), 2017, pp. 86--95.
[46] Tulkens, Stephan and Hilte, Lisa and Lodewyckx, Elise and Verhoeven, Ben and Daelemans, Walter,
"A dictionary-based approach to racism detection in dutch social media," arXiv preprint ar X
iv:1608.08738, 2016.
[47] Ross, Bjorn, and Rist, Michael, and Carbonell, Guillermo and Cabrera, Benjamin and Kurowsky, Nils
and Wojatzki, Michael, "Measuring the reliability of hate speech annotations: The case of the European
refugee crisis," arXiv preprint ar X iv:1701.08118, 2017.
[48] J. G. R. d. Sousa, "Feature extraction and selection for automatic hate speech detection on Twitter,"
2019.

75
[49] Schmidt, Anna, and Wiegand, Michael, "A survey on hate speech detection using natural language
processing," in Proceedings of the Fifth International Workshop on Natural Language Processing for
Social Media, 2017, pp. 1--10.
[50] Silva, Leandro and Mondal, Mainack and Correa, Denzil and Benevenuto, Fabricio and Weber, Ingmar,
"Analyzing the targets of hate in online social media," in Tenth International AAAI Conference on Web
and Social Media, 2016.
[51] Gamback, Bjorn, and Sikdar, Utpal Kumar, "Using convolutional neural networks to classify hate-
speech," in Proceedings of the first workshop on abusive language online, 2017, pp. 85--90.
[52] Malmasi, Shervin and Zampieri, Marcos, "Detecting hate speech in social media," in arXiv preprint
arXiv:1712.06427, 2017.
[53] Biere, Shanita and Bhulai, Sandjai and Analytics, Master Business, Hate Speech Detection Using
Natural Language Processing Techniques, Master’s dissertation, Vrije Universiteit Amsterdam, 2018.
[54] Djuric, Nemanja and Zhou, Jing and Morris, Robin and Grbovic, Mihajlo and Radosavljevic, Vladan
and Bhamidipati, Narayan, "Hate speech detection with comment embeddings," in Proceedings of the
24th international conference on world wide web, 2015, pp. 29--30.
[55] Li, Bofang and Drozd, Aleksandr and Guo, Yuhe and Liu, Tao and Matsuoka, Satoshi and Du,
Xiaoyong, "Scaling word2vec on the big corpus," Data Science and Engineering, vol. 4, pp. 157--175,
2019.
[56] Devlin, Jacob, and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina, "Bert: Pre-training of
deep bidirectional transformers for language understanding," in arXiv preprint arXiv:1810.04805,
2018.
[57] Patihullah, Junanda and Winarko, Edi, "Hate speech detection for Indonesia tweets using word
embedding and gated recurrent unit," IJCCS (Indonesian Journal of Computing and Cybernetics
Systems), vol. 13, 2019.
[58] Sebastian and Riehle, Dennis M and H{ohenberger, Steffen and Becker, "Discussing the value of
automatic hate speech detection in online debates," Multikonferenz Wirtschaftsinformatik (MKWI
2018): Data-Driven X-Turning Data in Value, Leuphana, Germany, 2018.

76
[59] Nandi, Rabindra Nath and Zaman, MM Arefin and Al Muntasir, Tareq and Sumit, Sakhawat Hosain
and Sourov, Tanvir and Rahman, Md Jamil-Ur, "Bangla news recommendation using doc2vec," in 2018
International Conference on Bangla Speech and Language Processing (ICBSLP), 2018, pp. 1--5.
[60] Gupta, Akanksha and Tripathy, BK, "Implementing GloVe for context-based k-means++ clustering,"
in 2017 International Conference on Intelligent Sustainable Systems (ICISS), 2017, pp. 1041--1046.
[61] Bohler, Henrik, and Asla, Petter and Marsi, Erwin and Saetre, Rune, "Idi@ ntnu at semeval-2016 task
6: Detecting Stance in tweets using shallow features and glove vectors for word representation," in
Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), 2016, pp.
445--450.
[62] Steven Bird, Ewan Klein, and Edward Loper, "Natural Language Processing with Python: Analysing
text with natural language Toolkit".
[63] A. Gaydhani, V. Doma, S. Kendre, and L. Bhagwat, "Detecting Hate Speech and Offensive Language
on Twitter using Machine Learning: An N-gram and TFIDF based Approach," in arXiv Prepr no.
arXiv:1809.08651, 2018.
[64] N. Chetty and S. Alathur, "Hate speech review in the context of online social networks," Aggress.
Violent Behav, vol. 40, p. 108–118, 2018.
[65] Waseem, Zeerak, and Hovy, Dirk, "Hateful symbols or hateful people? predictive features for hate
speech detection on Twitter," in Proceedings of the NAACL student research workshop, 2016.
[66] Kiilu, Kelvin Kiema and Okeyo, George and Rimiru, Richard and Ogada, Kennedy, "Using Naive
Bayes algorithm in the detection of hate tweets," International Journal of Scientific and Research
Publications, vol. 8, 2018.
[67] Topic, Goran and vSmuc, Tomislav and vSojat, Zorislav and Skala, Karolj, "Reimplementation of the
random forest algorithm," Parallel Numerics 2005, Theory and Applications, 2005.
[68] Sigurbergsson, Gudbjartur Ingi and Derczynski, Leon, "Offensive Language and Hate Speech Detection
for Danish," arXiv preprint ar X iv:1908.04531, 2019.
[69] Badjatiya, Pinkesh and Gupta, Shashank and Gupta, Manish and Varma, Vasudeva, "Deep learning for
hate speech detection in tweets," in Proceedings of the 26th International Conference on World Wide
Web Companion, 2017, pp. 759--760.

77
[70] Pitsilis, Georgios K and Ramampiaro, Heri and Langseth, Helge, "Effective hate-speech detection in
Twitter data using recurrent neural networks," Applied Intelligence, vol. 48, pp. 4730--4742, 2018.
[71] Arman Daliri, "Deep Learning Algorithms Topic: Examining some deep learning algorithms," 2019.
[72] Zhang, Ziqi, and Luo, Lei, "Hate speech detection: A solved problem? the challenging case of long-tail
on Twitter," Semantic Web, vol. 10, pp. 925--945, 2019.
[73] Corazza, Michele and Menini, Stefano and Arslan, Pinar and Sprugnoli, Rachele and Cabrio, Elena and
Tonelli, Sara and Villata, Serena, "Comparing different supervised approaches to hate speech
detection," 2018.
[74] Mossie, Zewdie, "Social Media Dark Side Content Detection using Transfer Learning Emphasis on
Hate and Conflict," in Companion Proceedings of the Web Conference 2020, 2020, pp. 259--263.
[75] Ajay Shrestha, Ausif Mahmood, "Review of Deep Learning Algorithms and Architectures," Digital
Object Identifier 10.1109/ACCESS, 2017.
[76] Patihullah, Junanda, and Winarko, Edi, "Hate Speech Detection for Indonesia Tweets Using Word
Embedding And Gated Recurrent Unit," IJCCS (Indonesian Journal of Computing and Cybernetics
Systems), vol. 13, pp. 43--52, 2019.
[77] Rana, Rajib, "Gated recurrent unit (GRU) for emotion classification from noisy speech," arXiv preprint
ar X iv:1612.07778, 2016.
[78] Kelvin Kiilu, George Okeyo, "Using Naïve Bayes Algorithm in the detection of Hate Tweets,"
International Journal of Scientific and Research Publications (IJSRP), 2018.
[79] Kumar, R., Reganti, A.N., Bhatia, A., & Maheshwari, T, "Aggression-annotated corpus of Hindi-
English code-mixed data," arXiv preprint arXiv, 2018.
[80] Yodit Teshome, Sentence level opinion mining for Amharic language, Debre Berhan, 2019.
[81] Diego Benitio Sanchez, "Design and Development Of Hate Speech Detector I Social Networks Based
on Deep Learning Technologies," 2019.

78
 Appendixes
9.1 Amharic hate speech data set labeling guideline

The main goal of this labeling instruction is to identify hate, offensive, and free speeches
based on the definition we are going to provide on these appendices using worldwide and
countrywide hate and offensive speech definition because we label both hate and offensive
speech as hate speech.

 Hate speech is a kind of expression that deliberately promotes hatred,

discrimination, or attack against a person or a discernable group of identity, based
on ethnicity, religion, race, gender, or disability; any social interactive method that
facilitates the creation and sharing of information for more than one person at one
time and Social networking through the internet.

The following 8 characteristics clearly define hate speech for annotators

The type of speech which is:-

1. Attack targeted specific religion, gender, ethnic group, and political group or
view.
2. Spread hatred.
3. Suggests calling of violence, killing, or intimidating a specific group.
4. Discriminate or dehumanizing.
5. References to the alleged inferiority or superiority of some target groups.
6. Consists of a combination of hateful expression, which is an insult, threats, or
denigrating toward a target’s groups.
7. Accusing or Condemning people based on their target groups.
8. It uses a direct word that the society denounced as hatred, hostile, or
disrespectful nickname of the target group.

79
  Offensive speech is a speech which negatively characterizes or offends an
individual or specific group of peoples by which could make them upset, annoyed,
angry, or disgusting.

Offensive speeches will be characterized by the following characteristics if:-

1. It contains insulting and disgusting word quote for other people's posts to
condemn the posts or comments.
2. Refers to a given target with mocking intent.
3. Considered the target as unkind or unpleasant people.
4. It contains insulting, dirty, disgusting, or upsetting words.
5. Contain violent or insulting words but not possible to explicitly identify a target
group in the post/comment.
6. It contains defamation, which is a false accusation person or attack on a person's
character.
7. Associated the target with a negative feature or quality typical human flaws.
 Free Speech is a type of speech that does not consist of the characteristics of posts
specified by offensive and hate speech.

80
9.2 The calculation for inter-annotator agreement

We have five annotators to annotate 1000 dataset into 2 classes as we have shown in table

5.1, here we will calculate the Fleiss kappa value for inter-annotator agreement

(585+580+562+554+576) (515+420+438+446+424)
𝑃𝑐 = = 0.5714 𝐴𝑛𝑑 𝑃𝑓 = = 0.4286
5000 5000

𝑃𝑒1 = 𝑃𝑐 2 = 0.3264 𝐴𝑛𝑑 𝑃𝑒2 = 𝑃𝑓 2 = 0.1836

𝑃𝑒 = 𝑃𝑒1 + 𝑃𝑒2 = 0.501

(585)2 + (415)2 − 1000 (580)2 + (420)2 − 1000

P(A1) = = 0.513 𝐴𝑛𝑑 P(A2) =
1000 × 999 1000 × 999

= 0.513

(562)2 + (438)2 − 1000 (554)2 + (446)2 − 1000

P(A3) = = 0.507 𝐴𝑛𝑑 P(A4) =
1000 × 999 1000 × 999

= 0.505

(576)2 + (424)2 − 1000

P(A5) = = 0.511
1000 × 999

(𝑃(𝐴1) + 𝑃(𝐴2) + 𝑃(𝐴3) + 𝑃(𝐴4) + 𝑃(𝐴5))

𝑃= = 0.509
5

Finally, our Fleiss Kappa value results in

(𝑃−𝑃𝑒1) (0.509−0.1836)
𝑃= = = 0.664
1−𝑃𝑒 1−0.501

81