Fake News Detection on Pakistani News Using Machine

Learning and Deep Learning

Azka Kishwara , Adeel Zafara

a Riphah International University, Islamabad Pakistan

Abstract

Fake news causes a huge impact on the reader’s mind, therefore it has become
a major concern. Identifying fake news or differentiating between fake and au-
thentic news is quite challenging. The trend of fake news in Pakistan has grown
a lot in the last decade. This research aims to develop the first comprehensive
fake news detection dataset for Pakistani news by using multiple fact-checked
news APIs. This research also evaluates the developed dataset by using mul-
tiple state-of-the-art artificial intelligence techniques. Five machine learning
techniques namely Naive Bayes, KNN, Logistic Regression, SVM, and Decision
Trees are used. While two deep learning techniques CNN and LSTM are used
with GloVe and BERT embeddings. The performance of all the applied models
and embeddings is compared based on precision, F1-score, accuracy, and recall.
The results show that LSTM initialized with GloVe embeddings has performed
best. The research also analyzes the misclassified samples by comparing such
samples with human judgments.
Keywords: NLP, Fake News, BERT, GloVe, LSTM, CNN

1. Introduction

In the past few decades, the internet has become accessible to almost every-
one which eventually increased the use of the internet in our daily lives. With

∗ Correspondingauthor
Email addresses: azkakish22@gmail.com (Azka Kishwar), adeel.zafar@riphah.edu.pk
(Adeel Zafar)

Preprint submitted to Expert Systems with Applications April 15, 2022

Electronic copy available at: https://ssrn.com/abstract=4090739

the increase of internet access to everyone, the use of print media has become
very less while online news has become a trend. Nowadays, news spread very
rapidly. But this news is not always authentic. The fake news can be of three
major types: partial authentic, deceptive, and fabricated [1].
Fake news can cause a great impact on the reader’s mind and thus can
result in wrong decisions. It can have vast financial, as well as health effects [2].
Therefore, there is a huge need of detecting fake news.
The most common way of detecting fake news is human judgments but that
cannot be reliable in some cases. A human needs to have the complete domain
knowledge to identify fake news correctly [3]. Therefore, there should be some
automated way to detect fake news.
For automated fake news detection by applying any artificial intelligence al-
gorithm, a dataset is one of the most important factors. Some of these famous
datasets are LIAR [4], Fake-or-Real news [5], Twitter News [6] and Weibo News
[6]. These datasets cover news on multiple topics, but none of the datasets has
focused on the news of any specific region or country. Some techniques for de-
tecting fake news are applied on web scrapped data for Pakistan but the dataset
is not publicly available and contains only 344 labeled samples [7]. These limited
samples are not enough for developing and evaluating any benchmark artificial
intelligence algorithm. Therefore, the need arises to develop a comprehensive
fake news detection dataset specifically for Pakistani news.
This research aims to develop the first comprehensive fake news detection
dataset for Pakistani news and to evaluate the developed dataset by using multi-
ple state-of-the-art machine learning and deep learning techniques with different
embeddings. The performance of these applied machine learning, deep learn-
ing, and different embeddings are analyzed and compared based on different
performance metrics including accuracy, precision, recall, and F1-score. Five
state-of-the-art machine learning models namely SVM, Logistic Regression, De-
cision Trees, Naı̈ve Bayes, and KNN are applied. While two state-of-the-art
deep learning models CNN and LSTM along with GloVe word embeddings and
BERT sentence embeddings are applied. The research also analyzes the mis-

Electronic copy available at: https://ssrn.com/abstract=4090739

classified samples and compares these samples with human judgments. Fig. 1
shows the complete methodology of the research.

Figure 1: Workflow of the Steps Performed in Methodology

1.1. Research Questions

Following are the research questions addressed in this research.

Electronic copy available at: https://ssrn.com/abstract=4090739

 1. How to develop a comprehensive fake news detection dataset related to
Pakistan?
2. How machine learning and deep learning techniques can help to improve
the performance of the dataset?

1.2. Contributions

Following are the major contributions of the research

1. Developed first comprehensive fake news detection dataset which covers

the news related to Pakistan
2. In depth experimentation and analysis of multiple AI techniques along
with performance comparison of all applied techniques for evaluating the
developed dataset
3. Analysis of misclassified samples and their comparison with human judge-
ments

This paper is organized into multiple sections. The related work is discussed
in section 2 while section 3 covers the details of developed dataset and its eval-
uation along with feature extraction, and experimental setup of the applied
models. The results are discussed in section 4 and section 5 covers findings,
conclusion and future work.

2. Background and Related Work

This section provides a brief overview of the need and existing work done
for fake news detection using multiple techniques. The section also covers the
currently available datasets and their limitations.

2.1. Need for Automated Fake News Detection

Fake news and its detection is a hot topic among researchers. Many re-
searchers have highlighted the need of automated news detection in multiple
fields. The need of research for the automated deception detection in multiple
languages such as Asian languages is mentioned [8]. Additionally, the need for

Electronic copy available at: https://ssrn.com/abstract=4090739

some automated tools to assist both content creators and consumers in eval-
uating the credibility of online news is also highlighted [5]. Along with these
the need for systematic fact-checking, content verification, and filtering of social
media streams is also discussed [1].

2.2. Datasets for Fake News Detection

Multiple open-source datasets are used for the classification. Most of the
datasets include news from the political sector. Few of the models are created
based on the data scraped from online news websites. One of the most used
and publicly available fake news datasets named Liar, Liar Pants on Fire is
developed. This dataset is commonly known as LIAR and contains 12.8K short
news statements which are labeled into six categories [9]. Few other datasets
named as Fake-or-Real news, Twitter, BuzzFeedNews, and Weirdo are also used
by multiple researchers [2, 1, 10, 11, 12, 13]. Fake news detection is also applied
on many web scrapped datasets including multiple countries such as Pakistani
media news [7].

2.3. Important Features

A lot of work is done on identifying the important features which can help
improve the model accuracy. Discourse and pragmatics that is to use the lan-
guage to accomplish communication are used [14]. Word level features such as
lexical and semantic features for a headline, profanity, or slang are also consid-
ered important [15]. Auxiliary information including social engagements of a
person on social media have been considered as important features and can help
in improving the accuracy [2]. Speaker proles such as speaker title, party affili-
ation, credit history, and location are also considered as important features [6].
Multiple features from the text are extracted and then used for classification.
These features include term frequency (TF) and Inverse Document Frequency
(IDF) [7].

Electronic copy available at: https://ssrn.com/abstract=4090739

2.4. Fake News Detection using Machine Learning and Deep Learning Tech-
niques

With the increase in technology, artificial intelligence has become a need

in almost every field. Making the machines learn and use them for multiple
purposes has become quite common. There are a lot of applications where
artificial intelligence is playing its role effectively. Such applications includes
brain tumor detection, seismic activity prediction [16], prediction of playability
of game, procedural content generation [17], named-entity recognition [18], fake
news detection [19] and much more.
Few of the researchers have worked on modeling the temporal information of
the real-world news and performed in-depth experimentation on multiple real-
world large scale datasets [20]. Deep neural classification models are also used
where different pre-trained models such as BERT, GPT2, Roberta, and Fun-
nel transformers are used as embeddings. These models are tested on three
famous fake news datasets including COVID-19, ISOT, and LIAR [21]. Few
researchers have developed an early fake news detection technique named as
Propagation2Vec which uses propagation networks and assigns importance levels
and attention weights to different nodes [22]. Attention-based neural networks
multi-model named as CARN is also developed which extract relevant informa-
tion from different sources by maintaining the target modality [23]. Similarly,
few other researchers have also worked on multi-modal neural networks which
use social media information along with the news by considering the consistency
of data. Additionally, they have also used different textual and visual features
for detecting fake news [24]. Few other researchers have used a knowledge-based
framework for the detection of suspicious news by extracting multiple semantic
and sentiment-based features from the news text [25].

2.5. Limitations of Existing Techniques

Multiple machine learning and deep learning techniques mentioned above

are applied for the detection of fake news on multiple publicly available datasets
[2, 1, 10, 11, 12, 13]. These datasets contain the news of different topics and

Electronic copy available at: https://ssrn.com/abstract=4090739

fields but do not cover any specific area or region. There is no comprehensive
fake news dataset that covers the news of any specific region or country such
as Pakistan. One such dataset for Pakistani news has been developed but the
dataset is not publicly available. Additionally, it contains only 344 labeled
samples which is not enough for developing and evaluating different artificial
intelligence techniques.
Apart from the dataset limitation, there is no comparative analysis of differ-
ent embeddings applied with multiple techniques for the detection of fake news.
There is no way to choose the best-performing model among multiple applied.

3. Material and Methods

3.1. Benchmark Dataset

Different fake news detection benchmarked datasets are developed including

LIAR and Fake-or-Real etc. These datasets cover labeled news data of multi-
ple topics majorly including politics [9, 12]. But none of these benchmarked
datasets contains news of any specific area such as Pakistan. The first dataset
specific to news related to Pakistan is created [7] but the dataset is not pub-
licly available and it contains only 344 labeled news articles scraped from the
web. As the dataset contains very few numbers of samples, it is inappropriate
to use this dataset as a benchmark for developing and evaluating artificial in-
telligence algorithms for detecting fake news. Therefore, there is a huge need to
introduce a comprehensive dataset that can assist in developing and evaluating
multiple AI algorithms. This research developed the first comprehensive fake
news dataset which consists of news related to Pakistan.

3.1.1. Data Gathering

The news data related to Pakistan is gathered using multiple resources.
Common keywords related to Pakistan and Pakistani news are used for searching
the news related to Pakistan. Table 1 shows the list of keywords used to search
for Pakistani news data.

Electronic copy available at: https://ssrn.com/abstract=4090739

 Table 1: Keywords used for Data Gathering Related to Pakistani News
Pakistan Pakistani Pakistan News Pakistani News
Pakistan Zind- Pakistani Zind- Punjab Pak-
abad abad istan
Balochistan
Sindh Khyber Kashmir Azad Jammu and
Pakhtunkhwa Kashmir
Islamabad Karachi Lahore Peshawar
Multan Hyderabad Murree Rawalpindi
Pakistan
Gilgit Islamic Repub- Pakistan Israel China-Pakistan
lic of Pakistan Economic Corridor
Imran Khan Nawaz Sharif Pervaiz Shahid Afridi
Musharraf
Malala Yousaf Arif Alvi Cricket Pak- Pakistan Cricket
Zai istan Team

3.1.2. Sources Used for Data Gathering

Following are some of the key sources used for gathering the data.

Google Fact Checker. The claim search API of Google’s fact checker APIs1 has
been used to gather the fact checked news data related to Pakistani news. The
news data gathered by this API includes news data for both the real and fake
class. The API takes the search string as input to search for the fact checked
news claims related to the search term.

PolitiFact. PolitiFact2 is one the most famous fact checking website operated
by Poynter institute. The website contains multiple fact checked news data
categorized into one of the following six categories

1 https://toolbox.google.com/factcheck/apis
2 https://www.politifact.com/

Electronic copy available at: https://ssrn.com/abstract=4090739

 • True

• Mostly True

• Half True

• Mostly False

• False

• Pants on Fire

The fact checked news data related to Pakistan has been scrapped from the
website. The news data gathered by PolitiFact includes news data for both the
real and fake class.

TheNewsAPI. The news API3 is one of the most famous, simple and easy-to-
use REST API. It provides hundreds and thousands of authentic news published
over worldwide resources. The news data gathered by this API includes news
data for the real class only. The API takes the search string as input to search
for the news related to that search term.

FactCheck. FactCheck4 is nonprofit website which provides fact checked news as

well as counter agreements against fake and misleading claims made by different
politicians. The news data gathered by FactCheck website includes news data
for both the real and fake class.

Kaggle. Kaggle is one the world largest and most famous data science com-
munity which help to achieve multiple data science goals. There are multiple
challenges related to fake news detection on Kaggle. All of these challenges
provide state-of-the-art datasets to perform fake news detection. The news
records related to Pakistan are included in the developed dataset. The news
data gathered by these datasets includes news data for both the real and fake

3 https://newsapi.org/
4 https://www.factcheck.org/

Electronic copy available at: https://ssrn.com/abstract=4090739

class. Relevant news text is filtered out by checking if the news text contains any
of the shortlisted keyword mentioned in table. Following are the used Kaggle
competitions.

• Fake News5

• Fake and Real News Dataset6

• Fake News Detection Challenge KDD 20207

• Fake News Detection8

AFP Factcheck. AFP is one the leading global news agency. Similarly, its digital
verification service has also become leading global fact-checking organization
[26]. The fact checking section of the website9 contains the fact checked news
data. The fact checked news data related to Pakistan has been scrapped from
the website. The news data gathered by AFP Factcheck includes news data for
both the real and fake class.

NotAllowedTo. NotAllowedTo10 is listed as one of the fake news websites [27]

which contains notorious news. The news data related to Pakistan has been
scrapped from the website. The news data gathered by NotAllowedTo website
includes news data for fake class only.

Prntly. Prntly11 is one of the America’s top fake news website [28]. The news
data related to Pakistan has been scrapped from the website. The news data
gathered by Prntly website includes news data for fake class only.

5 https://www.kaggle.com/c/fake-news/data
6 https://www.kaggle.com/clmentbisaillon/fake-and-real-news-dataset
7 https://www.kaggle.com/c/fakenewskdd2020/data
8 https://www.kaggle.com/c/fake-detection/data
9 https://factcheck.afp.com/
10 https://notallowedto.com/
11 https://prntly.com/

10

Electronic copy available at: https://ssrn.com/abstract=4090739

RealNewsRightNow. RealNewsRightNow12 is also listed as one of the fake news
websites [27]). The news data related to Pakistan has been scrapped from the
website. The news data gathered by RealNewsRightNow website includes news
data for fake class only.

3.1.3. Problems Faced in Data Gathering

Data gathering for the fake news detection dataset includes gathering the
data for both authentic news as well as fake news. Multiple factors make the
data gathering process one of the most challenging tasks. Table 2 shows some of
the fact-checking websites13 along with the problems faced against each website.
Following are some of the key challenges faced during the data gathering
process.

Authentic Source of News. The news data either it is fake or real must be from
some authentic source. There are a lot of official Pakistani news websites which
provide real and authentic news but there are very few authentic resources for
gathering the fake news data.

Limited fact-checking news resources. There are very limited resources which
provides fact-checked news data. Almost all of the fact-checking websites con-
tains fact checked international news. The fact checked news data related to
Pakistan is very limited. Additionally, most of the fact checking websites does
not provide search option to search for specific news.

Limited Rest APIs for Gathering News. There are very limited resources which
provides well structured and documented REST APIs to gather news data.
Most of the resources either do not have APIs or they don’t maintain them
anymore. One such example is PolitiFact14 , the API for news gathering have
been removed and is not maintained by them anymore. Following is the list of

12 https://realnewsrightnow.com/
13 https://guides.stlcc.edu/fakenews/factchecking
14 https://www.politifact.com/

11

Electronic copy available at: https://ssrn.com/abstract=4090739

 Table 2: Problems Faced against Fact-checking Websites
Website Included in Problems Faced
Data Gathering
https: No
//apnews. • No search option to search for specific fact
com/hub/ checked news
ap-fact-check
https://www. Yes
factcheck. • Label is not clearly mentioned
org/
• Need to scrap the data as no API is avail-
able for data gathering

• Less data related to Pakistan

https:// No
fullfact.org/ • The content does not contain the news
data. It includes the evidences to either
prove it real or fake

https://www. Yes
politifact. • Less data related to Pakistan
com/
https://www. No
snopes.com/ • Very limited data related to Pakistan

• Content contains evidences for fact check

instead of news text

https://www. No
washingtonpost. • No search option available for searching
com/news/ specific fact checked news
fact-checker/

12

Electronic copy available at: https://ssrn.com/abstract=4090739

resources which provides well structured APIs and are used in data gathering
processing.

• Google Fact Checker15

• TheNewsAPI 16

Limited News Data due to Specific Search Area. Most of the fact checking web-
site are international websites and thus contains international fact checked news.
Therefore, when the fact checked news data is limited to only data related to
Pakistan, the data gathered becomes very less. One such example is PolitiFact17
which contains only 169 fact-checks related to Pakistan18 . Tough, the website
is one of the most popular fact-checking websites and contains thousands of fact
checked news.

Issues faced in Web Scarping. There are a lot of news websites which does not
provide any API to gather the data. Therefore, the news data has been scrapped
from those websites. Some websites have restricted the data scrapping as the
scrapped data does not contain any content except html of the loading page.
Therefore, the HTML content of such websites have manually extracted using
the page sources and then the data has been scrapped from the hardcoded
HTML content. One such example of these websites is NotAllowedTo19 .

Class Imbalancing. Class imbalancing means that the data belong to the classes
is not balanced. As fake news detection is a binary class problem, so the news
text belongs to either real or fake class. The data gathered is highly imbalanaced
as the real news data is much more than the fake news data. Section explains
the data statistics in detail. Classes are imbalanced because of very limited fact-
checking resources. Multiple official Pakistani news websites are available which

15 https://toolbox.google.com/factcheck/apis
16 https://newsapi.org/
17 https://www.politifact.com/
18 https://www.politifact.com/search/factcheck/?q=pakistan
19 https://notallowedto.com/

13

Electronic copy available at: https://ssrn.com/abstract=4090739

contains real and authentic news data. But the authentic source for gathering
fake news data is very limited. Additionally, the fact-checking websites also
provide very limited data due to the constrained search area.

3.1.4. Data Cleaning

Data cleaning is the process of identifying and cleaning the incomplete, in-
correct, irrelevant and inaccurate data20 . Following steps are performed while
cleaning the data

Discard duplicate records. The news data gathered from multiple resources con-
tains a lot of duplicate records where the news text is same. So as the first step
in data cleaning, all the duplicate records are discarded which results in only
unique news data records in the dataset.

Discard Missing Data. The most important thing in the news data for fake
news detection is the text of news. Therefore, as the next step of data cleaning,
all the news record where the text of news is missing are discarded. This results
in the dataset which contains unique and complete text news data.

Attribute Selection. The gathered data contains additional metadata like URL,
title, review date, publisher site, publisher name, claim date, claimant, content,
published at, and author information. But in a real-life scenario, this additional
information along with the news text might not always be available. Therefore,
all this additional metadata is ignored and all the experimentation is performed
on only news text only.

News Text Preprocessing. The raw news text contains raw data which might
affect the performance of classification models. Therefore, the raw news text is
preprocessed before feeding into the classification models. Following steps are
performed while preprocessing the raw news text.

20 https://towardsdatascience.com/what-is-data-cleaning-how-to-process-data-for-

analytics-and-machine-learning-modeling-c2afcf4fbf45

14

Electronic copy available at: https://ssrn.com/abstract=4090739

 • Removing IP Addresses

• Removing URLs

• Removing Punctuation

• Text Tokenization

• Removing Stopwords

• Stemming

Transforming Labels into Binary Class. The textual rating of the news record
is considered as the label for the classifier. As the news data is gathered from
multiple resources, therefor the textual rating attribute contains different values.
These textual ratings can be categorized into 49 categories. Fake news detection
is one of the binary class classification problem. In binary classification, there
should be only two class labels21 . So to achieve this news labels are transformed
into binary class. True, Partly True and Half True labels are transformed into
true(real) class while all other labels are transformed to false(fake) class.

3.1.5. Dataset Statistics

Once the data is cleaned and preprocessed, the dataset is completed. The
dataset includes 11.99K labeled samples. Table 3 shows the statistics of the
developed dataset. The table shows that the classes are imbalanced. The real
news data is much more than fake news data.

Table 3: Statistics of developed Datasets

Total True Sam- False Training Testing
Samples ples Samples Samples Samples
11990 10053 1937 9592 2398

Fig. 2 shows a sample of news data of both classes from the developed
dataset.

21 (https://machinelearningmastery.com/types-of-classification-in-machine-learning

15

Electronic copy available at: https://ssrn.com/abstract=4090739

 Figure 2: True and Fake News Sample from Dataset

80% of the dataset is used for training the algorithms while 20% of the
dataset is used for testing the developed algorithms. As the dataset contains
imbalanced classes, therefore the dataset is split using stratified sampling.

3.2. Dataset Evaluation

The developed dataset is evaluated by classifying the fake and real news
using multiple machine learning and deep learning algorithms. To achieve this,
first, the raw dataset labels are converted into binary classes. True, Partly True,
and Half True labels are transformed into true(real) classes while all other labels
are transformed into false(fake) classes.

3.3. Feature Extraction

The performance of AI models is highly dependent on the features used.

Therefore, multiple features such as lexical, sentimental and n-gram features are
extracted from the text and are used with machine learning algorithms.Additionally,
GloVe and BERT embeddings are used with deep learning algorithms.

3.4. Artificial Intelligence Approaches

Multiple machine learning and deep learning models are developed to evalu-
ate the dataset. This section explains the experimental setup and details of the
developed models.

3.4.1. Machine Learning Approaches

SVM, Decision Trees, Logistic Regression, KNN and Naı̈ve Bayes models are
used from the machine learning approaches. Uni-gram and Bi-gram features are

16

Electronic copy available at: https://ssrn.com/abstract=4090739

used with Naı̈ve Bayes model while for all other machine learning models, lexical
and sentiment features are used.

3.4.2. Deep Learning Approaches

1D-CNN and LSTM are used from deep learning approaches. Both the mod-
els are used with both GloVe and BERT embeddings.The models are compiled
with ADAM optimizer with a learning rate set to 0.001 while binary cross-
entropy is used as a loss function. The final output layer contains one neuron
and sigmoid as the activation function. The models are trained over 10 epochs
using the batch size of 64.

4. Results and Discussion

This section explains the in-depth analysis of results of all the applied mod-
els along with the performance comparisons. The performance of all the models
is compared and is evaluated using different performance metrics including ac-
curacy, precision, recall and F1-score. The best performing models are shown
in bold.

4.1. Machine Learning Approaches

Table 4 shows the results of all the machine learning models. Fig. 3 shows
the performance comparison of machine learning models.
The results show that KNN with lexical and sentiment features is the best
performing machine learning model by achieving the highest F1-score and ac-
curacy of 89%. The results also show that adding sentiment features along with
lexical features does not have a significant effect on the performance of the SVM
model.

4.2. Deep Learning Approaches

Table 5 shows the results of both deep learning models along with the applied
embeddings. Fig. 4 shows the performance comparison of applied deep learning
models.

17

Electronic copy available at: https://ssrn.com/abstract=4090739

 Table 4: Performance comparison of Machine Learning Techniques on developed dataset
Tech- Feature Accuracy Precision Recall F1-
nique Score
SVM Lexical 0.845 0.817 0.845 0.792
SVM Lexical + 0.845 0.818 0.845 0.792
Sentimental
Logistic Lexical + 0.846 0.820 0.846 0.796
Regres- Sentimental
sion
Decision Lexical + 0.874 0.875 0.874 0.874
Trees Sentimental
Naive Uni-gram 0.855 0.876 0.855 0.801
Bayes
Naive Bi-gram 0.659 0.6226 0.659 0.563
Bayes
KNN Lexical + 0.896 0.897 0.896 0.896
Sentimen-
tal

Figure 3: Performance Comparison of Machine Learning Techniques on Developed Dataset

18

Electronic copy available at: https://ssrn.com/abstract=4090739

 Table 5: Performance comparison of CNN on Developed dataset
Tech- Feature Accuracy Precision Recall F1-
nique Score
CNN GloVe 0.936 0.935 0.936 0.934
CNN BERT 0.930 0.933 0.930 0.931
CNN BERT En- 0.912 0.911 0.912 0.911
coded Input
LSTM GloVe 0.944 0.945 0.944 0.945
LSTM BERT 0.862 0.855 0.862 0.824
LSTM BERT En- 0.898 0.894 0.898 0.896
coded Input

Figure 4: Performance Comparison of Deep Learning Techniques on Developed Dataset

The results shows that the LSTM initialized with pre-trained GloVe word
embeddings is the best performing model by achieving 0.94 F1-score and accu-
racy. Overall, CNN has better performance in each case than LSTM.
Fig. 5 shows the comparison of F1-scores of different embeddings applied
with both CNN and LSTM. The result shows that GloVe word embedding has
outperformed with both CNN and LSTM. While BERT embeddings have shown
better performance with CNN than LSTM. This is because the text of news in
the dataset in mostly short statements, therefore applying word embedding

19

Electronic copy available at: https://ssrn.com/abstract=4090739

instead of sentence embeddings results in better performance.

Figure 5: Comparison of Different Embeddings on Developed Dataset

4.3. Comparison of Best Performing Models

Table 6 shows the best performing models of both machine learning and deep
learning techniques. Fig. 6 shows the performance bar chart of both the best
models of machine learning and deep learning techniques. The results show that
LSTM has performed better on the developed dataset. Overall deep learning
techniques have better performance rather than machine learning techniques.
While among deep learning techniques, GloVe embedding has shown better
performance with both CNN and LSTM by achieving the F1-score of 0.93 and
0.94 respectively.

4.4. Analysis of Misclassified Examples

The results above show that the overall best performing model is LSTM
pre-trained with GloVe word embeddings. It has achieved 0.94 F1-score and
accuracy.
Few random misclassified samples where either real class is predicted as fake
or fake class is predicted as real are available in Appendix 5.1. Total number of
misclassified samples are 133.

20

Electronic copy available at: https://ssrn.com/abstract=4090739

 Table 6: Performance Comparison of Best Performing Models on Developed Dataset
Model Accu- Pre- Re- F1- Confusion Ma-
racy cis- call Score trix
ion
KNN 0.896 0.897 0.896 0.896
 
268 119
 
131 1880

LSTM 0.944 0.945 0.944 0.945

 
336 51
 
82 1929

Figure 6: Comparison of Best Performing Models on Developed dataset

First five rows of the table show the samples which are actually fake but
predicted as real by the model. While the last five rows of the table show the
samples, which are actually real but predicted as fake by the model. Most of
the news text which are predicted as fake but are real looks like real news. A
human might also predict this news as real because the content of the news
sounds so right. Similarly, most the news text which are predicted as real but

21

Electronic copy available at: https://ssrn.com/abstract=4090739

are fake looks like fake news. And in the same way a human might predict this
news as fake because the content sounds fake.
For analyzing these misclassified examples, a survey is taken where 57 hu-
mans are provided with these 10 misclassified news texts and they are asked
to guess whether the news text is real or fake. Table 7 and Table 8 shows the
results of the survey about fake and real news respectively.

Table 7: Count of Survey against Fake News Misclassified Examples

News Actual Predicted Correctly In-correctly
Number Class Class Predicted as Predicted
Fake by Human as Real by
Human
N1 Fake Real 23 34
N2 Fake Real 21 36
N3 Fake Real 28 29
N4 Fake Real 17 40
N5 Fake Real 12 45

Table 8: Count of Survey against Real News Misclassified Examples

News Actual Predicted Correctly In-correctly
Number Class Class Predicted Predicted as
as Real by Fake by Human
Human
N6 Real Fake 19 38
N7 Real Fake 25 32
N8 Real Fake 16 45
N9 Real Fake 18 39
N10 Real Fake 28 29

Fig. 7 shows the bar chart of response count of the human survey. The
results show that the news predicted as fake by the model are mostly predicted

22

Electronic copy available at: https://ssrn.com/abstract=4090739

as fake by most of the humans as well. Similarly, the news predicted as real by
the model is mostly predicted as real by mostly humans as well. This human
judgement shows that the human can also predict the news wrong. The content
and meaning of the news matter a lot while detecting the class of news. These
results show that the human judgement is also weak in case of predicting the
news. When asked by human, about the factors which they consider while
predicting the news only using news text, most of them focused on the look and
meaning of content of the news and the common sense.

Figure 7: Bar Chart of Human Survey against Misclassified News

5. Conclusion and Future Work

The key findings of this research includes the development and evaluation
of first comprehensive Pakistani fake news detection dataset. The results shows
that LSTM initialized with GloVe Embeddings has shown the best performance
on the dataset by achieving almost 0.94 F1-score. It is also seen that GloVe em-
beddings has shown better performance than BERT embeddings. The compar-
ison of misclassified samples with human judgements shows that human judge-
ments are also weak as misclassified samples are wrongly predicted by human
as well.
In future, the developed dataset can be enhanced to make it balanced be-
tween both classes and then it can be evaluated as a balanced dataset. Addi-
tionally, the performance of the dataset can be compared using multiple other

23

Electronic copy available at: https://ssrn.com/abstract=4090739

evaluation metrics suitable for imbalanced class classification such as AUC, ROC
and G-mean.

References

[1] V. Rubin, Deception Detection and Rumor Debunking for Social Media,
2016, pp. 342–364.

[2] K. Shu, A. Sliva, S. Wang, J. Tang, H. Liu, Fake news detection on social
media: A data mining perspective, SIGKDD Explor. Newsl. 19 (1) (2017)
22–36. doi:10.1145/3137597.3137600.
URL https://doi.org/10.1145/3137597.3137600

[3] V. Rubin, N. Conroy, Discerning truth from deception: Human judgments

& automation efforts, First Monday 17. doi:10.5210/fm.v17i3.3933.

[4] Y. Chen, N. Conroy, V. Rubin, News in an online world: The need for an
” automatic crap detector ”, Vol. 6?10, 2015.

[5] Y. Chen, V. Rubin, Perceptions of clickbait: A q - methodology approach,

Proceedings of the Annual Conference of CAIS / Actes du congrès annuel
de l’ACSIdoi:10.29173/cais1046.

[6] Y. Long, Q. Lu, R. Xiang, M. Li, C.-R. Huang, Fake news detection through
multi-perspective speaker profiles, in: Proceedings of the Eighth Interna-
tional Joint Conference on Natural Language Processing (Volume 2: Short
Papers), Asian Federation of Natural Language Processing, Taipei, Taiwan,
2017, pp. 252–256.
URL https://www.aclweb.org/anthology/I17-2043

[7] I. Kareem, S. Awan, Pakistani media fake news classification using ma-
chine learning classifiers, 2019, pp. 1–6. doi:10.1109/ICIC48496.2019.
8966734.

24

Electronic copy available at: https://ssrn.com/abstract=4090739

 [8] V. Rubin, Pragmatic and cultural considerations for deception detection in
asian languages, ACM Transactions on Asian Language Information Pro-
cessing 13. doi:10.1145/2605292.

[9] W. Wang, ”liar, liar pants on fire”: A new benchmark dataset for fake news
detection, 2017, pp. 422–426. doi:10.18653/v1/P17-2067.

[10] N. Ruchansky, S. Seo, Y. Liu, Csi: A hybrid deep model for fake news
detection, 2017, pp. 797–806. doi:10.1145/3132847.3132877.

[11] Y. Wang, F. Ma, Z. Jin, Y. Yuan, G. Xun, K. Jha, L. Su, J. Gao, Eann:
Event adversarial neural networks for multi-modal fake news detection,
Association for Computing Machinery, New York, NY, USA, 2018, pp.
849–857. doi:10.1145/3219819.3219903.
URL https://doi.org/10.1145/3219819.3219903

[12] J. Y. Khan, M. T. I. Khondaker, A. Iqbal, S. Afroz, A benchmark study on

machine learning methods for fake news detection, ArXiv abs/1905.04749.

[13] Z. Jiawei, B. Dong, P. Yu, Fakedetector: Effective fake news detection

with deep diffusive neural network, 2020, pp. 1826–1829. doi:10.1109/
ICDE48307.2020.00180.

[14] V. Rubin, N. Conroy, Y. Chen, Towards news verification: Deception de-

tection methods for news discourse, 2015. doi:10.13140/2.1.4822.8166.

[15] V. Rubin, N. Conroy, Y. Chen, S. Cornwell, Fake news or truth? using

satirical cues to detect potentially misleading news, 2016. doi:10.18653/
v1/W16-0802.

[16] B. Aslam, A. Zafar, U. Khalil, U. Qureshi, Seismic activity prediction of the

northern part of pakistan from novel machine learning technique, Journal
of Seismology 25. doi:10.1007/s10950-021-09982-3.

[17] A. Zafar, H. Mujtaba, S. Ashiq, M. O. Beg, A constructive approach

for general video game level generation, in: 2019 11th Computer Science

25

Electronic copy available at: https://ssrn.com/abstract=4090739

 and Electronic Engineering (CEEC), 2019, pp. 102–107. doi:10.1109/
CEEC47804.2019.8974328.

[18] A. Kishwar, K. Batool, Comparison of named entity recognition models

based on neural network in biomedical, in: 2021 International Bhurban
Conference on Applied Sciences and Technologies (IBCAST), 2021, pp.
426–431. doi:10.1109/IBCAST51254.2021.9393197.

[19] Y. Yang, L. Zheng, Z. Jiawei, Q. Cui, Z. Li, P. Yu, Ti-cnn: Convolutional

neural networks for fake news detection, 2018.

[20] C. Song, K. Shu, B. Wu, Temporally evolving graph neural network for
fake news detection, Information Processing Management 58 (6) (2021)
102712. doi:https://doi.org/10.1016/j.ipm.2021.102712.
URL https://www.sciencedirect.com/science/article/pii/
S0306457321001965

[21] M. Samadi, M. Mousavian, S. Momtazi, Deep contextualized text

representation and learning for fake news detection, Informa-
tion Processing Management 58 (6) (2021) 102723. doi:https:
//doi.org/10.1016/j.ipm.2021.102723.
URL https://www.sciencedirect.com/science/article/pii/
S0306457321002077

[22] A. Silva, Y. Han, L. Luo, S. Karunasekera, C. Leckie, Propagation2vec:

Embedding partial propagation networks for explainable fake news early
detection, Information Processing Management 58 (5) (2021) 102618.
doi:https://doi.org/10.1016/j.ipm.2021.102618.
URL https://www.sciencedirect.com/science/article/pii/
S030645732100114X

[23] C. Song, N. Ning, Y. Zhang, B. Wu, A multimodal fake news detection

model based on crossmodal attention residual and multichannel convolu-
tional neural networks, Information Processing Management 58 (1) (2021)

26

Electronic copy available at: https://ssrn.com/abstract=4090739

 102437. doi:https://doi.org/10.1016/j.ipm.2020.102437.
URL https://www.sciencedirect.com/science/article/pii/
S0306457320309304

[24] J. Xue, Y. Wang, Y. Tian, Y. Li, L. Shi, L. Wei, Detect-

ing fake news by exploring the consistency of multimodal data,
Information Processing Management 58 (5) (2021) 102610.
doi:https://doi.org/10.1016/j.ipm.2021.102610.
URL https://www.sciencedirect.com/science/article/pii/
S0306457321001060

[25] A. G. Martı́n, A. Fernández-Isabel, C. González-Fernández, C. Lan-

cho, M. Cuesta, I. Martı́n de Diego, Suspicious news detection
through semantic and sentiment measures, Engineering Applica-
tions of Artificial Intelligence 101 (2021) 104230. doi:https:
//doi.org/10.1016/j.engappai.2021.104230.
URL https://www.sciencedirect.com/science/article/pii/
S0952197621000774

[26] Afp fact check, https://factcheck.afp.com/about-us, accessed: 2021-

07-04.

[27] Fake news sites, https://gist.github.com/Criipi/

a3a7357466821f2ec62ce42b2529394b, accessed: 2021-07-04.

[28] Prntly, https://en.wikipedia.org/wiki/Prntly, accessed: 2021-07-04.

5.1. Appendices

27

Electronic copy available at: https://ssrn.com/abstract=4090739

 Table 9: Samples of Misclassified Examples by LSTM
News News Text Act- Pre-
Num- ual dicted
ber Class Class
N1 Ban on animal sacrifice in Jammu and Kashmir Fake Real
before Eid al-Adha
N2 Helpline number 7837018555 is related to the Fake Real
‘free ride’ facility provided by the Hyderabad
Police for the women travelling in the city dur-
ing nights.
N3 The official Twitter handle of ’Office of the Fake Real
Union Territory of Ladakh’ has been renamed
as ’Gilgit-Baltistan, Ladakh (U.T.), India’.
N4 Pakistan government has imposed large taxes on Fake Real
the registration of motorcycles and rickshaws in
southern Sindh province
N5 Says Sen. John McCain spent the July 4 week- Fake Real
end in Islamabad, Pakistan, selling ”F-16s and
advanced weapons to the folks who harbored
Osama bin Laden.”
N6 Mohammad Hafeez said Pakistan was lacking in Real Fake
a system to groom talented cricketers.
N7 Pakistani activist Karima Baloch has been Real Fake
found dead in Toronto.
N8 Pakistani Muslims destroyed a Hindu temple in Real Fake
northwestern Pakistan Wednesday.
N9 Sir Gangaram built horse train in Pakistans Real Fake
Punjab to transport men, machinery in 1898
N10 Foreign Minister Qureshi says the ”onus lies on Real Fake
India” to reverse steps in Kashmir.

28

Electronic copy available at: https://ssrn.com/abstract=4090739