https://bit.

ly/2MWyuD7

Text Mining

Social
Sepuluh Nopember
Institute of Technology (ITS)

Sentiment
29-31 Oct. 2019

Analysis
Using WEKA Tools

Dr Shuzlina Abdul Rahman Ts Dr Sofianita Mutalib

Associate Professor Senior Lecturer
Centre for Information Systems Studies, Faculty of Computer & Mathematical Sciences,
Universiti Teknologi MARA (UiTM), Shah Alam, MALAYSIA
Learning Outcomes from
Workshop

• At the end of the workshop, participants

should be able to use WEKA for the Sentiment
below tasks:
Analysis
• pre-process textual data
• develop classification model using
machine learning algorithms
INTRODUCTION

WHAT IS SENTIMENT WHY SA IS HOW TO APPLY SA IN

ANALYSIS (SA)? INTERESTING? WEKA?
 What is
Sentiment
Analysis
(SA)?
[1]

SA - process which focuses on analyzing people’s opinions, feelings, and

attitudes towards a specific product, organization or service.
 Why SA is
interesting?

[2] https://www.pewresearch.org/fact-tank/2019/04/04/indonesians-optimistic-
about-their-countrys-democracy-and-economy-as-elections-near/
Basic Concepts
• A document can be described by a set of
representative keywords called index terms.
Concepts • Different index terms have varying relevance when
of Text used to describe document contents.
Analytics? • This effect is captured through the assignment of
numerical weights to each index term of a document.
(e.g.: frequency, term-frequency-index document
frequency or tf-idf)
• DBMS Analogy
• Index Terms à Attributes
• Weights à Attribute Values

7 Han et al, 2011

 Basic Concepts
• Index Terms (Attribute) Selection - tokenization:
• Stop list – irrelevant set of words (a, the, of,…)
• Word stem – (drug: drugs, drugged) viewed as different occurrences of the same
word
• Index terms weighting methods
• Terms r Documents Frequency Matrices – measures the no. of
occurrences of term t in the document d

Han et al, 2011

8
 Text Categorization
• Pre-given categories and labeled document examples (Categories may form
hierarchy)
• Classify new documents ; e.g. Google mail apps
• A standard classification (supervised learning ) problem

Positive
Categorization
Positive
Past Reviews System
Negative

… …
Positive
Neutral
Neutral
New Reviews Negative
Han et al, 2011
9
 Data Acquisition and Preparation

Tokenization

Text Pre-processing Feature selection

How To Feature Transformation

apply SA in
Weka Tools? Text Classification using
Random Forest (RF)
Naïve Bayes (NB)

Machine Learning Logistic Regression (LR)

Support Vector Machine (SVM)

Analysis of the results

 WEKA

Machine Learning Software in Java

https://www.kaggle.com/datasets https://archive.ics.uci.edu/ml/datasets.php
WEKA 3 • Weka is a collection of
machine learning
algorithms for data mining
tasks. It contains tools for
data preparation,
classification, regression,
clustering, association rules
mining, and visualization.
• https://www.cs.waikato.ac.
nz/ml/weka/
• Two versions:
• Stable ver. (3.8)
• Developer ver. (3.9)
Waikato Environment for Knowledge
Analysis (WEKA)
In 1993, the University of Waikato in New Zealand began development of
the original version of Weka

It is released as open source software under the GNU GPL. It is written in

JAVA and provides a API that is well documented and promotes integration
into your own applications

WEKA provides a collection of data mining, machine learning algorithms

and processing tools

WEKA is an environment for comparing learning algorithms

 WEKA GUI Chooser

Data
Mining
Tool

https://www.cs.waikato.ac.nz/ml/weka/index.html
WEKA Interfaces

E-BOOKS
 https://www.cs.waikato.ac.nz/ml/weka/documentation.html

Select and run classification Run association algorithms to

and regression algorithms to extract insights from your Visualize the relationship
operate on your data between attributes.
data

Load a dataset and Run attribute selection

manipulate the data into a Select and run clustering algorithms on your data to
form that you want to work algorithms on your dataset select those attributes that
with are relevant to the feature
you want to predict.
 .arff
.csv .dat

WEKA
Data
Format
 WEKA 3

LET’S BEGIN
Dataset - Product Reviews
• Amazon Kindle
• https://www.kaggle.com/bharadwaj6/kindle-
reviews
• Amazon Kindle Store category from May 1996
- July 2014.
• Contains total of 982619 entries.
• Each reviewer has at least 5 reviews and each
product has at least 5 reviews in this dataset.
Download and open files and
save (.csv) in Excel

You need to save the

file in .csv (Comma
delimited)
 Data Preparation
• Transform
• Merge
• Rename

Text Pre-processing
• Tokenization
• Attribute selection
Outline
Text Classification using Machine Learning
• Random Forest (RF)
• Naïve Bayes (NB)
• Logistic Regression (LR)
• Support Vector Machine (SVM)

Analysis of the results

Data Preparation
• For this exercise
• 10 attributes
• 2500 instances
• Two primary attributes
• overall
• reviewText
• Delete other attributes
• no, asin, helpful,
reviewTime,
reviewID,
reviewerName,
summary, and
unixReviewTime
 Data Preparation
• Then, Load your dataset into Weka
• Click ‘Open File..’ -> find your data ‘.csv’
-> click ‘Open’
Data Preparation
• Overall is a overall rating based on the
reviews.
• We have reviews in the range of [1-5].
• Let label rating = 1 or 2 as the
“Negative” review .
• Lets consider "3" as the “Neutral”
review.
• And rating = 4 or 5 as “Positive”
reviews.
• Then, run again the dataset in Weka.
Data Preparation
• Overall is a overall rating based
on the reviews.
• We have reviews in the range of
[1-5].
• Let label rating = 1 or 2 as the
“Negative” review .
• Lets consider "3" as the
“Neutral” review.
• And rating = 4 or 5 as “Positive”
reviews.
Data Preparation
• Change from numeric to nominal
• Steps:
• Filter>>Unsupervised>> Attribute >> NumericToNominal
• Change ”attributeIndices” to first
• Click OK >> Apply
 Merge the values of
attribute “overall”, for
Data Preparation ‘1’ and ‘2’ and
‘4’ and ‘5’
• Merge TWO attributes’ value;
• Steps:
• Filter>>Unsupervised>> Attribute >> MergeTwoValues
• Change ”attributeIndices” to first
• Change “firstValueIndex” to 1 or first
• Change “secondValueIndex” to 2
• Click OK >> Apply

RECALL our objective is to:

• Overall is a overall rating based on the reviews.
• We have reviews in the range of [1-5].
• Let label rating = 1 or 2 as the “Negative” review .
• Lets consider "3" as the “Neutral” review.
• And rating = 4 or 5 as “Positive” reviews.
Data Preparation
• Rename the new label
• Steps:
• Filter>>Unsupervised>> Attribute >> RenameNominalValues
• Change ”selectedAttributes” to 1
• Type in “valueReplacements” with 1_2:neg, 3:neu, 4_5:pos
• Click OK >> Apply
Data Preparation
• To set attribute “overall” as
class label
• Steps:
• Click edit, go to header
column one, right hand
click, choose “Attribute as
class”
• Save file as
kindle_review_labelled.arff
Data Preparation with Excel (alternative)
Go to Excel. Find and Replace
‘Overall’ number into label.
 Data Preparation

• Transform
• Merge
• Rename

Text Pre-processing

• Tokenization
• Attribute selection
Outline
Text Classification using Machine Learning

• Random Forest (RF)

• Naïve Bayes (NB)
• Logistic Regression (LR)
• Support Vector Machine (SVM)

Analysis of the results

Text Pre-processing (Tokenization)
• Process text (attribute review)
from Nominal to String
• Steps;
• Choose
Filter>>Unsupervised>>attribute
>>Nominal to String
• Change “attributeIndexes” to 1
• Click OK, then Apply
Text Pre-processing (Tokenization)
• Process text (attribute review) to create
Index Terms
• Steps;
• Choose
Filter>>Unsupervised>>attribute>>StringToWordVector”
• Change “IDFTransform” to True
• Change “TFTransorm” to True
• Change “attributeIndices” to 1
• Change “lowerCaseTokens” to True
• Change “outputWordCounts” to ‘True’,
• Goto “stemmer” and Choose ”IteratedLovinsStemmer”
• Goto “stopwordsHandler” and Choose “Rainbow”
• Goto “tokenizer” and Choose “WordTokenizer”, type in
delimiters as follows:
.,;:'"()?!@#$%^&*(){}[]/|\1234567890-=+_)
• WordToKeep = 200
• Click OK, then Apply
• Word Tokenizer used for tokenizing a set of delimiter characters
• Filter -> Choose ‘Unsupervised’ -> ‘Attribute’ -> click
‘StringToWordVector’

• 200 attributes to
be kept for each
class
• WordToKeep
= 200
Text Pre-processing (Tokenization): Output
• Selection of attribute also can be done in ‘Classify’ tab.
• On ‘Classify’ tab -> ‘Classifier’ -> ‘Meta’ -> ‘AttributeSelectedClassifier’

Classifier -> choose ‘J48’

Evaluator -> choose
‘GainRatioAttributeEval’
search -> choose ‘Ranker’

Cont…
• Click on ‘Ranker’ -> change ‘numToSelect’ =100
which means Weka will select the top 100 attributes by the rank
Alternative Approach:
On ‘Select attribute’ tab >>choose ‘AttributeSelection’>>
‘CorrelationAttributeEval’; choose “Ranker”
Click Start ; always ensure the Class label is the correct one.

CorrelationAttributeEval
• Evaluates the worth of an attribute by
measuring the correlation (Pearson’s)
of the attributes with the class.
• Nominal attributes are considered on
a value by value basis by treating each
value as an indicator. An overall
correlation for a nominal attribute is
arrived at via a weighted average.
• ‘Test options’ – for exercise we are going to use percentage split by
default = 66% -> click ‘Start’
Text preprocessing: Save reduced dataset

In the result list, go to the

run result and right hand
click, choose “Save reduced
data” and name the file.
Alternative technique of Tokenization NGram
• Open the original 2500 dataset (or retrieved the save file).
• Perform the similar process to get below result:
Text preprocessing – NGram Tokenization and Stop Words
• NGramTokenizer used to split a string into an n-gram with min and max grams.
• Filter -> unsupervised -> ‘StringToWordVector’

IDFTransform, TFTransform,
lowerCaseTokens,
outputWordCounts ->
change to ‘True’,
stopwordsHandler =
‘MultiStopWords’
Tokenizer =
‘NGramTokenizer’,
WordToKeep = 200,
Click OK & Apply
Text preprocessing – NGramTokenization and Stop Words
• Check the Select Attribute and the word ranked in it.
Attribute evaluator -> choose ‘CorrelationAttributeEval’ ->click ‘Start’
Tokenization:
Use Own Dictionary and Stop Words
Best Practice - Always save files in .arff format
• Preprocess -> click ‘Save’ -> name your file .arff -> click ‘Save’
 Data Preparation

• Transform
• Merge
• Rename

Text Pre-processing

• Tokenization
• Attribute selection
Outline
Text Classification using Machine Learning

• Random Forest (RF)

• Naïve Bayes (NB)
• Logistic Regression (LR)
• Support Vector Machine (SVM)

Analysis of the results

 • Text Classification based on
Sentiment
• Sentiment is labelled based on
case study
Text
Classification
Text Classification – RandomForest (RF)
Test Options: Cross-validation 10 folds
Save Models
• Classify -> in ‘Result list’ -> right click on ‘RandomForest’-> click ‘Save result buffer’

• Name your model and save as “.model”

• Repeat step for each model.
Text Classification – Naïve Bayes (NB)
Test Options: Cross-validation 10 folds
Text Classification – Logistic Regression (LR)
Test Options: Cross-validation 10 folds
Text Classification – Support Vector Machine (SVM)
Test Options: Cross-validation 10 folds
To use SVM classification, you need to download the LibSVM first.
Go to Weka GUI -> Tools -> click on Package manager

Download the latest version and install

Text Classification – Support Vector Machine (SVM)
Test Options: Cross-validation 10 folds
To use SVM classification, you need to download the LibSVM first.
Go to Weka GUI -> Tools -> click on Package manager
Text Classification – RandomForest (RF)
Test Options: Percentage split by default = 66% training, 34% testing
Text Classification – Naïve Bayes (NB)
Test Options: Percentage split by default = 66% training, 34% testing
Text Classification – Logistic Regression (LR)
Test Options: Percentage split by default = 66% training, 34% testing
Text Classification – Support Vector Machine (SVM)
Test Options: Percentage split by default = 66% training, 34% testing
 Data Preparation

• Transform
• Merge
• Rename

Text Pre-processing

• Tokenization
• Attribute selection
Outline
Text Classification using Machine Learning

• Random Forest (RF)

• Naïve Bayes (NB)
• Logistic Regression (LR)
• Support Vector Machine (SVM)

Analysis of the results

 Analysis of the Results
• This result is based on the average weight.

Methods Ten fold cross-validation Split percentage by default =

66% training, 34% testing
TP Rate FP Rate Precision Recall TP Rate FP Rate Precision Recall

Random Forest (RF) 0.611 0.256 0.566 0.611 0.586 0.265 0.535 0.586
Naïve Bayes (NB) 0.505 0.271 0.530 0.505 0.502 0.265 0.531 0.502
Logistic Regression (LR) 0.612 0.222 0.593 0.612 0.576 0.235 0.563 0.576

Support Vector Machine (SVM) 0.640 0.235 0.614 0.640 0.638 0.236 0.637 0.638
Classifier Evaluation Metrics:
Precision and Recall
• Precision: exactness – what % of tuples that the classifier labeled as positive are actually positive?
*+ *+
• 𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = =
*+,-+ +.

• Recall: completeness – what % of positive tuples did the classifier label as positive?
*+ *+
• 𝑟𝑒𝑐𝑎𝑙𝑙 = =
*+,-1 +

Actual C1 ~C1
• Perfect score is 1.0 class\Predicte
d class
• Inverse relationship between C1 True False
precision & recall Positives
(TP)
Negatives
(FN)
~C1 False True
Positives Negatives
(FP) (TN)
62
Basic Measures for Text Retrieval

Relevant Relevant &

Retrieved Retrieved

All Documents

• Precision: the percentage of retrieved documents that are in fact relevant to the query (i.e.,
“correct” responses)
| {Relevant} Ç {Retrieved} |
precision =
| {Retrieved} |

• Recall: the percentage of documents that are relevant to the query and were, in fact, retrieved

| {Relevant} Ç {Retrieved} |
recall =
| {Relevant} |
63
References
• [1] Sentiment Analysis, https://devopedia.org/sentiment-analysis

• [2] Indonesians optimistic about their country’s democracy and economy as

elections near, APRIL 4, 2019, https://www.pewresearch.org/fact-
tank/2019/04/04/indonesians-optimistic-about-their-countrys-democracy-and-
economy-as-elections-near/

• [3] Jiawei Han, Micheline Kamber, and Jian Pei, Data Mining: Concepts and
Techniques, 3rd edition, Morgan Kaufmann, 2011.

• [4] AN OVERVIEW OF SENTIMENT ANALYSIS,

https://jgateplus.com/home/2019/01/16/an-overview-of-sentiment-analysis/
 How to Assign Weights
• Two-fold heuristics based on frequency
• TF (Term frequency)
• More frequent within a document à more relevant to semantics
• e.g., “query” vs. “commercial”

• IDF (Inverse document frequency)

• Less frequent among documents à more discriminative
• e.g. “algebra” vs. “science”

65
 TF Weighting
• Weighting:
• More frequent => more relevant to topic
• e.g. “query” vs. “commercial”

• Raw TF= f(t,d): how many times term t appears in doc d

• Normalization:
• Document length varies => relative frequency preferred
• e.g., Maximum frequency normalization

66
 IDF Weighting
• Ideas:
vLess frequent among documents à more discriminative
• Formula:

•
• n — total number of docs
• k — # docs with term t appearing
(the DF document frequency)

67
 TF-IDF Weighting
• TF-IDF weighting : weight(t, d) = TF(t, d) * IDF(t)
• Frequent within doc à high tf à high weight
• Selective among docs à high idf à high weight
• Recall VS model
• Each selected term represents one dimension
• Each doc is represented by a feature vector
• Its t-term coordinate of document d is the TF-IDF weight
• This is more reasonable
• Just for illustration …
• Many complex and more effective weighting variants exist in practice

68
Regex
• https://www.w3schools.com/python/python_regex.asp