2015 7th International Conference on Intelligent Human-Machine Systems and Cybernetics

Large Scale Text Classification using Map Reduce and Naive Bayes Algorithm for
Domain Specified Ontology Building

Joan Santoso∗† , Eko Mulyanto Yuniarno∗ , and Mochamad Hariadi∗

∗ Department of Electrical Engineering, Faculty of Industrial Technology
Institut Teknologi Sepuluh Nopember
Surabaya, Indonesia
† Department of Informatics Engineering
Sekolah Tinggi Teknik Surabaya
Surabaya, Indonesia
Email: joan.santoso@gmail.com, ekomulyanto@ee.its.ac.id, mochar@ee.its.ac.id

Abstract—Internet that covers a large information gives an
opportunity to obtain knowledge from it. Internet contains
large unstructured and unorganized data such as text, video,
and image. Problems arise on how to organize large amount of
data and obtain a useful information from it. This information
can be used as knowledge in the intelligent computer system.
Ontology as one of knowledge representation covers a large
area topic. For constructing domain specified ontology, we use
large text dataset on Internet and organize it into specified
domain before ontology building process is done. We try
to implement naive bayes text classifier using map reduce
programming model in our research for organizing our large
text dataset. In this experiment, we use animal and plant
domain article in Wikipedia online encyclopedia as our dataset.
Our proposed method can achieve highest accuracy with score
about 98.8%. This experiment shows that our proposed method
provides a robust system and good accuracy for classifying
document into specified domain in preprocessing phase for
domain specified ontology building.
Keywords-Naive Bayes; Map Reduce; Big Data; Text Classi-
fication; Domain Specified Text Classification;
I. I NTRODUCTION
Internet provides a large resource of data or information.
In everyday life, people use Internet for sharing information
on personal blog or website, upload and watch videos,
download or upload pictures on social media. Recently large
data on the Internet can grow because of user online activity.
Internet contains large unstructured and unorganized data
such as text, video, and image.
Big Data delivers a new paradigm in processing a large
unstructured and complex data efficiently. Big data problems
in [1] are defined as how to process a large amount of data
where the data is complex and difficult to be traditionally
processed by system. These paradigms are given to challenge
us to organize large data on the Internet, especially text
data and obtain information to represent as knowledge in
the intelligent computer system.
Many type of knowledge representation is developed
recently for representing specified domain information or
topics. Ontology is defined as representation concept and
each relation. To build some domain specified ontology, we
need to obtain data that belong to those domain. Internet
provides large data with multiple domain information. We
need to manage those data by classifying them into specified
domain before we extract and build the ontology. We use
text classification algorithm to organize text data on Internet
before our domain specified ontology building process is
done.
Text classification is a process to categorize text in
specific label or class. Many research are developed to
provide automatic classification using Machine Learning
algorithm. Zheng et.al. in [2] use for classifying Chinese
News Document. Rennie et.al. in [3] provides optimization
for Naive Bayes in text classification. Main objective for
our research is to develop a robust text classification system
that can handle large data by implementing naive bayes
text classification algorithm using map reduce programming
model.
Map Reduce defined in [12] is a technique used to perform
Big Data Processing and proposed in [13]. We can write Map
Reduce in functional programming style so it is possible to
run parallel. Map Reduce method divides into 2 step, i.e.
map and reduce. Map function is needed to divide existing
work into a smaller process by generating intermediate key
and value pairs. Reduce function is needed to combine
several intermediate values that have same intermediate key.
We divide our research paper into five section. First
section delivers about our background and objective in this
research. Our related works can be seen in second section.
For third section, we define about our proposed method.
Fourth section is used to define our experiment dataset and
experiment scenario. Last section is used for providing our
research conclusion and our further research.
II. R ELATED W ORKS
Text classification is used in many application such as
sentiment analysis [4], emotion identification from twitter
[5], spam filtering in [6] and [7]. Other researchs in text

978-1-4799-8646-0/15 $31.00 © 2015 IEEE 428

DOI 10.1109/IHMSC.2015.24
classification for Indonesian Language are developed in [8]
and [9]. Rachmania et.al. in [8] provides classification for
Indonesian news document and Laksana et.al. in [9] provides
classification for Indonesian tweets in twitter. Some research
that use Naive Bayes as classifier can be found at [10], [7],
and [11]. By looking at this, we also try to use Naive Bayes
as classifier algorithm in our research. We use this algorithm
because Naive bayes is widely used in text classification and
have some advantages, that are simple to implement and
have a good performance in text classification.
Several Map Reduce performance for handling big data
problem are defined in [1]. Yahoo in 2009 successfully sort
500GB of data with only 59 seconds using Map Reduce.
By looking at this advantage, Map Reduce is taken into
many research about developing machine learning algorithm
in map reduce way. Research in [14] provides many machine
algorithms that can be implemented using Map Reduce, such
as Naive Bayes, k-Means, Logistic Regression, Principal
Components Analysis, Neural Network, etc. We design our
Naive Bayes algorithm in map reduce programming style
because we believe our classifier can provides us a good
speed and accuracy in classification process.
III. P ROPOSED M ETHOD
We present our proposed method in this section. Our
proposed method is divided into two parts. First part is
preprocessing for our data. Second part presents about text
classification that we us in this research. Architecture system
for our work can be looked at figure 1.
Figure 1. Architecture System
Our classification process is divided into two phases.
First phase is learning from annotated document to build
429
classifier knowledge. Second phase is classifying new data
into specific class. Preprocessing is used for each process
either classification or learning phase.
A. Preprocessing
Our preprocessing in this research consists of 4 processes.
We can see detail of our preprocessing process in figure 2.
First process is about content extraction. This step is used
to extract website content by removing unimportant part in
web page. We use some handmade rule in this web page
extraction process.
Figure 2. Preprocessing Process
Second process is tokenization. We use this process to
break up document into token such as word, number, or
punctuation. We develop some regular expressions to help
our tokenization process. Third process is stopword removal.
Some common words that usually appear in document will
be removed using some dictionary list from [15]. Fourth
or last process in our preprocessing phase is stemming.
This process is used to map different morphological variant
of word into base word. We use Indonesian Language
Stemmer in [15] for our stemming process. We implement
this preprocessing algorithm using map reduce in algorithm
1 and 2.
B. Naive Bayes Text Classification
Our text classification phase is divided into two phases.
First phase is learning process to build some knowledge and
second phase is document classification. Feature extraction
is needed to help classifier to learn or classify document.
We use standard bag-of words representation as feature in
our classification algorithm. We can see detail of our text
classification process in figure 3.
Algorithm 1 Map Function for Preprocessing Algorithm
1: Input : filename,document
2: Output : Intermediate key, Intermediate value
3: token list ← getT oken(document)
4: for each token t in token list do
5: if t not in stopword then
6: content ← content + getStemmingResult(t)
7: end if
8: end for
9: emitIntermediate(f ilename, content)

Algorithm 2 Reduce Function for Preprocessing Algorithm

1: Input : key,list(value)
2: Output : key,value
3: for each value v in list(value) do
4: content ← content + v
5: end for
6: emit(key, content) Figure 3. Text Classification Process

And for computing prior probability P(c), we can use 4.

Naive Bayes algorithm is widely use as text classification
algorithm. This classifier uses bayes theorem with some Nc
naive assumption about class conditional independence for P(c) = (4)
N
each feature. Naive Bayes classifier uses probability to find
Where :
output target class with maximum posterior probability. We
can define naive bayes classifier model in 1. We can compute • Nc : total target class c in dataset

P(ci |D) using Bayes Theorem in 2 as follows. • N : total instance in dataset

ŷ = argmax P(ci |D) (1) Document D represents bag-of word feature. We can
ci ∈C define document D={w1 , w2 , w3 , ..., wn } where n is total
Where : number of features in document. For computing posterior
• D : document
probability P(D|c), we can use 5 and 6. We use laplace
• ci : target class ci in C
smoothing in 6 that is a smoothing algorithm which is widely
• C : all target class in dataset
used in text classification.
• P(ci |D) : posterior probability class ci if given 
document D P(D|c) = P(wi |c) (5)
wi ∈D

. Where :
P(D|c) P(c)
P(c|D) = (2) • D : document D
P(D) • c : target class
• wi : word in document D
Usually some research in naive bayes classifier is dropping • P(wi |c) : posterior probability word wi in D if given
denominator so this equation can define into 3. class c

P(c|D) = P(D|c) P(c) (3)

count(wi ,c)+1
Where : P(wi |c) = (6)
count(c) + |V |
• D : document
• c : target class Where :
• P(D|c) : posterior probability document D if given • |V |: Vocabulary size
target class c • count(wi ,c) : total term wi in class c
• P(c) : prior probability for target class c in dataset • count(c) : total term in class c

430
 Our naive bayes classifier implementation in map reduce Algorithm 6 Reduce Function for Naive Bayes Classifica-
is based on research in [14]. This implementation for naive tion Algorithm
bayes algorithm is divided into two phases. First phase 1: Input : key,list(value)
produces a knowledge from training data. Second phase 2: Output : key,value
performs classification itself. We implement our training 3: maxV alue ← −1
phase using algorithm 3 and 4. 4: for each value in list(value) do
5: if value.probability ≥ maxV alue then
Algorithm 3 Map Function for Naive Bayes Learning 6: maxV alue ← value.probability
Algorithm 7: tclass ← value.class
1: Input : label,document 8: end if
2: Output : Intermediate key,Intermediate value 9: end for
3: emitIntermediate(tclass + ” ” + label, 1) 10: emit(key, tclass)
4: for each term in document do
5: emitIntermediate(label + ” ” + term, 1)
6: end for A. Dataset
Our dataset are articles from Wikipedia which is an online
encyclopedia. We use 2000 articles for our experiment. We
Algorithm 4 Reduce Function for Naive Bayes Learning use 2000 Wikipedia articles in this experiment from two
Algorithm domains, i.e. animal and plant. We can see statistic for each
1: Input : key,list(value) dataset that we used in table I.
2: Output : key,value
Table I
3: for v in list(value) do T EXT DATASET FOR T RAINING
4: result ← result + v
5: end for No Dataset Articles
1 Plant Domain 1000
6: emit(key, result)
2 Animal Domain 1000
3 Total 2000
For classification process, we use map reduce program-
ming style for labeling our large text dataset. Detail process We use these data in table I as training dataset for our
in our text classification implementation can be seen in classifier. For our experiment, we use 3 testing datasets
algorithm 5 and 6. that consist of several documents belong to animal or plant
domain. These three testing datasets are used in our three
Algorithm 5 Map Function for Naive Bayes Classification experiment scenarios. We can see statistic for our testing
Algorithm dataset in table II.
1: Input : document,learning knowledge, list of class
Table II
2: Output : Intermediate key, Intermediate value T EXT DATASET FOR E XPERIMENTS
3: nb model ← buildClassif ier(learning knowledge)
4: for each c in list of class do No Experiment Article
1 First Scenario 250
5: prior ← nb model.getP riorP robability(c) 2 Second Scenario 500
6: for each token in document do 3 Third Scenario 1000
7: temp ← nb model.getP osterior(c, token)
8: posterior ← posterior x temp
9: end for B. Experiment Result
10: probability ← prior x posterior Our learning process shows a fast process in running time,
11: emitIntermediate(document, (class, probability)) i.e. 31 second for learning 2000 articles. We divide our
12: end for experiment scenario into three parts. For first experiment
scenario, we try classifying using 250 documents. Second
experiment scenario is used to classify 500 documents. Last
IV. E XPERIMENTS experiment scenario is used to classify 1000 documents. Our
We present our experiment in this section. We use single experiment accuracy and running time can be seen in table
node machine with hadoop configured in pseudo distributed III.
mode for our experiment. We divide this section into two First experiment shows that our classifier can achieve
sections. First is about our dataset and second is about our 98.4% for classifying document into animal and plant
experiment result. domain. Second experiment shows that our classifier can

431
 Table III
E XPERIMENTS R ESULT
No Experiment Accuracy Running Time
1 First Scenario 98.4% 1 mins 31 sec
2 Second Scenario 98.8% 2 mins 18 sec
3 Third Scenario 98.1 % 3 mins 35 sec
achieve 98.8% in document classification result. Third exper-
iment can achieve 98.1% as experiment result. Our proposed
system can achieve highest accuracy value 98.8% in second
experiment. This result shows our proposed method can
provides a robust system for text classification for specifying
domain specified article in preprocessing phase for domain
specified ontology building.
V. C ONCLUSION AND F URTHER R ESEARCH
Internet provides a lot of data. Many data type in Internet
is unstructured, such as image, video, text, and sound.
We believe that text classification can be used to organize
large unstructured data on Internet into structured data in
specified domain. Our proposed method using Naive Bayes
algorithm in Map Reduce programming model as classifier
can achieve highest accuracy with score about 98.8%. This
experiment shows that our proposed method provides a
robust system and good accuracy for classifying document
into specified domain as preprocessing phase for domain
specified ontology building.
Some of articles usually not only belong to one topic but
sometimes it can also belong to another topics. We will try
to improve our algorithm so it can classify document with
multi label class as result in our further research. We will
also try to implement real time text classification algorithm
to classify document into specified domain. This process
is done as preprocessing in automatic domain specified
ontology building process for our further research.
ACKNOWLEDGMENT
We would like to thank partners in Institut Teknologi
Sepuluh November Surabaya for their helpful comments. We
also thank partners from computational linguistic research
group in Sekolah Tinggi Teknik Surabaya for providing us
some tools and several data in this research.
R EFERENCES
[1] T. White, Hadoop: The definitive guide. ” O’Reilly Media,
Inc.”, 2012.
[2] G. Zheng and Y. Tian, “Chinese web text classification system
model based on naive bayes,” in E-Product E-Service and E-
Entertainment (ICEEE), 2010 International Conference on.
IEEE, 2010, pp. 1–4.
[3] J. D. Rennie, L. Shih, J. Teevan, D. R. Karger et al., “Tackling
the poor assumptions of naive bayes text classifiers,” in ICML,
vol. 3. Washington DC), 2003, pp. 616–623.
432
[4] B. Pang, L. Lee, and S. Vaithyanathan, “Thumbs up?: sen-
timent classification using machine learning techniques,” in
Proceedings of the ACL-02 conference on Empirical methods
in natural language processing-Volume 10. Association for
Computational Linguistics, 2002, pp. 79–86.
[5] W. Wang, L. Chen, K. Thirunarayan, and A. P. Sheth,
“Harnessing twitter” big data” for automatic emotion iden-
tification,” in Privacy, Security, Risk and Trust (PASSAT),
2012 International Conference on and 2012 International
Confernece on Social Computing (SocialCom). IEEE, 2012,
pp. 587–592.
[6] I. Androutsopoulos, J. Koutsias, K. V. Chandrinos,
G. Paliouras, and C. D. Spyropoulos, “An evaluation
of naive bayesian anti-spam filtering,” arXiv preprint
cs/0006013, 2000.
[7] V. Metsis, I. Androutsopoulos, and G. Paliouras, “Spam
filtering with naive bayes-which naive bayes?” in CEAS,
2006, pp. 27–28.
[8] A. Rachmania, J. Jaafar, and N. Zamin, “Likelihood calcula-
tion classification for indonesian language news documents,”
in Information Technology and Electrical Engineering (ICI-
TEE), 2013 International Conference on, Oct 2013, pp. 149–
154.
[9] J. Laksana and A. Purwarianti, “Indonesian twitter text
authority classification for government in bandung,” in
Advanced Informatics: Concept, Theory and Application
(ICAICTA), 2014 International Conference of, Aug 2014, pp.
129–134.
[10] I. Ahmed, D. Guan, and T. C. Chung, “Sms classification
based on naive bayes classifier and apriori algorithm frequent
itemset,” International Journal of Machine Learning and
Computing, vol. 4, pp. 183–187, 2014.
[11] N. M. A. Lestari, I. K. G. D. Putra, and A. K. A. Cahyawan,
“Personality types classification for indonesian text in partners
searching website using naı̈ve bayes methods,” International
Journal of Computer Science Issues (IJCSI), vol. 10, no. 1,
2013.
[12] A. Cuzzocrea, I.-Y. Song, and K. C. Davis, “Analytics over
large-scale multidimensional data: the big data revolution!”
in Proceedings of the ACM 14th international workshop on
Data Warehousing and OLAP. ACM, 2011, pp. 101–104.
[13] J. Dean and S. Ghemawat, “Mapreduce: Simplified data
processing on large clusters, osdi04: Sixth symposium on
operating system design and implementation, san francisco,
ca, december, 2004,” S. Dill, R. Kumar, K. McCurley, S.
Rajagopalan, D. Sivakumar, ad A. Tomkins, Self-similarity in
the Web, Proc VLDB, 2001.
[14] C. Chu, S. K. Kim, Y.-A. Lin, Y. Yu, G. Bradski, A. Y.
Ng, and K. Olukotun, “Map-reduce for machine learning
on multicore,” Advances in neural information processing
systems, vol. 19, p. 281, 2007.
[15] F. Tala, “A study of stemming effects on information retrieval
in bahasa indonesia,” 2003.