(IJCSIS) International Journal of Computer Science and Information Security,Vol. 8, No. 2, 2010
Clustering Unstructured Data (Flat Files)
An Implementation in Text Mining Tool
, Atika Mustafa
and Anis Noor Ali
Department of Computer ScienceFAST
National University of Computer and Emerging SciencesKarachi, Pakistan
With the advancement of technology and reducedstorage costs, individuals and organizations are tending towardsthe usage of electronic media for storing textual information anddocuments. It is time consuming for readers to retrieve relevantinformation from unstructured document collection. It is easierand less time consuming to find documents from a largecollection when the collection is ordered or classified by group orcategory. The problem of finding best such grouping is still there.This paper discusses the implementation of k-Means clusteringalgorithm for clustering unstructured text documents that weimplemented, beginning with the representation of unstructuredtext and reaching the resulting set of clusters. Based on theanalysis of resulting clusters for a sample set of documents, wehave also proposed a technique to represent documents that canfurther improve the clustering result.
Information Extraction (IE); Clustering, k-MeansAlgorithm; Document Classification; Bag-of-words; DocumentMatching; Document Ranking; Text Mining
Text Mining uses unstructured textual information andexamines it in attempt to discover structure and implicit
meanings ―hidden‖ within the text [
6]. Text mining concernslooking for patterns in unstructured text .A cluster is a group of related documents, and clustering,also called unsupervised learning is the operation of groupingdocuments on the basis of some similarity measure,automatically without having to pre-specify categories . Wedo not have any training data to create a classifier that haslearned to group documents. Without any prior knowledge of number of groups, group size, and the type of documents, theproblem of clustering appears challenging .Given N documents, the clustering algorithm finds
,number of clusters and associates each text document to thecluster. The problem of clustering involves identifying numberof clusters and assigning each document to one of the clusterssuch that the intra-documents similarity is maximum comparedto inter-cluster similarity.
Figure 1. Document Clustering
One of the main purposes of clustering documents is toquickly locate relevant documents . In the best case, theclusters relate to a goal that is similar to one that would beattempted with the extra effort of manual label assignment. Inthat case, the label is an answer to a useful question. Forexample, if a company is operating at a call center where usersof their products submit problems, hoping to get a resolution of their difficulties, the queries are problem statements submittedas text. Surely, the company would like to know about thetypes of problems that are being submitted. Clustering can helpus understand the types of problems submitted . There is alot of interest in the research of genes and proteins using publicdatabases. Some tools capture the interaction between cells,molecules and proteins, and others extract biological facts fromarticles. Thousands of these facts can be analyzed forsimilarities and relationships . Domain of the inputdocuments used in the analysis of our implementation,discussed in the following sections, is restricted to ComputerScience (CS).II.
Before clustering algorithm is used, it is necessary to givestructure to the unstructured textual document. The documentis represented in the form of vector such that the words (alsocalled features) represent dimensions of the vector andfrequency of the word in document is the magnitude of thevector. i.e.
A Vector is of the form<(t1,f1),(t2,f2),(t
where t1,t2,..,tn are the terms/words(dimension of the vector)
and f1,f2,…,fn are the corresponding frequencies or
magnitude of the vector components.