International Conference on Computer Science and Information Technology 2008

Handling Noisy Data using Attribute Selection and Smart Tokens

J. Jebamalar Tamilselvi Dr. V. Saravanan

PhD Research Scholar Associate Professor & HOD
Department of Computer Application Department of Computer Application
Karunya University Karunya University
Coimbatore – 641 114,Tamilnadu, INDIA Coimbatore – 641 114, Tamilnadu, INDIA
E-mail: jebamalar@karunya.edu E-mail: saravanan@karunya.edu

Abstract initial step in data cleaning. There are many

Data cleaning is a process of identifying or
determining expected problem when integrating data
from different sources or from a single source. There
are so many problems can be occurred in the data
warehouse while loading or integrating data. The
main problem in data warehouse is noisy data. This
noisy data error is due to the misuse of abbreviations,
data entry mistakes, duplicate records and spelling
errors. The proposed algorithm will be efficient in
handling the noisy data by expanding abbreviation,
removing unimportant characters and eliminating
duplicates. The attribute selection algorithm is used
for the attribute selection before the token formation.
An attribute selection algorithm and token formation
algorithm is used for data cleaning to reduce a
complexity of data cleaning process and to clean data
flexibly and effortlessly without any confusion. This
research work uses smart token to increase the speed
of the mining process and improve the quality of the
data.
1. Introduction
The data cleaning process is used to improve the
quality of the data before the mining process. There
are so many error will be introduced while integrating
the data warehouses or while loading a single data
warehouse by the misuse of data entry problem. One
of the main errors in data warehouse is noisy data. The
noisy data is a random error or variance in a measured
variable. The noisy data errors are due to the misuse of
abbreviations, data entry mistakes, duplicate records
and spelling errors [16].
An attribute selection is very important to reduce
the time of the data cleaning process. An attribute
selection algorithm is effective in reducing attribute,
removing irrelevant attribute, increasing speed of the
data cleaning process, and improving result in clarity.
An intelligent attribute selection is proposed as an
approaches available for selecting the attributes for the
mining process to reduce dimensionality of the data
warehouse. However, the recent increase of
dimensionality of data poses difficulty with respect to
efficiency and effectiveness in data cleaning process.
The efficiency and effectiveness of attribute selection
method is demonstrated through extensive
comparisons with this proposed method using real-
world data of high dimensionality [3], [5].
The token based approach will be applied in the
selected attribute fields only. This attribute is selected
based on the certain criteria. This attribute selection is
mainly for the data cleaning process. The similarity
function with long sting will take more time for the
comparison process as well as it requires multi-pass
approach. The proposed token formation algorithm is
used to form a token for the selected attribute fields.
The token based approach is proposed to reduce the
time for the comparison process and to increase the
speed of the data cleaning process [8], [9]. This
research paper deals about developing an approach to
handle noisy data using Attribute selection and Smart.
2. Related Work
The large data file is sorted on designated fields to
bring potentially identical records together. However,
sorting is based on “dirty” fields, which may fail to
bring matching records together, and its time
complexity is quadratic in the number of records. This
sorting technique is inefficient while dealing with
large data file [BD, 83]. The merge/purge problem in a
large database is solved by forming keys from some
selected fields, sorting the entire data set on the keys,
clustering the sorted records and using a scanning
window of a fixed size to reduce the number of
comparisons [6], [7].
The several steps are used to clean the data
warehouse [10]. The first step is Scrub dirty data
fields. This step attempts to remove typographical

978-0-7695-3308-7/08 $25.00 © 2008 IEEE 770

DOI 10.1109/ICCSIT.2008.62