2020 International Conference on Computers, Information Processing and Advanced Education (CIPAE)
A Discretization Method for Industrial Data Based
2020 International Conference on Computers, Information Processing and Advanced Education (CIPAE) | 978-1-7281-8223-0/20/$31.00 ©2020 IEEE | DOI: 10.1109/CIPAE51077.2020.00060
on Big Data Technology
Xiang Wan*, Cheng Wang, Zhengming Tang, Haijun Sun, Shan Gao, Lei Qiao
Wuhan Second Ship Design and Research Institute, Hubei Wuhan 430064, China
*Corresponding author e-mail: wanwuhan@yeah.net
Abstract—A parallel improvement of the traditional
K-Means clustering algorithm is achieved based on the
Mapreduce architecture, and the new parallelized clustering
algorithm is used to realize the discretization of industrial big
data in this paper. The new algorithm streamlines the
calculation process, meanwhile, saves the computational
overhead caused by data analysis and communication
consumption caused by information transfer.
Keywords—Big data, industrial data, K-Means clustering
algorithm
I. INTRODUCTION
Industrial data come from a wide variety of sources and
diversity, hence there are significant differences between
industrial parameters, and sometimes the differences can
even reach several orders of magnitude. If the differences
between parameter values is not processed properly in
advance, the workload of parameter identification and
difference processing in subsequent data mining will be
increased, which will adversely affect the efficiency and
accuracy of mining work. Therefore, it is necessary to
perform a discretization operation on industrial data so that
it can present a unified model which is conducive to data
mining.
By replacing the initial data with several groups, clusters,
intervals, labels, etc., data discretization reduces the
differences between the initial data parameters and reduces
the size of the initial data. Consequently, the efficiency of
subsequent mining work is improved, meanwhile the mining
results are more practical and easier to understand due to the
unification of mining object patterns. Common
discretization methods include group discretization,
histogram discretization, clustering discretization, etc. In
contrast to the first two methods which need to manually
formulate grouping rules and set grouping parameters,
clustering method is more objective. Thus we prefer use
clustering method to realize the discretization of industrial
data.
Fig.1 The Flow of K-Means Clustering Algorithm
978-1-7281-8223-0/20/$31.00 ©2020 IEEE
DOI 10.1109/CIPAE51077.2020.00060
Authorized licensed use limited to: Bauman Moscow State Technical University. Downloaded on September 28,2023 at 10:34:45 UTC from IEEE Xplore. Restrictions apply.
II. THE TRADITIONAL CLUSTERING ALGORITHM
The clustering algorithm groups data with common
attributes and characteristics into a cluster according to
certain characteristics and attributes of the data. The data of
the same cluster is highly homogeneous. On the contrary,
the data of different clusters is highly heterogeneous to
achieve the discretization of the data. The clustering method
does not set clustering parameters and discrete targets in
advance, thus it belongs to the unsupervised learning
method with good objectivity and reliability.
K-Means algorithm is a clustering algorithm based on
centroid division, in addition, the average value of data in a
cluster is usually used as the centroid of the cluster. In
K-Means algorithm, the initial data set D is divided into k
clusters, which are C1, C2, ..., Ck. Besides, any data in the
initial data set D belongs to only one cluster, then for 1İ i
and jİ k, there is Ci  D and Ci ģCj= Ø. The distance
between the data is characterized by Euclidean distance:
d (m, ci ) m, ci (1)
In Equation (1), ci is the centroid of the cluster Ci, and m
is any data in the cluster Ci. The clustering quality function
is defined as:
k
E ¦ ¦ d (m, c )
i 1 mCi
i
2
(2)
By analyzing the quality function, it is evidently that the
ultimate goal of K-Means clustering method is to make the
data in the clusters highly similar, and the boundaries
between clusters are clear and obvious.
The calculation process of K-Means clustering method is
shown in Fig.1.
204
 ķ Determine the number of clusters as k, and select k
items of data from the initial data set D as the initial
centroids ci;
ĸ For each item of data xp in the data set D, calculate
the distance from the k centroids by Equation (1)
respectively, and divide it into the cluster with the smallest
distance;
Ĺ After all the data is divided into their respective
clusters, calculate the clustering quality E according to
Equation (2), and recalculate the new centroids ci* of each
cluster according to Equation (3),
nq
1
ci*
nq
¦x
q 1
q
(3)
Among Equation (3), xq is any item of data in cluster Ci,
and nq is the total items number of data in cluster Ci;
ĺThen calculate the clustering quality E* according to
Equation (2) and the new centroid ci*.If
E*  E d H (4)
then judge the quality function to converge, otherwise
repeat steps ĸ ~ ĺ to iterate until the convergence
conditions are met. Sometimes, ci *=ci or ci*  ci d N is
also used to determine the convergence condition, that
means if the centroid has not changed or the centroid change
has a small displacement, the iteration ends.
K-Means clustering algorithm is simple and easy to
understand. Its main calculation time consumption is
focused on calculating the distance between the data and the
centroid. However, if the traditional K-Means clustering
algorithm is directly applied to the discretization of
industrial big data, the amount of data to be processed is too
large, and a large amount of distance calculation needs to be
completed. Repeated calculations cause a lot of
computational and communication overhead. Therefore,
how to realize the parallelization of the traditional K-Means
clustering algorithm and improve the computing efficiency
has become an area of intense investigation [1, 2].
III. THE OPTIMIZED PARALLEL ALGORITHM
A. Mapreduce Structure
In view of the time complexity of calculation step with
distance from each item of data to the centroid is the largest,
while the calculation process is relatively independent and
does not interfere with each other, that reflects good locality,
we implement calculation parallelization of this calculation
process based on the Mapreduce architecture, and reduces
the calculation time consumption greatly.
As a scalable parallel programming model with
outstanding scalability, fault tolerance, and applicability, the
Mapreduce structure has a good effect on the processing of
various types of data (including structured data,
semi-structured data and unstructured data). The central
calculation idea of Mapreduce structure is “divide and rule”.
Through the parallel processing of data blocks, the purpose
of collaborative computing of massive data is achieved [3,
205
Authorized licensed use limited to: Bauman Moscow State Technical University. Downloaded on September 28,2023 at 10:34:45 UTC from IEEE Xplore. Restrictions apply.
4].
The parallel computing of jobs and tasks in the
Mapreduce architecture is achieved through the regulation
and management of master and slave nodes [5]. The master
control node Job Tracker in the Mapreduce architecture is
responsible for the coordination and regulation of
computing tasks, and the slave node Task Tracker is
responsible for specific analysis and calculation. The loose
coupling between the nodes in the Mapreduce architecture
makes it versatile and convenient to use. Through parallel
calculating in the Mapreduce structure, the calculation
process is streamlined, and the computational overhead
caused by data analysis and the communication
consumption caused by information transfer are saved.
B. The Parallelization Improvements
The parallelization of the original K-Means algorithm is
realized in the Mapreduce platform according to the
independence of the distance calculation process from the
data to the cluster centroid. The calculation task of the
distance between each data and the cluster centroid center is
realized in the Map stage, and the calculation of the new
centroids is performed in the Reduce stage. The
parallelization algorithm flow is shown in Fig.2.
ķ Formally organize the initial data set D. Each item of
data is arranged in rows and the row numbers are listed;
ĸ Determine the number of clusters as k and randomly
select the initial centroids ci;
Ĺ In the Map phase, all the data are randomly
distributed to each node for calculation of the distance d(m,
ci ) between each item of data and each initial centroid ci.
The closest centroid is selected and classified into this type
of cluster. The intermediate result of the output key-value
pair is output;
ĺ In the Shuffle process, the intermediate key-value
pairs need to be processed, and the data of same clusters are
summarized into a set;
Ļ At the beginning of the Reduce phase, merge
calculations are performed on the data of homogeneous
clusters, so that the new centroid ci* is determined by
calculating the average of each data in the homogeneous
clusters, and then the clustering quality E* is determined by
Equation (2);
ļ By determining the convergence conditions, the flow
direction is determined. If the convergence conditions are
met, the algorithm ends and the results are output. On the
contrary, if the convergence conditions are not met, iterative
calculations are started until the convergence conditions are
met.
Ignore the impact of information transmission and
resource allocation, compare and analyze the time
complexity of the K-Means algorithm and the parallelization
algorithm. Set the total number of data in the data set as p,
the calculation number of iterations required to reach the
convergence criterion is q, the calculation time required to
calculate the distance from an item of data to a centroid is t,
then the time complexity can be simply expressed as T=pqtk
in K-Means algorithm. While in the parallelization
algorithm, the data set is divided into n blocks. A Map task
is responsible for the calculation of a data block. Assuming
 that there are a total of a node, each node is responsible for
the completion of (n/a) tasks. The time complexity can be
expressed by T*=pqtk/a. It can be seen that the calculation
efficiency is greatly improved.
Fig.2 The Flow of Parallelization Algorithm
IV. CALCULATION RESULTS AND CONCLUSIONS
A. Calculation Results
Taking industrial data as the research object, the data is
discretized using the parallelization algorithm, and the
number of clusters is set to 3 and 4. The discretization
results are shown in Fig.3 and 4.
Fig.3 Schematic diagram of discretization results when the cluster number
is 3
206
Authorized licensed use limited to: Bauman Moscow State Technical University. Downloaded on September 28,2023 at 10:34:45 UTC from IEEE Xplore. Restrictions apply.
Fig. 4 Schematic diagram of discretization results when the cluster number
is 4
B. Conclusions
By discretizing the industrial data, the data is divided
into 3 and 4 clusters. As shown in the clustering results, the
data in each cluster has a high degree of similarity, and the
densely distributed data is classified as one cluster, the data
with sparse edges are classified into one category, thus the
goal of data classification is achieved basically. Through the
example verification, the parallelization algorithm proposed
in this paper can be used for the discretization of industrial
big data.
ACKNOWLEDGMENTS
This work was financially supported by Natural Science
Foundation of Hubei Province (2019CFB281).
REFERENCES
[1] ZHANG Wang, WANG Hui, Parallel K-Means Clustering Algorithm
in Web Personalized Service, Microelectronics & Computer.vol.
24,no.10,pp.65-67,2007.
[2] LU Yi-qing, LIN Jin-xian, Parallel PSO combined with K-means
clustering algorithm based on MPI, Journal of Computer Applications.
Vol.31,no.2,pp.428-431,2011.
[3] Li Cheng-hua, Zhang Xin-fang, Jin Hai, Xiang Wen, MapReduce: a
New Programming Model for Distributed Parallel Computing,
Computer Engineering&Science. Vol.33, no.3, pp.129-135, 2011.
[4] LIU Zhi-hui, ZHANG Quan-ling, Research overview of big data
technology, Journal of Zhejiang University (Engineering Science).vol.
48,no.6,pp.957-972,2014.
[5] TAO Xue-jiao, HU Xiao-feng, LIU Yang, Overview of Big Data
Research, Journal of System Simulation. No.S1,pp.142-146,2013.