See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/326053560

Data quality in big data processing: Issues, solutions and open problems

Conference Paper · August 2017

DOI: 10.1109/UIC-ATC.2017.8397554

CITATIONS READS
14 3,274

4 authors, including:

Pengcheng Zhang Jerry Gao

Hohai University San Jose State University
105 PUBLICATIONS   980 CITATIONS    302 PUBLICATIONS   4,987 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Smart Cities View project

BiG Data Quality Assurance Modeling Research View project

All content following this page was uploaded by Jerry Gao on 27 July 2020.

The user has requested enhancement of the downloaded file.

 Data Quality in Big Data Processing: Issues, Solutions and Open Problems
Pengcheng Zhang1, Fang Xiong1, Jerry Gao2,3
1
College of Computer and Information, Hohai University, Nanjing, China
2
San Jose State University, San Jose, CA, 3Taityan University of Technology, China
Email Address: {pchzhang@hhu.edu.cn; jerry.gao@sjsu.edu}
Abstract—With the rapid development of social networks
and Internet of things, Big Data age has arrived. The
increasing number of data have brought great value to the
public and enterprises, how to manage and use Big Data
better has become the focus of all walks of life. However, the
Big Data 4V characteristics has brought a lot of problems to
the processing of Big Data. The key to Big Data processing is
to solve the problem of data quality, to ensure data quality
as a prerequisite for Big Data to play the value. The
recommendation system and the prediction system are the
successful application of the Big Data technology. In this
paper, we study the recommendation system and prediction
system in Big Data environment, and try to find out the data
quality of data collection, data preprocessing, data storage
and data analysis in Big Data processing. Through the
elaboration and analysis of the problem, the corresponding
solution is put forward. At the end of the paper we have
raised some open questions.
Keywords- Big data, Big data processing, Data Quality，
Recommendation system, Prediction system.
I. INTRODUCTION
With the development of mobile applications, the
amount of information data is exploding, the concept of
Big Data is noticed by the industry and the academic
community of universal. there is a different definition
about the Big data’s concept, such as Wikipedia [2] on the
"Big Data" interpretation is: big data involved in the
amount of data to a large scale cannot be artificial, in a
reasonable time to intercept, management, processing, and
sorting into human beings can read the information. Baidu
Encyclopedia of the "big data" is defined as: big data
refers to the amount of data involved in the huge scale
cannot through the current mainstream software tools, in a
reasonable time to achieve the capture, management,
processing, and organize to help business Decision-
making for more aggressive purposes.
No matter what kind of definition are permeated with
the characteristics of big data, namely: Volume, Velocity,
Variety, Veracity. As the amount of data in each area
increases rapidly, the efficient processing of data has
become a major challenge. How to correctly understand
the big data, how to use big data for people's production
and life to provide quality services is the new challenges
that the researchers faced. Big data processing has several
stages what includes collection, pre-processing, storage
and analysis. Data quality is the most important aspect of
big data processing, and big data can be supported by a
large number of embedded data analysis and statistical
analysis tools [1]. In order to provide reliable data analysis,
it is necessary to use a variety of tools to manage big data
algorithms. The key issue is to ensure that the data quality
is high and the data source is reliable.
Big data’s research is in its infancy, experts and
scholars of various countries on each aspects of Big Data
work is actively exploring. The recommendation system is
the successful application of Big Data processing
technology. The Big Data recommendation system has
gradually become the research hotspot in the information
field. The Big Data collection user feedback, purchase
record and even social data are analyzed and excavated
between the customer and the commodity Correlation.
With the deepening of the information revolution, the
prediction of the big data age is easier, and human’s life is
being greatly changed by big data. The most common
application’s case is "forecasting the stock market",
"forecasting the flu", "predicting consumer behavior"
Predictive analysis is the core function of Big Data. Big
Data scale is expanding all the time. It’s a great challenge
for the data storage and analysis. This paper takes the
recommendation system and the prediction system as an
example to elaborate and analyze the data quality of data
acquisition, data preprocessing, data storage and data
analysis in Big Data research, and put forward the
corresponding solution.
The rest of this article is arranged as follows. Section II
review the relevant work. The third section introduces the
related concepts involved in this paper, including the
characteristics of Big Data, Big Data processing process,
and Big Data application system. The fourth section
analyzes and describes the problems encountered in the
process of Big Data processing from the perspective of
data quality. Section 5 presents a solution to the problems
encountered. Section 6 discusses the relevance of the
problem to Big Data characteristics and data quality, and
analyzes some of the current open issues. Finally
summarize the full text.
II. RELATED SURVEY
As the big data presents new features, its data quality is
also facing many challenges. How to deal with a large
number of real-time data has become a key issue in
research and application. Data quality of Big Data is a hot
topic in academic research. As shown in Table I, data
quality standards mainly involve four dimensions:
Availability, Usability, Reliability, and Relevance.
Availability is defined as the degree of user access to data
and related information. It is divided into accessibility, and
timeliness. The concept of usability means whether the
data is useful and satisfying the user's needs or not.
Reliability is whether we can trust data, including accuracy,
consistency, and integrity. Relevance is used to describe
the degree of correlation between data content and user
expectations or requirements. [10].
TABLE I. BIG DATA QUALITY QUALITY STANDARD
Dimensions Elements
Accessibility
Availability
Timeliness
Usability Credibility
Accuracy
Consistency
Reliability
Integrity
Completeness
Relevance Fitness
Many research scholars have commented on the
problems and challenges of big data. Jacek Ma'slankowski
has published a paper [11]. he focused on the data quality
problems in the data collection process. the article
analyzes the data acquisition process problem through
comparing the traditional statistical survey technology and
Big Data technology to collect data. The case of research
show that there are many hurdles in data quality when
using Big Data technology. These obstacles are identified
and described in this article. Toly Chen et al. [13]
reviewed data differences due to improper data
preprocessing (including huge data, incomplete data
normalization, subjective data linearization or non-
linearization, bias weighting, and loss of information
Discretization, etc.) and corrected these differences. Jie Li
illustrates the status of Big Data storage and management,
analyzes the problem, and finally presents a solution [7].
Analyzing Big Data is a challenging task because it
involves large distributed file systems that should be fault-
tolerant, flexible, and extensible [15].
After investigating the literature, some scholars have
studied the challenges and problems encountered in Big
Data, and put forward some opinions on the problems
existing in data collection, data preprocessing, data storage
and data analysis, and some Can solve the problem of the
solution. However, very few people have a clear analysis
and description of each stage of the problem. Based on the
previous scholars' research, this paper takes the Big Data
application system as an example, including the
recommendation system and the prediction system to
illustrate the four stages of data acquisition, data
preprocessing, data storage and data analysis in Big Data
processing data quality issues. Followed by a
corresponding solution for each question.
III. RELATED CONCEPT
A. Characteristics of Big Data
In 2001, Gartner's Doug Laney put forward a view that
Big Data has three dimensions: Volume, Variety and
Velocity. As a result, IDC defines it: Big Data technology
describes a new generation of technologies and
architectures designed to economically extract value from
a wide variety of data by achieving high-speed capture,
discovery and analysis. Gartner updates it as "Big Data is
large, high speed and high quality information assets that
require new forms of processing to achieve enhanced
decision making, insight discovery and process
optimization." [3][5]. IBM defines Big Data with four
features by adding Veracity to the three features.
Figure 1. 4Vs of Big Data
Volume: With the rapid development of information
technology, data began to explosive growth, storage units
from the past GB to TB, until the PB, EB. Taobao nearly
400 million members of the commodity trading data
generated about 20TB every day; Facebook about 1 billion
users log data generated more than 300TB every day.
There is an urgent need for intelligent algorithms,
powerful data processing platforms and new data
processing technologies to analyze, analyze, predict and
process such large-scale data in real time [4].
Variety: Big data is from a variety of data sources.
Data types and formats are increasingly rich, including
structured, semi-structured and unstructured and other
forms of data. Incompatible data formats, misaligned data
structure analysis of semantic representation and
inconsistent data may result in the spread of major
challenges [5].
Velocity: Unlike traditional archives, broadcasts,
newspapers and other traditional data carriers, Big Data
exchange and dissemination through the Internet, cloud
computing, etc. than the traditional media information
exchange and dissemination of fast. Big Data and massive
data of the important difference, in addition to Big Data
larger than the data, the Big Data on the processing of data
response speed more stringent requirements.
Veracity: With the interest of new data sources such
as social data, corporate content, transaction and
application data, the limitations of traditional data sources
are broken, and the accuracy of Big Data tends to change,
and what data is found to be truly valid in Big Data Is very
important. Enterprises increasingly need effective restraint
means to ensure its authenticity and security.
B. Big Data processing
Through big data analysis, a lot of valuable
information can be extracted from the massive data, while
excellent technical support is essential. With the
development of the computer industry, different big data
analysis technology has also been the continuous
development. Data integration, conversion and other
technologies have a lot of relying on the tools. As shown
in the figure2, the processing of Big data is divided into
the following four processes:
Figure 2. The processing of the Big Data
Data collection: Big Data collection is the
foundation of the whole process. With the development of
Internet technology and application and the popularity of
various terminal devices, the range of data producers is
increasing, the output of data is more and more, between
the data correlation and data become more and more
complex, which is Big Data in the "big" embodiment, so
we need to improve the data acquisition speed and
accuracy requirements. Big Data acquisition mainly
includes system log collection method, network data
acquisition method and other data acquisition methods.
Data preprocessing: Big Data preprocessing is
mainly used to complete the already received data to
identify, extract and clean the operation. Because the data
is variety gathered by the data acquisition steps is not
conducive to the subsequent data analysis. And some data
is invalid data which need to be removed, otherwise it
will affect the accuracy and reliability of data analysis.
Therefore, you need to unify the data format and remove
invalid data.
Data storage: Big Data storage and processing is
not only large-scale, more requires its transmission and
processing of the response speed. Due to the different data
source, with the characteristics of data diversity,
traditional database storage technology cannot adapt to
Big Data storage. Big Data storage or processing system
must be a variety must have better compatibility to
loading to kinds of data and hardware and software
platform.
Data analysis: After the completion of the data
collection and processing, system need analyze the data. It
is well known that is a fact what Big Data is not just
having big data, and the most important reality is the
analysis of Big Data, only through analysis to get a lot of
intelligent, in-depth, valuable information. As more and
more applications involve Big Data, and these Big Data
attributes are presented with the growing complexity of
Big Data, so Big Data analysis methods in the field of Big
Data is particularly important.
C. Big Data Application System
Recommendation system: Recommendation system
was proposed by Robert Armstrong in AAAI in 1995,
while Marko Balabanovic et al. introduced the
personalized recommendation system LIRA1.
Recommendation system based on the user's interest
characteristics and purchase behavior, to the user to
recommend the user interested in information and goods.
So far, the recommendation system has been widely used
in many fields. With the boom development of e-
commerce, recommendation system plays an irreplaceable
role in the Internet. It is reported that Amazon's 35% of
the turnover of the site from their own recommendation
system; Taobao, Jingdong and other e-commerce platform
recommendation system is relatively successful. With the
rise of social networking, Twitter, Sina microblogging’s
friends recommend the system, Google, Netease's news
recommendation system is also favored by the user. On
the other hand, the location of the information and other
information can be very accurately obtained with the
process of using the mobile Internet users. The user can
search according to their current location Restaurants,
hotels, cinemas, tourist attractions and other information
services in Meetup and where the location-based service
site. The recommendation system in the big data
environment is an extension of the traditional
recommendation system. Because the big data
environment’s is more complex than the traditional
environment to provide the environment and data
characteristics, only by extracting and predicting the
users’ preference contained in big data’s environment can
produce validly more accurate recommendation [24].
prediction system: Big Data prediction are based on
existing data analysis, based on data’s development and
dissemination to assess the upcoming trends and
conditions. Big Data is the application of core data
prediction, and traditional predictive tools cannot handle
the size, speed, and complexity inherent in Big Data [25].
The weather forecast is one of the major beneficiaries of
Big Data, and big data will be of great benefit to weather
forecasts [26,27]. As the Internet bring the convenience of
predicting the popularity of Big Data. During the World
Cup, Baidu predicted the 64 games entirely, the accuracy
rate of 67%, into the knockout after the accuracy rate of
94%. Google successfully predicts the number of flu
patients based on the user's search log summary. Some
big data companies and big data service companies (such
as EDITD) use mobile social media data to predict future
fashion.
IV. ISSUES OF BIG DATA PROCESSING
A. Issues of Data Collection
At present, the recommendation system, prediction
system analysis data is mainly from the Internet. Facebook,
Baidu and other companies need to deal with more than
10PB of data per month, Google deal with hundreds of PB
of data, these data are exponential growth to create [17].
Many social networks, such as Twitter, provide an
accessible API to encourage developers to take advantage
of the resources on their social networks for data mining
and to extract valuable information. In the process of data
collection, the increasingly complex social network and its
massive content data brings great challenges to the
acquisition of social network data. The following are some
of the issues that must be considered in the data collection
process:
Issue A-1. Less data collected and low recall rates
In the process of data collection, the whole network
data collection is difficult to achieve because the network
connectivity is difficult to guarantee and other reasons. and
the dynamic nature of the social network determines that
we usually only get every moment or more time network
snapshot. A large number of social network event
monitoring system in the realization of the collection of
thematic messages seriously dependent on the social
network built-in search engine, the quality and quantity of
data can’t be guaranteed. In the process of collecting data,
there is a problem that data loss is easy, and there is a
problem that the data collected is low and the recall rate is
low, and finally the data analysis is difficult.
Issue A-2. Data sparseness
The sparseness of data is one of the most common
problems in the recommendation system. The sparseness
of data mainly refers to the sparseness of the user-object
scoring matrix that is the interaction between the user and
the item is too little. As of the end of 2014, Taobao has
registered nearly 500 million members, active users over
120 million, the number of online products reached 1
billion. But the average user to browse the number of
goods is relatively small, the average number of users per
visit to the baby does not exceed 800. In fact, the average
number of items per user may not exceed 20. In this scale,
any two users browse the intersection of goods are
relatively small. The ratio of the items selected by the user
to the entire project space is very low, reducing the
predictive performance of the various recommended
strategies greatly. If the data set is very sparse, only
contains a very small amount of user behavior data, the
recommended algorithm will be reduced the accuracy
greatly, it is easy to lead to the proposed algorithm over-
fitting, and affecting the performance of the algorithm.
B. Issues of Data Preprocessing
In [18] Witten IH et al. mentioned that the data
preprocessing describes the processing of raw data to
prepare for any other type of processing. Due to the
diversity of data sources, the collected data sets have
unpredictable quality levels in terms of redundancy, noise,
consistency, etc. [13]. In the data preprocessing to collect
the relevant data preprocessing calculation, the data
processing results act as a recommendation system
mathematical form of input. The study of big data
preprocessing is very complex and it contains a wealth of
strategies and techniques. Big data sources’ authenticity,
accuracy, completeness, timeliness and other research is a
very critical first step in the big data processing stage. The
problems encountered during the preprocessing process
are as follows:
Issue B-1. Noise data
Whether it is a recommendation system or a prediction
system, there are often a variety of noise, such as data loss
or data anomalies in collecting data. In the original data
source will be mixed with a variety of noise data, on the
one hand, there are some abnormal data in the process of
data collection and reporting. on the other hand， it also
includes the system on-line operation generated cheating
data. Hardware errors, programming errors, or garbledness
in the program may also cause noise data. The noise data
will lead to abnormalities in the recommended results, and
processing noise data is a key issue in the data
preprocessing phase.
Issue B-2. Incomplete data
If you want to analyze the data of a year's earthquake
and make some predictions ， we can find that period
Twitter, microblogging and other social media data will
surge, these sites accumulate a lot of data in a short period
of time, but these data are difficult to reflect all the
problems. Because there are a large number of people in
large city, smart phones are more popular and covered, and
most of the earthquakes are concentrated in large cities,
and those are poorly collected in relatively remote areas
and the most severely affected areas almost have no
statistics related to the data. This can cause some data to
be missed. Data integrity is generated due to node failure
and network transmission instability.
C. Issues of Data Storage
According to IDC, global data generation in 2011
was only 1.8ZB (or 1.8 trillion GB), and global data is
expected to grow 50 times to 2020. The data in the era of
big data are diversified, not only structured data, but also
semi-structured and unstructured data, and with the
development of social networks and mobile networks,
unstructured data is increasing. Big data storage’ focus is
to collect the different structure of the data through the
pretreatment, improve the quality of data, storing it, and
the establishment of the corresponding database for
management. Big data storage problems are mainly
focused on the following aspects:
Issue C-1. Limitations of Storage Technology
For Big Data applications (such as recommendation
and prediction systems), the biggest problem is the need
for large amounts of data to produce results, but this has
caused problems with data storage. Large-scale storage
reaches the PB level usually, and sometimes even the EB
level, by 2020, our total data will grow 44 times over 2009.
The Big Data management, query and analysis of the
storage technology put forward higher requirements. Due
to the large amount of data, complex structure, storage
standards will also be a revolutionary change. Big
Datatypes are complex, often structured, semi-structured
and non-structural coexistence, Big Data storage needs
achieve a variety of types of data unified storage, the
current storage technology there are still some limitations
[11].
Issue C-2. Data timeliness
Timeliness refers to content that is strongly related to
time, such as news, current affairs, and so on. The
recommendation system requires real-time analysis of the
user's browsing history and accurate recommendation of
the corresponding content. This requires that the storage
system must be able to maintain a high response speed,
because the response delay is the result of the system will
push the "expired" content to the user, resulting in invalid
recommendation. Weather forecast size from day to day,
there are harsh time requirements. If we know the
predicting the next day, then the forecast is worthless.
Other areas of Big Data to predict the application of the
characteristics of "timeliness" also have higher
requirements, such as the stock market, real-time pricing.
D. Issues of Data Analysis
The goal of data analysis is to use effective methods to
accurately identify and predict the relationship between
data values and data. Over the past few decades,
researchers have responded to the ever-increasing amount
of data by accelerating the analysis algorithm. What are
the problems with Big Data analysis for large-scale data
applications such as social networks?
Issue D-1. Accuracy
Accuracy is the ultimate goal of Big Data projections.
Dynamic variables at different time points, and any
variable triggers an entire system change or even a
butterfly effect(未改). If a variable has a decisive effect on
the results and is difficult to capture, it is difficult to
predict, such as human factors. Big Data forecast
applications are mostly very unstable areas but there are
fixed laws, such as weather, stock market, disease. This
requires the system to accurately capture the data for each
variable and adjust the forecast in real time. And some of
the data of the law is elusive, resulting in the accuracy of
the forecast there is a big problem. Such as earthquake
prediction.
Issue D-2. Scalability
Recommendation systems need to recommend
thousands of products or even more than one million. The
massive user history log poses a huge challenge to the
traditional recommendation algorithm. With the rapid
growth of the number of users and the rapid growth of the
number of goods, the massive user log needs to wait for
the recommendation system to calculate, analyze and
excavate. Although the proposed algorithm may be
simpler, the computational complexity is high and usually
requires very high-dimensional user data The Through the
statistical or likelihood calculation of high-dimensional
matching to form a virtual community, which also led to
even the most basic recommendation algorithm is often
very time-consuming, and in practice, scalability is poor,
with the increase in the number of users and goods, The
complexity of the algorithm is polynomial growth, the
algorithm performance is getting worse.
V. SOLUTIONS OF BIG DATA PROCESSING
In the previous chapter, we classify the types of data
quality problems that exist in the process and analyze the
problems. The main content of this chapter is to propose a
solution.
A. Solutions of Data Collection
In big data, the quality of the data source symbolizes
the quality of the data. In order to deal with the barriers to
the data collection phase of big data, the following
conclusions need to be noticed.
Solution A-1. Increase acquisition coverage
We can combine built-in search and meta-search-based
approach to increase the coverage of the acquisition and
get more information. Meanwhile this method is not only
for a social network, they are applicable for most social
networks. Using a based-on PageRank friend
recommendation algorithm is feasible. It integrates the
user’s interactive information and social information in the
recommended process. The FP-Growth association mining
algorithm under Mahout improves the accuracy, recall and
forecast coverage of the recommendation system by
correlating the actual business data of the social network.
Solution A-2. Using dimension reduction techniques
and processing algorithms
Using dimensionality reduction technology can
effectively alleviate the sparseness of data. Among the
many dimensionality reduction methods, the most relevant
methods are principal component analysis and singular
value decomposition. It can also be used by some diffusion
algorithms, from the original first-order association (how
many similar scores or common purchases of two users) to
second-order or even higher-order associations (assuming
that the relevance or similarity itself is spreadable), You
can also add some default scoring, thus improving the
resolution of similarity. In general, it will be sparser if the
Big Datais bigger. It is now thought that algorithms that
can deal with sparse data (such as diffusion, iterative
optimization, transfer similarity, etc.) are more valuable.
B. Solutions of Data Preprocessing
Data preprocessing has a critical impact on data
analysis, including the recommended results and
prediction results. In order to improve the quality of data,
the emphasis on data preprocessing is necessary. This
article provides two solutions.
Solution B-1. Remove noise operation
Data collection and reporting of abnormal data, they
need to combine the database table structure and the actual
scene to do filtering, such as null check, numerical
exception, type anomalies, data to heavy. In addition, for
the "artificial" noise data, such as brush click, brush list
and other acts, these key data will affect the follow-up
algorithm effect seriously. It needs to have some anti-
cheating strategy to clear or down, such as session analysis,
combined with the cookie, ip, behavior and the number of
times and other rules to filter. In the data preprocessing
phase to remove the noise operation, the effect is often
better than the complex optimization algorithm.
Solution B-2. Clustering Algorithm
Hathaway and Bezdek developed a method based on
incomplete dissimilarity clustering incomplete relation
data [22]. Li et al. A hybrid method based on the interval
attribute of the nearest neighbor information is proposed,
and the hybrid method of incomplete data clustering is
carried out by genetic algorithm and fuzzy c mean [23]. [8]
The data set is divided into a complete data set and a non-
complete data set, and then the affinity data clustering
algorithm is used to cluster the complete data set.
According to the similarity measure, the incomplete data is
divided into corresponding the cluster. In order to improve
the efficiency of the algorithm, a distributed clustering
algorithm based on cloud computing technology is
designed.
C. Solutions of Data Storage
In order to solve the problem of data storage, we can
use the following two ways.
Solution C-1. Using cloud storage technology and
tiered storage
Cloud storage [19] by means of the cluster
application, grid technology or distributed file system and
other functions, the network a large number of storage
devices work together through the application software
together. Cloud storage is focused on providing Internet-
based online storage services, users do not need to
consider storage capacity, storage device type, data storage
location, and data availability. Hierarchical storage for the
application of capacity, use and performance requirements,
selecting different storage media to reduce the total cost.
Hierarchical storage in cloud storage classifies and
migrates to different storage tiers. Storage hierarchical
management strategies simplify storage resource
management greatly.
Solution C-2. Spark platform
Spark [20] introduces the RDD data model and the
memory-based computing model so that it can well adapt
to the data mining of large data, and is superior to Hadoop
in iterative computing. In order to solve the
recommendation and predict the timeliness, a real-time
context algorithm based on Spark is proposed. The
algorithm combines the filtering of the situation on the
basis of cooperative filtering, and uses Kafka as the real-
time message transceiver to deal with the real-time flow
data with Spark Streaming, which enhances the accuracy
and timeliness of the algorithm.
D. Solutions of Data Analysis
After the data analysis can reflect the important value
of all large data. Traditional data analysis can’t handle
massive amounts of data. There is a need for further
improvements in the accuracy and scalability of the data.
Solution D-1. Useful algorithms
In order to improve the accuracy of Big Data mining
and prediction capabilities, to solve the traditional data
mining technology can’t adapt to Big Data processing
environment. Many companies have begun to use Na07ve
Bayesian classifiers, association rule mining, decision
trees and other well-known algorithms to analyze their
data and predict their potential customers and business
decisions. Accurate classification results may help the
company to obtain reliable predictions intelligently
through the viable business. [9] proposed an association
rule mining to improve the Na07ve Bayesian classifier,
mainly by combining the relevant attributes to adapt to the
naive Bayesian classifier in the independent hypothesis to
reduce the attributes. In [16], the large-scale data mining
architecture and prediction model based on BP neural
network are established by using cloud service and Big
Data processing technology. Finally, the prediction results
can be obtained.
Solution D-2. Clustering Algorithm
In order to solve the scalability problem, according
to the literature, the use of clustering algorithm can
solve the scalability of the recommendation system
effectively and improve the recommending accuracy. In
[21], a K-means clustering algorithm with improving
center-of-gravity selection method and distance
measurement is proposed. [14] studied the application
of model-based collaborative filtering technology, in
particular, it proposed a cluster CF framework and two
clustering algorithms CF: Project-based fuzzy
clustering collaborative filtering (IFCCF) and trust-
aware clustering collaborative filtering (TRACCF). In
the literature [6], the PCA-GAKM algorithm is proposed.
First, the data is preprocessed by principal component
analysis. Then, the K-means clustering algorithm is
improved by the fusion genetic algorithm. Finally, the
TOP-N recommendation mechanism is used to
generate the recommended list.
 VI. DISCUSSION
Big data provides more opportunities for the Internet,
but also brings more problems or even traps. This paper
mainly from the data quality point of view, to recommend
the system and the forecasting system as an example, put
forward the big data processing process existing problems
TABLE II.
The existing issues
Less data collected and low
recall rates
Collection
The interaction between the user and the
Data sparseness
Abnormal data and cheating data at the time
Noise data
Preprocessing
Incomplete data
Limitations of Storage The total amount of data is large and the type
Technology
Storage
Data timeliness
Accuracy
Analysis
High data dimension, high computational
Scalability
Although some results have been made for the study of
data quality, such as the validity, reliability and availability
of data, the research on Big Datais still in its infancy. Big
Dataneeds to solve many problems, such as: the data of the
multi-source, the quality of differentiation, how to obtain
high-quality large data, how to integrate the existing multi-
source data, how to detect and repair data and other issues.
Some of the problems we have mentioned can be easily
overcome, and these practical problems are common in
traditional data processing. However, some issues are
closely related to the characteristics of large data, and so
far, are still unresolved issues. Big issues caused by
several public issues will serve as the main content of this
section to explain the dilemma that may be faced by large
data. Here are some open problems [12]:
Open problem1. Inconsistent data
Big data clouds will soon be applied to various areas,
the resulting data is absolutely inconsistent. There are four
inconsistencies in time, text, space, and functional
dependencies that must address data inconsistencies at the
data level, information level, and knowledge level in order
to better analyze the data. Big Dataanalysis is a new era of
data analysis, there are still data inconsistencies. But
traditional inconsistency solutions do not necessarily apply
to large data. Data is more prone to inconsistencies
because data is captured or generated by different sensors
and systems. How the effect of data inconsistency is an
open question of Big Dataanalysis.
Open problem2. The bottleneck of data mining
algorithm
in four aspects. In the big data processing work, there are
Big Dataof large-scale data, data generation fast, complex
data types and so on. These features bring a lot of data
quality problems to Big Dataprocessing. Table II shows
the relationship between data processing and data quality
and 4V characteristics.
ISSUES, DATA QUALITY,4V CHARACTERISTIC
Primary Cause Data Quality 4V Characteristic
Network connectivity and dynamics Availability Volume
Relevance Variety
item is less
Usability Variety
of acquisition
Data distribution is not balanced, the Veracity
Reliability
network transmission is unstable Velocity
Usability Volume
is complex
Response time is long Availability Variety
The rule of the data is elusive Reliability Velocity
Usability Volume
complexity
Most data mining algorithms in big data analysis will
be designed for parallel computing. However, once the
data mining algorithm is designed or modified for parallel
computing, the exchange of information between different
data mining processes can cause bottlenecks. One of them
is a synchronization problem, because even if the same
algorithm is used to handle the same amount of data, the
programs of different programs will make their work
foolish at different times. The bottleneck of data mining
algorithms will be an open issue in big data analysis,
which suggests that we need to take this into account when
developing and designing new data mining algorithms for
big data analysis.
Open problem3. Security issues
Since environmental data and human behavior will be
collected by a large number of data analysis, how to
protect them will also be an open problem because there is
no safe way to handle the collected data, and big data
analysis cannot be a reliable system. Although we have to
tighten the security of big data analysis before we can
collect more data from all over the world, the reality is that
so far, much research has focused on the security issues of
big data analysis. According to our observation, the
security of big data analysis can be divided into four
aspects: input, data analysis, output and communication
with other systems. For input, it can be considered as a
collection of data related to sensors, handheld devices and
even networking devices. One of the important security
issues in the big data analysis input section is to ensure
that the sensor is not compromised by the attack. For
analysis and input, it can be considered as a security issue
 for such a system. In order to communicate with other
systems, the security problem lies in the bigdata analysis
of communication with other external systems. Because of
these potential problems, security has become one of the
open issues of big data analysis.
CONCLUSION AND FUTURE WORK
We have entered the era of large data, through data
collection, processing, storage, so that data is better
analyzed. As a result of the substantial increase in the
volume of data in various fields, it has become a major
challenge in efficiently processing data. We will need to
address these issues, thereby improving the quality of the
data. This paper takes the recommendation system and the
forecasting system as an example to analyze and describe
the possible problems in the process of Big Dataprocessing.
We reviewed the concept of large data, Big
Dataprocessing, forecasting systems and recommendation
systems. Data quality, data preprocessing, data storage,
data analysis and other data quality problems in Big
Dataresearch are expounded and analyzed. For each stage,
we ask possible questions and propose solutions, and
finally analyze some open questions.
Our Big Dataanalysis is unavoidable. Existing Big
Datatechnologies and tools still have some limitations that
cannot completely solve Big Dataproblems. In the follow-
up study, we need to study the big data problem, try to
solve the problem of data quality through better solution.
REFERENCES
[1] Louridas, P., Ebert, C.: Embedded analytics and statistics for big
data. IEEE Software 30, 33–39 (2013)
[2] Wikipedia contributors. "Big data." Wikipedia, The Free
Encyclopedia. Wikipedia, The Free Encyclopedia, 31 Mar. 2017.
Web. 31 Mar. 2017.
[3] G. J. and . E. Reinsel, "“Extracting Value from Chaos”, IDC’s
Digital Universe Study, sponsored by EMC,"2011.
[4] A. A. TOLE, "Big Data Challenges," Database Systems Journal,
vol. vol. IV, no. no. 3, pp. 31-40, 2013
[5] S. Kaisler, F. Armour and J. A. Espinosa, "Big Data: Issues and
Challenges Moving Forward," Hawaii International Conference on
System Sciences, no. 46th, 2013
[6] WANG Z,YU X,FENG N,et al.An improved collaborative movie
recommendation system using computational intelligence [J ].
Journal of Visual Languages & Computing,20 1 4,2 5(6):6 6 7-6 7
5.
[7] Li J, Xu Z, Jiang Y, et al. The overview of big data storage and
management[C]// IEEE, International Conference on Cognitive
Informatics & Cognitive Computing. IEEE, 2014:510-513.
[8] Leng Y L. Incomplete Big Data Distributed Clustering[J]. Applied
Mechanics & Materials, 2014, 687-691:1496-1499.
View publication stats
[9] Yang T, Qian K, Dan C T L, et al. Improve the Prediction
Accuracy of Naïve Bayes Classifier with Association Rule
Mining[C]// IEEE, International Conference on Big Data Security
on Cloud. IEEE, 2016:129-133.
[10] Cai L, Zhu Y. The Challenges of Data Quality and Data Quality
Assessment in the Big Data Era [J]. Data Science Journal, 2015, 14
(1): 21-3.
[11] Maślankowski J. Data Quality Issues Concerning Statistical Data
Gathering Supported by Big Data Technology[J]. Communications
in Computer & Information Science, 2014, 424(1):92-101.
[12] Tsai C W, Lai C F, Chao H C, et al. Big data analytics: a survey[J].
Journal of Big Data, 2015, 2(1):21.
[13] Chen T, Honda K. Solving data preprocessing problems in existing
location-aware systems[J]. Journal of Ambient Intelligence &
Humanized Computing, 2015:1-7.
[14] BIRTOLO C, RONCA D. Advances in clustering collaborative
filtering by means of fuzzy C-means and trust[J].Expert Systems
with Applications,20 1 3, 40(1 7):6 9 9 7-700 9.
[15] Duggal P S, Paul S. Big Data Analysis: Challenges and
Solutions[C]// International Conference on Cloud, Big Data and
Trust 2013, Nov 13-15, RGPV. 2013.
[16] Yang F, Xiao-Yan A I, Zhang Y H, et al. New mining architecture
and prediction model for big data[J]. Electronic Design
Engineering, 2016.
[17] Martin Hilbert, Priscila López, The world’s technological capacity
to store, communicate, and compute information, Science 332
(6025) (2011) 60–65.
[18] Witten IH, Frank E, Hall MA (2011) Data mining: practical
machine learning tools and techniques. Morgan Kaufmann,
Burlington.
[19] Zeng W, Zhao Y, Ou K, et al. Research on cloud storage
architecture and key technologies[C]// International Conference on
Interaction Sciences: Information Technology, Culture and Human
2009, Seoul, Korea, 24-26 November.DBLP,2009:1044-1048.
[20] Xie L, Zhou W, Li Y. Application of Improved Recommendation
System Based on Spark Platform in Big Data Analysis[J].
Cybernetics & Information Technologies, 2017, 16.
[21] GHAZANFAR M A, PRüGEL-BENNETT A. Leveraging
clustering approaches to solve the graysheep users problem in
recommender systems [J]. Expert Systems with Applications,20 1
4,4 1 (7 ): 3 2 6 1-3 2 7 5.
[22] J.Hathaway, C.Bezdek, "Clustering incomplete relational data
using the non-Euclidean relational fuzzy c-means algorithm,
"Pattern Recognition Letters, vol.23, no. 1, pp.151–160, 2002.
[23] D.Li, H.Gu, L.Zhang, "a hybrid genetic algorithm-fuzzy c-means
approach for incomplete data clustering based on nearest-neighbor
intervals," Soft Computing, vol.17, no. 10,pp.1787-1796, 2013.
[24] Li C, Lan M, Zou B, et al. Big Data and Recommendation
System[J]. Big Data Research, 2015, 14(1):39-43..
[25] Madden S (2012) From databases to big data. IEEE Internet
Comput 16(3):4–6
[26] Hamm S (2013) How big data can boost weather forecasting.
http://readwrite.com/2013/02/28/how-big-data-can-boost-weather-
forecasting#awesm=ou64ZEaKe2HtUu. Accessed 20 Nov 2014
[27] Han J, Kamber M, Pie J (2012) Data mining: concepts and
techniques. Elsevier, Inc., San Francisco