Big Data Challenges

Nasser T* and Tariq RS

Article · September 2015

The subject of this study is the challenges of Big Data and to realize those challenges two
important questions need to be answered what is Big Data? And what are the characteristics
of Big Data? The term “Big Data” is a little bit an inaccurate designation because it means
that the preexisting data is small while it is really not and also it indicates that the data size is
the one challenge we have [1]. Simply Big Data refers to the data and information which
can’t be handled or processed through the current traditional software systems. Big Data is
large sets of structured and unstructured data which needs be processed by advanced
analytics and visualization techniques to uncover hidden patterns and find unknown
correlations to improve the decision making process.
Background
Challenges of big
data
Data integration and aggregation: The stream of Big Data is heterogeneous, so it is not
enough to capture it and save in our repository. For example if we take the data of several
scientific experiments, it would be useless to save them as bunch of data sets.It is not likely
that someone will find this data or include it in any analysis. However if the data has
adequate metadata, it might be used but the challenge still arise from the differences on the
experimental
details and the hosting data record structure. Data analysis is a sophisticated process and
more than simply finding, identifying, understanding and citing data. Perform data analysis in
large scale requires automating all these steps. This needs to express different data structures
and semantics in form that computer can understand and then resolve automatically. A lot of
work has been conducted in the field of data integration, however still more additional efforts
required achieving automatic error-free different solution.
Big Data: Issues, Challenges and Techniques in Business Intelligence
Conference Paper · December 2015
Background topic

Data is growing exponentially as it is being generated and recorded from everyone and
everywhere for example online social networks, sensor devices, health records, human
genome sequencing, phone logs, government records, professionals such as scientists,
journalists, writers etc [1]. Formation of such huge amount of data from multiple sources with
high volume and velocity by variety of digital devices gives birth to the term Big Data. As
the big data grows with high velocity (speed), it becomes very complex to handle, manage
and analyze by using existing traditional systems. Data stored within the data warehouses is
different from the big data. The former one is cleaned, managed, known and trusted and the
later one includes all the warehouse data as well as the data which these warehouses are not
capable to store [2]. The big data problem means that a single machine can no longer process
or even hold all of the data that we want to analyze. The only solution we have is to distribute
the data over large clusters. An example of a large cluster is one of Google's data centers that
contain tens of thousands of machines.

Challenge of big data

3.1 Lack of Big Data Professionals
Most recently devised big data processing tools and algorithms include MapReduce, Hadoop,
Dryad, Apache Spark, Apache mahout, Tableau [9,11] etc. But besides the development these
high processing, complex technologies for big data processing, organizations need highly
skilled professionals to handle and make use of these tools according to the needs of an
organization. No doubt there are experts around the big data as well, but looking at current
scenario, a special kind of training should be given to these naïve experts so that they become
proficient to deal with the big data from different dimensions including data modeling, data
architecture, data integration etc [2]. According to a report by McKinesey & Company [10],
the US might realize the requirement of 140,000 to 190,000 skilled persons for data analysis
as well as more than one million managers and analysts with advanced analytical knowledge
and skills to make correct and accurate decisions.
Conclusion and
recommendation
5. Conclusions and Future directions
As we live in the era of big data, here comes the need of modern, high performance and
capable equipments along with scalable techniques and algorithms to deal with the issues and
challenges which must come across while playing with the large data-sets. Big data analytics
is one of the reasons for the universal success of any business organization. Organizations
lagging behind in big data analytics are likely to be visually and physically handicapped as
they would suffer with monitory losses in terms of their future customers and better future
investments. The birth of big data revealed the shortcomings of existing data mining
technologies which in turn raised new challenges. In this paper, we have presented a brief
overview of big data along with its key properties, also identified some challenges of big
data. A very brief introduction and a comparison for most popular big data processing
frameworks; Hadoop MapReduce and Apache Spark is presented which helps young
researchers and data scientists to analyze the big data and uncover hidden, unknown patterns.

Big Data Challenges

By Cynthia Harvey, Posted June 5, 2017

Challenge of big data

7. Organizational resistance
It is not only the technological aspects of big data that can be challenging — people can be an
issue too.
In the NewVantage Partners survey, 85.5 percent of those surveyed said that their firms were
committed to creating a data-driven culture, but only 37.1 percent said they had been
successful with those efforts. When asked about the impediments to that culture shift,
respondents pointed to three big obstacles within their organizations:
 Insufficient organizational alignment (4.6 percent)
 Lack of middle management adoption and understanding (41.0 percent)
 Business resistance or lack of understanding (41.0 percent)