Professional Documents
Culture Documents
The Ultimate
Guide to Big Data
for Businesses
E-guide
In this e-guide
In this e-guide:
The ultimate guide to big data for
businesses p. 2 Big data is the fuel for today's analytics applications. This in-depth big
.
Why is big data important for businesses?
p. 3
data guide explains how businesses can benefit from it and what they
What are the different types of big data?
need to do to use it effectively.
. p. 4
Further reading p. 18
Page 1 of 18
E-guide
In this e-guide
What are the business benefits of big data? Big data environments can be used to process, manage and analyze many different
. p. 7
types of data. The data riches now available to organizations include customer
What are the common big data challenges?
. p. 8
databases and emails, internet clickstream records, log files, images, social network
posts, sensor data, medical information and much more.
Key elements of big data enviornments
. . p. 10
Companies increasingly are trying to take advantage of all that data to help drive
Best practices for big data management
and analytics p. 13 better business strategies and decisions. In a survey of IT and business executives from
Big data technologies and tools p. 14 85 large companies conducted by consultancy NewVantage Partners in late 2020,
What are the future trends in big data?
91.9% said they were accelerating the pace of their investments in big data and related
. p. 17 AI initiatives, while 96% reported successful outcomes from such projects.
Further reading p. 18
However, even many of those blue-chip companies are struggling to maximize the
business potential of their big data environments. Only 39.3% of the survey
respondents said their organizations were managing data as a business asset, and just
Page 2 of 18
E-guide
24% said they had created a data-driven organization, according to a report on the
In this e-guide annual survey that was published in January 2021.
The ultimate guide to big data for
businesses p. 2
To help companies both large and small get more value from the data flowing into
their systems, this comprehensive guide to big data for businesses explains what it is,
Why is big data important for businesses?
. p. 3 its business benefits, the challenges it poses and best practices for using it effectively.
What are the different types of big data? You'll also find examples of big data use cases and an overview of big data technology.
. p. 4
Throughout the guide, there are hyperlinks to related articles that cover those topics
The many V’s of big data p. 5 more deeply and offer expert advice on managing big data programs.
.
Page 3 of 18
E-guide
operational issues, detect fraudulent transactions and manage supply chains, among
In this e-guide other uses.
The ultimate guide to big data for
businesses p. 2
If done well, the end results include more effective marketing and advertising
campaigns, improved business processes, increased revenue, reduced costs and
Why is big data important for businesses?
. p. 3 stronger strategic planning -- all of which can lead to better financial results and
What are the different types of big data? competitive advantages over business rivals. In addition, big data contributes to
. p. 4
breakthroughs in medical diagnoses and treatments, scientific research and smart city
The many V’s of big data p. 5 initiatives, law enforcement and other government programs.
.
Page 4 of 18
E-guide
In this e-guide
The many V's of big data
The ultimate guide to big data for
businesses p. 2 Big data commonly is characterized by a set of V's, using words that begin with v to
Why is big data important for businesses? explain its attributes. Doug Laney, a former Gartner analyst who now works at
. p. 3 consulting firm West Monroe, first defined three V's -- volume, variety and velocity --
What are the different types of big data? in 2001. Many people now use an expanded list of five V's to describe big data, with
. p. 4
these characteristics included:
The many V’s of big data p. 5
.
• Volume. There's no minimum size level that constitutes big data, but it typically
Big data examples and use cases p. 6 involves a large amount of data -- terabytes or more.
What are the business benefits of big data? • Variety. As mentioned above, big data includes various data types that may be
. p. 7
processed and stored in the same system.
What are the common big data challenges? • Velocity. Sets of big data often include real-time data and other information
. p. 8
that's generated and updated at a fast pace.
Key elements of big data enviornments • Veracity. This refers to how accurate and trustworthy different data sets are,
. . p. 10
something that needs to be assessed upfront.
Best practices for big data management • Value. Organizations also must understand the business value that sets of big
and analytics p. 13
data can provide to use it effectively.
Big data technologies and tools p. 14
What are the future trends in big data? Another V that's often applied to big data is variability, which refers to the multiple
. p. 17
meanings or formats that the same data can have in different source systems. Lists
Further reading p. 18 with as many as 10 V's have also been created.
Page 5 of 18
E-guide
In this e-guide
Big data examples and use cases
The ultimate guide to big data for
businesses p. 2 Market research firm IDC estimated that 64.2 zettabytes of data -- or 64 billion TB --
Why is big data important for businesses? was created or replicated worldwide in 2020, and it predicts that number will grow to
. p. 3 180 zettabytes by 2025. About 10% of the 2020 total was mainstream enterprise data,
What are the different types of big data? and not all of that was big data. But overall, the amount of enterprise data is growing
. p. 4
twice as fast as consumer data is, according to IDC. Clearly, that includes the increasing
The many V’s of big data p. 5
. volumes of big data being generated and collected by many businesses.
Big data examples and use cases p. 6
That data can be used for a variety of batch and stream processing applications, as well
What are the business benefits of big data?
. p. 7
as interactive querying, machine learning, predictive modeling and more. Ronald
Schmelzer, principal analyst and managing partner at AI research and advisory firm
What are the common big data challenges?
. p. 8 Cognilytica, outlined eight common use cases for big data in an article, along with
Key elements of big data enviornments examples of them by industry. His list includes the following uses:
. . p. 10
Best practices for big data management • getting a 360-degree view of customers to help optimize marketing, increase
and analytics p. 13 sales and upgrade customer service;
Big data technologies and tools p. 14 • improving customer acquisition and retention, which likewise is enabled by
better understanding customer needs and preferences;
What are the future trends in big data?
. p. 17 • strengthening fraud prevention and cybersecurity protections by better
identifying suspicious transactions and security threats;
Further reading p. 18
• improving business forecasts and processes, optimizing product pricing and
increasing operational efficiency;
• developing personalization and recommendation systems for corporate
websites, streaming services and online advertising;
Page 6 of 18
E-guide
Page 7 of 18
E-guide
In this e-guide
What are the future trends in big data? Because of its very nature, big data tends to be challenging to process, manage and use
. p. 17
effectively. Big data environments typically are complex, with multiple systems and
Further reading p. 18 tools that need to be well orchestrated to work smoothly together. The data itself is
also complex, particularly when data sets are large and varied or involve streaming
data.
Page 8 of 18
E-guide
Further reading p. 18
Page 9 of 18
E-guide
In this e-guide
Further reading p. 18
Big data management and analytics initiatives involve various components and
functions. These are some of their core aspects that need to be factored into project
plans upfront.
Page 10 of 18
E-guide
Big data architecture. The traditional data warehouse can be incorporated into big
In this e-guide data architectures to store structured data. More commonly, though, architectures
The ultimate guide to big data for feature data lakes, which can store different data sets in their native formats and
businesses p. 2 typically are built on technologies like Spark, Hadoop, NoSQL databases and cloud
Why is big data important for businesses? object storage services. Other architectural layers support data management and
. p. 3
analytics processes, as explained in an article on designing big data architectures by
What are the different types of big data? tech writer Mary K. Pratt. A solid architecture also provides the underpinnings that
. p. 4
data engineers need to create big data pipelines to funnel data into repositories and
The many V’s of big data p. 5
. analytics applications.
Big data examples and use cases p. 6
Big data analytics. Big data systems are primarily used for analytics applications, which
What are the business benefits of big data?
. p. 7
can range from straightforward BI and reporting to various forms of advanced analytics
done by data science teams. Machine learning in particular has benefitted from the
What are the common big data challenges?
. p. 8 availability of big data -- once mostly a scientific pursuit, it's now widely used by
Key elements of big data enviornments businesses to find patterns and anomalies in large data sets. An article by Kathleen
. . p. 10
Walch, another principal analyst and managing partner at Cognilytica, further explains
Best practices for big data management how big data and machine learning algorithms can be used together to make analytics
and analytics p. 13
more effective.
Big data technologies and tools p. 14
What are the future trends in big data? Big data collection. Before sets of big data can be processed and analyzed, they need
. p. 17
to be collected, often from both internal systems and external data sources. That can
Further reading p. 18
be a complicated undertaking because of the amount of data, its variety and the
number of different sources that may be involved. Data security and privacy issues add
to the challenges, even more so now that businesses need to comply with GDPR, CCPA
Page 11 of 18
E-guide
and other regulations. Read more about collecting big data and best practices for
In this e-guide managing the process in an article by Pratt.
The ultimate guide to big data for
businesses p. 2
Big data integration and preparation. Integrating data sets is also a crucial task in big
data environments, and it adds new requirements and challenges compared to
Why is big data important for businesses?
. p. 3 traditional data integration processes. For example, the volume, variety and velocity
What are the different types of big data? characteristics of big data may not lend themselves to conventional extract, transform
. p. 4
and load procedures. As a result, data management teams often must adopt new
The many V’s of big data p. 5 integration techniques for big data. Once data is integrated and ready for use, it needs
.
to be prepared for analysis, a process that includes data discovery, cleansing,
Big data examples and use cases p. 6
modeling, validation and other steps. In data lakes that store data in its raw form, data
What are the business benefits of big data?
. p. 7
preparation is often done by data scientists or data engineers to fit the needs of
individual analytics applications.
What are the common big data challenges?
. p. 8
Big data governance. Effective data governance is also vital to help ensure that
Key elements of big data enviornments
. . p. 10 collections of big data are consistent and get used properly, in compliance with privacy
Best practices for big data management regulations and internal data standards alike. But governing big data poses new
and analytics p. 13
challenges for data governance managers because of the wide variety of data they
Big data technologies and tools p. 14 often need to oversee now. Frequently done as part of data governance programs,
What are the future trends in big data? data quality management is an important facet of big data deployments, too. And
. p. 17
likewise, the combination of big data and data quality requires new processes for
Further reading p. 18
identifying and fixing errors and other quality issues.
Page 12 of 18
E-guide
In this e-guide
Best practices for big data management and analytics
The ultimate guide to big data for
businesses p. 2 An enterprise big data strategy that lays out a vision, goals and guidelines is a critical
Why is big data important for businesses? starting point for organizations. In an article on developing a strategy for big data,
. p. 3 Walch recommended the following four steps:
What are the different types of big data?
. p. 4 1. Define your company's business objectives to ensure that the strategy is
The many V’s of big data p. 5 aligned with them.
. 2. Identify available data sources and assess the current state of data usage in
Big data examples and use cases p. 6 business processes.
What are the business benefits of big data?
3. Identify, prioritize and document big data use cases that meet your business
. p. 7 objectives.
What are the common big data challenges?
4. Formulate a project roadmap that includes a gap analysis of your data
. p. 8 architecture and existing technologies, and then reprioritize the planned use
Key elements of big data enviornments
cases if necessary.
. . p. 10
Best practices for big data management Farmer suggested six big data best practices in another article. Among other things,
and analytics p. 13
that includes focusing on business needs over technology capabilities, collecting and
Big data technologies and tools p. 14 storing data for possible future uses, managing sets of big data in an iterative way for
What are the future trends in big data? different analytics applications, and considering use of the cloud to ease deployments
. p. 17
and potentially lower costs.
Further reading p. 18
Page 13 of 18
E-guide
In this e-guide
What are the future trends in big data? Big data technologies and tools
. p. 17
Further reading p. 18 The big data era began in earnest when the Hadoop distributed processing framework
was first released in 2006, providing an open source platform that could handle diverse
sets of data. A broad ecosystem of supporting technologies was built up around
Hadoop, including the Spark data processing engine. In addition, various NoSQL
Page 14 of 18
E-guide
databases were developed, offering more platforms for managing and storing data
In this e-guide that SQL-based relational databases weren't equipped to handle.
The ultimate guide to big data for
businesses p. 2
While Hadoop's built-in MapReduce processing engine has been partially eclipsed by
Spark and other newer technologies, it and other Hadoop components are still used by
Why is big data important for businesses?
. p. 3 many organizations. Overall, the technologies that now are common options for big
What are the different types of big data? data environments include the following categories:
. p. 4
The many V’s of big data p. 5 • Processing engines. Examples include Spark, Hadoop MapReduce and stream
. processing platforms like Flink, Kafka, Samza, Storm and Spark's Structured
Big data examples and use cases p. 6 Streaming module.
• Storage repositories. Examples include the Hadoop Distributed File System and
What are the business benefits of big data?
. p. 7 cloud object storage services like Amazon Simple Storage Service and Google
Cloud Storage.
What are the common big data challenges?
. p. 8 • NoSQL databases. Examples include Cassandra, Couchbase, CouchDB, HBase,
MarkLogic Data Hub, MongoDB, Redis and Neo4j.
Key elements of big data enviornments
. . p. 10 • SQL query engines. Examples include Drill, Hive, Presto and Trino.
• Data lake and data warehouse platforms. Examples include Amazon Redshift,
Best practices for big data management
and analytics p. 13 Delta Lake, Google BigQuery, Kylin and Snowflake.
• Commercial platforms and managed services. Examples include Amazon EMR,
Big data technologies and tools p. 14
Azure HDInsight, Cloudera Data Platform and Google Cloud Dataproc.
What are the future trends in big data?
. p. 17
Learn about the features and capabilities of 15 open source big data tools, including
Further reading p. 18
many of the technologies listed above, and read a comparison of Hadoop and Spark
that examines their architectures, processing capabilities, performance and other
Page 15 of 18
E-guide
attributes. Another article details a set of useful big data analytics features to look for
In this e-guide in tools.
The ultimate guide to big data for
businesses p. 2
Further reading p. 18
Page 16 of 18
E-guide
In this e-guide
What are future trends in big data?
The ultimate guide to big data for
businesses p. 2 Increasingly, organizations are running big data systems in the cloud, often using
Why is big data important for businesses? vendor-managed platforms that provide big data as a service to simplify deployments
. p. 3 and ongoing management. As Cognilytica's Schmelzer wrote in an article about top big
What are the different types of big data? data trends, moving to the cloud enables businesses to "deal with almost limitless
. p. 4
amounts of new data and pay for storage and compute capability on demand without
The many V’s of big data p. 5
. having to maintain their own large and complex data centers."
Big data examples and use cases p. 6
He also listed the following as notable trends:
What are the business benefits of big data?
. p. 7
• increasing data diversity, driven in particular by growing data volumes from IoT
What are the common big data challenges? devices that are leading more organizations to adopt edge computing to better
. p. 8
handle processing workloads;
Key elements of big data enviornments • further increases in enterprise use of machine learning and other AI
. . p. 10
technologies, both for data analytics and to enable chatbots to provide better
Best practices for big data management customer support with more personalized interactions; and
and analytics p. 13
• wider adoption of DataOps practices for managing data flows, as well as a
Big data technologies and tools p. 14 heightened focus on data stewardship to help organizations deal with data
What are the future trends in big data? governance, security and privacy issues.
. p. 17
Further reading p. 18
Further reading
Page 17 of 18
E-guide
In this e-guide
Further reading p. 18
Page 18 of 18