Unit 2

1. Aggregate data models are a type of data

model that is designed to store and manage
pre-aggregated data. This means that the data
has already been processed and summarized,
making it faster and easier to query and
analyze. Aggregate data models are often used
in data warehousing and analytics applications.

There are two main types of aggregate data models:

● Relational aggregate data models: These

models are similar to traditional relational
database models, but they use specialized
tables and indexes to store and manage
aggregate data.

● NoSQL aggregate data models: These

models are designed for non-relational
databases, such as document databases and
graph databases. They offer a variety of
different ways to store and manage aggregate
data, depending on the specific needs of the
application.

Benefits of using aggregate data models:

● Improved performance: Aggregate data

models can significantly improve the
performance of data queries and analysis
operations. This is because the data has
already been processed and summarized, so
the database does not have to spend time
performing these operations on the fly.

● Reduced storage requirements: Aggregate

data models can reduce the amount of storage
space required to store data. This is because
the aggregate data is typically much smaller
than the original source data.

● Simplified data analysis: Aggregate data

models can simplify data analysis by providing
pre-computed summaries of the data. This
makes it easier for users to identify trends and
patterns in the data.

Use cases for aggregate data models:

● Reporting: Aggregate data models are often

used to generate reports on key business
metrics, such as sales, revenue, and customer
churn.

● Analytics: Aggregate data models can be

used to perform complex analytics operations,
such as trend analysis, forecasting, and
predictive analytics.

● Machine learning: Aggregate data models can

be used to train machine learning models. This
is because the aggregate data typically contains
a large number of features and data points,
which is essential for training accurate machine
learning models.

Examples of aggregate data models:

● Star schema: A star schema is a type of

relational aggregate data model that is
commonly used in data warehousing
applications. It consists of a fact table
surrounded by dimension tables. The fact table
contains the aggregate data, while the
dimension tables contain the detailed data that
is used to slice and dice the aggregate data.

● Snowflake schema: A snowflake schema is a

type of relational aggregate data model that is
similar to a star schema, but it has additional
dimension tables that are normalized. This
makes the snowflake schema more efficient for
querying and analyzing complex data sets.

● Materialized views: Materialized views are a

type of aggregate data model that is used in
NoSQL databases. They are pre-computed
views of the data that are stored in the
database. Materialized views can significantly
improve the performance of data queries and
analysis operations.

Overall, aggregate data models are a powerful tool for

storing, managing, and analyzing large data sets. They
can be used to improve the performance of data
queries and analysis operations, reduce storage
requirements, and simplify data analysis.

2. Aggregates in big data are pre-computed

summaries of large data sets. They are used to
improve the performance of data queries and analysis
operations, reduce storage requirements, and simplify
data analysis.

Aggregates can be created on any type of data, but

they are most commonly used on numerical data.
Some common aggregate functions include:

● Count: The number of values in a data set.

● Sum: The sum of all values in a data set.

● Average: The average of all values in a data

set.

● Median: The middle value in a data set when

the values are sorted in ascending order.

● Mode: The most frequent value in a data set.

Aggregates can also be created on categorical data.

For example, you could count the number of
customers in each country or the number of products
sold by each category.

Aggregates are typically stored in aggregate data

models, such as star schemas, snowflake schemas,
and materialized views. These data models are
designed to make it easy to query and analyze
aggregate data.

Aggregates are used in a wide variety of big data

applications, including:

● Reporting: Aggregates are used to generate

reports on key business metrics, such as sales,
revenue, and customer churn.

● Analytics: Aggregates are used to perform

complex analytics operations, such as trend
analysis, forecasting, and predictive analytics.

● Machine learning: Aggregates are used to

train machine learning models.

● Data visualization: Aggregates are used to

create data visualizations, such as charts and
graphs.

3.
Key-value data models and document data models are
two types of non-relational data models. They are both
designed to store and manage large amounts of data
efficiently and flexibly.

Key-value data models store data as a collection of

key-value pairs. The key is a unique identifier for the
data item, and the value is the data item itself. Key-
value data models are very simple and efficient, and
they are well-suited for storing data that needs to be
accessed quickly and easily, such as cache data or
session data.

Document data models store data as documents. A

document is a semi-structured data structure that can
contain a variety of different data types, such as text,
numbers, images, and nested data structures.
Document data models are more flexible than key-
value data models, but they can be less efficient for
certain types of queries.

Here is a table that compares key-value data models

and document data models:

Feature Key-value data Document data

model model

Data Simple, Flexible, semi-

model efficient structured

Use Cache data, Complex data

cases session data, structures,
simple object nested data,
storage flexible
querying

Examples Redis, MongoDB,

Memcached, CouchDB,
DynamoDB Elasticsearch

Which data model to choose?

The best data model for your application depends on

your specific needs. If you need to store and retrieve
data quickly and easily, then a key-value data model
may be a good choice. If you need to store and query
complex data structures, then a document data model
may be a better choice.

Here are some examples of when you might use each

data model:

● Key-value data model:

○ Caching user session data

○ Storing product metadata in an e-

commerce application

○ Storing real-time data from IoT devices

● Document data model:

○ Storing user profiles in a social media

application

○ Storing product catalogs in an e-

commerce application

○ Storing log data from a distributed

system

4. Relationships in big data are the connections

between different data points. They can be explicit or
implicit, and they can be complex and difficult to
identify. Relationships in big data can be used to
improve the accuracy and effectiveness of data
analysis and machine learning models.

There are two main types of relationships in big data:

● Explicit relationships: These are relationships

that are explicitly defined in the data. For
example, a customer ID field in a sales
database might be explicitly linked to a
customer name field.

● Implicit relationships: These are relationships

that are not explicitly defined in the data, but
can be inferred from the data. For example, two
customers who have purchased the same
products repeatedly are likely to have some
kind of relationship.

Relationships in big data can be identified using a

variety of techniques, including:

● Correlation analysis: This technique looks for

statistical relationships between different data
points. For example, if you find that two data
points are highly correlated, it suggests that
there is some kind of relationship between
them.

● Natural language processing (NLP): This

technique can be used to extract relationships
from text data. For example, NLP could be used
to identify the relationships between different
characters in a novel.

● Graph analysis: This technique can be used

to model relationships between data points as a
graph. Graph analysis can be used to identify
complex relationships that would be difficult to
find using other methods.

Relationships in big data can be used for a variety of

purposes, including:

● Improving the accuracy of data analysis: By

understanding the relationships between
different data points, analysts can make more
accurate predictions and inferences. For
example, a bank might use relationships in
customer data to predict which customers are
most likely to churn.

● Improving the effectiveness of machine

learning models: Machine learning models can
be trained on relationships in data to improve
their performance. For example, a
recommendation system might use
relationships between user data and product
data to recommend products to users.

● Identifying new opportunities and risks: By

understanding the relationships between
different data points, organizations can identify
new opportunities and risks that they might not
otherwise be aware of. For example, a retailer
might use relationships in sales data to identify
new product trends.

5. Graph databases are a type of NoSQL database

that are designed to store and manage data in the
form of graphs. Graphs are made up of nodes and
edges, where nodes represent entities and edges
represent relationships between entities. Graph
databases are well-suited for storing and querying
complex data that has a high degree of connectivity.

Graph databases offer a number of advantages over

traditional relational databases for certain types of
applications. These advantages include:

● Performance: Graph databases can be very

efficient for querying data that is highly
connected. This is because graph databases
can directly traverse the relationships between
nodes, without having to perform expensive
joins.

● Flexibility: Graph databases are more flexible

than relational databases for modeling complex
data. This is because graph databases do not
require a predefined schema.

● Scalability: Graph databases can scale to

handle very large data sets. This is because
graph databases can distribute data across
multiple servers and perform parallel queries.

Graph databases are used in a variety of applications,

including:

● Social network analysis: Graph databases are

commonly used to store and analyze social
network data, such as the relationships between
users on social media platforms.

● Recommendation systems: Graph databases

can be used to build recommendation systems
that suggest products, services, or content to
users based on their past behavior and the
relationships between different users and items.

● Fraud detection: Graph databases can be

used to detect fraudulent activity, such as credit
card fraud and insurance fraud.

● Knowledge graphs: Graph databases can be

used to store and query knowledge graphs,
which are large networks of data that represent
the real world.

Some popular graph databases include:

● Neo4j

● Amazon Neptune

● OrientDB

● ArangoDB

● Dgraph

Graph databases are a powerful tool for storing and

managing complex data. They can be used to improve
the performance, flexibility, and scalability of data-
driven applications.

6.

Schemaless databases are a type of NoSQL database

that do not require a predefined schema to store data.
This means that data can be stored in flexible and
dynamic formats, such as JSON documents, key-value
pairs, graphs, or columns. Schemaless databases are
often used for applications that need to handle large
volumes of unstructured or semi-structured data, such
as social media, e-commerce, or analytics.

Benefits of schemaless databases:

● Flexibility: Schemaless databases allow for

greater flexibility in data modeling, as there are
no constraints on the structure of the data. This
makes them ideal for storing data that can vary
in format and content.

● Scalability: Schemaless databases are

designed for scalability, as they can handle
large amounts of unstructured data with ease.
This makes them suitable for handling big data
and real-time data processing.

● Reduced complexity: Schemaless databases

can reduce the complexity of data modeling and
development. This is because there is no need
to define a schema upfront.

Challenges of schemaless databases:

● Performance: Schemaless databases can be

slower than traditional relational databases for
certain types of queries. This is because
schemaless databases need to dynamically
parse the data at query time.

● Data integrity: Schemaless databases can be

more difficult to maintain data integrity, as there
are no constraints on the structure of the data.
This can lead to data inconsistencies and
errors.

● Data governance: Schemaless databases can

be more difficult to govern, as there is no central
repository for metadata. This can make it
difficult to understand and manage the data.

Use cases for schemaless databases:

● Social media: Schemaless databases are

often used to store and manage social media
data, such as user profiles, posts, and
relationships.

● E-commerce: Schemaless databases are

often used to store and manage e-commerce
data, such as product catalogs, customer
orders, and inventory.

● Analytics: Schemaless databases are often

used to store and analyze large data sets, such
as customer data, sensor data, and financial
data.

Some popular schemaless databases include:

● MongoDB

● CouchDB

● DynamoDB

● Redis

● Cassandra

7. A materialized view in big data is a precomputed

copy of a query result. It is a physical table that is
stored in the database and is updated automatically
whenever the data in the underlying tables changes.
Materialized views can significantly improve the
performance of data queries, especially those that are
frequently executed and involve complex
aggregations.

Materialized views are particularly useful for big data

applications because they can help to reduce the
amount of data that needs to be processed and
scanned for each query. This can lead to significant
performance gains, especially for queries that involve
large data sets or complex aggregations.

Here are some of the benefits of using materialized

views in big data:

● Improved performance: Materialized views

can significantly improve the performance of
data queries by precomputing the results of
frequently executed queries. This can reduce
the amount of data that needs to be processed
and scanned for each query, leading to faster
response times.

● Reduced load on the database: Materialized

views can reduce the load on the database by
offloading some of the processing work from the
database server to the materialized view. This
can improve the overall scalability and
performance of the database system.

● Simplified data analysis: Materialized views

can simplify data analysis by providing pre-
computed summaries of the data. This can
make it easier for analysts to identify trends and
patterns in the data without having to write
complex queries.

Materialized views can be used in a variety of big data

applications, including:

● Data warehousing: Materialized views are

commonly used in data warehousing
applications to improve the performance of
reporting and analytical queries.

● Real-time analytics: Materialized views can be

used to power real-time analytics dashboards
and applications by providing pre-computed
summaries of the data.

● Machine learning: Materialized views can be

used to train machine learning models by
providing pre-processed features and
aggregations of the data.

8. Distribution models in big data are techniques used

to distribute data across multiple servers or nodes.
This is done to improve the performance, scalability,
and reliability of big data applications.

There are two main types of distribution models:

● Sharding: Sharding divides data into smaller,

more manageable chunks, which are then
distributed across multiple servers. This can
improve performance by allowing multiple
servers to process data in parallel.

● Replication: Replication copies data across

multiple servers. This can improve reliability by
ensuring that data is still available even if one
server fails.

In practice, it is common to use a combination of

sharding and replication to distribute data in big data
applications. This allows organizations to balance the
need for performance, scalability, and reliability.

Here are some of the benefits of using distribution

models in big data:

● Improved performance: Sharding and

replication can improve the performance of big
data applications by allowing multiple servers to
process data in parallel. This is especially
beneficial for applications that need to process
large data sets or complex queries.

● Increased scalability: Distribution models can

help organizations to scale their big data
applications to meet increasing demands. By
distributing data across multiple servers,
organizations can increase the overall
processing capacity and throughput of their
applications.

● Improved reliability: Replication can improve

the reliability of big data applications by
ensuring that data is still available even if one
server fails. This is especially important for
applications that need to be available 24/7.

Here are some examples of how distribution models

are used in big data:

● Apache Hadoop: Apache Hadoop is a popular

open-source big data processing framework
that uses sharding and replication to distribute
data across multiple nodes.

● Amazon Web Services (AWS): AWS offers a

variety of big data services that use distribution
models, such as Amazon DynamoDB, Amazon
Redshift, and Amazon Elastic MapReduce
(EMR).

● Google Cloud Platform (GCP): GCP also

offers a variety of big data services that use
distribution models, such as BigQuery, Cloud
Dataproc, and Cloud Data Fusion.

Distribution models are an essential part of big data

applications. By distributing data across multiple
servers, organizations can improve the performance,
scalability, and reliability of their applications.

Here are some additional considerations for choosing

a distribution model for your big data application:

● The type of data: Some types of data, such as

relational data, are better suited for sharding,
while other types of data, such as graph data,
are better suited for replication.

● The access patterns: If your application needs

to frequently access all of the data, then
replication may be a better choice than
sharding. If your application only needs to
access a subset of the data at a time, then
sharding may be a better choice.

● The budget: Replication can be more

expensive than sharding, as it requires more
storage space.

9. Sharding is a database partitioning technique that

splits a database into smaller, more manageable
pieces called shards. These shards are then
distributed across multiple servers, which can improve
performance and scalability. Sharding is often used in
big data applications, where the amount of data can be
too large to be stored and processed on a single
server.

There are two main types of sharding:

● Horizontal sharding: Horizontal sharding splits

the data horizontally, meaning that each shard
contains a subset of all of the data. This is the
most common type of sharding and is well-
suited for large datasets.

● Vertical sharding: Vertical sharding splits the

data vertically, meaning that each shard
contains a specific set of columns or tables.
This is less common than horizontal sharding,
but can be useful for certain applications, such
as those that need to frequently query specific
subsets of the data.

Sharding can be implemented in a variety of ways.

One common approach is to use a hash function to
assign each row of data to a shard. Another approach
is to use a range-based partitioning scheme, where
each shard contains a specific range of values.

Sharding can provide a number of benefits for big data

applications, including:

● Improved performance: Sharding can improve

performance by distributing the workload across
multiple servers. This can be especially
beneficial for applications that need to process
large datasets or complex queries.

● Increased scalability: Sharding can help

organizations to scale their big data applications
to meet increasing demands. By adding more
servers, organizations can increase the overall
processing capacity and throughput of their
applications.

● Reduced costs: Sharding can help

organizations to reduce the cost of their big data
infrastructure. By distributing the data across
multiple servers, organizations can avoid the
need to purchase expensive, high-end servers.

However, there are also some challenges associated

with sharding, including:

● Increased complexity: Sharding can add

complexity to the design and management of
big data applications. Organizations need to
carefully consider how to shard the data and
how to distribute the workload across the
servers.

● Data consistency: Sharding can make it more

difficult to maintain data consistency.
Organizations need to implement mechanisms
to ensure that all of the shards are up-to-date
and that all of the data is accessible to all of the
servers.

● Query complexity: Sharding can make it more

complex to write queries that access data from
multiple shards. Organizations need to use a
query optimization tool to ensure that their
queries are executed efficiently.

Overall, sharding is a powerful technique for improving

the performance, scalability, and cost-effectiveness of
big data applications. However, it is important to
carefully consider the challenges associated with
sharding before implementing it.

Here are some examples of how sharding is used in

big data:

● Apache Hadoop: Apache Hadoop uses

sharding to distribute data across multiple
nodes. This allows Hadoop to process large
datasets in parallel, which improves
performance.

● Amazon DynamoDB: Amazon DynamoDB is a

fully managed NoSQL database service that
uses sharding to distribute data across multiple
servers. This makes DynamoDB highly scalable
and fault-tolerant.

● Google BigQuery: Google BigQuery is a fully

managed, petabyte-scale analytics data
warehouse that uses sharding to distribute data
across multiple servers. This makes BigQuery
very fast and scalable.

10. Master-slave replication is a database replication

technique that maintains one master database and
one or more slave databases. The master database is
the primary database where all writes are performed.
The slave databases are read-only copies of the