Q.1]EXPLAIN CAP theorem ?
Ans:- The CAP theorem, originally introduced
as the CAP principle, can be used to explain
some of the competing requirements in a
distributed system with replication. It is a
tool used to make system designers aware of
the trade-offs while designing networked
shared-data systems.
The three letters in CAP refer to three
desirable properties of distributed systems
with replicated data: consistency (among
replicated copies), availability (of the system
for read and write operations) and partition
tolerance (in the face of the nodes in the
system being partitioned by a network
fault).
Consistency –
Consistency means that the nodes will have
the same copies of a replicated data item
visible for various transactions. A guarantee
that every node in a distributed cluster
returns the same, most recent and a
successful write. Consistency refers to every
client having the same view of the data.
There are various types of consistency
models.
Availability –
Availability means that each read or write
request for a data item will either be
processed successfully or will receive a
message that the operation cannot be
completed. Every non-failing node returns a
response for all the read and write requests
in a reasonable amount of time. The key
word here is “every”.
Partition Tolerance –
Partition tolerance means that the system
can continue operating even if the network
connecting the nodes has a fault that results
in two or more partitions, where the nodes
in each partition can only communicate
among each other. That means, the system
continues to function and upholds its
consistency guarantees in spite of network
partitions. Network partitions are a fact of
life.
Q.2]What is YARN and explain it ?
Ans:- Certainly! YARN (Yet Another Resource
Negotiator) is a core component of Apache
Hadoop, designed for resource management
and job scheduling in a distributed computing
environment. For a big data analyst,
understanding YARN's architecture is vital as it
orchestrates resource allocation and
facilitates efficient data processing.
Components of YARN:
1.)ResourceManager (RM):i.)The central
authority overseeing resource allocation in
the cluster.
ii.)Consists of two main components:
a.)Scheduler: Allocates resources to various
applications based on predefined policies
b.)ApplicationManager: Manages the
lifecycle of application masters.
2.)NodeManager (NM):- i.)Runs on each
node in the cluster and manages resources
available on that node. ii.)Monitors resource
utilization (CPU, memory, etc.) and reports it
to the Resource Manager.
3.)ApplicationMaster (AM):-i.)Manages the
execution of a specific application within the cluster.
ii.)Negotiates resources with the ResourceManager
via the ApplicationManager.
Workflow:-
a.)Submit Application:-A user submits a
job/application to the ResourceManager.
b.)Application Execution:-The Application
Master coordinates with the Resource
Manager to acquire necessary resources for
execution.
c.)Task Execution:-NodeManagers create
containers and execute application tasks
within these containers.
d.)Resource Deallocation:-Once tasks are
completed, resources are released and made
available for other applications.
Q.1]EXPLAIN CAP theorem ?
Ans:- The CAP theorem, originally introduced
as the CAP principle, can be used to explain
some of the competing requirements in a
distributed system with replication. It is a
tool used to make system designers aware of
the trade-offs while designing networked
shared-data systems.
The three letters in CAP refer to three
desirable properties of distributed systems
with replicated data: consistency (among
replicated copies), availability (of the system
for read and write operations) and partition
tolerance (in the face of the nodes in the
system being partitioned by a network
fault).
Consistency –
Consistency means that the nodes will have
the same copies of a replicated data item
visible for various transactions. A guarantee
that every node in a distributed cluster
returns the same, most recent and a
successful write. Consistency refers to every
client having the same view of the data.
There are various types of consistency
models.
Availability –
Availability means that each read or write
request for a data item will either be
processed successfully or will receive a
message that the operation cannot be
completed. Every non-failing node returns a
response for all the read and write requests
in a reasonable amount of time. The key
word here is “every”.
Partition Tolerance –
Partition tolerance means that the system
can continue operating even if the network
connecting the nodes has a fault that results
in two or more partitions, where the nodes
in each partition can only communicate
among each other. That means, the system
continues to function and upholds its
consistency guarantees in spite of network
partitions. Network partitions are a fact of
life.
Q.2]What is YARN and explain it ?
Ans:- Certainly! YARN (Yet Another Resource
Negotiator) is a core component of Apache
Hadoop, designed for resource management
and job scheduling in a distributed computing
environment. For a big data analyst,
understanding YARN's architecture is vital as it
orchestrates resource allocation and
facilitates efficient data processing.
Components of YARN:
1.)ResourceManager (RM):i.)The central
authority overseeing resource allocation in
the cluster.
ii.)Consists of two main components:
a.)Scheduler: Allocates resources to various
applications based on predefined policies
b.)ApplicationManager: Manages the
lifecycle of application masters.
2.)NodeManager (NM):- i.)Runs on each
node in the cluster and manages resources
available on that node. ii.)Monitors resource
utilization (CPU, memory, etc.) and reports it
to the Resource Manager.
3.)ApplicationMaster (AM):-i.)Manages the
execution of a specific application within the cluster.
ii.)Negotiates resources with the ResourceManager
via the ApplicationManager.
Workflow:-
a.)Submit Application:-A user submits a
job/application to the ResourceManager.
b.)Application Execution:-The Application
Master coordinates with the Resource
Manager to acquire necessary resources for
execution.
c.)Task Execution:-NodeManagers create
containers and execute application tasks
within these containers.
d.)Resource Deallocation:-Once tasks are
completed, resources are released and made
available for other applications.
Q.1]EXPLAIN CAP theorem ?
Ans:- The CAP theorem, originally introduced
as the CAP principle, can be used to explain
some of the competing requirements in a
distributed system with replication. It is a
tool used to make system designers aware of
the trade-offs while designing networked
shared-data systems.
The three letters in CAP refer to three
desirable properties of distributed systems
with replicated data: consistency (among
replicated copies), availability (of the system
for read and write operations) and partition
tolerance (in the face of the nodes in the
system being partitioned by a network
fault).
Consistency –
Consistency means that the nodes will have
the same copies of a replicated data item
visible for various transactions. A guarantee
that every node in a distributed cluster
returns the same, most recent and a
successful write. Consistency refers to every
client having the same view of the data.
There are various types of consistency
models.
Availability –
Availability means that each read or write
request for a data item will either be
processed successfully or will receive a
message that the operation cannot be
completed. Every non-failing node returns a
response for all the read and write requests
in a reasonable amount of time. The key
word here is “every”.
Partition Tolerance –
Partition tolerance means that the system
can continue operating even if the network
connecting the nodes has a fault that results
in two or more partitions, where the nodes
in each partition can only communicate
among each other. That means, the system
continues to function and upholds its
consistency guarantees in spite of network
partitions. Network partitions are a fact of
life.
Q.2]What is YARN and explain it ?
Ans:- Certainly! YARN (Yet Another Resource
Negotiator) is a core component of Apache
Hadoop, designed for resource management
and job scheduling in a distributed computing
environment. For a big data analyst,
understanding YARN's architecture is vital as it
orchestrates resource allocation and
facilitates efficient data processing.
Components of YARN:
1.)ResourceManager (RM):i.)The central
authority overseeing resource allocation in
the cluster.
ii.)Consists of two main components:
a.)Scheduler: Allocates resources to various
applications based on predefined policies
b.)ApplicationManager: Manages the
lifecycle of application masters.
2.)NodeManager (NM):- i.)Runs on each
node in the cluster and manages resources
available on that node. ii.)Monitors resource
utilization (CPU, memory, etc.) and reports it
to the Resource Manager.
3.)ApplicationMaster (AM):-i.)Manages the
execution of a specific application within the cluster.
ii.)Negotiates resources with the ResourceManager
via the ApplicationManager.
Workflow:-
a.)Submit Application:-A user submits a
job/application to the ResourceManager.
b.)Application Execution:-The Application
Master coordinates with the Resource
Manager to acquire necessary resources for
execution.
c.)Task Execution:-NodeManagers create
containers and execute application tasks
within these containers.
d.)Resource Deallocation:-Once tasks are
completed, resources are released and made
available for other applications.
Q.1]EXPLAIN CAP theorem ?
Ans:- The CAP theorem, originally introduced
as the CAP principle, can be used to explain
some of the competing requirements in a
distributed system with replication. It is a
tool used to make system designers aware of
the trade-offs while designing networked
shared-data systems.
The three letters in CAP refer to three
desirable properties of distributed systems
with replicated data: consistency (among
replicated copies), availability (of the system
for read and write operations) and partition
tolerance (in the face of the nodes in the
system being partitioned by a network
fault).
Consistency –
Consistency means that the nodes will have
the same copies of a replicated data item
visible for various transactions. A guarantee
that every node in a distributed cluster
returns the same, most recent and a
successful write. Consistency refers to every
client having the same view of the data.
There are various types of consistency
models.
Availability –
Availability means that each read or write
request for a data item will either be
processed successfully or will receive a
message that the operation cannot be
completed. Every non-failing node returns a
response for all the read and write requests
in a reasonable amount of time. The key
word here is “every”.
Partition Tolerance –
Partition tolerance means that the system
can continue operating even if the network
connecting the nodes has a fault that results
in two or more partitions, where the nodes
in each partition can only communicate
among each other. That means, the system
continues to function and upholds its
consistency guarantees in spite of network
partitions. Network partitions are a fact of
life.
Q.2]What is YARN and explain it ?
Ans:- Certainly! YARN (Yet Another Resource
Negotiator) is a core component of Apache
Hadoop, designed for resource management
and job scheduling in a distributed computing
environment. For a big data analyst,
understanding YARN's architecture is vital as it
orchestrates resource allocation and
facilitates efficient data processing.
Components of YARN:
1.)ResourceManager (RM):i.)The central
authority overseeing resource allocation in
the cluster.
ii.)Consists of two main components:
a.)Scheduler: Allocates resources to various
applications based on predefined policies
b.)ApplicationManager: Manages the
lifecycle of application masters.
2.)NodeManager (NM):- i.)Runs on each
node in the cluster and manages resources
available on that node. ii.)Monitors resource
utilization (CPU, memory, etc.) and reports it
to the Resource Manager.
3.)ApplicationMaster (AM):-i.)Manages the
execution of a specific application within the cluster.
ii.)Negotiates resources with the ResourceManager
via the ApplicationManager.
Workflow:-
a.)Submit Application:-A user submits a
job/application to the ResourceManager.
b.)Application Execution:-The Application
Master coordinates with the Resource
Manager to acquire necessary resources for
execution.
c.)Task Execution:-NodeManagers create
containers and execute application tasks
within these containers.
d.)Resource Deallocation:-Once tasks are
completed, resources are released and made
available for other applications.
Q.1]EXPLAIN CAP theorem ?
Ans:- The CAP theorem, originally introduced
as the CAP principle, can be used to explain
some of the competing requirements in a
distributed system with replication. It is a
tool used to make system designers aware of
the trade-offs while designing networked
shared-data systems.
The three letters in CAP refer to three
desirable properties of distributed systems
with replicated data: consistency (among
replicated copies), availability (of the system
for read and write operations) and partition
tolerance (in the face of the nodes in the
system being partitioned by a network
fault).
Consistency –
Consistency means that the nodes will have
the same copies of a replicated data item
visible for various transactions. A guarantee
that every node in a distributed cluster
returns the same, most recent and a
successful write. Consistency refers to every
client having the same view of the data.
There are various types of consistency
models.
Availability –
Availability means that each read or write
request for a data item will either be
processed successfully or will receive a
message that the operation cannot be
completed. Every non-failing node returns a
response for all the read and write requests
in a reasonable amount of time. The key
word here is “every”.
Partition Tolerance –
Partition tolerance means that the system
can continue operating even if the network
connecting the nodes has a fault that results
in two or more partitions, where the nodes
in each partition can only communicate
among each other. That means, the system
continues to function and upholds its
consistency guarantees in spite of network
partitions. Network partitions are a fact of
life.
Q.2]What is YARN and explain it ?
Ans:- Certainly! YARN (Yet Another Resource
Negotiator) is a core component of Apache
Hadoop, designed for resource management
and job scheduling in a distributed computing
environment. For a big data analyst,
understanding YARN's architecture is vital as it
orchestrates resource allocation and
facilitates efficient data processing.
Components of YARN:
1.)ResourceManager (RM):i.)The central
authority overseeing resource allocation in
the cluster.
ii.)Consists of two main components:
a.)Scheduler: Allocates resources to various
applications based on predefined policies
b.)ApplicationManager: Manages the
lifecycle of application masters.
2.)NodeManager (NM):- i.)Runs on each
node in the cluster and manages resources
available on that node. ii.)Monitors resource
utilization (CPU, memory, etc.) and reports it
to the Resource Manager.
3.)ApplicationMaster (AM):-i.)Manages the
execution of a specific application within the cluster.
ii.)Negotiates resources with the ResourceManager
via the ApplicationManager.
Workflow:-
a.)Submit Application:-A user submits a
job/application to the ResourceManager.
b.)Application Execution:-The Application
Master coordinates with the Resource
Manager to acquire necessary resources for
execution.
c.)Task Execution:-NodeManagers create
containers and execute application tasks
within these containers.
d.)Resource Deallocation:-Once tasks are
completed, resources are released and made
available for other applications.
Q.1]EXPLAIN CAP theorem ?
Ans:- The CAP theorem, originally introduced
as the CAP principle, can be used to explain
some of the competing requirements in a
distributed system with replication. It is a
tool used to make system designers aware of
the trade-offs while designing networked
shared-data systems.
The three letters in CAP refer to three
desirable properties of distributed systems
with replicated data: consistency (among
replicated copies), availability (of the system
for read and write operations) and partition
tolerance (in the face of the nodes in the
system being partitioned by a network
fault).
Consistency –
Consistency means that the nodes will have
the same copies of a replicated data item
visible for various transactions. A guarantee
that every node in a distributed cluster
returns the same, most recent and a
successful write. Consistency refers to every
client having the same view of the data.
There are various types of consistency
models.
Availability –
Availability means that each read or write
request for a data item will either be
processed successfully or will receive a
message that the operation cannot be
completed. Every non-failing node returns a
response for all the read and write requests
in a reasonable amount of time. The key
word here is “every”.
Partition Tolerance –
Partition tolerance means that the system
can continue operating even if the network
connecting the nodes has a fault that results
in two or more partitions, where the nodes
in each partition can only communicate
among each other. That means, the system
continues to function and upholds its
consistency guarantees in spite of network
partitions. Network partitions are a fact of
life.
Q.2]What is YARN and explain it ?
Ans:- Certainly! YARN (Yet Another Resource
Negotiator) is a core component of Apache
Hadoop, designed for resource management
and job scheduling in a distributed computing
environment. For a big data analyst,
understanding YARN's architecture is vital as it
orchestrates resource allocation and
facilitates efficient data processing.
Components of YARN:
1.)ResourceManager (RM):i.)The central
authority overseeing resource allocation in
the cluster.
ii.)Consists of two main components:
a.)Scheduler: Allocates resources to various
applications based on predefined policies
b.)ApplicationManager: Manages the
lifecycle of application masters.
2.)NodeManager (NM):- i.)Runs on each
node in the cluster and manages resources
available on that node. ii.)Monitors resource
utilization (CPU, memory, etc.) and reports it
to the Resource Manager.
3.)ApplicationMaster (AM):-i.)Manages the
execution of a specific application within the cluster.
ii.)Negotiates resources with the ResourceManager
via the ApplicationManager.
Workflow:-
a.)Submit Application:-A user submits a
job/application to the ResourceManager.
b.)Application Execution:-The Application
Master coordinates with the Resource
Manager to acquire necessary resources for
execution.
c.)Task Execution:-NodeManagers create
containers and execute application tasks
within these containers.
d.)Resource Deallocation:-Once tasks are
completed, resources are released and made
available for other applications.
Q.3] Real-time Streaming Platforms for Big
Data ?
Ans:- Real-time analytics can keep you up-to-
date on what’s happening right now, such as
how many people are currently reading your
new blog post and whether someone just
liked your latest Facebook status. For most
use cases, real-time is a nice-to-have feature
that won’t provide any crucial insights.
However, sometimes real-time is a must.
a.)Apache Flink:-Apache Flink is an open-
source streaming platform that’s extremely
fast at complex stream processing. In fact, it’s
able to process live streams within
milliseconds because it can be programmed
to only process new, changed data as it goes
through rows of big data in real-time. In this
way, Flink easily enables the execution of
batch and stream processing at a large scale
to offer real-time insights, so it’s no wonder
this platform is known for offering low latency
and high performance.
b.)Apache Spark:-Another open-source data
processing framework that’s known for its
speed and ease of use is Spark. This platform
runs in-memory on RAM on clusters and isn’t
tied to Hadoop’s MapReduce two-stage
paradigm, which adds to its lightning-fast
performance when it comes to big data
processing. c.)Apache Samza:-Samza is an
open-source distributed stream-processing
framework that lets users build applications
that can process big data in real-time from
several sources. It’s based on Apache Kafka
and YARN, but it can also run as a standalone
library. LinkedIn originally developed Samza,
but since then, other big brands have started
using it—such as eBay, Slack, Redfin,
Optimizely, and TripAdvisor.
Q.4] What is Streaming Data Architecture?
Ans:-a.)A streaming data architecture is a
framework of software components built to
ingest, process, and analyze data streams –
typically in real time or near-real time. Rather
than writing and reading data in batches, a
streaming data architecture consumes data
immediately as it is generated, persists it to
storage, and may include various additional
components per use case – such as tools for
real-time processing, data manipulation, and
analytics.
b.)Streaming architectures must account for
the unique characteristics of data streams,
which tend to generate massive amounts of
data (terabytes to petabytes) that it is at best
semi-structured and requires significant pre-
processing and ETL to become useful.
c.)Stream processing is a complex challenge
rarely solved with a single database or ETL
tool – hence the need to “architect” a
solution consisting of multiple building blocks.
Part of the thinking behind Upsolver SQLake
is to replace these point products with an
integrated platform that delivers with self-
orchestrating declarative data pipelines. We’ll
demonstrate how this approach manifests
within each part of the streaming data supply
chain.
Q.3] Real-time Streaming Platforms for Big
Data ?
Ans:- Real-time analytics can keep you up-to-
date on what’s happening right now, such as
how many people are currently reading your
new blog post and whether someone just
liked your latest Facebook status. For most
use cases, real-time is a nice-to-have feature
that won’t provide any crucial insights.
However, sometimes real-time is a must.
a.)Apache Flink:-Apache Flink is an open-
source streaming platform that’s extremely
fast at complex stream processing. In fact, it’s
able to process live streams within
milliseconds because it can be programmed
to only process new, changed data as it goes
through rows of big data in real-time. In this
way, Flink easily enables the execution of
batch and stream processing at a large scale
to offer real-time insights, so it’s no wonder
this platform is known for offering low latency
and high performance.
b.)Apache Spark:-Another open-source data
processing framework that’s known for its
speed and ease of use is Spark. This platform
runs in-memory on RAM on clusters and isn’t
tied to Hadoop’s MapReduce two-stage
paradigm, which adds to its lightning-fast
performance when it comes to big data
processing. c.)Apache Samza:-Samza is an
open-source distributed stream-processing
framework that lets users build applications
that can process big data in real-time from
several sources. It’s based on Apache Kafka
and YARN, but it can also run as a standalone
library. LinkedIn originally developed Samza,
but since then, other big brands have started
using it—such as eBay, Slack, Redfin,
Optimizely, and TripAdvisor.
Q.4] What is Streaming Data Architecture?
Ans:-a.)A streaming data architecture is a
framework of software components built to
ingest, process, and analyze data streams –
typically in real time or near-real time. Rather
than writing and reading data in batches, a
streaming data architecture consumes data
immediately as it is generated, persists it to
storage, and may include various additional
components per use case – such as tools for
real-time processing, data manipulation, and
analytics.
b.)Streaming architectures must account for
the unique characteristics of data streams,
which tend to generate massive amounts of
data (terabytes to petabytes) that it is at best
semi-structured and requires significant pre-
processing and ETL to become useful.
c.)Stream processing is a complex challenge
rarely solved with a single database or ETL
tool – hence the need to “architect” a
solution consisting of multiple building blocks.
Part of the thinking behind Upsolver SQLake
is to replace these point products with an
integrated platform that delivers with self-
orchestrating declarative data pipelines. We’ll
demonstrate how this approach manifests
within each part of the streaming data supply
chain.
Q.3] Real-time Streaming Platforms for Big
Data ?
Ans:- Real-time analytics can keep you up-to-
date on what’s happening right now, such as
how many people are currently reading your
new blog post and whether someone just
liked your latest Facebook status. For most
use cases, real-time is a nice-to-have feature
that won’t provide any crucial insights.
However, sometimes real-time is a must.
a.)Apache Flink:-Apache Flink is an open-
source streaming platform that’s extremely
fast at complex stream processing. In fact, it’s
able to process live streams within
milliseconds because it can be programmed
to only process new, changed data as it goes
through rows of big data in real-time. In this
way, Flink easily enables the execution of
batch and stream processing at a large scale
to offer real-time insights, so it’s no wonder
this platform is known for offering low latency
and high performance.
b.)Apache Spark:-Another open-source data
processing framework that’s known for its
speed and ease of use is Spark. This platform
runs in-memory on RAM on clusters and isn’t
tied to Hadoop’s MapReduce two-stage
paradigm, which adds to its lightning-fast
performance when it comes to big data
processing. c.)Apache Samza:-Samza is an
open-source distributed stream-processing
framework that lets users build applications
that can process big data in real-time from
several sources. It’s based on Apache Kafka
and YARN, but it can also run as a standalone
library. LinkedIn originally developed Samza,
but since then, other big brands have started
using it—such as eBay, Slack, Redfin,
Optimizely, and TripAdvisor.
Q.4] What is Streaming Data Architecture?
Ans:-a.)A streaming data architecture is a
framework of software components built to
ingest, process, and analyze data streams –
typically in real time or near-real time. Rather
than writing and reading data in batches, a
streaming data architecture consumes data
immediately as it is generated, persists it to
storage, and may include various additional
components per use case – such as tools for
real-time processing, data manipulation, and
analytics.
b.)Streaming architectures must account for
the unique characteristics of data streams,
which tend to generate massive amounts of
data (terabytes to petabytes) that it is at best
semi-structured and requires significant pre-
processing and ETL to become useful.
c.)Stream processing is a complex challenge
rarely solved with a single database or ETL
tool – hence the need to “architect” a
solution consisting of multiple building blocks.
Part of the thinking behind Upsolver SQLake
is to replace these point products with an
integrated platform that delivers with self-
orchestrating declarative data pipelines. We’ll
demonstrate how this approach manifests
within each part of the streaming data supply
chain.
Q.3] Real-time Streaming Platforms for Big
Data ?
Ans:- Real-time analytics can keep you up-to-
date on what’s happening right now, such as
how many people are currently reading your
new blog post and whether someone just
liked your latest Facebook status. For most
use cases, real-time is a nice-to-have feature
that won’t provide any crucial insights.
However, sometimes real-time is a must.
a.)Apache Flink:-Apache Flink is an open-
source streaming platform that’s extremely
fast at complex stream processing. In fact, it’s
able to process live streams within
milliseconds because it can be programmed
to only process new, changed data as it goes
through rows of big data in real-time. In this
way, Flink easily enables the execution of
batch and stream processing at a large scale
to offer real-time insights, so it’s no wonder
this platform is known for offering low latency
and high performance.
b.)Apache Spark:-Another open-source data
processing framework that’s known for its
speed and ease of use is Spark. This platform
runs in-memory on RAM on clusters and isn’t
tied to Hadoop’s MapReduce two-stage
paradigm, which adds to its lightning-fast
performance when it comes to big data
processing. c.)Apache Samza:-Samza is an
open-source distributed stream-processing
framework that lets users build applications
that can process big data in real-time from
several sources. It’s based on Apache Kafka
and YARN, but it can also run as a standalone
library. LinkedIn originally developed Samza,
but since then, other big brands have started
using it—such as eBay, Slack, Redfin,
Optimizely, and TripAdvisor.
Q.4] What is Streaming Data Architecture?
Ans:-a.)A streaming data architecture is a
framework of software components built to
ingest, process, and analyze data streams –
typically in real time or near-real time. Rather
than writing and reading data in batches, a
streaming data architecture consumes data
immediately as it is generated, persists it to
storage, and may include various additional
components per use case – such as tools for
real-time processing, data manipulation, and
analytics.
b.)Streaming architectures must account for
the unique characteristics of data streams,
which tend to generate massive amounts of
data (terabytes to petabytes) that it is at best
semi-structured and requires significant pre-
processing and ETL to become useful.
c.)Stream processing is a complex challenge
rarely solved with a single database or ETL
tool – hence the need to “architect” a
solution consisting of multiple building blocks.
Part of the thinking behind Upsolver SQLake
is to replace these point products with an
integrated platform that delivers with self-
orchestrating declarative data pipelines. We’ll
demonstrate how this approach manifests
within each part of the streaming data supply
chain.
Q.3] Real-time Streaming Platforms for Big
Data ?
Ans:- Real-time analytics can keep you up-to-
date on what’s happening right now, such as
how many people are currently reading your
new blog post and whether someone just
liked your latest Facebook status. For most
use cases, real-time is a nice-to-have feature
that won’t provide any crucial insights.
However, sometimes real-time is a must.
a.)Apache Flink:-Apache Flink is an open-
source streaming platform that’s extremely
fast at complex stream processing. In fact, it’s
able to process live streams within
milliseconds because it can be programmed
to only process new, changed data as it goes
through rows of big data in real-time. In this
way, Flink easily enables the execution of
batch and stream processing at a large scale
to offer real-time insights, so it’s no wonder
this platform is known for offering low latency
and high performance.
b.)Apache Spark:-Another open-source data
processing framework that’s known for its
speed and ease of use is Spark. This platform
runs in-memory on RAM on clusters and isn’t
tied to Hadoop’s MapReduce two-stage
paradigm, which adds to its lightning-fast
performance when it comes to big data
processing. c.)Apache Samza:-Samza is an
open-source distributed stream-processing
framework that lets users build applications
that can process big data in real-time from
several sources. It’s based on Apache Kafka
and YARN, but it can also run as a standalone
library. LinkedIn originally developed Samza,
but since then, other big brands have started
using it—such as eBay, Slack, Redfin,
Optimizely, and TripAdvisor.
Q.4] What is Streaming Data Architecture?
Ans:-a.)A streaming data architecture is a
framework of software components built to
ingest, process, and analyze data streams –
typically in real time or near-real time. Rather
than writing and reading data in batches, a
streaming data architecture consumes data
immediately as it is generated, persists it to
storage, and may include various additional
components per use case – such as tools for
real-time processing, data manipulation, and
analytics.
b.)Streaming architectures must account for
the unique characteristics of data streams,
which tend to generate massive amounts of
data (terabytes to petabytes) that it is at best
semi-structured and requires significant pre-
processing and ETL to become useful.
c.)Stream processing is a complex challenge
rarely solved with a single database or ETL
tool – hence the need to “architect” a
solution consisting of multiple building blocks.
Part of the thinking behind Upsolver SQLake
is to replace these point products with an
integrated platform that delivers with self-
orchestrating declarative data pipelines. We’ll
demonstrate how this approach manifests
within each part of the streaming data supply
chain.
Q.3] Real-time Streaming Platforms for Big
Data ?
Ans:- Real-time analytics can keep you up-to-
date on what’s happening right now, such as
how many people are currently reading your
new blog post and whether someone just
liked your latest Facebook status. For most
use cases, real-time is a nice-to-have feature
that won’t provide any crucial insights.
However, sometimes real-time is a must.
a.)Apache Flink:-Apache Flink is an open-
source streaming platform that’s extremely
fast at complex stream processing. In fact, it’s
able to process live streams within
milliseconds because it can be programmed
to only process new, changed data as it goes
through rows of big data in real-time. In this
way, Flink easily enables the execution of
batch and stream processing at a large scale
to offer real-time insights, so it’s no wonder
this platform is known for offering low latency
and high performance.
b.)Apache Spark:-Another open-source data
processing framework that’s known for its
speed and ease of use is Spark. This platform
runs in-memory on RAM on clusters and isn’t
tied to Hadoop’s MapReduce two-stage
paradigm, which adds to its lightning-fast
performance when it comes to big data
processing. c.)Apache Samza:-Samza is an
open-source distributed stream-processing
framework that lets users build applications
that can process big data in real-time from
several sources. It’s based on Apache Kafka
and YARN, but it can also run as a standalone
library. LinkedIn originally developed Samza,
but since then, other big brands have started
using it—such as eBay, Slack, Redfin,
Optimizely, and TripAdvisor.
Q.4] What is Streaming Data Architecture?
Ans:-a.)A streaming data architecture is a
framework of software components built to
ingest, process, and analyze data streams –
typically in real time or near-real time. Rather
than writing and reading data in batches, a
streaming data architecture consumes data
immediately as it is generated, persists it to
storage, and may include various additional
components per use case – such as tools for
real-time processing, data manipulation, and
analytics.
b.)Streaming architectures must account for
the unique characteristics of data streams,
which tend to generate massive amounts of
data (terabytes to petabytes) that it is at best
semi-structured and requires significant pre-
processing and ETL to become useful.
c.)Stream processing is a complex challenge
rarely solved with a single database or ETL
tool – hence the need to “architect” a
solution consisting of multiple building blocks.
Part of the thinking behind Upsolver SQLake
is to replace these point products with an
integrated platform that delivers with self-
orchestrating declarative data pipelines. We’ll
demonstrate how this approach manifests
within each part of the streaming data supply
chain.