Warner DP 203 Slides

Exam DP-203: Data Engineering on
Click to edit Master

Microsoft titleCrash
Azure style
Course
Tim Warner
Tim Warner
Click to edit Master title style
• Based in Nashville, TN, US

• Central time zone
• MCT, former MVP
• Twitter: @TechTrainerTim
• Badge:
TechTrainerTim.com
Agenda
• Introduction
• Design and implement data storage (15-20%)
• Develop data processing (40-45%)
• Secure, monitor, and optimize data storage and data
processing (30–35%)
Course materials
timw.info/dp203
Course Expectations
• We'll learn by doing – at least 80 percent demo
• Case study approach
• Please review the recordings…several times!
• I’m here to answer your questions – take advantage of
this
• Use the Q&A panel
Break Schedule (Central Time Zone)
• 07:00am - Start
• 08:00am - 7-minute break
• 12:00pm - Finish
Session Recordings
Session Recordings
Session Recordings
Mobile Browser: learning.oreilly.com
O'Reilly Mobile App
Exam DP-203 Latest OD Changes
What is an Azure Data Engineer?
• Design and implement the management, monitoring,
security, and privacy of data using the full stack of data
services
• “Builds and tunes data pipelines”
• “Implements, monitors, and optimizes data platforms”
• “Has solid knowledge of SQL, Python, or Scala”
• The Azure Data Scientist consumes the data the
Engineer provides
Azure Data Engineer Associate
DP-203
DP-203
Data Engineering on Microsoft
Azure
Data Engineering on Microsoft Azure
1-year validity period

Azure Data Fundamentals
DP-900
Azure Data Scientist Associate
DP-100
Azure Data Analyst Associate
DA-100
Azure Cosmos DB Developer
DP-420
Tim'stoCertification
Click edit Master Study Model
title style
Thank you!
• Course materials: timw.info/dp203

• Twitter: @TechTrainerTim
• Pluralsight: timw.info/ps
• Web: timw.info
Data Fundamentals
Data Types
Structured Semi-structured Unstructured

Table
Data Workload Types
Online Transactional Processing (OLTP) Online Analytical Processing (OLAP)
Customer
CustomerID CustomerName CustomerPhone
Orders
OrderID CustomerID OrderDate
Data is periodically loaded,

aggregated and stored in a cube
Data is stored one transaction at a
time
Data Processing Types
Batch
Streaming
Data Processing
Data
processing
Functions Cognitive Services
Raw
Data
Databricks Other tools

Cleaned and
transformed data
ETL
Transform
Basic filtering and

transformations
Extract Load
Discard sensitive data
Azure Data Factory
Azure Stream Analytics

ELT
Load Transform
Complex
processing
Extract
Azure Data Factory
Azure Synapse
Data Analytics
On-premises data
SQL Server, Oracle, Data ingestion Data storage Data processing Data visualization
fileshares, SAP
Cloud data
Azure, AWS, GCP
SaaS data
Salesforce, Dynamics
Non-Binary Data Formats
• CSV
• Good for bandwidth-sensitive data loads
• JSON
• Clear, structured format with optional validation
Binary Data Formats
• Optimized for splitting across compute nodes
• Parquet, ORC: Columnar store
• Fast read performance (compression) for analytical
workloads
• Avro: Row-based store that includes JSON
• Schematized
• Optimized for write performance
Blob Storage and

Data Lake
Azure Blob Storage
Block blobs Page blobs Append blobs
Has a maximum size of 4.7TB Can hold up to 8TB of data The maximum size is just over
195GB
Best for storing large, discrete, Is organized as a collection of
binary objects that changes fixed sized-512 byte pages Is a block blob that is used to
infrequently optimize append operations
Used to implement virtual disk
Each individual block can store storage for virtual machines Each individual block can store
up to 100MB of data up to 4MB of data
A block blob can contain up to
50000 blocks
Blob Storage
ADLS Gen 2
A repository of data Organises data into Supports POSIX and It is compatible with
for your Modern Data directories for RBAC permissions Hadoop Distributed
Warehouse improved file access File System
Store
Azure Data Lake Storage
High performance data lake available in
all 54 Azure regions
Data Lake Storage Gen 2
Azure Data Lake Storage Gen 2
Access Tiers & Lifecycle Management
Azure SQL Products

Relational Database Tables
Customers
CustomerID CustomerName CustomerPhone Data is stored in a table
100 Muisto Linna XXX-XXX-XXXX
101 Noam Maoz XXX-XXX-XXXX
Table consists of rows and columns
102 Vanja Matkovic XXX-XXX-XXXX
103 Qamar Mounir XXX-XXX-XXXX
104 Zhenis Omar XXX-XXX-XXXX All rows have same # of columns
105 Claude Paulet XXX-XXX-XXXX
106 Alex Pettersen XXX-XXX-XXXX
Each column is defined by a datatype
107 Francis Ribeiro XXX-XXX-XXXX
ACID Principle
Normalization
Customers Orders
CustomerID CustomerName CustomerPhone OrderID CustomerName CustomerPhone
100 Muisto Linna XXX-XXX-XXXX AD100 Noam Maoz XXX-XXX-XXXX
101 Noam Maoz XXX-XXX-XXXX AD101 Noam Maoz XXX-XXX-XXXX
102 Vanja Matkovic XXX-XXX-XXXX AD102 Noam Maoz XXX-XXX-XXXX
103 Qamar Mounir XXX-XXX-XXXX AX103 Qamar Mounir XXX-XXX-XXXX
104 Zhenis Omar XXX-XXX-XXXX AS104 Qamar Mounir XXX-XXX-XXXX
105 Claude Paulet XXX-XXX-XXXX AR105 Claude Paulet XXX-XXX-XXXX
106 Alex Pettersen XXX-XXX-XXXX MK106 Muisto Linna XXX-XXX-XXXX
Data is normalized to:

Reduce storage Avoid data duplication Improve data quality
Table Relationships
Customers Orders
CustomerID CustomerName CustomerPhone OrderID CustomerID SalesPersonID
100 Muisto Linna XXX-XXX-XXXX AD100 101 200
101 Noam Maoz XXX-XXX-XXXX AD101 101 200
102 Vanja Matkovic XXX-XXX-XXXX AD102 101 200
103 Qamar Mounir XXX-XXX-XXXX AX103 103 201
104 Zhenis Omar XXX-XXX-XXXX AS104 103 201
105 Claude Paulet XXX-XXX-XXXX AR105 105 200
106 Alex Pettersen XXX-XXX-XXXX MK106 105 201
In a normalized database schema:

Primary Keys and Foreign keys are used to define No data duplication exists (other than key values in 3rd Data is retrieved by joining tables together
relationships Normal Form (3NF) in a query
SQL Statement Categories
DML DDL DCL
Data Manipulation Language Data Definition Language Data Control Language
Used to query and manipulate Used to define database Used to manage security
data objects permissions
SELECT, INSERT, UPDATE, CREATE, ALTER, DROP, GRANT, REVOKE, DENY
DELETE RENAME
Azure Synapse
PolyBase
Data Warehouse Star Schema
Data Warehouse Snowflake Schema
Azure Synapse
Azure Synapse SQL Pool (DW) Architecture
Synapse SQL Pool Types
Azure Synapse Table Distribution Modes
https://timw.info/0jl
Azure Synapse Table Distribution Modes
https://timw.info/0jl
Slowly Changing Dimensions (SCD)
Azure Databricks
Azure Databricks
Lambda Architecture
Lambda Architecture with Databricks
Kappa Architecture
Kappa Architecture with Databricks
Data Security
Network security
Securing your network from attacks and unauthorized access is an important
part of any architecture
Network security
Internet protection Firewalls DDoS protection
groups
Assess the resources that To provide inbound The Azure DDoS Protection Network Security Groups
are internet-facing, and to protection at the service protects your Azure allow you to filter network
only allow inbound and perimeter, there are applications by scrubbing traffic to and from Azure
outbound communication several choices: traffic at the Azure resources in an Azure
where necessary. Make • Azure Firewall network edge before it can virtual network. An NSG
sure you identify all • Azure Application impact your service’s can contain multiple
resources that are allowing Gateway availability inbound and outbound
inbound network traffic of security rules
• Azure Storage Firewall
any type
Identity and access
Authentication Azure Active Directory features
This is the process of establishing the
identity of a person or service looking to Single sign-on Apps & device Identity services
access a resource. Azure Active Directory Enables users to management Manage Business
is a cloud-based identity service that remember only one You can manage your to business (B2B)
provide this capability ID and one cloud and identity services
password to access on-premises apps and and Business-to-
multiple devices and Customer (B2C)
applications the access to your identity services
Authorization organizations resources
This is the process of establishing what
level of access an authenticated person
or service has. It specifies what data
they're allowed to access and what they
can do with it. Azure Active Directory
also provides this capability
Encryption
Encryption at rest Encryption on Azure
Data at rest is the data that has been
stored on a physical medium. This could Raw encryption Database encryption Encrypting secrets
be data stored on the disk of a server, Enables the Enables the encryption Azure Key Vault is a
data stored in a database, or data stored encryption of: of databases using: centralized cloud
in a storage account • Azure Storage • Transparent Data service for storing
Encryption your application
• V.M. Disks
secrets
• Disk Encryption
Encryption in transit
Data in transit is the data actively moving
from one location to another, such as
across the internet or through a private
network. Secure transfer can be handled
by several different layers
Encryption
Encryption at rest Encryption on Azure
Data at rest is the data that has been
stored on a physical medium. This could Raw encryption Database encryption Encrypting secrets
be data stored on the disk of a server, Enables the Enables the encryption Azure Key Vault is a
data stored in a database, or data stored encryption of: of databases using: centralized cloud
in a storage account • Azure Storage • Transparent Data service for storing
Encryption your application
• V.M. Disks
secrets
• Disk Encryption
Encryption in transit
Data in transit is the data actively moving
from one location to another, such as
across the internet or through a private
network. Secure transfer can be handled
by several different layers
Azure SQL Database Firewall Rules
Azure SQL Database DDM
Azure SQL Database Always Encrypted
Azure Data Factory
Power BI
What are data streams
Data streams: Data streams are used to:
In the context of analytics, data streams
are event data generated by sensors or Analyze data: Understand systems: Trigger actions:
other sources that can be analyzed by Continuously Understand component Trigger specific
another technology analyze data to or actions when
detect issues and system behavior under certain thresholds
understand or various conditions to are identified
Data stream processing approach: respond to them fuel further
There are two approaches. Reference enhancements
data is streaming data that can be of said system
collected over time and persisted in
storage as static data. In contrast,
streaming data have relatively low
storage requirements. And run
computations in sliding windows
Event processing
The process of consuming data streams, analyzing them, and deriving actionable insights
out of them is called Event Processing and has three distinct components:
Examples include sensors or processes that generate data continuously such as a

Event producer heart rate monitor or a highway toll lane sensor
An engine to consume event data streams and deriving insights from them.
Depending on the problem space, event processors either process one incoming
Event processor event at a time (such as a heart rate monitor) or process multiple events at a time
(such as a highway toll lane sensor)
An application which consumes the data and takes specific action based on the
Event consumer insights. Examples of event consumers include alert generation, dashboards, or even
sending data to another event processing engine
Processing events with Azure Stream
Analytics
Microsoft Azure Stream Analytics is an event processing engine. It enables the consumption
and analysis of high volumes of streaming data in real time
Source Ingestion Analytical engine Destination
Sensors Event Hubs Stream Analytics Query Azure Data Lake

Language
Systems IoT Hubs Cosmos DB
.NET SDK
Applications Azure Blob Store SQL Database
Blob Store
Power BI
Create an Event Hub
Create an event hub namespace Create an event hub
1. In the Azure portal, select NEW, type 1. After the deployment is complete, click the xx-name-eh event hub on the dashboard
Event Hubs, and then select Event Hubs
from the resulting search. Then select 2. Then, under Entities, select Event Hubs
Create 3. To create the event hub, select the + Event Hub button. Provide the name socialstudy-eh,
2. Provide a name for the event hub, and and then select Create
then create a resource group. Specify xx- 4. To grant access to the event hub, we need to create a shared access policy. Select the socialstudy-eh
name-eh and xx-name-rg respectively, event hub when it appears, and then, under Settings, select Shared access policies
XX- represent your initials to ensure
uniqueness of the Event Hub name and 5. Under Shared access policies, create a policy with MANAGE permissions by selecting + Add. Give the
Resource policy the name of xx-name-eh-sap, check MANAGE, and then select Create
Group name 6. Select your new policy after it has been created, and then select the copy button for the
3. Click the checkbox to Pin to the CONNECTION STRING – PRIMARY KEY entity
dashboard, then select the Create 7. Paste the CONNECTION STRING – PRIMARY KEY entity into Notepad, this is needed later in the
button exercise
8. Leave all windows open
Azure Stream Analytics workflow
Complex event processing of Stream Data in Azure
Input Complex Event Output

Adapter Processor Adapter
Start a Stream Analytics Job
Azure Data Factory components
Consumes Activity Pipeline
Data set
(e.g. hive, stored proc., (Schedule, monitor,
(e.g. table, file) Is a logical
Produces copy ) grouping of manage)
Represents a data
item(s) stored in
Linked service Runs on

(e.g. SQL Server, Hadoop
Cluster)
Control flow Parameters Integration runtime

Azure Data Factory components
Linked Service Triggers
@ Parameters
IR Integration
Runtime
Data
Pipeline Control
Lake Store CF
Flow
1
Activities
Azure
Databricks
Dataset
Azure Monitor
Data Pipelines
Azure Diagnostics
Log Analytics
Lambda architectures from a real time
mode perspective
Speed Layer:
The Speed layer processes data streams in
real or near real time. This works well when
the aim is to minimize the latency of the
data ingestion to analysis:
1. New data ingested from sources
4. Real time views of the data created
Serving Layer:
The serving layer is optional in the
real-time architecture and acts as the
storage output of either the Batch or Speed
layer that is used by client applications to
access the results
of the data-sets
Architect a stream processing pipeline
with Azure Stream Analytics
Design a stream processing pipeline
with Azure Databricks
Automate an enterprise business
intelligence architecture
Exam DP-203
Item Types
Multiple Choice
Multiple Choice
Repeated Scenario
You need to move an Azure VM to another hardware
host.
Solution: You redeploy the VM.
Does this solution meet the goal?
a. Yes
b.No
Repeated Scenario
You need to move an Azure VM to another hardware
host.
Solution: You create a proximity placement group.
Does this solution meet the goal?
a. Yes
b.No
Select and Place
Build List and Reorder
Active Screen
Case Study
Performance-Based Lab
Microsoft Online Testing

Microsoft Online Testing Process
timw.info/online

Warner DP 203 Slides

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Warner DP 203 Slides

Uploaded by

Copyright:

Available Formats

Exam DP-203: Data Engineering on

Click to edit Master

• Based in Nashville, TN, US

Data Engineering on Microsoft Azure

1-year validity period

• Course materials: timw.info/dp203

Structured Semi-structured Unstructured

Data is periodically loaded,

Databricks Other tools

Basic filtering and

Discard sensitive data

Azure Data Factory

Azure Stream Analytics

Azure Data Factory

Blob Storage and

Azure SQL Products

Data is normalized to:

In a normalized database schema:

Examples include sensors or processes that generate data continuously such as a

Source Ingestion Analytical engine Destination

Sensors Event Hubs Stream Analytics Query Azure Data Lake

Input Complex Event Output

Linked service Runs on

Control flow Parameters Integration runtime

Solution: You redeploy the VM.

Does this solution meet the goal?

Solution: You create a proximity placement group.

Does this solution meet the goal?

Microsoft Online Testing

You might also like