Azure Data Factory V2 Demos and Monitoring

Learning Objectives
• Data Factory Demo - lookup
• Data Factory Demo - filter
• Data Factory Demo - for each activity
• Data Factory Demo - conditional split
• Monitoring
Data Factory Demo - lookup
Problem : In data Factory which activity allows performing lookup matches against
existing database records?
Solution : Lookup activity
• It can read data stored in a database or file system and pass it to subsequent copy or
transformation activities. Data obtained from ADF's Lookup activity can only be used on
an object level. In other words, you can use ADF's Lookup activity's data to determine
object names (table, file names, etc.) within the same pipeline dynamically.
• Lookup activity can read from a variety of database and file-based sources, you can find
the list of all possible data sources here.
• Lookup activity can work in two modes:
• Singleton mode - Produces first row of the related dataset
• Array mode - Produces the entire dataset
We will look into both modes of Lookup activity in this post.
Pipeline - Azure Data Factory Lookup
Activity Singleton Mode
Azure Data Factory Lookup Activity Singleton Mode

• First demo will be creating Lookup activity to read the first row of SQL query
from Source database and using it in subsequent Stored Procedure activity,
which we will be storing in a log table inside the Destination database.
• For the purpose of this exercise, we will create a pipeline PL_ControlFlow1
and view in Source database to extract all table names, using the below
query:
CREATE VIEW [dbo].[VW_TableList]
AS SELECT TABLE_SCHEMA+'.'+TABLE_NAME AS Name
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_TYPE='BASE TABLE‘
GO C:\Himanshi\
Resources\Resources\SourceDB
Azure Data Factory Lookup Activity
Singleton Mode(Cont.)
• In Destination DB . C:\Himanshi\
Resources\Resources\DestDB Sc
TableName : @activity('Lookup_AC').output.firstRow.name
Demo2 : Azure Data Factory Lookup
Activity Array Mode
• @activity('Lookup_AC').output.value
Demo – Filter Activity

• Items : @activity('Lookup_AC').output.value
• Condition : @startswith(string(item().name),'dbo.P')
Why Monitoring is Important ?
• Is there any latencies?
• Are there queues for storage transaction requests?
• Overall health of service?
• Am I going to miss any service-level agreements?
• Am I having any security issues?
• How is CPU, or our DTU, or RU utilizations
• Are there any unexpected errors.
• How my Indexes are performing? Any fragmentation?
Azure Monitor Service
Metrics Explorer
Log Analytics
Thanks