Azure Data Factory –

Loading Azure SQL Database or Azure

Data Warehouse From ADL
This document will describe how to set up an Azure Data Factory that can be used to transfer data from
Azure Data Lake Storage (ADLS) to Azure SQL Databases.

This document assumes that you have already set up the various linked services needed for your
pipeline as described in the Pre-requisites document.

Rev B – 1/17/2020
Rev A – 9/26/2019

Send feedback to MaestroTeam@microsoft.com

Contents
Before You Start..........................................................................................................................................3
Linked Services............................................................................................................................................3
Partitioned Upload......................................................................................................................................3
Before You Start
This document assumes you want to load an Azure SQL Database table or Azure SQL Warehouse table
from an Azure Data Lake source using partitioned loading operations.

Before you can set up your Data Movement pipeline you will have to complete the steps described in
our ADF Pre Requisites document. These include:

A. Creating an Azure Data Factory

B. Generating a Service Principal
C. Configuring the target database
a. Uploading the support SQL
b. Generating partition schema and partition functions
c. Creating the target table
D. Ensure you have access to the ADLA/S resources you will use
E. Create the Linked Services you need

After your data factory has been created you can generate your pipelines

Linked Services
The primary linked services your data movement pipeline will need are

 Linked Service for Structured Stream reading

 Linked Service for Data Lake Storage Access
 Linked Service for Azure SQL Database Access
 Linked Service for Blob storage

These will allow you to run an ADLS to Azure SQL data movement pipeline.

Partitioned Upload
The Data Movement process we use here leverages the partition ability of Azure SQL databases. This
requires a partition column (LoadDate) to be added to the table, and a partition schema and partition
function to be defined. Additionally, we expect certain SQL procedures and functions to be uploaded.
These operations are also described in the pre requisites document referenced above.

For more information on how partitioned tables work – we recommend reading these two web pages by
Cathrine Wilhelmsen - Table Partitioning In Sql Server and Partition Switching

[The next version of this document will contain more details on the generation process]