Rayis Imayev

June 20th, 2019

Continuous Integration and Delivery (CI/CD)

in Azure Data Factory
"Coming together is a beginning,
staying together is progress,
and working together is success.“

Henry Ford
Continuous Integration and Delivery (CI/CD) in Azure Data Factory

Session plan:
o Introduction to the Azure Data Factory (ADF)
o Continuous Integration and Delivery (CI/CD) overview
o ADF pipeline case study
o Merging ADF with CI/CD
o ADF integration with a GitHub repository
o Azure DevOps review
o Creating integration of GitHub based ADF code and DevOps pipelines
o ADF pipeline amendments and affect on DevOps pipeline
o Conclusion
Introduction to the Azure Data Factory (ADF)
Cloud platform to compose data-driven workflows with
steps to move and transform data.

 Control Flow
 Pipeline
 Activity
 Datasets
 Linked Services
 Triggers

 Mapping Data Flows

 Source/Sink
 Transformation Tasks

Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev
Continuous Integration and Delivery overview
CI/CD process helps to establish a good
software development practice and
aims to build a healthy relationship
between development, quality
assurance, and other supporting teams.

1. Develop code
2. Unit test
3. Integrate
4. Acceptance test
5. Deploy to production

Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev
ADF pipeline case study (Demo)
Sample ADF pipeline

Copy all files across multiple containers

from one file-based storage store to
another.

Activities:
- Get Metadata
- For Each Loop container
- Copy Data
- … more items

Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev
Merging ADF with CI/CD
Questions:

1. Can multiple people work within the same ADF?

2. Can I integrate my ADF changes into other people's ADF code?
3. Can I have a process to frequently ship the ADF solution code to various environments?
4. Can a continuous deployment be automated to a production environment?

Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev
ADF integration with a GitHub repository (Demo)

1. Setting a GitHub Repository for your Azure Data Factory

2. Saving your ADF code in a GitHub repository
3. Adding changes to the existing ADF code
4. Publishing changes to a live Data Factory and creating ARM templates

http://datanrg.blogspot.com/2019/02/azure-data-factory-integration-with.html

Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev
Azure DevOps review
Plan, track, and discuss work across your teams.

Collaborate on code development using free Git public

and private repositories, pull requests, and code review.

Azure Pipelines helps you implement a build, test, and

deployment pipeline for any app.

Using testing services for your apps.

Code once and share packages across your organization

Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev
Creating DevOps project to deploy ADF code (Demo)

1. New DevOps project

2. New Release Pipeline
3. Variable groups for my deployment solution
4. Adding Git artifact for the deployment
5. Deployment stages and deployment steps within each stage
6. Release pipeline variables
7. Testing a deployment release

http://datanrg.blogspot.com/2019/02/continuous-integration-and-delivery.html

Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev
ADF code change and its affect in DevOps pipeline (Demo)

1. Changing ADF code

2. Merging the ADF code to your GitHub repository
3. Publishing / Creaing ARM templates to deploy ADF code
4. Monitoring ADF code deployment
5. Validating code changes in other environments

http://datanrg.blogspot.com/2019/02/continuous-integration-and-delivery.html

Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev
Continuous Integration and Delivery (CI/CD) in Azure Data Factory

Session plan:
o Introduction to the Azure Data Factory (ADF)
o Continuous Integration and Delivery (CI/CD) overview
o ADF pipeline case study
o Merging ADF with CI/CD
o ADF integration with a GitHub repository
o Azure DevOps review
o Creating integration of GitHub based ADF code and DevOps pipelines
o ADF pipeline amendments and affect on DevOps pipeline
o Conclusion
Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev
Conclusion:
1) ADF code deployment could be automated with the help of Azure DevOps.
2) Don’t forget to include Disable/Enable deployment steps if your ADF has triggers.
3) Make sure to use Azure Key Vault to store your secrets and reuse them during deployment.

Links:
http://datanrg.blogspot.com/2019/02/continuous-integration-and-delivery.html
https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment
https://docs.microsoft.com/en-us/azure/devops/learn/what-is-devops

Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev
 Thank you for attending my session!

Rayis Imayev
Toronto, Canada

Consultant / Big Data Integration & Analytics

Senior BI & ETL Developer at Mosaic North America

http://datanrg.blogspot.com/
https://twitter.com/RayisImayev

Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev