CHECKLIST 2023

Migrating Your Data

Warehouse to the Cloud:
Five Best Practices
By James Kobielus
 Migrating Your Data Warehouse
to the Cloud: Five Best Practices
By James Kobielus
M
odern data runs in cloud-native
environments. Migration of legacy
on-premises data to the cloud is
becoming necessary because enterprises are facing
several critical pain points with their on-premises
deployments, including siloed, on-premises
data stores that prevent unified rollup of data
and analytics, limited technical support staff
to manage on-premises data, and scalability
constraints resulting from inflexible architectures of
on-premises data platforms.
Legacy on-premises data warehouses (DWs)
are increasingly showing their age. Over the
past decade, enterprises everywhere have been
migrating to cloud-based DWs that offer greater
scalability, performance, flexibility, and cost-
effectiveness. Typically accessed on a fully managed,
pay-as-you-go basis, the modern cloud DW can
1   TDWI RESEARCH
Five best practices for migrating your data
warehouse to the cloud:
1 Modernize enterprise data warehousing
environments around cloud-native
principles
2 Modularize enterprise data warehousing
assets for maximum reuse
3 Convert legacy enterprise data warehousing
environments
4 Automate enterprise migration to modern
environments
5 Augment enterprise teamwork of data
warehousing professionals to operationalize
migrated assets
tdwi.org
 TDWI CHECKLIST REPORT: MIGRATING YOUR DATA WAREHOUSE TO THE CLOUD: FIVE BEST PRACTICES
support both structured and unstructured data
assets. In addition to supporting conventional
ETL-based data integration, the modern cloud DW
can also provide real-time and low-latency analytics,
automated data onboarding and integration,
orchestrated data pipelines, data engineering and
preparation, self-service data delivery, embedded
analytics models, and other sophisticated features.
This TDWI Checklist shares five best practices for
migrating your enterprise DW to the cloud and
presents recommendations for aligning your DW
migrations with your overall cloud modernization
strategies and business imperatives.
Modernize enterprise
1 data warehousing
environments around
cloud-native principles
Migrating to a modern cloud-based data platform can
accelerate business transformation. It can expand
data accessibility; scale data ingestion, storage,
processing, and delivery; and help your organization
harness new data types for innovative applications.
Migrations involve transferring the capabilities,
services, and assets of legacy data platforms partially
or entirely to newer, more modern platforms. Usually,
the target platform will replace the legacy platform,
with the latter being turned off or repurposed after
the migration is complete.
When the data integration and data warehousing
environments are being transferred to a new cloud
platform, the migration project may have any or all
of the following modernization objectives:
2   TDWI RESEARCH
• Convergence. Migrate one or more DWs to
support unified data governance and deliver
a “single version of truth” to all downstream
business intelligence, reporting, and other
analytics applications.
• Enhancement. Migrate one or more DWs to a
fully managed cloud computing environment to
reduce infrastructure and operating costs, boost
performance and scalability, enable usage-based
pricing, and improve reliability and availability.
• Evolution. Migrate one or more DWs to a
cloud-based platform with the flexibility to
evolve into a larger multicloud, virtualized,
mesh, or cloud-to-edge data architecture. The
target cloud DW should be able to ingest and
store new types of structured, semistructured,
and unstructured data; support streaming,
low-latency, and real-time data applications;
and enable unification of data integration and
machine learning pipelines.
Migration is often far more than “lifting and
shifting” existing DW system footprints to the new
environment. Beyond such essential tasks as copying
data from legacy databases and loading it onto
the target platform, there are numerous details to
iron out before switching over to the target cloud
platform. These necessary migration chores
often include:
• Revising data schemas
• Rewriting stored procedures and user-defined
functions
• Tweaking query plans to boost performance
• Changing database drivers and job-scheduling
tools
tdwi.org
 TDWI CHECKLIST REPORT: MIGRATING YOUR DATA WAREHOUSE TO THE CLOUD: FIVE BEST PRACTICES
• Implementing new connectors to synchronize
data reliably between the original sources and
the new target cloud platform
The architecture of the legacy on-premises DW
platform might not be suited to take the best
advantage of cloud resources and services. To ensure
a successful DW modernization, the first steps in any
migration project should be:
• Migrate and tweak ETL code, data engineering
workflows, analytics applications, and machine-
learning model libraries to work with the new
cloud DW platform
• Identify business requirements for the DW as
critical application infrastructure
• Assess the degree to which legacy DW systems
can continue to meet these requirements
• Highlight the technology constraints, limitations,
and risk factors that can be eliminated using
modern cloud DW architectures
• Refactor reporting, analytics, data governance,
and other application requirements to align with
emerging requirements and with the capabilities
of the target cloud DW platform
Migration of a DW may also involve migration of the
associated data engineering pipeline functions to a
more modern, cloud-based data integration platform.
Here are a few key reasons enterprises might want
to undertake migration/modernization of the data
integration pipeline to the cloud in parallel with DW
migration/modernization:
3   TDWI RESEARCH
• Interoperability. Cloud data integration
pipelines optimize connectivity for API-based
cloud DWs. They enable mass ingestion from
databases, files, applications, and streaming
sources. They support data storage in
cloud platforms and broad metadata-aware
connectivity through APIs into PaaS, SaaS, and
other cloud services.
• Performance. Cloud data integration pipelines
are better suited than on-premises ETL/data
integration platforms to some use cases, such
as fast loading, staging, and processing of
data through mass ingestion and pushdown
optimization. They support elastic serverless
processing and they eliminate the need to
implement time-consuming, expensive,
and complex upgrades to the ETL, data
integration, and other capabilities. They also
support advanced pushdown optimization for
loading data from sources to cloud DWs with
transformations.
• Agility. Cloud data integration pipelines
make it easier for users to explore and try new
capabilities and services as the cloud provider
introduces them rather than requiring users
to install new software versions in their own
environments, as with on-premises ETL and
data integration.
• Accessibility. Cloud data integration pipelines
make it easier to democratize data integration
tasks to a wider range of users in a cloud-native
platform compared to on-premises ETL and data
integration tools.
tdwi.org
 TDWI CHECKLIST REPORT: MIGRATING YOUR DATA WAREHOUSE TO THE CLOUD: FIVE BEST PRACTICES
Modularize enteprise
2 data warehousing assets
for maximum reuse
Migration projects are smoother and less disruptive
if the legacy DW platform’s assets are transferred
as modular capabilities to the new modern cloud
architecture.
Modularity enables enterprises to start small with
well-defined use cases for the cloud DW migration.
Organizations can selectively and iteratively migrate
specific business domains—along with associated
data objects, integration code, ETL mappings,
stored procedures, sessions, and other DW and data
integration assets—in a fashion that minimizes
disruption to the business.
At the very least, you should migrate DW data sets
only after they’ve been made correct, consistent,
and compliant. To ensure a smoothly modular
migration of legacy DW assets, the target cloud DW
architecture should be architected in conformance
with these tenets:
• Visibility. The target cloud data environment
should support discovery, search, and
management of all assets through a searchable
data catalog. This catalog should provide a
central point of management of all DW and data
integration assets.
• Openness. The target DW should present open
computing APIs for access, programming,
manipulation, monitoring, and management of
all DW and data integration functions.
4   TDWI RESEARCH
• Programmability. The target environment
should support self-service, visual, low-code
development and operationalization of DW
capabilities as composable containerized
microservices.
• Containerization. The target environment
should enable orchestration of DW capabilities
as microservices within a containerized cloud
computing fabric.
• Elasticity. The target environment should
support elastic provisioning of computing,
storage, memory, and network resources to
support dynamic changes to DW workloads and
usage patterns.
Convert legacy
3 enterprise data
warehousing
environments
Migration may involve conversion of data,
integration code, and other artifacts to new formats
that can be more efficiently processed by the
target platform.
Conversion is one of the central requirements for
any enterprise trying to leverage its substantial
investment in data, mappings, ETL code, and other
assets from legacy DWs. Enterprises may choose
to move all or only a portion of legacy data over
to their new modern cloud DW. Whatever their
overarching business requirements may be, this
presents an opportunity to ensure that whatever data
is migrated remains trustworthy, clean, consistent,
and compliant with all relevant mandates.
tdwi.org
 TDWI CHECKLIST REPORT: MIGRATING YOUR DATA WAREHOUSE TO THE CLOUD: FIVE BEST PRACTICES
This critical task requires a keen focus on whether
the corresponding ETL logic is sufficient as-is or
needs to be converted to make it fit for both the
migration and for operationalization in the target
cloud DW. In many enterprises, ETL mappings have
been built over many years, may number in the
thousands, and certainly represent a substantial
investment of time and money. Ideally, enterprise
data professionals should be able to reuse as much of
that logic as possible instead of starting from scratch.
When undertaking a migration of DW assets to a
cloud-computing environment, enterprises should
make sure that they also have ETL and other data
integration tools that are cloud-native and can
continue to be of service after the DW has been
switched over to the target cloud environment. Using
a data integration tool that enables reuse of ETL
code and other integration logic can significantly
accelerate successful migration of an on-premises
DW to the cloud.
Enterprises should explore tools for converting and
migrating their ETL and data integration assets with
full fidelity and interoperability to a cloud-native
environment. The assets to be migrated and possibly
converted generally include data objects, metadata,
stored procedures, ETL mappings, data-engineering
workflows, integration code, analytics libraries,
parameter files, custom scripts, schedulers, and
access controls.
5   TDWI RESEARCH
Automate enterprise
4 migration to modern
environments
Migrations are faster and less prone to human
error if all necessary activities are automated to
the maximum extent possible and if management
of the target DW and data integration environments
is also automated.
Migrating a DW to the cloud is a complex project
that can be delivered on time, within budget, and
with no unforeseen glitches if the appropriate level
of automation is used in its execution. Automated
conversion can be a significant cost-saving tactic
in DW migrations, eliminating much of the need
for manual conversion or wholesale rewriting
of ETL mappings and other assets. It can also
greatly accelerate the migrations while reducing
the incidence of errors being introduced into the
migrated data integration and DW assets.
At the start of the migration, DW professionals
should estimate their likely level of effort, identify
relevant sources and targets, characterize the task
breakdowns and dependencies, and prepare a road
map under which the project will be executed in
phased sprints. This analysis should also identify the
tasks that can be automated and precisely how that
will be carried out and supervised.
Where migration automation is concerned, data
professionals should scope several tasks. Prior to
the conversion, they should automate scanning and
assessment of the scope of the assets to be migrated
from the legacy on-premises ETL and DW platforms
to that of the target platform, and preparation of
the target cloud environment to ingest, store, and
manage the assets being migrated. When automating
tdwi.org
 TDWI CHECKLIST REPORT: MIGRATING YOUR DATA WAREHOUSE TO THE CLOUD: FIVE BEST PRACTICES

conversion of assets, they should validate and test • Select appropriate ETL transformations
that the assets are being converted and/or rewritten
with full fidelity for migration and processing within • Anticipate and proactively resolve data quality
the target cloud environment. and noncompliance issues

Finally, data professionals should automate the

productionization of the assets that have been
migrated. This involves loading, organizing,
Augment enterprise
and listing converted assets in the target cloud
environment. It also entails optimization of the
5 teamwork of data
performance of the target cloud DW functions that warehousing
rely on the migrated assets and it requires use of
AI/ML-, metadata-, and rule-driven approaches to
professionals to
automate ETL and other data pipeline workloads for operationalize
consistency and repeatability.
migrated assets
At the very least, automate the following functions in
Migrations require tight coordination and complex
the target cloud platform:
dependencies among diverse data, IT, and other
stakeholders pulling together toward common
• Validate data as it’s ingested into the data
objectives.
integration pipeline

Migration of DWs from legacy on-premises platforms

• Document data integration pipeline traffic
to modern clouds requires close collaboration
patterns
among DW and data integration professionals. To be
successful, all these technical stakeholders require
• Correlate data integration pipeline events
training and skills enhancement to understand
and work with the target cloud DW, the enabling
• Identify data integration root causes
ETL and data engineering tools, and the migration
tools needed to manage conversion, movement, and
• Remediate data integration pipeline issues
optimization of the migrated assets.

• Optimize ETL and other data integration

One of the key staff augmentation strategies
pipeline processing
in DW and data integration migrations is to
enlist consultants, system integrators, and other
• Map business glossary terms to underlying
professional services partners. They may have
DW data
the expertise needed to drive these projects while
also training enterprise technical staff, identifying
• Tag and classify data stored in the DW
the best tools and platforms, and developing
standardized processes for managing it all.
• Predict source-to-target ETL mappings

6   TDWI RESEARCH tdwi.org

 TDWI CHECKLIST REPORT: MIGRATING YOUR DATA WAREHOUSE TO THE CLOUD: FIVE BEST PRACTICES
Augmentation of the technical staff’s collaborative
productivity is a key benefit of the automation
features built into DW and data integration
migration tools. When used effectively, these tools
can deliver such benefits as:
• Accelerated data integration and DW migrations
with fewer scheduling challenges
• Greater data integration and DW asset reusability
with less conversion effort
• Improved data integration and DW developer and
administrator productivity
• Faster data integration and DW code generation
with fewer errors
• Higher-quality data integration and DW code
generation, testing, and validation with less
rework
• Contextual recommendations on monitoring,
management, and troubleshooting pipeline
issues, as well as profiling, matching, merging,
cleansing, exception handling, issue resolution,
and other stewardship functions
Concluding thoughts
Migrating data from an on-premises DW to a
cloud-based DW can be a complex project with
many moving parts. When identifying the optimal
target architecture for the DW migration, your
enterprise should:
• Architect the target cloud DW so that it has the
flexibility to evolve into a more comprehensive
7   TDWI RESEARCH
cloud data platform capable of ingesting and
storing new types of structured, semistructured,
and unstructured data, and providing a scalable
platform for unification of data integration and
MLOps pipelines
• Adopt data engineering, validation, security,
protection, and governance tools optimized for
the target cloud DW
• Make necessary investments in enabling cloud
infrastructure for DW modernization, especially
AI/ML-automated data engineering pipelines and
searchable data catalogs
• Deploy a cloud DW that boosts the density and
capacity utilization of CPU, memory, storage,
and other hardware utilization on the back-end
cloud platforms
• Deliver consumable cloud DW functions at low
latency via a consistent, secure, web-native API
that can be called from any client application on
a pay-as-you-go basis
When undertaking the migration of a legacy
on-premises DW to this target cloud DW, your
enterprise should:
• Migrate ETL, data integration, and other data
engineering capabilities to a more modern
cloud-based data integration pipeline platform in
parallel with migration of the on-premises legacy
DW to a modern cloud-based architecture
• Selectively and iteratively migrate specific
business domains—along with associated data
objects, integration code, ETL mappings, stored
procedures, sessions, and other DW and data
integration assets—in a fashion that minimizes
disruption to the business
tdwi.org
 TDWI CHECKLIST REPORT: MIGRATING YOUR DATA WAREHOUSE TO THE CLOUD: FIVE BEST PRACTICES

• Adopt tools for converting data objects,

metadata, stored procedures, ETL mappings,
About our sponsor
data-engineering workflows, integration code,
analytics libraries, parameter files, custom
scripts, schedulers, access controls, and other
data integration assets with full fidelity and
snowflake.com
interoperability to a cloud-native environment
Snowflake enables every organization to mobilize
• Use AI/ML-, metadata-, and rule-driven
their data with Snowflake’s Data Cloud. Customers
approaches to automate ETL and other data
use the Data Cloud to unite siloed data, discover and
pipeline workloads for consistency and
securely share data, and execute diverse analytic
repeatability
workloads. Wherever data or users live, Snowflake
delivers a single data experience that spans multiple
• Provide DW and data integration technical
clouds and geographies. Thousands of customers
stakeholders with the necessary training and
across many industries, including 543 of the 2022
skills enhancement to understand and work
Forbes Global 2000 (G2K) as of October 31, 2022, use
with your target cloud DW, the enabling ETL and
the Snowflake Data Cloud to power their businesses.
data engineering tools, and the migration tools
needed to manage conversion, movement, and
Learn more at snowflake.com.
optimization of the migrated assets

8   TDWI RESEARCH tdwi.org

 TDWI CHECKLIST REPORT: MIGRATING YOUR DATA WAREHOUSE TO THE CLOUD: FIVE BEST PRACTICES
About the author
James Kobielus is senior director of
research for data management at
TDWI. He is a veteran industry analyst,
consultant, author, speaker, and
blogger in analytics and data
management. He focuses on advanced
analytics, artificial intelligence, and cloud
computing. Kobielus has held positions at Futurum
Research, SiliconANGLE Wikibon, Forrester
Research, Current Analysis, and the Burton Group
and also served as senior program director, product
marketing for big data analytics, for IBM, where he
was both a subject matter expert and a strategist on
thought leadership and content marketing programs
targeted at the data science community. You can
reach him by email (jkobielus@tdwi.org), on
Twitter (@jameskobielus), and on LinkedIn
(https://www.linkedin.com/in/jameskobielus/).
A Division of 1105 Media
6300 Canoga Avenue, Suite 1150
Woodland Hills, CA 91367
E info@tdwi.org
tdwi.org
9   TDWI RESEARCH
About TDWI Research
TDWI Research provides industry-leading research
and advice for data and analytics professionals
worldwide. TDWI Research focuses on modern data
management, analytics, and data science approaches
and teams up with industry thought leaders and
practitioners to deliver both broad and deep
understanding of business and technical challenges
surrounding the deployment and use of data and
analytics. TDWI Research offers in-depth research
reports, commentary, assessment, inquiry services,
and topical conferences as well as strategic planning
services to user and vendor organizations.
About TDWI Checklist Reports
TDWI Checklist Reports provide an overview of
success factors for a specific project in business
intelligence, data warehousing, analytics, or a related
data management discipline. Companies may use
this overview to get organized before beginning a
project or to identify goals and areas of improvement
for current projects.
© 2023 by TDWI, a division of 1105 Media, Inc. All rights reserved. Reproductions in whole or
part are prohibited except by written permission. Email requests or feedback to info@tdwi.org.
Product and company names mentioned herein may be trademarks and/or registered trademarks
of their respective companies. Inclusion of a vendor, product, or service in TDWI research does
not constitute an endorsement by TDWI or its management. Sponsorship of a publication should
not be construed as an endorsement of the sponsor organization or validation of its claims.
tdwi.org