How Not to

Make a Mess of
Data Consolidation
Building the foundations of a data-led financial institution

criticalsoftware.com
info@criticalsoftware.com
White Paper How Not to Make a Mess of Data Consolidation

Data: the Holy

Grail for modern
businesses
All manners of organisations rely on data to carry out their
business. And as digital transformation marches on, the
volumes of data used by businesses only continues to grow.

That growth is accelerating. Data volumes created on an

annual basis have doubled in the past two and a half years [1]
and are forecast to double again in the coming three years.

149

118

94

74

59

41
33
26
15.5 18
12.5
6.5 9
5
2

2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024

Figure 1 Data volumes created annually, in zettabytes.

© Critical Software. All rights reserved.

For instance, the data volumes generated by Internet of
Things (IoT) devices and sensors are expected to grow from
13.6 zettabytes in 2019 to 79.4 zettabytes in 2025 [2].

The financial sector is no exception. On the one hand, financial

services institutions – especially banks and insurance brokers
- own a huge asset. Data is the new oil, after all! On the
other, data that isn’t managed properly, isn’t accessible or
hasn’t been properly verified can be just as useless as not
having any data at all. How can organisations ensure that
they can trust their data in a world that picks-up speed
year by year in terms of data volumes generated?

Data consolidation is one means of managing vast quantities of

data. As financial institutions realise how data can transform and
grow their business, there is a growing need to implement data
consolidation processes, bringing all of that data to one location
while simultaneously removing redundant and erroneous data.

Although it seems the perfect solution on the surface, it’s

all too easy for data consolidation projects to go wrong,
with many projects failing to achieve the primary goal
of data consolidation – making data easier to use.

References:
[1] https://www.statista.com/statistics/871513/worldwide-data-created/
[2] https://www.statista.com/statistics/1017863/worldwide-iot-connected-devices-data-size/
White Paper How Not to Make a Mess of Data Consolidation
Common risks in
data consolidation
From the outside, migration
projects look always relatively
straightforward: take the existing
system and copy it onto a new
platform. However, we soon
find that we have to deal with
cleverly designed vendor lock ins,
version incompatibilities, complex
legacy systems with a significant
number of customisations and
regulatory requirements on
reporting in the finance world.
And that’s where the problems
start. Any kind of lift and shift or
greenfield approach in migrating
and consolidating data introduces
significant business risks in terms of
business continuity. The only feasible
approach in our experience is to
© Critical Software. All rights reserved.
build a new infrastructure landscape
around the existing legacy system,
using incremental change and
replacement, so that – step-by-
step - a new system emerges while
business continuity is maintained.
Another risk that comes with
improper data consolidation is the
lack of data quality and master
data governance. To ensure
better results once migration is
completed, it is not sufficient just
to copy data from one place to
the other. To avoid bringing old
problems to a new platform, the
focus must be on enhancing data
while migrating all that in line with
a solid (master) data governance.
All this requires the involvement of
the business side and availabilities
must be ensured as misalignments
are one of the biggest threads
for data consolidation projects.
Lack of data governance and
end-user involvement during the
migration project are two big
threats to the migration itself.
Financial services organisations
must also be cognisant of the
temptation to implement a ‘big
bang’ project. Modernising too
much, too fast will automatically
lead to problems and, inevitably,
failure. The size and complexity of
deliverables raise the probability
of failure dramatically. An
agile approach with multiple
sprints (units of time in which
project tasks are performed)
reduces this risk yet also serves
to achieve modernisation and
Cloud migration goals.
The success of data consolidation
projects therefore relies on how
a given project is carried out and
how data is migrated and stored.
White Paper How Not to Make a Mess of Data Consolidation

A data decision
to make
For data consolidation to be useful, it requires careful thought and consideration as to how
data – once migrated – will be stored and used. Will data find a home in a warehouse, a
lake or a hub, and what are the differences between these? Let’s define each in turn:

DATA WAREHOUSE DATA LAKE DATA HUB

A data warehouse is typically
used to store structured data and
ensure availability for pre-defined
and standardised reporting across
all users inside the organisation.
In the context of big data, data
lakes are selected to manage
appropriate information and have
proven to be a useful architecture.
However, when using them, data
scientists are needed to carry
out the meaningful evaluation
and consolidation of the data.
This naturally has an impact on
resourcing budgets through their
need for IT skills and expertise.
A data hub is a modern data
management architecture enabling
companies to consolidate and
exchange data. It offers the
possibility to perform differentiated
data analyses and to generate
comprehensive data science
scenarios, amongst other things.
There are different technologies
that can be used when setting
up a data hub architecture, all
based on Cloud infrastructures.

© Critical Software. All rights reserved.

 Gartner research found that 57% of data and analytics leaders are investing in data warehouses,
46% are using data hubs and 39% are using data lakes. But it must be remembered that they
are not interchangeable alternatives, and their purpose must be understood and aligned with
business requirements.

DATA HUBS: THE IDEAL SOLUTION?

The intention of a data hub is to provide simplicity and to deliver up-to-date business objectives, providing
a 360-degree view of any facet of the business and allowing employees to gain access to data they want,
when they need it. The data hub is the go-to place for the core data within an enterprise. It centralises the
enterprise's critical data, usually located across different applications, and enables seamless data sharing
between diverse endpoints. It also acts as the main source of trusted data within governance initiatives.

Data hubs enable the collection of data in a highly efficient way, equipped with technology allowing
for the flexible implementation of reports and KPIs from various sources and applications. Using
data hubs can allow companies to achieve unprecedented agility, enabling quick responses to
new requirements in their individual market segments and other data relevant changes.

But while data hubs provide some answers to the question of data consolidation, and may seem simple enough,
there are still a number of risks which financial institutions face when attempting to unite data into a single source.

Finance and risk INGESTION ANALYTICS

PERSISTENCE TRANSFORMATION
application
• Data from multiple • Stores, manages • Data quality rules, • Machine learning /
Advanced sources and maintains business rules and Artificial Intelligence
business objects compliance rules /
analytics • Pre-processes
policies
• Business insights
real-time & batch • Near-raw format are generated
read many

data
• KPIs for the
Business business
intelligence • Analytics / ML / AI
jobs
write once

Operational
applications
DATA MONITORING

Data warehouse
(structure for analytics) DATA SECURITY

Data lake
((un)structured, unrefined) DATA GOVERNANCE

Figure 2 Data hub overview.

White Paper How Not to Make a Mess of Data Consolidation

How a well-defined
migration framework
can help
At Critical Software, we see data consolidation as the process
that collects all relevant data for the entire organisation, a
department or a line of business from wherever it is stored;
removes any redundancies or inconsistences; and normalises
it in order to create a safe, accurate and clear single source of
truth. This can then be exploited by standard visualisation tools
or other applications in a safe, simple and consolidated way.

The described methodological approach, referred to by its acronym

BOT, is divided into three main phases – Build, Operate and Transfer.

1) BUILD: SET THE STAGE 2.1) OPERATE: INCEPTION & ARCHITECTURE 2.2) OPERATE: DELIVERY 3) TRANSFER

Assessment Research | Analysis | Design | Certification Data analytics | Develop | Scrum | Data science | Data engineering

Customer and stakeholder

Define project vision Research

Define desirable scope of the MVP
Define project prerequisites
(Governance, team, logistics, IT)
Design the new journeys Agile teams Knowledge sharing
Design architectural solution Discover, refine, govern, orchestrate and visualise data and transfer
Identify and analyse the data sources
Design user interface prototypes (if applicable)

Architecture Technical journey

CENTRES OF EXCELLENCE

Data analytics & Business intelligence | Artificial intelligence & Machine learning | User experience & User interface | Architecture & Development | Agile & Scrum | Security | Data engineering

Figure 3 Critical Software migration framework

© Critical Software. All rights reserved.

BUILD
With data consolidation, it is
common to have a clear notion of
what is already provided and how.
Typically, we suggest an assessment
to identify major pain points and
align and merge future goals. This
helps us define a project vision and
design an MVP (minimum viable
product) in terms of technologies,
architecture and data processing.
INCEPTION & ARCHITECTURE
The analysis team will refine
and prepare the backlog to be
integrated into the following sprints.
This is a mixed team between
business and client, with a project
owner from the customer side and
a business/technical analyst from
Critical supported by business
domain experts. The technical
leader will adjust the architecture to
be adequate to new requirements.
DELIVERY
The scrum teams will iterate within
each sprint on the implementation
of the backlog previously defined.
The main task to focus on is
on data-related-areas, but
also includes documentation,
prototyping, support to key users,
training, testing and bug fixing.
When managing this process, USER EXPERIENCE
organisations can take advantage
User experience should be at the
of a series of centres of excellence
heart of any data consolidation
(CoEs), optimising the data
build. How users can access
consolidation process through
relevant data using a consolidated
implementing efficient project
platform is integral to its business
management and adopting best
success. A platform that is complex
practices in terms of building
and does not possess the most
the consolidation platform.
up-to-date data would only
Some of these include:
damage an institution’s ability to
use data to their advantage.
AGILE
Through user experience design
(UxD) techniques like mapping,
Using Agile methodologies for a
easy-to-use interfaces can be built
data consolidation project enables
which clearly label data, allowing
organisations to ensure business
users to identify with ease the
continuity whilst their existing
information they are looking for.
landscape is still operational and
can provide results as it is used
to. As we are building the new
CONSOLIDATE ELEMENT-
landscape around the existing
BY-ELEMENT
legacy systems with incremental
change and replacement,
Using Agile allows for element-
organisations will benefit from
by-element consolidation. We
an unbroken data analytics
typically recommend migrating
service while their solution
frameworks either KPI by KPI or
will be transformed gradually.
market by market, or any other
Because of that, organisations
element-focused approach. In any
will achieve an earlier return on
case, we focus on certain areas and
investment, better visibility and
consolidate them step by step.
transparency, a more adaptive
approach and higher quality. Using this approach, financial
services companies can ensure
continuous operations but
also facilitate data cleansing
and data validation during
the consolidation process. This
requires significant effort and
involvement from the business side
but ensures that no uncleansed
data is migrated and that the
consolidated KPIs and reports
deliver business-validated results.
White Paper How Not to Make a Mess of Data Consolidation
The data governance
conundrum
Yet within this process of building
a data consolidation platform is
the need to establish a series of
data governance guidelines. These
assign responsibilities and limit
freedom as they introduce guidance
for users and data processors
on how to access and utilise the
data within the framework.
Data governance aims to ensure
that a dataset is following
certain rules so that it can be
processed by all consumers. A
data governance framework is a
set of data rules, organisational
role delegations and processes
aimed at bringing everyone in the
organisation onto the same page.
Most companies already have
some form of data governance for
individual applications, business
units, or functions, even if the
© Critical Software. All rights reserved.
processes and responsibilities
are informal. In practice, data
governance is about establishing
systematic, formal control over
these processes and responsibilities.
Doing so can help organisations
remain responsive, especially as
they grow to a size at which it is
no longer efficient for individuals
to perform cross-functional
tasks. Several of the overall
benefits of data management
can only be realised after the
institution has established
systematic data governance.
Key to an effective data governance
framework is compliance. Without
rigorous adherence to rules,
data governance – and even
the entire consolidation project
itself – could be put in jeopardy.
One way to ensure this is to
achieve significant buy-in from
across the institution, rather
than making governance the sole
responsibility of project managers.
This buy-in should not come at a
late stage of the data governance
process. Arguably, the earlier
stakeholders within the institution
are able to buy into the governance
structure taking shape, the better,
as they will have more time to
adapt to new ways of handling
data while increasing their
awareness of the norms of data
handling as they are established.
How can early engagement with
stakeholders be achieved? That is
where effective communication
comes in. By outlining clearly not
only what the data governance
framework entails, but also why it
is being implemented (to comply
with data protection regulations,
for instance), members of the
organisation are much more likely
to engage with the process – not
only ensuring data is maintained
in a safe and effective way, but
also ensuring that the wider
changes brought about by data
consolidation do not fail.
 PROCESSES

(WHEN)

Develop a value Aligning policies,

1 Mission 1
statement requirements & controls

Goals
Prepare a 2 Establishing decision rights
Focus Metrics &
roadmap
WHAT 2 areas Success Measures

Funding

Plan and fund 3 Establishing accountability

Data rules and

RULES & RULES OF 3
definitions
ENGAGEMENT Design the 4 Performing stewardship
program
(WHY)
Decision
4
rights
Deploy the 5 Managing change
program

5 Accountabilities

Govern the 6 Defining data

data
Control
6
mechanisms
Monitor, measure, 7 Issue resolution
report

Data (HOW) Specifying data quality

7 8
stakeholders requirements

PEOPLE & Data governance Data governance Building governance into

8 10 9
ORGANISATIONAL BODIES office processes technology

(WHO)
Data governance Stakeholder care and
9 10
office support

Figure 4 The DGI Data Governance Framework © The Data Governance Institute
White Paper How Not to Make a Mess of Data Consolidation

Driving change with

data consolidation
The key driver for consolidating data and traditional data warehouses should be future-orientated. Ensuring a
future proof data landscape to utilise data-driven technologies in the best possible way is a key goal for data
driven companies to implement solutions around artificial intelligence (AI) and Big Data.

The previously separated worlds of analytics roles and data roles are coming together, not only impacting on
technology but also the people and processes that support and use them. Consolidation also works to optimise
business – a financial institution which has a good grasp of the data it is using is far more able to adapt to its
customers and to the market, improving its offering and subsequently increasing revenues.

Therefore, for Critical Software, business involvement is key to any data consolidation project. Any kind of
solely technical approach for a data consolidation project is bound to fail. Adopting a holistic, business-led view
of a project allows organisations to embrace new data services while ensuring any legacy systems that need to
be maintained are migrated successfully.

At Critical Software, we see data consolidation as the process of putting an organisation’s data in an
integrated environment, delivering clear business benefits in the form of:

• Immediate access to relevant and correct data.

• Increased efficiency due to consolidated data.

• Reduced operational expenses.

• Better alignment and compliance with data protection laws.

• Better compliance with regulatory requirements, e.g. ECB or BaFIN.

Critical Software has been involved in various data consolidation projects and our migration framework,
combined with agile methodologies, has ensured their development efficacy and efficiency. Most importantly,
however, our approach has ensured business involvement throughout every phase of the project – avoiding any
messy communications and keeping the project on-track.

© Critical Software. All rights reserved.

To find out more about our
work, please get in touch:
info@criticalsoftware.com
We are CMMI Maturity Level 5 rated.
For a list of our certifications & standards 
visit our website.

criticalsoftware.com
info@criticalsoftware.com