Data Warehouse

In our days, organizations are turning to cloud-based technologies for convenient

data collection, reporting, and analysis. This is how Data Warehouse occurred, to
serve many purposes and help improving business intelligence. Since it has this
importance, it is important to understand what data warehouse is and why it is
evolving in the global marketplace.

In this article, we’ll provide an overview of Data Warehouse – explore key

concepts like data warehouse architecture, characteristics of data warehouse, Data
management, the benefits of data warehouse, and data warehouse applications.

Data Warehouse Definition

First, we need to define the Data Warehouse term:

It is to collect the data, organize it, and manage it from different data sources to
provide commercial and financial predictions or views that benefits the people
involved.

Depending on the definition of data warehouse, it is the central data warehouse file
of the stored data that is collected from a varied set of internal and external
sources.

It is a technique that combines structured, unstructured, and semi-structured data

from multiple sources to present a show for the analysts and the users to improve
the business intelligence.
So, it is usually used for the analytical and financial reports, and that helps keeping
the previous structs and facilitates analyzing the data to improve the commercial
procedures.

It is important to know the differences between databases and data warehouses,

such that people usually get lost between these two terms. Data base is a traditional
way of storing data, whereas data warehouse is a separate class of data structures
designed specially to analyze data, as it stores everything in one place instead of
many external individual data banks as we mentioned.

Notation about the difference between databases and data warehouses:

Databases and Data Warehouses are both databases that contain tables, indexes,
primary keys, queries, etc., but the essential difference between them is that
databases are designed to store and organize data, while data warehouses are
designed to store and analyze data where data is collected from different databases
and reorganized. This allows analyzing it and extracting important information
from it to help in making decisions. Therefore, data warehouses (data warehouses)
are used to store huge amounts of data for long periods, and these data can be
updated but cannot be modified because they are used only for analysis and study.
A database is a transactional system that monitors data in real time and updates it
in order to get only the most recent data.

The data warehouse is programmed to collect structured data over time.

For example, a database may contain only the most recent address of a customer,
while a data warehouse may contain all addresses of a customer over the past 10
years.
Explanation

Before getting into knowing the special methods of using this type of data
management we need to ensure that the automated data has been modeled correctly
in total. And that to provide a complete and spanning data model on the level of its
revising and computing.

Working on improving and providing the capabilities and abilities of the technique
related to modeling and designing data. Which suits the special requirements of
modeling with it.

And to ensure that specialists in data modeling and experts (in areas of business
closely related to data) are selected, while entrusting them with the responsibility
of ensuring the validity and comprehensiveness of outputs from models and
designs for data, and that working on modeling all data with a specialized authority
in the field of modeling in light of whether it is structured or unstructured, Create a
unified central environment to save and manage the outputs of the modeling and
design processes for its data, provided that it includes all the outputs of the
modeling and design during any stage of the modeling and design life cycle stage
for example:

1- creating a new design model

2- a check-in
3- Approved
4- Published

considering that the management of Lifecycle modeling and design with the help
of a workflow management system.
And it must provide an environment for centralizing search capabilities and
managing the use of data models and designs for authorized users to enter the
facilities authorized through this complex system.

Appropriate data modeling and design tools should also be used, bearing in mind
that the data models may be moved to a unified central custodian environment to
manage the modeling and design output of the proprietary data if this is required in
the future.

Accordingly, government agencies must work to adhere to standard modeling and

design methodologies and to adopt their data models and designs as an additional
reference for all initiatives related to the development of information systems
during all stages of design, development, and testing of those systems.
Evolution of data warehouses from data analytics to artificial intelligence and
machine learning

When data warehouses first appeared in the late 1980s, their purpose was to aid the
flow of data, from operating systems to decision support systems (DSSs). These
raw data warehouses required a huge amount of redundancy. Most organizations
had multiple Decision Support Systems (DSS) environments serving different
users. Although Decision Support Systems (DSS) environments use much of the
same data, data collection, erasure, and integration has often been replicated in
each environment.

As data warehouses become more efficient, they have emerged from information
stores that have supported traditional business intelligence (BI) platforms in
extensive analytics infrastructures that support a wide range of applications, such
as operational analytics and performance management.

Data warehouse iterations have advanced over time to provide incremental

additional value to the organization.

Supporting each of these five steps requires a growing and diverse set of data sets.
The last three steps in particular create a necessity for a broader set of data and
analytical capabilities.

Today, artificial intelligence and machine learning are transforming every asset of
industry, services, and organizations—and data warehouses are no exception. The
expansion of big data and the application of new digital technologies are changing
the requirements and capabilities of data warehouses.

The Autonomous Data Warehouse is the final step in this evolution, offering
companies the ability to extract more value from their data while reducing costs
and improving data warehouse reliability and performance.

Learn more about Autonomous Data Warehouses, then start using your own
Autonomous Data Warehouse.

What is a cloud data warehouse?

A cloud data warehouse uses the cloud to ingest and store data from different data
sources.

The original data repositories were built on local servers. These local data
repositories still have many advantages today. In some cases, it can provide
improved speed, security, and governance features. But local data warehouses are
inflexible and require complex forecasting to determine how to scale the data
warehouse to meet future needs. Managing these data warehouses can also be very
complex.

A cloud data warehouse uses the cloud to ingest and store data from different data
sources.
On the other hand, some of the advantages of cloud data warehouses include:

Flexibility, with separate computing and storage services

Expandable capabilities to address computing or storage requirements

Ease of use

Ease of management

Cost savings

The best cloud data warehouses are fully managed and self-directed, ensuring that
even beginners can build and use a data warehouse with just a few clicks. In
addition, most cloud data warehouses follow a pay-as-you-go model, which brings
additional cost savings to customers.

What is a modern data warehouse?

Whether a data warehouse is part of an IT, data engineering, business analytics, or

data science team, different users across the organization have different needs for a
data warehouse.

Modern data architecture meets these different needs by providing a way to

manage all types of data, workloads, and analysis. It consists of architecture
patterns with the necessary components integrated to work together in line with
industry best practices. Modern data warehouse includes:
 A converged database that simplifies the management of all types of data and
provides different ways of using data

Self-service data ingestion and transfer services

Support for SQL, machine learning, graph and spatial processing

Multiple analytics options make it easy to use data without moving it

Automated management of supply and expansion services and simple

management

A modern data warehouse can simplify data workflows in a way that other
repositories cannot. This means that everyone, from analysts and data engineers to
data scientists and IT teams, can do their jobs more effectively and pursue
innovative work that moves the organization forward, without infinite delays and
complexity.

Data Architecture:

To ensure that all components and parts of its data architecture are compatible with
the recognized standard patterns and in line with the principles of the subject
structure architecture applicable to companies specialized in building structures.

Therefore, the competent authority working to secure the capabilities and

capabilities of the data architecture, (supported by providing a preservation
environment for data architecture designs) should document its current data
architecture, and to define and formulate a roadmap to reach the target data
architecture based on the relevant standard patterns models that have been
formulated by the administration (and therefore all documents prepared for this
purpose must be included) among the details of the current data architecture,
design procedures and outputs wherever they exist.

In addition to sharing the outputs of the design, modernization and embedding

stages with the design team assigned within the statement management program
for the purpose of controlling them and ensuring their compatibility with the
design rules at the general level, and thus it can be shared and reused by any parties
that have a purpose in these innovative and modern effective systems.

The structure of the data warehouse is determined by the specific needs of the
organization. The common structures are characterized by the following:

Ease:

All data warehouses share a basic design in which metadata, summary data, and
raw data are stored within the central repository of the data warehouse. The
repository is fed with data sources on one side and accessed by users for analytics,
reporting and data collection on the other.

Ease with a staged operation area:

Operational data must be erased and processed before it is placed in the

warehouse. Although this can be done programmatically, many data warehouses
add a staging area for data before it enters the repository, to simplify data
preparation.

Coaxial system:

Adding data stores between the central repository and users allows an organization
to customize its data warehouse to serve different types of businesses. When the
data is ready for use, it is transferred to the appropriate data store.

isolated test environments:

Isolated test environments are private, tight, and secure areas that allow companies
to quickly and informally explore new data sets or data analysis methods, without
having to adapt or comply with formal data warehouse rules and protocol.

Types of Data Warehouse (DWH)

There are three main types of Data Warehouses (DWH) that are mainly used in
enterprise systems:

1- Enterprise Data Warehouse (EDW): As a central data warehouse, EDW

provides a holistic approach to organizing and presenting data.
 2- Operational Data Store (ODS): ODS is a suitable data store when neither
OLTP nor DWH can support business reporting requirements.
3- Data Mart: Data Market is designed for departmental data, such as sales,
finance, and supply chain.

Benefits for organizations

Now that we know what a data warehouse means and how it works, it's time to
learn about the benefits of data warehouses and exactly how they can help your
business grow and expand. Whether you own a digital marketing agency or have a
traditional setup, data warehousing can bring many benefits to your business.

In the modern, fast-paced world of intense competition, your ability as a business

to make polished decisions quickly is essential to outsmarting your opponents.
1. Save time:

DWH gives you access to all required data within minutes, so you and your
employees don't have to fear a deadline approaching. All you have to do is publish
your data model to get the data in a matter of seconds. Most storage solutions
allow you to do this without using a complex query or machine learning.

With data storage, your business won't have to rely on the availability of a 24/7
technical expert to troubleshoot issues associated with information retrieval. In this
way, you can save a lot of time.

2. Improves data quality:

Improved data quality helps ensure that your company's policies are based on
accurate information about your company's efforts.

By understanding the meaning of data storage, you can transform data from
multiple sources into a common arrangement. Thus, you can guarantee the
reliability and quality of your company's data. In this way, you can identify and
remove copied data, poorly logged data, and any other errors.

Implementing a data quality management and data integrity improvement program

can be costly and daunting for your business. You can easily use a data warehouse
to eliminate a number of these inconveniences while saving money and boosting
the overall efficiency of your organization.
3. Improves business intelligence:

You can use a data warehouse to collect, ingest, and extract data from any source
and set up a process to benefit from business analytics. As a result, your business
intelligence will improve by leaps and bounds, because it is possible to effortlessly
integrate data from distinct sources.

Let's face it: checking multiple data banks can be difficult and, at times,
inconvenient. But, with a data warehouse, everyone on your team can have an
integrated understanding of all relevant information at the right time.
EDW allows your sales and marketing teams to track and identify your dynamic
targets that have accounts on social networking sites. So, if you are running a
promotion targeting females in your mid-20s and working in the beauty industry,
your team can fetch profiles for your target audience using your data lake in a
matter of seconds. They won't even have to review worksheets and data banks.
4. Leads to data consistency:

Another important benefit of using centralized data stores is the parity of big data.
Your company could benefit from data warehousing or a data market with a similar
arrangement. Since data warehousing stores large amounts of data from various
sources, such as a transaction system, in a consistent manner, each source will
generate results that are synchronized with other sources.

This ensures improved quality and consistency of data. Thus, you and your team
can feel reassured that your data is correct, which leads to more informed
organizational decisions.
5. Boosts return of investment (ROI) :

According to a report by the International Data Corporation (IDC), using a data

warehouse generates an average 5-year ROI of 112 percent with an average
payback period of 1.6 years.

It enables you to increase your overall return on investment by leveraging the value
and business acumen cultivated in several data banks. As you increasingly use
standardized and structured information within the central store, you are making
more of your investment.

Thus, you can clarify, enumerate and verify the efficiency of your initiatives to the
top management in terms of improving ROI.

6. Stores historical data:

Since the data warehouse allows you to store large amounts of historical data from
databases, you can easily check the different time stages and tendencies that can be
a leader for your company. Thus, with the right real-time data in your hands, you
can make superior corporate decisions regarding your business strategies.

Moreover, forecasting the outcome of your business operations is an important

aspect of being a resourceful entrepreneur. It can be difficult to predict the future
without a concrete understanding of your historical achievements and failures.
For example, suppose you own a fashion brand. You are planning to launch a
promotional campaign for your new clothing line. Creating a central repository
allows you to access and analyze historical data from your past campaigns in order
to determine which approach has worked best, and how you can emulate it in
upcoming promotions.

You cannot expect to store and analyze this extensive historical data in any
traditional data bank. Thus, using EDW gives you an edge in your business
procedures.

7. Increases data security:

Did you know that data-related intricacies cost more businesses more than a
whopping $5 million each year?

But with data storage, you can save yourself the additional data security hassles.

As a company that deals with customer information regularly, your first and
foremost priority is to protect the information of current and potential consumers.
Hence, to avoid all future inconveniences, you have to take all necessary measures
to escape data breaches. With a storage solution, you can keep and protect all your
data sources. This will greatly reduce the risk of a data breach.

Data Warehouse enables security enhancement by introducing advanced security

features built into its setup. Consumer information is a valuable resource for any
company. But once safety becomes an issue, this information becomes the main
burden on you.
These are just some of the advantages that data storage should offer your business.
It provides you with enhanced business intelligence, powerful decision support,
superior business practices, and powerful analytics processing.

8. data cleansing:

Many companies use data storage for the purpose of leveraging historical data for
critical business decisions. Hence, it is necessary to ensure that only high-quality
data is loaded into the data warehouse. This can be done by making data
purification a part of the data storage process, where the data is purified, selected
and then stored. So data purification can help detect and remove invalid,
incomplete or outdated records from source datasets.

9. Data conversion and upload:

Data conversion involves modifying the data to a format compatible with the target
system, such as a database, to simplify data loading.

Many data warehouse management tools offer built-in transformations, such as

grouping, searching, joining, and filtering, (and these features are what makes them
superior and in demand in our time) making data processing easier to simplify the
step of data integration in a data warehouse. It also guarantees savings on money,
time, effort and storage space.
Now that we learned about the data warehouse benefits we need to choose data
warehouse tools.

How do you choose data warehouse tools?

A properly configured and built-in data warehouse architecture is indispensable for

a data-driven business. To perform queries and perform multifaceted analytics, you
need a powerful data warehouse design tool so that different teams across the
organization can easily access and use the data.

However, choosing a data warehouse software tool that fits all your business
requirements needs careful consideration. After all, switching from one DWH tool
to another can be tedious and turbulent. Therefore, the more you think about your
choice, the easier it will be for you to make things for yourself in the future.

Here are five key factors to consider when choosing a storage platform:

1. The cloud versus the on-premise:

When choosing a data warehousing software tool, the first point to consider is to
go for a cloud or on-premises data warehousing software. If you are looking for
low cost data warehouse software without servers and hardware and lower
maintenance costs, you should look for a cloud based data warehouse.

Conversely, if data security is a priority for your business, an on-premises data

warehouse architecture may be the right way to go, as it gives you complete
control over information security and access. Furthermore, on-premises data
warehouse solutions generally provide higher speed than cloud deployment
alternatives due to lower chances of latency issues.
2. performance:

When it comes to performance, access and processing speed are two important
considerations for any data storage tool. While searching, ask yourself which data
warehouse management tool will give you faster query performance? How fast can
data be extracted from source systems and loaded into destination systems? What
tool will help your data warehouse architecture maintain optimal performance?

Data integration tools in data warehousing offer various levels of performance

depending on how they are structured. To maintain optimal performance of your
data warehouse, use a tool that ensures that your data is thoroughly cleaned,
deduplicated, transformed, and loaded.

Also, choose a data warehouse software tool that supports frequently used source
data formats and target data structures. This will allow you to access various data
sets to quickly make a timely decision.

3. scalability:

If your company is rapidly expanding, you will want to choose a data warehouse
analytics tool that scales your business. For example, choose a tool that provides
fast and seamless batch sizing without constant monitoring to ensure compliance
with data set requirements.

You can specify the scalability of different data integration tools for data storage in
terms of cost, resource and simplicity. Some tools need more maintenance but are
cost effective. Similarly, you will find some DWH tools that are scalable
horizontally, which means that they provide optimal performance even if you add
more nodes to your data warehouse. Also, if these tools are properly optimized,
they can be relatively economical.

4. Automation capabilities:

The traditional approach to data storage has been replaced by an automated

alternative to meet the growing needs of data volume and enable faster time to
access information. DWA tools automate the repetitive steps involved in data
warehouse design, development, and deployment. To ensure that data is loaded
error-free into the data warehouse, the selected data store must be able to directly
automate the data cleaning process from defining the attributes of the source data
to validating it before it is loaded into the data warehouse.

Unlike traditional data warehousing tools, modern tools support workflow

automation and data model design patterns, such as Vault, Enmon and Kimball. It
provides automation at every step, from designing the data warehouse to mapping
and generating ETL code for loading information into the data warehouse. By
simplifying the process, modern data warehousing tools can drastically reduce the
time, expense, and risk of data warehousing projects.

5. Integrations:

Business expansion typically involves the integration of diverse data sources, such
as cloud sources, in-memory formats, and databases, resulting in growing
heterogeneous data volumes. It is necessary to define a DWH tool that can
integrate data from different applications and information systems in such a
scenario.
Data Warehouse Architectures
Independent Data Mart:

(Data marts: Mini-warehouses, limited in scope)

Separate ETL for each independent data mart

Data access complexity due to multiple data marts

Dependent Data Mart and Operational Data Store:

ODS provides option for obtaining current data

Single ETL for enterprise data warehouse (EDW)

Dependent data marts loaded from EDW

Logical Data Mart and Real-Time Data Warehouse:

ODS and data warehouse are one and the same

Near real-time ETL for Data Warehouse

Data marts are NOT separate databases, but logical views of the data warehouse

Easier to create new data marts

Three-Layer architecture
Data warehouse architecture types

The architecture of a data warehouse determines the arrangement of data in

different databases. Since data must be organized and cleaned to be of value, the
modern data warehouse architecture defines the most efficient method for
extracting information from raw data. Using a dimensional model, the raw data in
the staging area is extracted and transformed into a simple consumable storage
structure to provide valuable business information. Furthermore, unlike a cloud
data warehouse, the traditional data warehouse model requires on-premises
servers for all components of the warehouses to function.

When designing a corporate data warehouse, there are three different types of
models to consider:

Single-tier data warehouse

The single-tier data warehouse architecture structure produces a dense set of

data and reduces the volume of deposited data. Although useful for eliminating
redundancy, this type of warehouse design is not suitable for companies with
complex data requirements and many data flows. This is where multi-tiered data
warehouse architectures come in because they handle the most complex data
flows.

Two-tier data warehouse

In comparison, the data structure of the two-tier data warehouse model divides
tangible data sources from the same warehouse. In contrast to a single layer, a
two-tiered design uses a database system and server.
Small organizations where the server is used as a data market typically use this
type of data warehouse architecture. Although it is more efficient at storing and
organizing data, the two-tier architecture is not scalable. Moreover, it only
supports a token number of users.

Three-tiered data warehouse

The three-tier data warehouse architecture type is the most popular type for
modern DWH design because it produces a well-structured flow of data from raw
information to valuable insights.

The bottom layer in a data warehouse model usually consists of a data bank
server that creates an abstraction layer on data from many sources, such as
transaction data banks used for front-end uses.

The middle layer includes an Online Analytical Processing (OLAP) server. This level
changes the data to an arrangement more convenient for analysis and
multifaceted investigation from the user's perspective. Since it has an OLAP server
pre-built into the architecture, we can also call it an OLAP-focused data
warehouse.

The third and higher level is the client level that includes the tools and application
programming interface (API) used for high-level data analysis, query, and
reporting. However, people hardly include the fourth level in the data warehouse
architecture because it is often not considered an integral part of the other three
types.

The DW diagram below shows the three layers of a data warehouse:

As explained in more detail in the data warehouse diagram, these are the
different types of traditional data warehouse architecture. Now, let's learn about
the main components of a Data Warehouse (DWH) and how they help in building
and extending a Data Warehouse in detail.
The main components of DWH architecture
The different layers of a data warehouse or components in a DWH architecture
are:
1. data warehouse database

The central component of the DW architecture is a database that stores all

enterprise data and makes it manageable for reporting. This obviously
means that you need to choose the type of database you will use to store
the data in your repository.

Here are the four database types you can use:

Typical relational databases are the row-centric databases that you

probably use on a daily basis - for example, Microsoft SQL Server, SAP,
Oracle, IBM DB2.

Analytics databases are meticulously developed to store data to maintain

and manage analytics, such as Teradata and Greenplum.

Data warehouse applications are not exactly storage databases, but many
merchants now offer applications that offer data management software as
well as data storage hardware. For example, SAP Hana, Oracle Exadata, and
IBM Netezza.

Cloud-based databases can be hosted and retrieved on the cloud so you

don't have to buy any hardware to set up your data warehouse - eg
Amazon Redshift, Google BigQuery, Microsoft Azure SQL.
 2. Extract, Transform, and Load (ETL) Tools
ETL tools are central components of enterprise data warehouse design.
These tools help to extract data from various sources, transform it into a
suitable order, and load it into a data warehouse.
The ETL tool you choose will determine:
Time taken to extract data
Data mining approach
The type of transitions applied and the simplicity to do so
Define a business rule for data validation and purification to
improve end product analytics
Fill in misleading information
Determine the distribution of information from the primary
repository to your BI applications

3. Metadata
In the DW architecture, metadata describes the data warehouse database and
provides a framework for the data. It helps in creating, saving, processing and
making use of the data warehouse.

There are two types of metadata in data storage:

Technical metadata consists of information that developers and managers can use
when performing repository development and management tasks.
Business metadata includes information that provides an easy-to-understand
view of the data stored in the repository.
Metadata plays an important role for companies and technical teams to
understand the data in the repository and turn it into information.

Your data warehouse isn't a project, it's a process. To make your implementation
as efficient as possible, you need to take a really agile approach, which entails
having a data warehouse architecture based on metadata.
This is a visual approach to data warehousing that takes advantage of metadata-
rich data models to drive every aspect of the development process from
documenting source systems to copying schemas into a physical database and
facilitating mapping from source to destination.
The data warehouse schema is at the metadata level, which means you don't
have to worry about the quality of the code and how you'll encounter large
amounts of data. In fact, you can manage and control your data without getting
into the code.

Also, you can test data warehouse models concurrently before publishing and
copy your schema into any pilot database. The metadata-driven approach leads to
an iterative development culture and future development of your data
warehouse deployment, so that you can update your existing infrastructure with
new requirements without compromising the integrity and usability of your data
warehouse.

Combined with automation capabilities, a metadata-based data warehouse

design can simplify design, development, and deployment, leading to a robust
data warehouse implementation.
4. Data warehouse access tools

A data warehouse uses a database or a set of databases as a basis. Data

warehouse companies generally cannot work with databases without the
use of tools unless they have database administrators available. However,
this is not the case with all business units. This is why they use help from
many no-code data storage tools, such as:

Inquiry and reporting tools help users produce corporate reports for
analysis which can be in the form of spreadsheets, accounts or interactive
visuals.
Application development tools to assist in creating custom reports and
presenting them in custom interpretations for reporting purposes.
Data mining tools for data warehousing Organize procedures for
identifying matrices and links with massive amounts of data using
sophisticated statistical modeling methods.
OLAP tools help create a multidimensional data warehouse and allow
analysis of enterprise data from many perspectives.
 5. data warehouse bus
It defines the flow of data within the data storage bus architecture and includes
the data market. The data market is an access level that allows users to transfer
data. It is also used to segment the data that is produced for a particular user
group.

6. Data warehouse reporting layer

The reporting layer in a data warehouse allows end users to access the BI
interface or BI database architecture. The purpose of the reporting layer in a data
warehouse is to act as a dashboard to visualize data, generate reports, and output
any required information.

Data warehouse architecture best practices

Create data warehouse models that are optimized for retrieval of information in
both dimensional, abnormal, or mixed modes.
Define one approach to data warehouse designs such as top-down or bottom-up
approach and stick to it.
Always clean and transform data with the ETL tool before uploading data to the
data warehouse.
Create an automated data purge where all data is uniformly cleaned before
uploading.
Allow metadata to be shared between different components of a data
warehouse for a smooth extraction process.
Always ensure that data is properly integrated and not only consolidated when
moving from data warehouses to data warehouse. This will require 3NF
normalization of the data models.
Data warehouse design

When an organization sets out to design a data warehouse, it should begin by

defining its specific business requirements, agreeing on a scope, and formulating
a conceptual design. The organization can then create the logical and physical
design of the data warehouse. Logical design includes the relationships between
objects, and physical design involves the best way to store and retrieve objects.
The physical design also includes transfers, backups, and recovery processes.

Any data warehouse design must address the following:

Specific data content

Relationships within and between data sets

Systems environment that will support the data warehouse

 Types of data transformations required

Data refresh frequency

The design factor is based on the needs of the users. Most users are interested in
performing analytics and looking at data in aggregate, rather than dealing with
individual transactions. However, users often do not know what they want until a
specific need arises. Thus, the planning process should include adequate
explorations to anticipate needs. Finally, the design of the data warehouse should
allow room for expansion and evolution to keep pace with the evolving needs of
users.

Do I need a data pool?

Organizations use both data pools and data warehouses with large volumes of
data from different sources. The choice of when to use one or the other depends
on what the organization intends to do with the data. Here is a description of the
best way to use each:

Data pools store a large number of disparate and unfiltered data for later use for
a specific purpose. Data from line-of-business applications, mobile applications,
social media, Internet of Things (IoT) devices, etc., is collected as raw data in a
data pool. The structure, integration, selection, and coordination of different data
sets are extracted at the time of the analyzes by the person performing the
analyzes. When organizations need low-cost storage of unstructured and
unformatted data from multiple sources, and intend to use it for a specific
purpose in the future, a data pool may be the right choice.
Data warehouses are specifically aimed at data analysis. Analytical processing is
performed within the data warehouse on data that has been prepared for
analytics collected, contextualized, and transformed with the goal of creating
analytics-driven insights. Data warehouses are also adept at handling large
volumes of data from different sources. When organizations need advanced data
analytics or analytics that draws on legacy data from multiple sources across the
organization, a data warehouse can be the right choice.

What are the steps for creating an enterprise data warehouse?

Nowadays, many companies are starting to take an interest in building data

warehouses. In fact, building data warehouses is not a difficult thing. It is difficult
to build an enterprise-class data warehouse. This is a very difficult thing for
companies.. However, do not be discouraged, although it is difficult, we can also
build enterprise data warehouses through some methods. In this article, we will
introduce you to the steps for building data warehouses.

Building an enterprise level data warehouse is defining the subject matter, in fact,
the objective subject is defining the subject matter of data analysis or the front
end. The topic should reflect the relationship between each angle of analysis and
statistics of numerical data, to select the topic. This is very important, everyone
should pay attention.

The second step is to determine the measure. When we define the topic, we need
to look at technical indicators for analysis. In general, this is data value data, some
of which are not aggregated. Some may be called upon to provide useful
information to analysts. Measurements are indicators that are statistically
appropriate, and the design and calculation of complex significant indicators can
be carried out based on different measures.

The third step is to determine the actual grain size of the data. When we define
the measure, we need to consider the summary of the measure and the
polymerization of the different dimensions. If note note deduction of data in the
ETL process according to the unit "day", summed up the data per day, and the
particle size for the data warehouse is "day". If you can not confirm whether the
future analysis should be accurate in seconds, then we should follow the
Parthenius minimum, in the fact table in the data warehouse, the data is
summarized, the data is summarized in advance, ensuring the results of the
analysis results. efficiency.

The fourth step is to define the dimension, in fact the dimension is the different
angles of analysis. Based on different dimensions, you can see the position of
each metric or cross analysis based on all dimensions.

The fifth step is to create a fact table. After defining the factual and dimension
data, it will consider loading the fact table. Pen production of the business
system, the transaction log is the original data from the fact table to be
established. The specific approach is to bind the original table to the dimension
table and generate the fact table. When there is empty data, it is necessary to use
an external connection. After the connection, the proxy button for each tool is
connected to the fact table, and the fact table except for the dimension proxy
button, there is also measured data and there should be descriptive information..
etc.
Explain data warehouse examples:

The data warehouse has many real-world applications in the corporate world to
facilitate business decisions. Let's look at some examples of how they are used in
various industries to better understand the definition of a data warehouse.

In retail:
For the retail industry, a good example is the retail data market which includes
customer information from cash registers, mailing lists, websites, and comment
cards. Similarly, another suitable example of the application is the healthcare
sector which uses it to access patient reports, share important data with
insurance providers, forecast outcomes, etc.

In health care:

In healthcare, these central data stores are used to record patient information
from the various units of the medical unit. This may include personal patient
information, financial transactions with the hospital, and insurance data. All this is
integrated into the data warehouse and linked through the database schema.

in construction:

Similarly, in construction, builders demand data for every purchase made during
the construction schedule. This purchase should be credited to a source for
making financial decisions. The same goes for the wages of contract employees.
All this data will be recorded in the data warehouse and later used in business
intelligence by key decision makers to estimate the company's total spending on a
single construction site.

In finance:
Banks, insurance companies, commercial companies and other companies related
to the financial sector need accurate data at all times. This is only possible when
the data in the databases is validated correctly and appropriately connected to
other tables in the database.

These are just examples of how data warehouses are widely used in different
industries and for different purposes. Since it is just an organized store of raw
data, it can serve many purposes for the end user.

Data warehouse maintenance

There are certain steps that are taken to maintain the repository. One step is data
mining, which involves collecting large amounts of data from multiple source
points. After collecting a set of data, it goes through a process of data cleaning,
the process of combing through it for errors and correcting or excluding any data
found.

The cleaned data is then converted from a database format to a repository

format. Once stored in the repository, the data goes through sorting, merging and
summarization, so that it is easier to use. Over time, more data is added to the
repository as the different data sources are updated.

Today, companies can invest in cloud-based data storage software services from
companies including Microsoft, Google, Amazon, Oracle, and others.

Data warehouse design automation

The design of your data warehouse can be automated. It is essential that your
approach is correct. First, identify where your critical data is located, and which
data is relevant to your business intelligence initiatives.

Next, create a standardized metadata framework that provides important context

for this data in the data modeling stage. This framework will be able to match
your data warehouse model to the source system and ensure that the
relationships between entities are built appropriately with properly defined
primary and external keys. It would also prove that the tables are related
correctly and that the entity relationship types are set precisely.

Also, you must have processes in place that allow you to incorporate new sources
and other modifications into your source data model and republish it. An iterative
approach will provide a more detailed view of the data provided for business
intelligence purposes and the insights gained.

You can adopt a 3NF or dimensional modeling approach, depending on your

business intelligence requirements. The latter is better because it will help you
create a streamlined, unformatted structure for your data warehouse model.

While you're at it, here are some basic tips you should keep in mind:
Maintain consistent granularity in dimensional data models

Apply the correct SCD processing technology to your dimensional attributes

Simplify fact table loading using a metadata-based approach

Put processes in place to deal with early realities

Finally, team members can test the quality and integrity of the data models
before publishing them to the target database. Having automated data model
validation tool can provide significant time savings.

Following best practices when automating schema modeling will help you
seamlessly update your model and propagate changes across your data lines.

What is data mining?

mining data

Business data warehouse primarily for data mining. This includes searching for
patterns of information that will help them improve their business operations.

A good warehouse system makes it easier for different departments within a

company to access each other's data. For example, the marketing team can
evaluate the sales team's data in order to make decisions about how to modify
their sales campaigns.

Data Mining and its relationship to the data store

Data Mining is an end-user support technology, which aims to extract useful

information from information contained in a corporate database. In other words,
the origin of the information used by data mining algorithms is usually the
historical data contained in the data warehouse.
There should be integration between Data Mining technologies and the processes
involved in a data warehouse. That is, in order to be able to do business analysis,
there must be an agreement between Data Mining, Data Warehouse, and an
OLAP server.

Each time Data Warehouse provides new results, the company can re-implement
Data Mining to improve the decision-making process.

In short, data mining and Data Warehouse are fully compatible tools. Data
Warehouse provides Memory and Data Mining intelligence.

Five steps of data mining

The data mining process is divided into five steps:

The organization collects data and loads it into a data warehouse.

The data is then stored and managed, either on on-premises servers or in a

cloud service.

Business analysts, management teams, and IT professionals access and organize

data.

The application software sorts the data.

The end user presents the data in an easy-to-share format, such as a graph or a
table.
Advantages and disadvantages of data warehouses

Data storage aims to give a company a competitive advantage. It creates a source

of relevant information that can be tracked and analyzed over time in order to
help businesses make more informed decisions.

It can also drain company resources and overburden its current employees with
routine tasks intended to fuel the warehouse machine.

The Corporate Finance Institute identifies these potential disadvantages of

maintaining a warehouse:

Setting up and maintaining the repository takes significant time and effort.

Gaps in information, caused by human error, can take years to surface,

compromising the integrity and usefulness of information.

When using multiple sources, the inconsistency between them can lead to
information loss.

Advantages

It provides fact-based analysis about a company's past performance to inform

decision-making.

Serves as a historical archive of relevant data.

It can be shared across key departments for maximum benefit.

It reduces the minimum time required to collect all relevant data on a particular
topic.

Provides analysis tools.

Many reports and analyzes are defined by the user.

It allows you to directly access, analyze and monitor the organization's indicators.

It helps to identify the factors that affect the business of the company.

Allows for advancement and defining the future behavior of the organization.

Users can query data quickly and easily

Negatives

Repository set-up and maintenance is resource-heavy.

Input errors can damage the integrity of the archived information.

Using multiple sources can lead to data inconsistencies.

And at the end , we hope that we did clarify the most important and common
issues about the Data Warehouse.

Here are some sources on that we depended in this article:

- Rainer, R. Kelly; Cegielski, Casey G. (2012-05-01). Introduction to

Information Systems: Enabling and Transforming Business, 4t
- "9 Reasons Data Warehouse Projects Fail". [Link]. 4 December
2014. Retrieved 2017-04-30.
- "Exploring Data Warehouses and Data Quality". [Link]. Archived
from the original on 2018-07-26. Retrieved 2017-04-30.
- "What is Big Data?". [Link]. Archived from the original on 2017-
02-17. Retrieved 2017-04-30.
- Paul Gillin (February 20, 1984). "Will Teradata revive a market?". Computer
World. pp. 43, 48. Retrieved 2017-03-13.