Professional Documents
Culture Documents
Data Warehouse Concepts
Data Warehouse Concepts
Data Warehousing
Dr. Hany Saleeb
What is Data
Warehouse?
Defined in many different ways, but not rigorously.
A decision support database that is maintained
separately from the organizations operational database
Support information processing by providing a solid
platform of consolidated, historical data for analysis.
A data warehouse is a subject-oriented, integrated, timevariant, and nonvolatile collection of data in support of
managements decision-making process.W. H. Inmon
Data WarehouseSubjectOriented
Organized around major subjects, such as
customer, product, sales.
Focusing on the modeling and analysis of data
for decision makers, not on daily operations or
transaction processing.
Provide a simple and concise view around
particular subject issues by excluding data that
are not useful in the decision support process.
Data Warehouse
Integrated
Constructed by integrating multiple,
heterogeneous data sources
relational databases, flat files, on-line
transaction records
Data WarehouseTime
Variant
The time horizon for the data warehouse is significantly
longer than that of operational systems.
Operational database: current value data.
Data warehouse data: provide information from a
historical perspective (e.g., past 5-10 years)
Every key structure in the data warehouse
Contains an element of time, explicitly or implicitly
But the key of operational data may or may not contain
time element.
Data WarehouseNonVolatile
A physically separate store of data transformed
from the operational environment.
Operational update of data does not occur in
the data warehouse environment.
Does not require transaction processing, recovery,
and concurrency control mechanisms
Requires only two operations in data accessing:
initial loading of data and access of data.
OLAP
users
clerk, IT professional
knowledge worker
function
decision support
DB design
application-oriented
subject-oriented
data
current, up-to-date
detailed, flat relational
isolated
repetitive
historical,
summarized, multidimensional
integrated, consolidated
ad-hoc
lots of scans
unit of work
read/write
index/hash on prim. key
short, simple transaction
# records accessed
tens
millions
#users
thousands
hundreds
DB size
100MB-GB
100GB-TB
metric
transaction throughput
usage
access
complex query
Multidimensional Data
Sales volume as a function of
product, month, and
region
Dimensions: Product, Location, Time
Re
gi
on
Year
Product
City
Office
Month
Month Week
Day
Pr
od
u
TV
PC
VCR
sum
1Qtr
2Qtr
Date
3Qtr
4Qtr
sum
Country
ct
Cuboids Corresponding to
the Cube
all
0-D(apex) cuboid
product
product,date
date
country
product,country
1-D cuboids
date, country
2-D cuboids
3-D(base) cuboid
Multi-Tiered Architecture
other
Metadata
source
s
Operational
DBs
Extract
Transform
Load
Refresh
Monitor
&
Integrator
Data
Warehouse
OLAP Server
Serve
Analysis
Query
Reports
Data mining
Data Marts
Data Sources
Data Storage
Cube Operation
Cube definition and computation in DMQL
define cube sales[item, city, year]: sum(sales_in_dollars)
compute cube sales
(city)
(item)
(year)
(city, year)
(item, year)
Base table
Cust
C1
C2
C3
C4
C5
Region
Asia
Europe
Asia
America
Europe
Index on Region
Index on Type
Metadata Repository
Meta data is the data defining warehouse objects. It has
the following kinds
Description of the structure of the warehouse
schema, view, dimensions, hierarchies, derived data defn, data
mart locations and contents
Operational meta-data
data lineage (history of migrated data and transformation path),
currency of data (active, archived, or purged), monitoring
information (warehouse usage statistics, error reports, audit trails)
Business data
business terms and definitions, ownership of data, charging policies
Discovery-Driven
Exploration of Data Cubes
Hypothesis-driven: exploration by user, huge search space
Discovery-driven
pre-compute measures indicating exceptions, guide user
in the data analysis, at all levels of aggregation
Exception: significantly different from the value
anticipated, based on a statistical model
Visual cues such as background color are used to reflect
the degree of exception of each cell
Computation of exception indicator (modeling fitting and
computing SelfExp, InExp, and PathExp values) can be
overlapped with cube construction
Examples: Discovery-Driven
Data Cubes
Summary
Data warehouse
References (I)
K. Beyer and R. Ramakrishnan. Bottom-Up Computation of Sparse and Iceberg CUBEs. In Proc.
1999 ACM-SIGMOD Int. Conf. Management of Data (SIGMOD'99), 359-370, Philadelphia, PA,
June 1999.
S. Chaudhuri and U. Dayal. An overview of data warehousing and OLAP technology. ACM
SIGMOD Record, 26:65-74, 1997.
References (II)
K. Ross and D. Srivastava. Fast computation of sparse datacubes. In Proc. 1997 Int. Conf.
Very Large Data Bases, 116-125, Athens, Greece, Aug. 1997.
E. Thomsen. OLAP Solutions: Building Multidimensional Information Systems. John Wiley &
Sons, 1997.