Professional Documents
Culture Documents
Concepts and
Techniques
(3rd ed.)
Chapter 4
Jiawei Han, Micheline Kamber, and Jian Pei
University of Illinois at Urbana-Champaign &
Simon Fraser University
2011 Han, Kamber & Pei. All rights reserved.
1
Summary
2
A data warehouse is a subject-oriented, integrated, timevariant, and nonvolatile collection of data in support of
managements decision-making process.W. H. Inmon
Data warehousing:
Data WarehouseSubject-Oriented
Data WarehouseIntegrated
Data WarehouseNonvolatile
OLAP
users
clerk, IT professional
knowledge worker
function
decision support
DB design
application-oriented
subject-oriented
data
current, up-to-date
detailed, flat relational
isolated
repetitive
historical,
summarized, multidimensional
integrated, consolidated
ad-hoc
lots of scans
unit of work
read/write
index/hash on prim. key
short, simple transaction
# records accessed
tens
millions
#users
thousands
hundreds
DB size
100MB-GB
100GB-TB
metric
transaction throughput
usage
access
complex query
Note: There are more and more systems which perform OLAP
analysis directly on relational databases
Other
sources
Operational
DBs
Metadata
Extract
Transform
Load
Refresh
Monitor
&
Integrator
Data
Warehouse
OLAP Server
Serve
Analysis
Query
Reports
Data mining
Data Marts
Data Sources
Data Storage
Enterprise warehouse
collects all of the information about subjects spanning
the entire organization
Data Mart
a subset of corporate-wide data that is of value to a
specific groups of users. Its scope is confined to
specific, selected groups, such as marketing data mart
Virtual warehouse
A set of views over operational databases
Only some of the possible summary views may be
materialized
11
Data extraction
get data from multiple, heterogeneous, and external
sources
Data cleaning
detect errors in the data and rectify them when
possible
Data transformation
convert data from legacy or host format to warehouse
format
Load
sort, summarize, consolidate, compute views, check
integrity, and build indicies and partitions
Refresh
propagate the updates from the data sources to the
warehouse
12
Metadata Repository
Operational meta-data
Business data
Summary
14
item
time,location
time,item
location
supplier
item,location
time,supplier
location,supplier
item,supplier
time,location,supplier
time,item,location
time,item,supplier
1-D cuboids
2-D cuboids
3-D cuboids
item,location,supplier
item
time_key
day
day_of_the_week
month
quarter
year
branch
branch_key
branch_name
branch_type
location_key
units_sold
dollars_sold
avg_sales
item_key
item_name
brand
type
supplier_type
location
location_key
street
city
state_or_province
country
Measures
18
item
Sales Fact Table
time_key
item_key
branch_key
branch
location_key
branch_key
branch_name
branch_type
units_sold
dollars_sold
avg_sales
Measures
item_key
item_name
brand
type
supplier_key
supplier
supplier_key
supplier_type
location
location_key
street
city_key
city
city_key
city
state_or_province
country
19
Example of Fact
Constellation
time
time_key
day
day_of_the_week
month
quarter
year
item
Sales Fact Table
time_key
item_key
item_key
item_name
brand
type
supplier_type
branch_key
location_key
branch
branch_key
branch_name
branch_type
units_sold
dollars_sold
avg_sales
Measures
location
to_location
location_key
street
city
province_or_state
country
dollars_cost
units_shipped
shipper
shipper_key
shipper_name
location_key
shipper_type 20
A Concept Hierarchy:
Dimension (location)
all
all
Europe
region
country
city
office
Germany
Frankfurt
...
...
...
Spain
North_America
Canada
Vancouver ...
L. Chan
...
...
Mexico
Toronto
M. Wind
21
22
Specification of hierarchies
Schema hierarchy
day < {month < quarter;
week} < year
Set_grouping hierarchy
{1..10} < inexpensive
23
Multidimensional Data
Sales volume as a function of product,
month, and region
Re
gi
on
Year
Product
Product
City
Office
Month Week
Day
Month
24
Pr
od
u
TV
PC
VCR
sum
1Qtr
2Qtr
Date
3Qtr
4Qtr
sum
Country
ct
sum
25
date
country
product,country
1-D cuboids
date, country
2-D cuboids
26
Pivot (rotate):
Other operations
28
Shipping Method
Customer
CONTRACTS
AIR-EXPRESS
ORDER
TRUCK
Time
PRODUCT LINE
ANNUALY QTRLY
DAILY
CITY
PRODUCT ITEM
Product
PRODUCT GROUP
SALES PERSON
COUNTRY
DISTRICT
REGION
Location
Each circle is
called a
footprint
DIVISION
Promotion
Organization
29
Visualization
OLAP capabilities
Interactive manipulation
30
Summary
31
Top-down view
Choose the dimensions that will apply to each fact table record
Choose the measure that will populate each fact table record
33
Data Warehouse
Development: A
Recommended Approach
Multi-Tier Data
Warehouse
Distributed
Data Marts
Data
Mart
Data
Mart
Model refinement
Enterprise
Data
Warehouse
Model refinement
Information processing
Analytical processing
Data mining
Summary
37
[item,
city,
year]:
sum
(item)
(year)
Base table
Cust
C1
C2
C3
C4
C5
Region
Asia
Europe
Asia
America
Europe
Index on Region
Index on Type
Transform drill, roll, etc. into corresponding SQL and/or OLAP operations,
e.g., dice = selection + projection
Explore indexing structures and compressed vs. dense array structs in MOLAP
42
Greater scalability
Summary
44
Attribute-Oriented Induction
How it is done?
Attribute-Oriented Induction: An
Example
Example: Describe general characteristics of graduate
students in the University database
Gender
Jim
Initial
Woodman
Relation Scott
Major
M
F
Removed
Retained
Residence
Phone #
GPA
Vancouver,BC, 8-12-76
Canada
CS
Montreal, Que, 28-7-75
Canada
Physics Seattle, WA, USA 25-8-70
687-4598
3.67
253-9106
3.70
420-5232
3.83
Sci,Eng,
Bus
City
Removed
Excl,
VG,..
Gender Major
M
F
Birth_date
CS
Lachance
Laura Lee
Prime
Generalized
Relation
Birth-Place
Science
Science
Country
Age range
Birth_region
Age_range
Residence
GPA
Canada
Foreign
20-25
25-30
Richmond
Burnaby
Very-good
Excellent
Count
16
22
Birth_Region
Canada
Foreign
Total
Gender
M
16
14
30
10
22
32
Total
26
36
62
47
Presentation of Generalized
Results
Generalized relation:
Cross tabulation:
Visualization techniques:
50
Method:
Partition the set of relevant data into the target class and
the contrasting class(es)
Relevance Analysis:
Similarity:
Data generalization
Presentation of data summarization at multiple
levels of abstraction
Interactive drilling, pivoting, slicing and dicing
Differences:
OLAP has systematic preprocessing, query
independent, and can drill down to rather low level
AOI has automated desired level allocation, and
may perform dimension relevance analysis/ranking
when there are many relevant dimensions
AOI works on the data which are not in relational
forms
52
Summary
53
Summary
Multi-tiered architecture
References (I)
J. Han. Towards on-line analytical mining in large databases. ACM SIGMOD Record,
27:97-107, 1998.
55
References (II)
Surplus Slides
57