Data Preprocessing

Why preprocess the data?

Data cleaning
Data integration and transformation
Data reduction
Discretization and concept hierarchy generation
Summary
Why Data Preprocessing?
Data in the real world is dirty
incomplete: lacking attribute values, lacking certain attributes of
interest, or containing only aggregate data
noisy: containing errors or outliers
inconsistent: containing discrepancies in codes or names
No quality data, no quality mining results!
Quality decisions must be based on quality data
Data warehouse needs consistent integration of quality data
Required for both OLAP and Data Mining!
Why can Data be Incomplete?
Attributes of interest are not available (e.g., customer
information for sales transaction data)
Data were not considered important at the time of
transactions, so they were not recorded!
Data not recorder because of misunderstanding or
malfunctions
Data may have been recorded and later deleted!
Missing/unknown values for some data
Why can Data be Noisy/Inconsistent?
Faulty instruments for data collection
Human or computer errors
Errors in data transmission
Technology limitations (e.g., sensor data come at a faster
rate than they can be processed)
Inconsistencies in naming conventions or data codes (e.g.,
2/5/2002 could be 2 May 2002 or 5 Feb 2002)
Duplicate tuples, which were received twice should also be
removed
Major Tasks in Data Preprocessing
Data cleaning
Fill in missing values, smooth noisy data, identify or remove outliers, and
resolve inconsistencies
Data integration
Integration of multiple databases, data cubes, or files
Data transformation
Normalization and aggregation
Data reduction
Obtains reduced representation in volume but produces the same or
similar analytical results
Data discretization
Part of data reduction but with particular importance, especially for
numerical data
outliers=exceptions!
Forms of data preprocessing
Data Preprocessing
Why preprocess the data?
Data cleaning
Data integration and transformation
Data reduction
Discretization and concept hierarchy generation
Summary
Data Cleaning
Data cleaning tasks
Fill in missing values
Identify outliers and smooth out noisy data
Correct inconsistent data
How to Handle Missing Data?
Ignore the tuple: usually done when class label is missing (assuming the
tasks in classification)not effective when the percentage of missing values
per attribute varies considerably.
Fill in the missing value manually: tedious + infeasible?
Use a global constant to fill in the missing value: e.g., unknown, a new
class?!
Use the attribute mean to fill in the missing value
Use the attribute mean for all samples belonging to the same class to fill in
the missing value: smarter
Use the most probable value to fill in the missing value: inference-based
such as Bayesian formula or decision tree
How to Handle Missing Data?
Age Income Religion Gender
23 24,200 Muslim M
39 ? Christian F
45 45,390 ? F
Fill missing values using aggregate functions (e.g., average) or
probabilistic estimates on global value distribution
E.g., put the average income here, or put the most probable income
based on the fact that the person is 39 years old
E.g., put the most frequent religion here
Noisy Data
Noise: random error or variance in a measured variable
Incorrect attribute values may exist due to
faulty data collection instruments
data entry problems
data transmission problems
technology limitation
inconsistency in naming convention
Other data problems which requires data cleaning
duplicate records
incomplete data
inconsistent data
How to Handle Noisy Data?
Smoothing techniques
Binning method:
first sort data and partition into (equi-depth) bins
then one can smooth by bin means, smooth by bin median, smooth
by bin boundaries, etc.
Clustering
detect and remove outliers
Combined computer and human inspection
computer detects suspicious values, which are then checked by
humans
Regression
smooth by fitting the data into regression functions
Use Concept hierarchies
use concept hierarchies, e.g., price value -> expensive
Simple Discretization Methods: Binning
Equal-width (distance) partitioning:
It divides the range into N intervals of equal size: uniform grid
if A and B are the lowest and highest values of the attribute, the
width of intervals will be: W = (B-A)/N.
The most straightforward
But outliers may dominate presentation
Skewed data is not handled well.
Equal-depth (frequency) partitioning:
It divides the range into N intervals, each containing approximately
same number of samples
Good data scaling good handing of skewed data
Simple Discretization Methods: Binning
Example: customer ages
0-10 10-20
20-30 30-40 40-50 50-60 60-70 70-80
Equi-width
binning:
number
of values
0-22 22-31
44-48 32-38
38-44
48-55
55-62
62-80
Equi-width
binning:
Smoothing using Binning Methods
* Sorted data for price (in dollars): 4, 8, 9, 15, 21, 21, 24, 25, 26, 28, 29, 34
* Partition into (equi-depth) bins:
- Bin 1: 4, 8, 9, 15
- Bin 2: 21, 21, 24, 25
- Bin 3: 26, 28, 29, 34
* Smoothing by bin means:
- Bin 1: 9, 9, 9, 9
- Bin 2: 23, 23, 23, 23
- Bin 3: 29, 29, 29, 29
* Smoothing by bin boundaries: [4,15],[21,25],[26,34]
- Bin 1: 4, 4, 4, 15
- Bin 2: 21, 21, 25, 25
- Bin 3: 26, 26, 26, 34
Cluster Analysis
cluster
outlier
salary
age
Regression
x
y
y = x + 1
X1
Y1
(salary)
(age)
Example of linear regression
Inconsistent Data
Inconsistent data are handled by:
Manual correction (expensive and tedious)
Use routines designed to detect inconsistencies and
manually correct them. E.g., the routine may use the check
global constraints (age>10) or functional dependencies
Other inconsistencies (e.g., between names of the same
attribute) can be corrected during the data integration
process

Data Preprocessing
Why preprocess the data?
Data cleaning
Data integration and transformation
Data reduction
Discretization and concept hierarchy generation
Summary
Data Integration
Data integration:
combines data from multiple sources into a coherent store
Schema integration
integrate metadata from different sources
metadata: data about the data (i.e., data descriptors)
Entity identification problem: identify real world entities from
multiple data sources, e.g., A.cust-id B.cust-#
Detecting and resolving data value conflicts
for the same real world entity, attribute values from different sources
are different (e.g., J.D.Smith and Jonh Smith may refer to the same
person)
possible reasons: different representations, different scales, e.g., metric
vs. British units (inches vs. cm)
Handling Redundant
Data in Data Integration
Redundant data occur often when integration of multiple
databases
The same attribute may have different names in different databases
One attribute may be a derived attribute in another table, e.g.,
annual revenue
Redundant data may be able to be detected by correlation
analysis
Careful integration of the data from multiple sources may
help reduce/avoid redundancies and inconsistencies and
improve mining speed and quality
Data Transformation
Smoothing: remove noise from data
Aggregation: summarization, data cube construction
Generalization: concept hierarchy climbing
Normalization: scaled to fall within a small, specified
range
min-max normalization
z-score normalization
normalization by decimal scaling
Attribute/feature construction
New attributes constructed from the given ones
Normalization: Why
normalization?
Speeds-up some learning techniques (ex. neural
networks)
Helps prevent attributes with large ranges outweigh
ones with small ranges
Example:
income has range 3000-200000
age has range 10-80
gender has domain M/F


Data Transformation: Normalization
min-max normalization


e.g. convert age=30 to range 0-1, when min=10,max=80.
new_age=(30-10)/(80-10)=2/7
z-score normalization


normalization by decimal scaling


A A A
A A
A
min new min new max new
min max
min v
v _ ) _ _ ( ' +

=
A
A
dev stand_
mean v
v

= '
j
v
v
10
' = Where j is the smallest integer such that Max(| |)<1
' v
Data Preprocessing
Why preprocess the data?
Data cleaning
Data integration and transformation
Data reduction
Discretization and concept hierarchy generation
Summary
Data Reduction Strategies
Warehouse may store terabytes of data: Complex data
analysis/mining may take a very long time to run on the
complete data set
Data reduction
Obtains a reduced representation of the data set that is much smaller
in volume but yet produces the same (or almost the same) analytical
results
Data reduction strategies
Data cube aggregation
Dimensionality reduction
Data compression
Numerosity reduction
Discretization and concept hierarchy generation
Data Cube Aggregation
The lowest level of a data cube
the aggregated data for an individual entity of interest
e.g., a customer in a phone calling data warehouse.
Multiple levels of aggregation in data cubes
Further reduce the size of data to deal with
Reference appropriate levels
Use the smallest representation which is enough to solve the task
Queries regarding aggregated information should be
answered using data cube, when possible
Dimensionality Reduction
Feature selection (i.e., attribute subset selection):
Select a minimum set of features such that the probability distribution
of different classes given the values for those features is as close as
possible to the original distribution given the values of all features
reduce # of patterns in the patterns, easier to understand
Heuristic methods (due to exponential # of choices):
step-wise forward selection
step-wise backward elimination
combining forward selection and backward elimination
decision-tree induction
Heuristic Feature Selection
Methods
There are 2
d
possible sub-features of d features
Several heuristic feature selection methods:
Best single features under the feature independence assumption:
choose by significance tests.
Best step-wise feature selection:
The best single-feature is picked first
Then next best feature condition to the first, ...
Step-wise feature elimination:
Repeatedly eliminate the worst feature
Best combined feature selection and elimination:
Optimal branch and bound:
Use feature elimination and backtracking
Example of Decision Tree Induction
Initial attribute set:
{A1, A2, A3, A4, A5, A6}
A4 ?
A1?
A6?
Class 1
Class 2
Class 1
Class 2
> Reduced attribute set: {A1, A4, A6}
Data Compression
Original Data Compressed
Data
lossless
Original Data
Approximated
Principal Component Analysis or Karhuren-
Loeve (K-L) method
Given N data vectors from k-dimensions, find c <= k
orthogonal vectors that can be best used to represent data
The original data set is reduced to one consisting of N data vectors
on c principal components (reduced dimensions)
Each data vector is a linear combination of the c principal
component vectors
Works for numeric data only
Used when the number of dimensions is large
X1
X2
Y1
Y2
Principal Component Analysis
X1, X2: original axes (attributes)
Y1,Y2: principal components
significant component
(high variance)
Order principal components by significance and eliminate weaker ones
Numerosity Reduction:
Reduce the volume of data
Parametric methods
Assume the data fits some model, estimate model parameters,
store only the parameters, and discard the data (except possible
outliers)
Log-linear models: obtain value at a point in m-D space as the
product on appropriate marginal subspaces
Non-parametric methods
Do not assume models
Major families: histograms, clustering, sampling
Histograms
A popular data reduction
technique
Divide data into buckets
and store average (or sum)
for each bucket
Can be constructed
optimally in one
dimension using dynamic
programming
Related to quantization
problems.
0
5
10
15
20
25
30
35
40
10000 30000 50000 70000 90000
Histogram types
Equal-width histograms:
It divides the range into N intervals of equal size
Equal-depth (frequency) partitioning:
It divides the range into N intervals, each containing approximately
same number of samples
V-optimal:
It considers all histogram types for a given number of buckets and
chooses the one with the least variance.
MaxDiff:
After sorting the data to be approximated, it defines the borders of
the buckets at points where the adjacent values have the maximum
difference
Example: split 1,1,4,5,5,7,9,14,16,18,27,30,30,32 to three buckets
MaxDiff 27-18 and 14-9
Histograms
Clustering
Partitions data set into clusters, and models it by one
representative from each cluster
Can be very effective if data is clustered but not if data is
smeared
There are many choices of clustering definitions and
clustering algorithms, further detailed in Chapter 7
Cluster Analysis
cluster
outlier
salary
age
the distance between points in the
same cluster should be small
the distance between points in different
clusters should be large
Hierarchical Reduction
Use multi-resolution structure with different degrees of
reduction
Hierarchical clustering is often performed but tends to
define partitions of data sets rather than clusters
Parametric methods are usually not amenable to hierarchical
representation
Hierarchical aggregation
An index tree hierarchically divides a data set into partitions by value
range of some attributes
Each partition can be considered as a bucket
Thus an index tree with aggregates stored at each node is a
hierarchical histogram
Data Preprocessing
Why preprocess the data?
Data cleaning
Data integration and transformation
Data reduction
Discretization and concept hierarchy generation
Summary
Discretization
Three types of attributes:
Nominal values from an unordered set
Ordinal values from an ordered set
Continuous real numbers
Discretization:
divide the range of a continuous attribute into
intervals
why?
Some classification algorithms only accept categorical
attributes.
Reduce data size by discretization
Prepare for further analysis
Discretization and Concept
hierachy
Discretization
reduce the number of values for a given continuous attribute by
dividing the range of the attribute into intervals. Interval labels
can then be used to replace actual data values.
Concept hierarchies
reduce the data by collecting and replacing low level concepts
(such as numeric values for the attribute age) by higher level
concepts (such as young, middle-aged, or senior).
Discretization and concept
hierarchy generation for
numeric data
Binning/Smoothing (see sections before)
Histogram analysis (see sections before)
Clustering analysis (see sections before)
Entropy-based discretization
Segmentation by natural partitioning
Entropy-Based Discretization
Given a set of samples S, if S is partitioned into two intervals
S1 and S2 using boundary T, the information gain I(S,T) after
partitioning is

The boundary that maximizes the information gain over all
possible boundaries is selected as a binary discretization.
The process is recursively applied to partitions obtained
until some stopping criterion is met, e.g.,

Experiments show that it may reduce data size and improve
classification accuracy
) (
| |
| |
) (
| |
| |
) , (
2
2
1
1
S
S
S
S
Ent
S
Ent
S
T S I + =
o > ) , ( ) ( S T I S Ent
) ( log ) (
2
1
1 i
m
i
i
p p S Ent

=
=
Entropy:
Segmentation by natural
partitioning
The 3-4-5 rule can be used to segment numerical data into
relatively uniform, natural intervals.
* If an interval covers 3, 6, 7 or 9 distinct values at the most significant
digit, partition the range into 3 equiwidth intervals for 3,6,9 or 2-3-2 for 7
* If it covers 2, 4, or 8 distinct values at the most significant digit, partition
the range into 4 equiwidth intervals
* If it covers 1, 5, or 10 distinct values at the most significant digit, partition
the range into 5 equiwidth intervals
Users often like to see numerical ranges partitioned into
relatively uniform, easy-to-read intervals that appear intuitive
or natural. E.g., [50-60] better than [51.223-60.812]
The rule can be recursively applied for the resulting intervals
Concept hierarchy generation for
categorical data
Categorical attributes: finite, possibly large domain, with no ordering
among the values
Example: item type
Specification of a partial ordering of attributes explicitly at the schema
level by users or experts
Example: location is split by domain experts to street<city<state<country
Specification of a portion of a hierarchy by explicit data grouping
Specification of a set of attributes, but not of their partial ordering
Specification of only a partial set of attributes
Specification of a set of attributes
Concept hierarchy can be automatically generated based
on the number of distinct values per attribute in the
given attribute set. The attribute with the most distinct
values is placed at the lowest level of the hierarchy.
country
province_or_ state
city
street
15 distinct values
65 distinct values
3567 distinct values
674,339 distinct values
Association Rule Mining
Given a set of transactions, find rules that will predict the
occurrence of an item based on the occurrences of other
items in the transaction
Market-Basket transactions
TI D I tems
1 Bread, Milk
2 Bread, Diaper, Beer, Eggs
3 Milk, Diaper, Beer, Coke
4 Bread, Milk, Diaper, Beer
5 Bread, Milk, Diaper, Coke

Example of Association Rules
{Diaper} {Beer},
{Milk, Bread} {Eggs,Coke},
{Beer, Bread} {Milk},
Implication means co-occurrence,
not causality!
Definition: Frequent Itemset
Itemset
A collection of one or more items
Example: {Milk, Bread, Diaper}
k-itemset
An itemset that contains k items
Support count (o)
Frequency of occurrence of an itemset
E.g. o({Milk, Bread,Diaper}) = 2
Support
Fraction of transactions that contain an
itemset
E.g. s({Milk, Bread, Diaper}) = 2/5
Frequent Itemset
An itemset whose support is greater than
or equal to a minsup threshold
TI D I tems
1 Bread, Milk
2 Bread, Diaper, Beer, Eggs
3 Milk, Diaper, Beer, Coke
4 Bread, Milk, Diaper, Beer
5 Bread, Milk, Diaper, Coke

I assume that itemsets are
ordered lexicographically
Definition: Association Rule
Let D be database of transactions
e.g.:



Let I be the set of items that appear in the database,
e.g., I={A,B,C,D,E,F}
A rule is defined by X Y, where XcI, YcI, and
XY=C
e.g.: {B,C} {E} is a rule
Transaction IDItems Bought
2000 A,B,C
1000 A,C
4000 A,D
5000 B,E,F
Definition: Association Rule
Example:
Beer } Diaper , Milk {
4 . 0
5
2
| T |
) Beer Diaper, , Milk (
= = =
o
s
67 . 0
3
2
) Diaper , Milk (
) Beer Diaper, Milk, (
= = =
o
o
c
Association Rule
An implication expression of the
form X Y, where X and Y are
itemsets
Example:
{Milk, Diaper} {Beer}

Rule Evaluation Metrics
Support (s)
Fraction of transactions that
contain both X and Y
Confidence (c)
Measures how often items in Y
appear in transactions that
contain X
TI D I tems
1 Bread, Milk
2 Bread, Diaper, Beer, Eggs
3 Milk, Diaper, Beer, Coke
4 Bread, Milk, Diaper, Beer
5 Bread, Milk, Diaper, Coke

Rule Measures: Support and
Confidence
Find all the rules X Y with minimum
confidence and support
support, s, probability that a transaction
contains {X Y}
confidence, c, conditional probability that
a transaction having X also contains Y
Transaction IDItems Bought
2000 A,B,C
1000 A,C
4000 A,D
5000 B,E,F
Let minimum support 50%,
and minimum confidence
50%, we have
A C (50%, 66.6%)
C A (50%, 100%)
Customer
buys diaper
Customer
buys both
Customer
buys beer
TID date items_bought
100 10/10/99 {F,A,D,B}
200 15/10/99 {D,A,C,E,B}
300 19/10/99 {C,A,B,E}
400 20/10/99 {B,A,D}
Example
What is the support and confidence of the rule: {B,D}
{A}
Support:
percentage of tuples that contain {A,B,D} =
Confidence:

=
D} {B, contain that tuples of number
D} B, {A, contain that tuples of number
75%
100%
Remember:
conf(X Y) =
sup(X)
Y) sup(X
Association Rule Mining Task
Given a set of transactions T, the goal of association rule
mining is to find all rules having
support minsup threshold
confidence minconf threshold
Brute-force approach:
List all possible association rules
Compute the support and confidence for each rule
Prune rules that fail the minsup and minconf thresholds
Computationally prohibitive!
Mining Association Rules
Example of Rules:

{Milk,Diaper} {Beer} (s=0.4, c=0.67)
{Milk,Beer} {Diaper} (s=0.4, c=1.0)
{Diaper,Beer} {Milk} (s=0.4, c=0.67)
{Beer} {Milk,Diaper} (s=0.4, c=0.67)
{Diaper} {Milk,Beer} (s=0.4, c=0.5)
{Milk} {Diaper,Beer} (s=0.4, c=0.5)
TI D I tems
1 Bread, Milk
2 Bread, Diaper, Beer, Eggs
3 Milk, Diaper, Beer, Coke
4 Bread, Milk, Diaper, Beer
5 Bread, Milk, Diaper, Coke

Observations:
All the above rules are binary partitions of the same itemset:
{Milk, Diaper, Beer}
Rules originating from the same itemset have identical support but
can have different confidence
Thus, we may decouple the support and confidence requirements
Mining Association Rules
Two-step approach:
1. Frequent Itemset Generation
Generate all itemsets whose support > minsup

2. Rule Generation
Generate high confidence rules from each frequent
itemset, where each rule is a binary partitioning of a
frequent itemset

Frequent itemset generation is still computationally
expensive

Frequent Itemset Generation
null
AB AC AD AE BC BD BE CD CE DE
A B C D E
ABC ABD ABE ACD ACE ADE BCD BCE BDE CDE
ABCD ABCE ABDE ACDE BCDE
ABCDE
Given d items, there
are 2
d
possible
candidate itemsets
Frequent Itemset Generation
Brute-force approach:

Each itemset in the lattice is a candidate frequent itemset
Count the support of each candidate by scanning the database







Match each transaction against every candidate
Complexity ~ O(NMw) => Expensive since M = 2
d
!!!
TI D I tems
1 Bread, Milk
2 Bread, Diaper, Beer, Eggs
3 Milk, Diaper, Beer, Coke
4 Bread, Milk, Diaper, Beer
5 Bread, Milk, Diaper, Coke

N
Transactions
List of
Candidates
M
w
Computational Complexity
Given d unique items:
Total number of itemsets = 2
d
Total number of possible association rules:
1 2 3
1
1
1 1
+ =
(

|
.
|

\
|

|
.
|

\
|
=
+

=

d d
d
k
k d
j
j
k d
k
d
R
If d=6, R = 602 rules
Frequent Itemset Generation Strategies
Reduce the number of candidates (M)
Complete search: M=2
d
Use pruning techniques to reduce M

Reduce the number of transactions (N)
Reduce size of N as the size of itemset increases
Used by DHP and vertical-based mining algorithms

Reduce the number of comparisons (NM)
Use efficient data structures to store the candidates or
transactions
No need to match every candidate against every
transaction
Reducing Number of Candidates
Apriori principle:
If an itemset is frequent, then all of its subsets must also
be frequent

Apriori principle holds due to the following property of
the support measure:


Support of an itemset never exceeds the support of its
subsets
This is known as the anti-monotone property of support
) ( ) ( ) ( : , Y s X s Y X Y X > _
Example
TI D I tems
1 Bread, Milk
2 Bread, Diaper, Beer, Eggs
3 Milk, Diaper, Beer, Coke
4 Bread, Milk, Diaper, Beer
5 Bread, Milk, Diaper, Coke

s(Bread) > s(Bread, Beer)
s(Milk) > s(Bread, Milk)
s(Diaper, Beer) > s(Diaper, Beer, Coke)
Found to be
Infrequent
null
AB AC AD AE BC BD BE CD CE DE
A B C D E
ABC ABD ABE ACD ACE ADE BCD BCE BDE CDE
ABCD ABCE ABDE ACDE BCDE
ABCDE
Illustrating Apriori Principle
null
AB AC AD AE BC BD BE CD CE DE
A B C D E
ABC ABD ABE ACD ACE ADE BCD BCE BDE CDE
ABCD ABCE ABDE ACDE BCDE
ABCDE
Pruned
supersets
Illustrating Apriori Principle
Item Count
Bread 4
Coke 2
Milk 4
Beer 3
Diaper 4
Eggs 1
Itemset Count
{Bread,Milk} 3
{Bread,Beer} 2
{Bread,Diaper} 3
{Milk,Beer} 2
{Milk,Diaper} 3
{Beer,Diaper} 3
Itemset Count
{Bread,Milk,Diaper} 3

Items (1-itemsets)
Pairs (2-itemsets)

(No need to generate
candidates involving Coke
or Eggs)
Triplets (3-itemsets)
Minimum Support = 3
If every subset is considered,

6
C
1
+
6
C
2
+
6
C
3
= 41
With support-based pruning,
6 + 6 + 1 = 13
The Apriori Algorithm (the general
idea)
1. Find frequent 1-items and put them to L
k
(k=1)
2. Use L
k
to generate a collection of candidate itemsets C
k+1

with size (k+1)
3. Scan the database to find which itemsets in C
k+1
are
frequent and put them into L
k+1

4. If L
k+1
is not empty
k=k+1
GOTO 2
R. Agrawal, R. Srikant: "Fast Algorithms for Mining Association Rules",
Proc. of the 20th Int'l Conference on Very Large Databases, Santiago, Chile, Sept. 1994.
The Apriori Algorithm
Pseudo-code:
C
k
: Candidate itemset of size k
L
k
: frequent itemset of size k

L
1
= {frequent items};
for (k = 1; L
k
!=C; k++) do begin
C
k+1
= candidates generated from L
k
;
// join and prune steps
for each transaction t in database do
increment the count of all candidates in C
k+1
that are
contained in t
L
k+1
= candidates in C
k+1
with min_support (frequent)
end
return
k
L
k
;
Important steps in candidate generation:
Join Step: C
k+1
is generated by joining L
k
with itself
Prune Step: Any k-itemset that is not frequent cannot be a subset of a
frequent (k+1)-itemset
The Apriori Algorithm Example
TID Items
100 1 3 4
200 2 3 5
300 1 2 3 5
400 2 5
Database D itemset sup.
{1} 2
{2} 3
{3} 3
{4} 1
{5} 3
itemset sup.
{1} 2
{2} 3
{3} 3
{5} 3
Scan D
C
1
L
1
itemset
{1 2}
{1 3}
{1 5}
{2 3}
{2 5}
{3 5}
itemset sup
{1 2} 1
{1 3} 2
{1 5} 1
{2 3} 2
{2 5} 3
{3 5} 2
itemset sup
{1 3} 2
{2 3} 2
{2 5} 3
{3 5} 2
L
2
C
2
C
2
Scan D
C
3
L
3
itemset
{2 3 5}
Scan D
itemset sup
{2 3 5} 2
min_sup=2=50%
How to Generate Candidates?
Suppose the items in L
k
are listed in an order
Step 1: self-joining L
k
(IN SQL)
insert into C
k+1
select p.item
1
, p.item
2
, , p.item
k
, q.item
k

from L
k
p, L
k
q
where p.item
1
=q.item
1
, , p.item
k-1
=q.item
k-1
, p.item
k
< q.item
k
Step 2: pruning
forall itemsets c in C
k+1
do
forall k-subsets s of c do
if (s is not in L
k
) then delete c from C
k+1

Example of Candidates Generation
L
3
={abc, abd, acd, ace, bcd}
Self-joining: L
3
*L
3

abcd from abc and abd
acde from acd and ace
Pruning:
acde is removed because ade is not in L
3
C
4
={abcd}
{a,c,d} {a,c,e}
{a,c,d,e}
acd ace ade cde
\ \ X
X
How to Count Supports of
Candidates?
Why counting supports of candidates a problem?
The total number of candidates can be huge
One transaction may contain many candidates
Method:
Candidate itemsets are stored in a hash-tree
Leaf node of hash-tree contains a list of itemsets and counts
Interior node contains a hash table
Subset function: finds all the candidates contained in a transaction
Example of the hash-tree for C
3

Hash function: mod 3
H
1,4,.. 2,5,.. 3,6,..
H
Hash on 1
st
item
H H 234
567
H
145
124
457
125
458
159
345 356
689
367
368
Hash on 2
nd
item
Hash on 3
rd
item
Example of the hash-tree for C
3

Hash function: mod 3
H
1,4,.. 2,5,.. 3,6,..
H
Hash on 1
st
item
H H 234
567
H
145
124
457
125
458
159
345 356
689
367
368
Hash on 2
nd
item
Hash on 3
rd
item
12345
12345
look for 1XX
2345
look for 2XX
345
look for 3XX
Example of the hash-tree for C
3

Hash function: mod 3
H
1,4,.. 2,5,.. 3,6,..
H
Hash on 1
st
item
H H 234
567
H
145
124
457
125
458
159
345 356
689
367
368
Hash on 2
nd
item
12345
12345
look for 1XX
2345
look for 2XX
345
look for 3XX
12345
look for 12X
12345
look for 13X (null)
12345
look for 14X
C
AprioriTid: Use D only for first pass
The database is not used after the 1
st
pass.
Instead, the set C
k
is used for each step, C
k
= <TID, {X
k
}>
: each X
k
is a potentially frequent itemset in transaction
with id=TID.
At each step C
k
is generated from C
k-1
at the pruning step
of constructing C
k
and used to compute L
k
.
For small values of k, C
k
could be larger than the
database!
AprioriTid Example (min_sup=2)
TID Items
100 1 3 4
200 2 3 5
300 1 2 3 5
400 2 5
Database D
itemset sup.
{1} 2
{2} 3
{3} 3
{5} 3
L
1
itemset
{1 2}
{1 3}
{1 5}
{2 3}
{2 5}
{3 5}
itemset sup
{1 3} 2
{2 3} 2
{2 5} 3
{3 5} 2
L
2
C
2
C
3

itemset
{2 3 5}
itemset sup
{2 3 5} 2
TID Sets of itemsets
100 {{1},{3},{4}}
200 {{2},{3},{5}}
300 {{1},{2},{3},{5}}
400 {{2},{5}}
C
1

TID Sets of itemsets
100 {{1 3}}
200 {{2 3},{2 5},{3 5}}
300 {{1 2},{1 3},{1 5}, {2
3},{2 5},{3 5}}
400 {{2 5}}
C
1

C
3
TID Sets of itemsets
200 {{2 3 5}}
300 {{2 3 5}}
L
3
Methods to Improve Aprioris
Efficiency
Hash-based itemset counting: A k-itemset whose corresponding
hashing bucket count is below the threshold cannot be frequent
Transaction reduction: A transaction that does not contain any
frequent k-itemset is useless in subsequent scans
Partitioning: Any itemset that is potentially frequent in DB must be
frequent in at least one of the partitions of DB
Sampling: mining on a subset of given data, lower support threshold +
a method to determine the completeness
Dynamic itemset counting: add new candidate itemsets only when all
of their subsets are estimated to be frequent


Maximal Frequent Itemset
null
AB AC AD AE BC BD BE CD CE DE
A B C D E
ABC ABD ABE ACD ACE ADE BCD BCE BDE CDE
ABCD ABCE ABDE ACDE BCDE
ABCD
E
Border
Infrequent
Itemsets
Maximal
Itemsets
An itemset is maximal frequent if none of its immediate supersets
is frequent
Closed Itemset
An itemset is closed if none of its immediate supersets has
the same support as the itemset

TID Items
1 {A,B}
2 {B,C,D}
3 {A,B,C,D}
4 {A,B,D}
5 {A,B,C,D}
Itemset Support
{A} 4
{B} 5
{C} 3
{D} 4
{A,B} 4
{A,C} 2
{A,D} 3
{B,C} 3
{B,D} 4
{C,D} 3
Itemset Support
{A,B,C} 2
{A,B,D} 3
{A,C,D} 2
{B,C,D} 3
{A,B,C,D} 2
Maximal vs Closed Itemsets
TID Items
1 ABC
2 ABCD
3 BCE
4 ACDE
5 DE
null
AB AC AD AE BC BD BE CD CE DE
A B C D E
ABC ABD ABE ACD ACE ADE BCD BCE BDE CDE
ABCD ABCE ABDE ACDE BCDE
ABCDE
124 123
1234 245 345
12 124 24
4
123
2
3 24
34
45
12
2
24
4 4
2
3 4
2
4
Transaction Ids
Not supported by
any transactions
Maximal vs Closed Frequent Itemsets
null
AB AC AD AE BC BD BE CD CE DE
A B C D E
ABC ABD ABE ACD ACE ADE BCD BCE BDE CDE
ABCD ABCE ABDE ACDE BCDE
ABCDE
124 123
1234 245 345
12 124 24
4
123
2
3 24
34
45
12
2
24
4 4
2
3 4
2
4
Minimum support = 2
# Closed = 9
# Maximal = 4
Closed and
maximal
Closed but
not maximal
Maximal vs Closed Itemsets
Frequent
Itemsets
Closed
Frequent
Itemsets
Maximal
Frequent
Itemsets
Factors Affecting Complexity
Choice of minimum support threshold
lowering support threshold results in more frequent itemsets
this may increase number of candidates and max length of frequent
itemsets
Dimensionality (number of items) of the data set
more space is needed to store support count of each item
if number of frequent items also increases, both computation and
I/O costs may also increase
Size of database
since Apriori makes multiple passes, run time of algorithm may
increase with number of transactions
Average transaction width
transaction width increases with denser data sets
This may increase max length of frequent itemsets and traversals of
hash tree (number of subsets in a transaction increases with its
width)
Rule Generation
Given a frequent itemset L, find all non-empty subsets f c
L such that f L f satisfies the minimum confidence
requirement
If {A,B,C,D} is a frequent itemset, candidate rules:
ABC D, ABD C, ACD B, BCD A,
A BCD, B ACD, C ABD, D ABC
AB CD, AC BD, AD BC, BC AD,
BD AC, CD AB,

If |L| = k, then there are 2
k
2 candidate association rules
(ignoring L C and C L)
Rule Generation
How to efficiently generate rules from frequent
itemsets?
In general, confidence does not have an anti-monotone
property
c(ABC D) can be larger or smaller than c(AB D)

But confidence of rules generated from the same
itemset has an anti-monotone property
e.g., L = {A,B,C,D}:

c(ABC D) > c(AB CD) > c(A BCD)

Confidence is anti-monotone w.r.t. number of items on the
RHS of the rule
Rule Generation for Apriori
Algorithm
ABCD=>{ }
BCD=>A ACD=>B ABD=>C ABC=>D
BC=>AD BD=>AC CD=>AB AD=>BC AC=>BD AB=>CD
D=>ABC C=>ABD B=>ACD A=>BCD
Lattice of rules
ABCD=>{ }
BCD=>A ACD=>B ABD=>C ABC=>D
BC=>AD BD=>AC CD=>AB AD=>BC AC=>BD AB=>CD
D=>ABC C=>ABD B=>ACD A=>BCD
Pruned
Rules
Low
Confidence
Rule
Rule Generation for Apriori
Algorithm
Candidate rule is generated by merging two rules that
share the same prefix
in the rule consequent

join(CD=>AB,BD=>AC)
would produce the candidate
rule D => ABC

Prune rule D=>ABC if its
subset AD=>BC does not have
high confidence
BD=>AC CD=>AB
D=>ABC
Is Apriori Fast Enough?
Performance Bottlenecks
The core of the Apriori algorithm:
Use frequent (k 1)-itemsets to generate candidate frequent k-itemsets
Use database scan and pattern matching to collect counts for the candidate
itemsets
The bottleneck of Apriori: candidate generation
Huge candidate sets:
10
4
frequent 1-itemset will generate 10
7
candidate 2-itemsets
To discover a frequent pattern of size 100, e.g., {a
1
, a
2
, , a
100
}, one needs to
generate 2
100
~ 10
30
candidates.
Multiple scans of database:
Needs (n +1 ) scans, n is the length of the longest pattern
FP-growth: Mining Frequent Patterns
Without Candidate Generation
Compress a large database into a compact, Frequent-
Pattern tree (FP-tree) structure
highly condensed, but complete for frequent pattern mining
avoid costly database scans
Develop an efficient, FP-tree-based frequent pattern mining
method
A divide-and-conquer methodology: decompose mining tasks into
smaller ones
Avoid candidate generation: sub-database test only!
FP-tree Construction from a
Transactional DB
I tem frequency
f 4
c 4
a 3
b 3
m 3
p 3
min_support =3
TID Items bought (ordered) frequent items
100 {f, a, c, d, g, i, m, p} {f, c, a, m, p}
200 {a, b, c, f, l, m, o} {f, c, a, b, m}
300 {b, f, h, j, o} {f, b}
400 {b, c, k, s, p} {c, b, p}
500 {a, f, c, e, l, p, m, n} {f, c, a, m, p}
Steps:
1. Scan DB once, find frequent 1-itemsets (single
item patterns)
2. Order frequent items in descending order of
their frequency
3. Scan DB again, construct FP-tree
FP-tree Construction
root
TID freq. Items bought
100 {f, c, a, m, p}
200 {f, c, a, b, m}
300 {f, b}
400 {c, p, b}
500 {f, c, a, m, p}
I tem frequency
f 4
c 4
a 3
b 3
m 3
p 3
min_support =3
f:1
c:1
a:1
m:1
p:1
FP-tree Construction
root
I tem frequency
f 4
c 4
a 3
b 3
m 3
p 3
min_support =3
f:2
c:2
a:2
m:1
p:1
b:1
m:1
TID freq. Items bought
100 {f, c, a, m, p}
200 {f, c, a, b, m}
300 {f, b}
400 {c, p, b}
500 {f, c, a, m, p}
FP-tree Construction
root
I tem frequency
f 4
c 4
a 3
b 3
m 3
p 3
min_support =3
f:3
c:2
a:2
m:1
p:1
b:1
m:1
b:1
TID freq. Items bought
100 {f, c, a, m, p}
200 {f, c, a, b, m}
300 {f, b}
400 {c, p, b}
500 {f, c, a, m, p}
c:1
b:1
p:1
FP-tree Construction
root
I tem frequency
f 4
c 4
a 3
b 3
m 3
p 3
min_support =3
f:4
c:3
a:3
m:2
p:2
b:1
m:1
b:1
TID freq. Items bought
100 {f, c, a, m, p}
200 {f, c, a, b, m}
300 {f, b}
400 {c, p, b}
500 {f, c, a, m, p}
c:1
b:1
p:1
Header Table
I tem frequency head
f 4
c 4
a 3
b 3
m 3
p 3
Benefits of the FP-tree Structure
Completeness:
never breaks a long pattern of any transaction
preserves complete information for frequent pattern mining
Compactness
reduce irrelevant informationinfrequent items are gone
frequency descending ordering: more frequent items are more likely to
be shared
never be larger than the original database (if not count node-links and
counts)
Example: For Connect-4 DB, compression ratio could be over 100
Mining Frequent Patterns Using FP-
tree
General idea (divide-and-conquer)
Recursively grow frequent pattern path using the FP-tree
Method
For each item, construct its conditional pattern-base, and then its
conditional FP-tree
Repeat the process on each newly created conditional FP-tree
Until the resulting FP-tree is empty, or it contains only one path
(single path will generate all the combinations of its sub-paths, each of
which is a frequent pattern)
Mining Frequent Patterns Using the FP-tree
(contd)
Start with last item in order (i.e., p).
Follow node pointers and traverse only the paths containing p.
Accumulate all of transformed prefix paths of that item to form a
conditional pattern base
Conditional pattern base for p
fcam:2, cb:1
f:4
c:3
a:3
m:2
p:2
c:1
b:1
p:1
p
Construct a new FP-tree based
on this pattern, by merging all
paths and keeping nodes that
appear >sup times. This leads to
only one branch c:3
Thus we derive only one frequent
pattern cont. p. Pattern cp
Mining Frequent Patterns Using the FP-tree
(contd)
Move to next least frequent item in order, i.e., m
Follow node pointers and traverse only the paths containing m.
Accumulate all of transformed prefix paths of that item to form a
conditional pattern base
f:4
c:3
a:3
m:2
m
m:1
b:1
m-conditional
pattern base:
fca:2, fcab:1
{}
f:3
c:3
a:3
m-conditional FP-tree (contains only path fca:3)
All frequent patterns
that include m
m,
fm, cm, am,
fcm, fam, cam,
fcam


Properties of FP-tree for Conditional Pattern
Base Construction
Node-link property
For any frequent item a
i
,

all the possible frequent patterns that contain
a
i
can be obtained by following a
i
's node-links, starting from a
i
's head
in the FP-tree header
Prefix path property
To calculate the frequent patterns for a node a
i
in a path P, only the
prefix sub-path of a
i
in P need to be accumulated, and its frequency
count should carry the same count as node a
i
.
Conditional Pattern-Bases for the example
Empty Empty f
{(f:3)}|c {(f:3)} c
{(f:3, c:3)}|a {(fc:3)} a
Empty {(fca:1), (f:1), (c:1)} b
{(f:3, c:3, a:3)}|m {(fca:2), (fcab:1)} m
{(c:3)}|p {(fcam:2), (cb:1)} p
Conditional FP-tree Conditional pattern-base
Item
Why Is Frequent Pattern Growth
Fast?
Performance studies show
FP-growth is an order of magnitude faster than Apriori, and is also
faster than tree-projection
Reasoning
No candidate generation, no candidate test
Uses compact data structure
Eliminates repeated database scan
Basic operation is counting and FP-tree building
FP-growth vs. Apriori: Scalability With
the Support Threshold
0
10
20
30
40
50
60
70
80
90
100
0 0.5 1 1.5 2 2.5 3
Support threshold(%)
R
u
n

t
i
m
e
(
s
e
c
.
)
D1 FP-growth runtime
D1 Apriori runtime
Data set T25I20D10K
Alternative Methods for Frequent Itemset
Generation
Traversal of Itemset Lattice
General-to-specific vs Specific-to-general



Frequent
itemset
border
null
{a
1
,a
2
,...,a
n
}
(a) General-to-specific
null
{a
1
,a
2
,...,a
n
}
Frequent
itemset
border
(b) Specific-to-general
..
..
..
..
Frequent
itemset
border
null
{a
1
,a
2
,...,a
n
}
(c) Bidirectional
..
..
Alternative Methods for Frequent Itemset
Generation
Traversal of Itemset Lattice
Equivalent Classes



null
AB AC AD BC BD
CD
A B
C
D
ABC ABD ACD
BCD
ABCD
null
AB AC
AD BC BD CD
A B
C
D
ABC
ABD ACD BCD
ABCD
(a) Prefix tree (b) Suffix tree
Alternative Methods for Frequent Itemset
Generation
Traversal of Itemset Lattice
Breadth-first vs Depth-first



(a) Breadth first (b) Depth first
ECLAT
For each item, store a list of transaction ids (tids)



TID Items
1 A,B,E
2 B,C,D
3 C,E
4 A,C,D
5 A,B,C,D
6 A,E
7 A,B
8 A,B,C
9 A,C,D
10 B
Horizontal
Data Layout
A B C D E
1 1 2 2 1
4 2 3 4 3
5 5 4 5 6
6 7 8 9
7 8 9
8 10
9
Vertical Data Layout
TID-list
ECLAT
Determine support of any k-itemset by intersecting tid-lists
of two of its (k-1) subsets.






3 traversal approaches:
top-down, bottom-up and hybrid
Advantage: very fast support counting
Disadvantage: intermediate tid-lists may become too large
for memory
A
1
4
5
6
7
8
9
B
1
2
5
7
8
10

AB
1
5
7
8
Mining Frequent Closed Patterns:
CLOSET
Flist: list of all frequent items in support ascending order
Flist: d-a-f-e-c
Divide search space
Patterns having d
Patterns having d but no a, etc.
Find frequent closed pattern recursively
Every transaction having d also has cfa cfad is a frequent closed
pattern
J. Pei, J. Han & R. Mao. CLOSET: An Efficient Algorithm for
Mining Frequent Closed Itemsets", DMKD'00.
TID Items
10 a, c, d, e, f
20 a, b, e
30 c, e, f
40 a, c, d, f
50 c, e, f
Min_sup=2
Iceberg
Queries
Icerberg query: Compute aggregates over one or a set of
attributes only for those whose aggregate values is above
certain threshold
Example:
select P.custID, P.itemID, sum(P.qty)
from purchase P
group by P.custID, P.itemID
having sum(P.qty) >= 10
Compute iceberg queries efficiently by Apriori:
First compute lower dimensions
Then compute higher dimensions only when all the lower ones are
above the threshold
Multiple-Level Association
Rules
Items often form hierarchy.
Items at the lower level are expected
to have lower support.
Rules regarding itemsets at
appropriate levels could be quite
useful.
A transactional database can be
encoded based on dimensions and
levels
We can explore shared multi-level
mining
Food
bread milk
skim
Fraser
full fat white wheat
Wonder
....
Mining Multi-Level
Associations
A top_down, progressive deepening approach:
First find high-level strong rules:
milk bread [20%, 60%].
Then find their lower-level weaker rules:
full fat milk wheat bread [6%, 50%].
Variations at mining multiple-level association rules.
Level-crossed association rules:
full fat milk Wonder wheat bread
Association rules with multiple, alternative hierarchies:
full fat milk Wonder bread
Multi-Dimensional Association:
Concepts
Single-dimensional rules:
buys(X, milk) buys(X, bread)
Multi-dimensional rules: >2 dimensions or predicates
Inter-dimension association rules (no repeated predicates)
age(X,19-25) . occupation(X,student) buys(X,coke)
hybrid-dimension association rules (repeated predicates)
age(X,19-25) . buys(X, popcorn) buys(X, coke)
Categorical Attributes
finite number of possible values, no ordering among values
Quantitative Attributes
numeric, implicit ordering among values
Techniques for Mining Multi-
Dimensional Associations
Search for frequent k-predicate set:
Example: {age, occupation, buys} is a 3-predicate set.
Techniques can be categorized by how age is treated.
1. Using static discretization of quantitative attributes
Quantitative attributes are statically discretized by using
predefined concept hierarchies.
2. Quantitative association rules
Quantitative attributes are dynamically discretized into
binsbased on the distribution of the data.
3. Distance-based association rules
This is a dynamic discretization process that considers the
distance between data points.
Quantitative Association Rules
age(X,30-34) . income(X,24K -
48K)
buys(X,high resolution TV)
Numeric attributes are dynamically discretized
Such that the confidence or compactness of the rules
mined is maximized.
2-D quantitative association rules: A
quan1
. A
quan2
A
cat

Cluster adjacent
association rules
to form general
rules using a 2-D
grid.
Example:
ARCS (Association Rule Clustering
System)
How does ARCS work?

1. Binning

2. Find frequent predicate-
set

3. Clustering

4. Optimize

Limitations of ARCS
Only quantitative attributes on LHS of rules.
Only 2 attributes on LHS. (2D limitation)
An alternative to ARCS
Non-grid-based
equi-depth binning
clustering based on a measure of partial completeness.
Mining Quantitative Association Rules in Large
Relational Tables by R. Srikant and R. Agrawal.
Interestingness Measurements
Objective measures
Two popular measurements:
support; and
confidence

Subjective measures
A rule (pattern) is interesting if
it is unexpected (surprising to the user); and/or
actionable (the user can do something with it)
Computing Interestingness Measure
Given a rule X Y, information needed to compute rule
interestingness can be obtained from a contingency table
Y Y
X f
11
f
10
f
1+
X f
01
f
00
f
o+
f
+1
f
+0
|T|
Contingency table for X Y
f
11
: support of X and Y
f
10
: support of X and Y
f
01
: support of X and Y
f
00
: support of X and Y
Used to define various measures
support, confidence, lift, Gini,
J-measure, etc.
Drawback of Confidence
Coffee

Coffee
Tea 15 5 20
Tea 75 5 80
90 10 100
Association Rule: Tea Coffee

Confidence= P(Coffee|Tea) = 0.75
but P(Coffee) = 0.9
Although confidence is high, rule is misleading
P(Coffee|Tea) = 0.9375
Statistical Independence
Population of 1000 students
600 students know how to swim (S)
700 students know how to bike (B)
420 students know how to swim and bike (S,B)

P(S.B) = 420/1000 = 0.42
P(S) P(B) = 0.6 0.7 = 0.42

P(S.B) = P(S) P(B) => Statistical independence
P(S.B) > P(S) P(B) => Positively correlated
P(S.B) < P(S) P(B) => Negatively correlated
Statistical-based Measures
Measures that take into account statistical dependence
)] ( 1 )[ ( )] ( 1 )[ (
) ( ) ( ) , (
) ( ) ( ) , (
) ( ) (
) , (
) (
) | (
Y P Y P X P X P
Y P X P Y X P
t coef f icien
Y P X P Y X P PS
Y P X P
Y X P
Interest
Y P
X Y P
Lif t


=
=
=
=
|
Example: Lift/Interest
Coffee

Coffee
Tea 15 5 20
Tea 75 5 80
90 10 100
Association Rule: Tea Coffee

Confidence= P(Coffee|Tea) = 0.75
but P(Coffee) = 0.9
Lift = 0.75/0.9= 0.8333 (< 1, therefore is negatively associated)
There are lots of
measures proposed
in the literature

Some measures are
good for certain
applications, but not
for others

What criteria should
we use to determine
whether a measure
is good or bad?

What about Apriori-
style support based
pruning? How does
it affect these
measures?
Example: |-Coefficient
|-coefficient is analogous to correlation coefficient for
continuous variables
Y Y
X 60 10 70
X 10 20 30
70 30 100
Y Y
X 20 10 30
X 10 60 70
30 70 100
5238 . 0
3 . 0 7 . 0 3 . 0 7 . 0
7 . 0 7 . 0 6 . 0
=


= |
| Coefficient is the same for both tables
5238 . 0
3 . 0 7 . 0 3 . 0 7 . 0
3 . 0 3 . 0 2 . 0
=


= |
Properties of A Good Measure
Piatetsky-Shapiro:
3 properties a good measure M must satisfy:
M(A,B) = 0 if A and B are statistically independent

M(A,B) increase monotonically with P(A,B) when P(A)
and P(B) remain unchanged

M(A,B) decreases monotonically with P(A) [or P(B)]
when P(A,B) and P(B) [or P(A)] remain unchanged
Constraint-Based Mining
Interactive, exploratory mining giga-bytes of data?
Could it be real? Making good use of constraints!
What kinds of constraints can be used in mining?
Knowledge type constraint: classification, association, etc.
Data constraint: SQL-like queries
Find product pairs sold together in Vancouver in Dec.98.
Dimension/level constraints:
in relevance to region, price, brand, customer category.
Interestingness constraints:
strong rules (min_support > 3%, min_confidence > 60%).
Rule constraints
cheap item sales (price < $10) triggers big sales (sum > $200).
Rule Constraints in Association
Mining
Two kind of rule constraints:
Rule form constraints: meta-rule guided
mining.
P(x, y) ^ Q(x, w) takes(x, database systems).
Rule (content) constraint: constraint-based
query optimization.
sum(LHS) < 100 ^ min(LHS) > 20 ^ count(LHS) > 3 ^
sum(RHS) > 1000
Left-hand side
Right-hand side
Constrain-Based Association
Query
Database: (1) trans (TID, Itemset ), (2) itemInfo (Item, Type, Price)
A constrained asso. query (CAQ) is in the form of {(S1, S2 )|C },
where C is a set of constraints on S1, S2 including frequency constraint
A classification of (single-variable) constraints:
Class constraint: S c A. e.g. S c Item
Domain constraint:
Su v, u e { =, =, <, s, >, > }. e.g. S.Price < 100
vu S, u is e or e. e.g. snacks e S.Type
Vu S, or Su V, u e { _, c, ., =, = }
e.g. {snacks, sodas } _ S.Type
Aggregation constraint: agg(S) u v, where agg is in {min, max, sum,
count, avg}, and u e { =, =, <, s, >, > }.
e.g. count(S1.Type) = 1 , avg(S2.Price) < 100
Constrained Association Query
Optimization Problem
Given a CAQ = { (S1, S2) | C }, the algorithm should be :
sound: It only finds frequent sets that satisfy the given
constraints C
complete: All frequent sets satisfy the given constraints
C are found
A nave solution:
Apply Apriori for finding all frequent sets, and then to
test them for constraint satisfaction one by one.
A better approach:
Comprehensive analysis of the properties of constraints
and try to push them as deeply as possible inside the
frequent set computation.
Anti-monotone and Monotone
Constraints
A constraint C
a
is anti-monotone iff. for any pattern S
not satisfying C
a
, none of the super-patterns of S can
satisfy C
a

A constraint C
m
is monotone iff. for any pattern S
satisfying C
m
, every super-pattern of S also satisfies it
Property of
Constraints: Anti-
Monotone
Anti-monotonicity: If a set S violates the constraint,
any superset of S violates the constraint.
Examples:
sum(S.Price) s v is anti-monotone
sum(S.Price) > v is not anti-monotone
Application:
Push sum(S.price) s 1000 deeply into iterative frequent
set computation.
Characterization of
Anti-Monotonicity Constraints
S u v, u e { =, s, > }
v e S
S _ V
S _ V

min(S) s v
min(S) > v

max(S) s v
max(S) > v

count(S) s v
count(S) > v

sum(S) s v
sum(S) > v

avg(S) u v, u e {s, > }
(frequent constraint)
yes
no
no
yes

no
yes

yes
no

yes
no

yes
no

convertible
(yes)
Succinct Constraint
If a rule constraint is succinct set, we can directly
generate precisely the sets that satisfy it, before
support counting begins.
Example: min(I.price)>500, I is a set of products
Knowing the price of each product can directly evaluate
this predicate
Such constraints are precounting prunable
Property of Constraints:
Succinctness
Succinctness:
For any set S1 and S2 satisfying C, S1 S2 satisfies C
Given A1 is the sets of size 1 satisfying C, then any set S satisfying C are
based on A1 , i.e., it contains a subset that belongs to A1 ,
Example :
sum(S.Price ) > v is not succinct
min(S.Price ) s v is succinct
Optimization:
If C is succinct, then C is pre-counting prunable. The satisfaction of
the constraint alone is not affected by the iterative support counting.
Characterization of Constraints
by Succinctness
S u v, u e { =, s, > }
v e S
S _V
S _ V
S = V
min(S) s v
min(S) > v
min(S) = v
max(S) s v
max(S) > v
max(S) = v
count(S) s v
count(S) > v
count(S) = v
sum(S) s v
sum(S) > v
sum(S) = v
avg(S) u v, u e { =, s, > }
(frequent constraint)
Yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
weakly
weakly
weakly
no
no
no
no
(no)
Convertible Constraint
Suppose all items in patterns are listed in a total order
R
A constraint C is convertible anti-monotone iff a
pattern S satisfying the constraint implies that each
suffix of S w.r.t. R also satisfies C
Example: avg(I.price)500
If items are added to the itemset, sorted by price we
know that avg(I.price) can only increase with the
addition of a new item.
Similarly, there are convertible monotone constraints
Example of Convertible
Constraints: Avg(S) u V
Let R be the value descending order over the set of
items
E.g. I={9, 8, 6, 4, 3, 1}
Avg(S) > v is convertible monotone w.r.t. R
If S is a suffix of S
1
, avg(S
1
) > avg(S)
{8, 4, 3} is a suffix of {9, 8, 4, 3}
avg({9, 8, 4, 3})=6 > avg({8, 4, 3})=5
If S satisfies avg(S) >v, so does S
1

{8, 4, 3} satisfies constraint avg(S) > 4, so does {9, 8, 4, 3}