Processing 5

Query Optimization
Immanuel Trummer 
itrummer@cornell.edu
www.itrummer.org
Database Management
Systems (DBMS)
Application 1 Connections, Security, Utilities, ...
Application 2
DBMS Interface Query Processor
Query Parser Query Rewriter
... Query Optimizer Query Executor
Storage Manager
Data Access Buffer Manager
Transaction Manager Recovery Manager
[RG, Sec. 13, 14]

Slides by Immanuel Trummer, Cornell University
Data
Optimizer Overview
Plan
Enumeration
Query
SQL Query Query Plan
Optimizer
Cost
Prediction

Optimizer Overview
Plan
Enumeration
Query
Optimizer
Cost
Prediction

Cost Model Structure
Cost Estimate
Cost Model
Cardinality Model
Selectivity Model
Data Statistics

Cost Estimate
Cost Model
Cardinality Model
Selectivity Model
Data Statistics

Data Statistics
• Diﬀerent DBMS collect diﬀerent statistics
• Example statistics (collected in Postgres):
• Number of distinct values in column
• Ratio of SQL null values in column
• Most frequent values with associated frequency
• Histograms approximating column data distribution
• ...

Data Statistics in PG
• Force DBMS to create statistics via VACUUM ANALYZE
• Generated statistics are exploited by query optimizer
• Inexplicably bad performance? Might miss statistics
• Can access column statistics via pg_stats view
• E.g., tablename, attname, n_distinct, null_frac, ...

Cost Estimate
Cost Model
Cardinality Model
Selectivity Model
Data Statistics

Estimating Selectivity
• Selectivity: probability that row satisfies predicate
• Estimate via constraints and data statistics
• E.g., Selectivity(Column=Value) = 1/NrValues
• E.g., NrRows(Column=Value) ≤ 1 if Key Column
• Selectivity(A⋀B) = Selectivity(A) * Selectivity(B)

Which  
Simplifying Assumptions
Do We Make?

Estimating Selectivity
• Selectivity: probability that row satisfies predicate
• Estimate via constraints and data statistics
• E.g., Selectivity(Column=Value) = 1/NrValues

Assumes Uniform Data
• E.g., NrRows(Column=Value) ≤ 1 if Key Column
• Selectivity(A⋀B) = Selectivity(A) * Selectivity(B)

Assumes Independent Predicates
Cost Estimate
Cost Model
Cardinality Model
Selectivity Model
Data Statistics

Estimating Cardinality
• Assume that cardinality of single tables is given
• DBMS store that information for each base table
• Calculate cardinality product for tables in from clause
• = Number of result rows if predicates are always true
• Now multiply with selectivity estimates for all predicates
• Estimate how many rows pass predicate filter

Estimating Page Size
• Cost functions are based on number of data pages
• Calculate average byte size per record for each result
• Calculate how many records fit on one data page
• Pages cannot store fractional records → round down
• Divide number of rows by number of records per page
• Result is size, measured in pages

Cost Estimate
Cost Model
Cardinality Model
Selectivity Model
Data Statistics

Estimating Cost
• Generally only count cost of page reads and writes
• Sum up cost over all operators in the plan
• For each operator consider the following:
• For each input, how often is it read from disk?
• For each output, is it written to disk? Is it final result?
• Does the operator read/write intermediate results?

Cost Estimation
Example

Input Query and Database
SELECT Count(*)  
FROM Customers C JOIN Orders O USING (cid)  
JOIN Products P USING (pid)  
WHERE C.location = 'Ithaca' AND  
P.name = 'Book Ithaca is Gorges'
Customers(Cid, name, location)

Orders(Cid, Pid, date)
Products(Pid, name, price)

Query Plan Candidate
Aggregate
Count(*)
(On the Fly)
Hash Join 
(Output  ⨝O.Pid=P.Pid
Pipelined)
Scan Filter  
Data Flow
(Output  
)
Materialized
Index NL 
(Output  
)
⨝C.Cid=O.Cid 𝛔P.name='... Gorges'
Materialized
Scan Filter  
(Output   𝛔C.Location='Ithaca' Orders (O) Products (P)
Pipelined) (H as h I n d e x o n 
)
Cid Available
Customers (C) Slides by Immanuel Trummer, Cornell University
Selectivity Estimation
Element Property Value
Customers Cardinality 10,000
Orders Cardinality 100,000
Products Cardinality 5,000
Customers.Location # Values 100
Products.Name # Values 2,500
Foreign Key: Orders.Cid to Customers.Cid

Foreign Key: Orders.Pid to Products.Pid
Predicate Selectivity
𝛔C.Location='Ithaca' ?
C.Cid=O.Cid ?
𝛔P.name='... Gorges' ?
O.Pid=P.Pid ?
Selectivity Estimation
Element Property Value
Customers Cardinality 10,000
Orders Cardinality 100,000
Products Cardinality 5,000
Customers.Location # Values 100
Products.Name # Values 2,500
Foreign Key: Orders.Cid to Customers.Cid

Foreign Key: Orders.Pid to Products.Pid
Predicate Selectivity
𝛔C.Location='Ithaca' 1/100
C.Cid=O.Cid 1/10,000
𝛔P.name='... Gorges' 1/2,500
O.Pid=P.Pid 1/5,000
Cardinality Estimation
Aggregate
Count(*)
(On the Fly)
Hash Join  1/5,000

Pipelined)
Scan Filter  
Data Flow
(Output  
)
Materialized
Index NL  1/10,000 1/2,500
(Output  
)
Materialized
100,000 Rows 5,000 Rows
Scan Filter   1/100
)
10,000 Rows Cid Available
Aggregate
Count(*)
(On the Fly)

Pipelined)
Scan Filter  
Data Flow
(Output  
)
Materialized
Index NL  1/10,000 1/2,500
(Output  
)
Materialized
100 Rows 100,000 Rows 5,000 Rows
)
Aggregate
Count(*)
(On the Fly)

Pipelined)
Scan Filter  
Data Flow
1,000 Rows (Output  

)
Materialized
Index NL  1/10,000 1/2,500
(Output  
)
Materialized
100 Rows 100,000 Rows 5,000 Rows
)
Aggregate
Count(*)
(On the Fly)

Pipelined)
Scan Filter  
Data Flow
1,000 Rows 2 Rows (Output  

)
Materialized
Index NL  1/10,000 1/2,500
(Output  
)
Materialized
100 Rows 100,000 Rows 5,000 Rows
)
Aggregate
Count(*)
(On the Fly)
2/5 Rows
Pipelined)
Scan Filter  
Data Flow
1,000 Rows 2 Rows (Output  

)
Materialized
Index NL  1/10,000 1/2,500
(Output  
)
Materialized
100 Rows 100,000 Rows 5,000 Rows
)
Is That Realistic ... ?

Size Estimation
(Preparation)
Table Columns Size/Entry Rows/Page
Customers Cid, name, location ? ?
Orders Cid, Pid, date ? ?
Products Pid, name, price ? ?
Cid, name, location, pid,

C⨝O ? ?
date
Cid, C.name, location, pid,

C⨝O⨝P ? ?
date, P.name, price
Assume page size of 8,000 bytes,

4 bytes for ints and dates, 10 bytes for strings
Size Estimation
(Preparation)
Customers Cid, name, location 24 ?

C⨝O ? ?
date

C⨝O⨝P ? ?
date, P.name, price

Size Estimation
(Preparation)
Customers Cid, name, location 24 333

C⨝O ? ?
date

C⨝O⨝P ? ?
date, P.name, price

Size Estimation
(Preparation)
Customers Cid, name, location 24 333
Orders Cid, Pid, date 12 666
Products Pid, name, price 18 444

C⨝O 32 250
date

C⨝O⨝P 46 173
date, P.name, price

Size Estimation
Aggregate
Count(*)
(On the Fly)
2/5 Rows 1 Page
Pipelined)
Scan Filter  
Data Flow
4 Pages 1,000 Rows 2 Rows 1 Page (Output  

)
Materialized
Index NL  1/10,000 1/2,500
(Output  
)
⨝C.Cid=O.Cid 𝛔 P.name='... Gorges'
Materialized
1 Page 100 Rows 151 Pages 100,000 Rows 12 Pages 5,000 Rows
)
10,000 Rows 31 Pages Cid Available
Cost Estimation
Aggregate
Count(*)
(On the Fly)
2/5 Rows h
1 Page
m e m o ry 
(En o u g
Hash Join  1/5,000 for single-pass 
(Output  ⨝O.Pid=P.Pid partitioning)
Pipelined)
Scan Filter  
Data Flow

)
Materialized
Index NL  1/10,000 1/2,500
(Output  
)
Materialized
Pipelined) (H a s h I n d e x o n 
)
10,000 Rows 31 Pages Cid Available ucket,  
o n e p a g e p er b
(Count
Customers (C) o n e p e r e n t r y)
Cost Estimation
Aggregate
Count(*)
(On the Fly)
2/5 Rows h
1 Page
m e m o ry 
(En o u g
Pipelined)
Scan Filter  
Data Flow

)
Materialized
Index NL  1/10,000 1/2,500
(Output  
)
Materialized
(Output   𝛔C.Location='Ithaca' 31 Orders (O) Products (P)
)
(Count
Cost Estimation
Aggregate
Count(*)
(On the Fly)
2/5 Rows h
1 Page
m e m o ry 
(En o u g
Pipelined)
Scan Filter  
Data Flow

)
Materialized
Index NL  1/10,000 1/2,500
(Output  
)
1104 ⨝C.Cid=O.Cid 𝛔 P.name='... Gorges'
Materialized
)
(Count
Cost Estimation
Aggregate
Count(*)
(On the Fly)
2/5 Rows h
1 Page
m e m o ry 
(En o u g
Pipelined)
Scan Filter  
Data Flow

)
Materialized
Index NL  1/10,000 1/2,500
(Output   1104 ⨝C.Cid=O.Cid 13 𝛔P.name='... Gorges'
)
Materialized
)
(Count
Cost Estimation
Aggregate
Count(*)
(On the Fly)
2/5 Rows h
1 Page
m e m o ry 
(En o u g
(Output  15 ⨝O.Pid=P.Pid partitioning)
Pipelined)
Scan Filter  
Data Flow

)
Materialized
Index NL  1/10,000 1/2,500
)
Materialized
)
(Count
Cost Estimation
Aggregate 0 Count(*)
(On the Fly)
2/5 Rows h
1 Page
m e m o ry 
(En o u g
Pipelined)
Scan Filter  
Data Flow

)
Materialized
Index NL  1/10,000 1/2,500
)
Materialized
)
(Count
Cost Estimation
Aggregate 0 Count(*)
(On the Fly)
2/5 Rows h
1 Page
m e m o ry 
(En o u g
Pipelined)
Scan Filter  
Data Flow

)
Materialized
Index NL  1/10,000 1/2,500
)
Materialized
)
10,000 Rows 31 Pages Cid Available ucket,   1163
(Count
Optimizer Overview
Plan
Enumeration
Query
Optimizer
Cost
Prediction

Query Plan Space
• Decide order of operations and implementation
• Apply heuristic restrictions
• H1: Apply predicates/projections early
• H2: Avoid predicate-less joins
• H3: Focus on left-deep plans

H1: Early Predicates  
and Projections
• Processing more data is more expensive
• Want to reduce data size as quickly as possible
• Can do that by adding predicates (discarding rows)
• Can also do that by adding projections (discard columns)

Can We Improve This?
Count(*)
⨝O.Pid=P.Pid
Data Flow
𝛔C.Location='Ithaca' Orders (O) Products (P)

H2: Avoid Joins  
without Predicates
• Join result size: product of input cardinality * selectivity
• Selectivity is one when joining tables without predicates
• Often means very large join results, probably sub-optimal
• (Heuristic may discard optimal order in special cases)

H3: Use "Left-Deep Plans"
⨝
⨝ D
⨝ C
A B

Why Left-Deep Plans?
• Allows pipelining: joins pass on result parts in-memory
• Allows to use indices on join columns (of base tables)

Focus on "Left-Deep Plans"
⨝
n g
n i
l i
p e ⨝ D
P i
⨝ C
A B

Naive Plan Enumeration
• Generate every possible plan
• Estimate cost for each plan
• Select plan with minimal cost

Asymptotic Analysis
• Number of join orders can grow quickly O(nrTables! ).
• Number of join orders lower-bounds number of plans
• Enumerating plans is considered impractical

Principle of Optimality
• Want cheapest plan to join set of tables
• This plan joins table subsets "on the way"
• Assume we use sub-optimal plan for joining table subset
• Replacing by a better plan can only improve overall cost
• This is generally called the principle of optimality

Efficient Optimization
• Find optimal plans for (smaller) sub-queries first
• Sub-query: joins subsets of tables
• Compose optimal plans from optimal sub-query plans

Dynamic Programming
C⨝O⨝P

Dynamic Programming
C⨝O⨝P
C⨝O C⨝P O⨝P

Dynamic Programming
C⨝O⨝P
C⨝O C⨝P O⨝P
C O P

Dynamic Programming
C⨝O⨝P
C⨝O C⨝P O⨝P
Phase 1 C O P

Dynamic Programming
C⨝O⨝P
Phase 2 C⨝O C⨝P O⨝P
Phase 1 C O P

Dynamic Programming
Phase 3 C⨝O⨝P
Phase 2 C⨝O C⨝P O⨝P
Phase 1 C O P

Dynamic Programming
Phase 3 C⨝O⨝P
Phase 2
O(2 nrTables
C⨝O C⨝P
* ...)
O⨝P
Phase 1 C O P

Processing 5

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Processing 5

Uploaded by

Copyright:

Available Formats

Query Optimization

... Query Optimizer Query Executor

Transaction Manager Recovery Manager

[RG, Sec. 13, 14]

Slides by Immanuel Trummer, Cornell University

Slides by Immanuel Trummer, Cornell University

Slides by Immanuel Trummer, Cornell University

Slides by Immanuel Trummer, Cornell University

• Example statistics (collected in Postgres):

• Number of distinct values in column

• Ratio of SQL null values in column

• Most frequent values with associated frequency

• Histograms approximating column data distribution

Slides by Immanuel Trummer, Cornell University

• Force DBMS to create statistics via VACUUM ANALYZE

• Generated statistics are exploited by query optimizer

• Inexplicably bad performance? Might miss statistics

• Can access column statistics via pg_stats view

• E.g., tablename, attname, n_distinct, null_frac, ...

Slides by Immanuel Trummer, Cornell University

Slides by Immanuel Trummer, Cornell University

• Estimate via constraints and data statistics

• E.g., Selectivity(Column=Value) = 1/NrValues

• E.g., NrRows(Column=Value) ≤ 1 if Key Column

• Selectivity(A⋀B) = Selectivity(A) * Selectivity(B)

Slides by Immanuel Trummer, Cornell University

Slides by Immanuel Trummer, Cornell University

• Estimate via constraints and data statistics

• E.g., Selectivity(Column=Value) = 1/NrValues

• Selectivity(A⋀B) = Selectivity(A) * Selectivity(B)

Slides by Immanuel Trummer, Cornell University

• DBMS store that information for each base table

• Calculate cardinality product for tables in from clause

• = Number of result rows if predicates are always true

• Now multiply with selectivity estimates for all predicates

• Estimate how many rows pass predicate filter

Slides by Immanuel Trummer, Cornell University

• Calculate average byte size per record for each result

• Calculate how many records fit on one data page

• Pages cannot store fractional records → round down

• Divide number of rows by number of records per page

• Result is size, measured in pages

Slides by Immanuel Trummer, Cornell University

Slides by Immanuel Trummer, Cornell University

• Sum up cost over all operators in the plan

• For each operator consider the following:

• For each input, how often is it read from disk?

• For each output, is it written to disk? Is it final result?

• Does the operator read/write intermediate results?

Slides by Immanuel Trummer, Cornell University

Slides by Immanuel Trummer, Cornell University

Customers(Cid, name, location)

Slides by Immanuel Trummer, Cornell University

Foreign Key: Orders.Cid to Customers.Cid

Foreign Key: Orders.Cid to Customers.Cid

Hash Join 1/5,000

Hash Join 1/5,000

Hash Join 1/5,000

1,000 Rows (Output

Hash Join 1/5,000

1,000 Rows 2 Rows (Output

1,000 Rows 2 Rows (Output

Slides by Immanuel Trummer, Cornell University

Customers Cid, name, location ? ?

Hash Join  1/5,000

Hash Join  1/5,000

Hash Join  1/5,000

1,000 Rows (Output  

Hash Join  1/5,000

1,000 Rows 2 Rows (Output  

1,000 Rows 2 Rows (Output  

4 Pages 1,000 Rows 2 Rows 1 Page (Output  

4 Pages 1,000 Rows 2 Rows 1 Page (Output  

4 Pages 1,000 Rows 2 Rows 1 Page (Output  

4 Pages 1,000 Rows 2 Rows 1 Page (Output  

4 Pages 1,000 Rows 2 Rows 1 Page (Output  

4 Pages 1,000 Rows 2 Rows 1 Page (Output  

4 Pages 1,000 Rows 2 Rows 1 Page (Output  

4 Pages 1,000 Rows 2 Rows 1 Page (Output  