How Query Engines Work Sample

How Query Engines Work
An Introductory Guide
Andy Grove
This book is for sale at http://leanpub.com/how-query-engines-work
This version was published on 2021-04-10
This is a Leanpub book. Leanpub empowers authors and publishers with the Lean Publishing
process. Lean Publishing is the act of publishing an in-progress ebook using lightweight tools and
many iterations to get reader feedback, pivot until you have the right book and build traction once
you do.
© 2020 - 2021 Andy Grove

Tweet This Book!
Please help Andy Grove by spreading the word about this book on Twitter!
The suggested hashtag for this book is #howqueryengineswork.
Find out what other people are saying about the book by clicking on this link to search for this
hashtag on Twitter:
#howqueryengineswork
Contents
1. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2.1 Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
3. What Is a Query Engine? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

3.1 Why Are Query Engines Popular? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
3.2 What This Book Covers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
3.3 Source Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
3.4 Why Kotlin? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
4. Apache Arrow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
4.1 Arrow Memory Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
4.2 Inter-Process Communication (IPC) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
4.3 Arrow Flight Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
4.4 Compute Kernels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
4.5 Query Engines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
5. Choosing a Type System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

5.1 Row-Based or Columnar? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
5.2 Interoperability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
5.3 Type System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
6. Data Sources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
6.1 Data Source Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
6.2 Data Source Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
7. Logical Plans & Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

7.1 Printing Logical Plans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
7.2 Serialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
7.3 Logical Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
7.4 Column Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
7.5 Literal Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
7.6 Binary Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
7.7 Comparison Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
CONTENTS
7.8 Boolean Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

7.9 Math Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
7.10 Aggregate Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
7.11 Logical Plans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
7.12 Scan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
7.13 Projection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
7.14 Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
7.15 Aggregate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
8. Building Logical Plans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

8.1 Building Logical Plans The Hard Way . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
8.2 Building Logical Plans using DataFrames . . . . . . . . . . . . . . . . . . . . . . . . . 9
9. Physical Plans & Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

9.1 Physical Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
9.2 Column Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
9.3 Literal Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
9.4 Binary Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
9.5 Comparison Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
9.6 Math Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
9.7 Aggregate Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
9.8 Physical Plans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
9.9 Scan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
9.10 Projection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
9.11 Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
9.12 Hash Aggregate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
10. Query Optimizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

10.1 Rule-Based Optimizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
10.2 Projection Push-Down . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
10.3 Cost-Based Optimizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
11. Query Execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

11.1 Apache Spark Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
11.2 KQuery Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
11.3 Removing The Query Optimizer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
12. SQL Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

12.1 Tokenizer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
12.2 Pratt Parser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
12.3 Parsing SQL Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
12.4 Parsing a SELECT statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
12.5 SQL Query Planner . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
12.6 Translating SQL Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
CONTENTS
12.7 Planning SELECT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

12.8 Planning for Aggregate Queries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
13. Parallel Query Execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

13.1 Combining Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
13.2 Smarter Partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
13.3 Partition Keys . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
14. Distributed Query Execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

14.1 Distributed Resource Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
14.2 Kubernetes Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
14.3 Serializing a Query Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
14.4 Serializing Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
14.5 Choosing a Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
14.6 Streaming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
14.7 Custom Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
14.8 Distributed Query Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
14.9 Caching Intermediate Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
14.10 Materialized Views . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
14.11 Stability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
15. Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
15.1 Unit Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
15.2 Integration Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
15.3 Fuzzing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
16. Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
16.1 Measuring Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
16.2 Measuring Scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
16.3 Concurrency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
16.4 Automation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
16.5 Comparing Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
16.6 Publishing Benchmark Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
17. Further Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

17.1 Open Source Projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
17.2 YouTube . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
17.3 Sample Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
1. Acknowledgments
This content is not available in the sample book. The book can be purchased on Leanpub at http:
//leanpub.com/how-query-engines-work.
2. Introduction
Since starting my first job in software engineering, I have been fascinated with databases and query
languages. It seems almost magical to ask a computer a question and get meaningful data back
efficiently. After many years of working as a generalist software developer and an end-user of data
technologies, I started working for a startup that threw me into the deep end of distributed database
development. This is the book that I wish had existed when I started on this journey. Although this
is only an introductory-level book, I hope to demystify how query engines work.
My interest in query engines eventually led to me becoming involved in the Apache Arrow project,
where I donated the initial Rust implementation in 2018, then donated the DataFusion in-memory
query engine in 2019, and finally, donated the Ballista distributed compute project in 2021. I do not
plan on building anything else outside the Arrow project and am now continuing to contribute to
these projects within Arrow.
The Arrow project now has many active committers and contributors working on the Rust
implementation, and it is has improved significantly compared to my initial contribution.
Although Rust is a great choice for a high performance query engine, it is not ideal for teaching the
concepts around query engines, so I recently built a new query engine, implemented in Kotlin, as
I was writing this book. Kotlin is a very concise language and easy to read, making it possible to
include source code examples in this book. I would encourage you to get familiar with the source
code as you work your way through this book and consider making some contributions. There is no
better way to learn than to get some hands-on experience!
The query engine covered in this book was originally intended to be part of the Ballista project (and
was for a while) but as the project evolved, it became apparent that it would make more sense to
keep the query engine in Rust and support Java, and other languages, through a UDF mechanism
rather than duplicating large amounts of query execution logic in multiple languages.
Now that Ballista has been donated to Apache Arrow, I have updated this book to refer to the query
engine in the companion repository simply as “KQuery”, short for Kotlin Query Engine, but if anyone
has suggestions for a better name, please let me know!
Thanks to the Leanpub publishing platform, updates to this book will be made available free
of charge as they become available, so please check back occasionally or follow me on twitter
(@andygrove73) to receive notifications when new content is available.
2.1 Feedback
Please visit this book’s Leanpub forum at https://leanpub.com/how-query-engines-work/feedback
to provide feedback, or send me a DM on Twitter at @andygrove73.
3. What Is a Query Engine?
3.1 Why Are Query Engines Popular?

3.2 What This Book Covers

3.3 Source Code

3.4 Why Kotlin?

4. Apache Arrow
4.1 Arrow Memory Model

4.2 Inter-Process Communication (IPC)

4.3 Arrow Flight Protocol

4.4 Compute Kernels

4.5 Query Engines

DataFusion
Apache Arrow 3
Ballista
C++ Query Engine

5. Choosing a Type System
5.1 Row-Based or Columnar?

5.2 Interoperability
5.3 Type System

6. Data Sources
6.1 Data Source Interface

6.2 Data Source Examples

Comma-Separated Values (CSV)

JSON
Parquet
Orc
7. Logical Plans & Expressions
7.1 Printing Logical Plans

7.2 Serialization
7.3 Logical Expressions

7.4 Column Expressions

7.5 Literal Expressions

7.6 Binary Expressions

Logical Plans & Expressions 7
7.7 Comparison Expressions

7.8 Boolean Expressions

7.9 Math Expressions

7.10 Aggregate Expressions

7.11 Logical Plans

7.12 Scan
7.13 Projection
Logical Plans & Expressions 8
7.14 Selection
7.15 Aggregate
8. Building Logical Plans
8.1 Building Logical Plans The Hard Way

8.2 Building Logical Plans using DataFrames

9. Physical Plans & Expressions
9.1 Physical Expressions

9.2 Column Expressions

9.3 Literal Expressions

9.4 Binary Expressions

9.5 Comparison Expressions

9.6 Math Expressions

Physical Plans & Expressions 11
9.7 Aggregate Expressions

9.8 Physical Plans

9.9 Scan
9.10 Projection
9.11 Selection
9.12 Hash Aggregate

10. Query Optimizations
10.1 Rule-Based Optimizations

10.2 Projection Push-Down

10.3 Cost-Based Optimizations

11. Query Execution
11.1 Apache Spark Example

11.2 KQuery Examples

11.3 Removing The Query Optimizer

12. SQL Support
12.1 Tokenizer
12.2 Pratt Parser

12.3 Parsing SQL Expressions

12.4 Parsing a SELECT statement

12.5 SQL Query Planner

12.6 Translating SQL Expressions

SQL Support 15
12.7 Planning SELECT

12.8 Planning for Aggregate Queries

13. Parallel Query Execution
13.1 Combining Results

13.2 Smarter Partitioning

13.3 Partition Keys

14. Distributed Query Execution
14.1 Distributed Resource Scheduling

14.2 Kubernetes Overview

14.3 Serializing a Query Plan

14.4 Serializing Data

14.5 Choosing a Protocol

14.6 Streaming
Distributed Query Execution 18
14.7 Custom Code

14.8 Distributed Query Planning

14.9 Caching Intermediate Results

14.10 Materialized Views

14.11 Stability
15. Testing
15.1 Unit Testing

15.2 Integration Testing

15.3 Fuzzing
16. Benchmarks
16.1 Measuring Performance

16.2 Measuring Scalability

16.3 Concurrency
16.4 Automation
Hardware Configuration
Environment
Benchmarks 21
Benchmark Configuration
Benchmark Results
16.5 Comparing Benchmarks

16.6 Publishing Benchmark Results

17. Further Resources
17.1 Open Source Projects

17.2 YouTube
17.3 Sample Data


How Query Engines Work Sample

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

How Query Engines Work Sample

Uploaded by

Copyright:

Available Formats

How Query Engines Work

This version was published on 2021-04-10

© 2020 - 2021 Andy Grove

3. What Is a Query Engine? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

5. Choosing a Type System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

7. Logical Plans & Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

7.8 Boolean Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

8. Building Logical Plans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

9. Physical Plans & Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

10. Query Optimizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

11. Query Execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

12. SQL Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

12.7 Planning SELECT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

13. Parallel Query Execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

14. Distributed Query Execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

17. Further Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

3.1 Why Are Query Engines Popular?

3.2 What This Book Covers

3.3 Source Code

3.4 Why Kotlin?

4.1 Arrow Memory Model

4.2 Inter-Process Communication (IPC)

4.3 Arrow Flight Protocol

4.4 Compute Kernels

4.5 Query Engines

C++ Query Engine

5.1 Row-Based or Columnar?

5.3 Type System

6.1 Data Source Interface

6.2 Data Source Examples

Comma-Separated Values (CSV)

7.1 Printing Logical Plans

7.3 Logical Expressions

7.4 Column Expressions

7.5 Literal Expressions

7.6 Binary Expressions

7.7 Comparison Expressions

7.8 Boolean Expressions

7.9 Math Expressions

7.10 Aggregate Expressions

7.11 Logical Plans

8.1 Building Logical Plans The Hard Way

8.2 Building Logical Plans using DataFrames

9.1 Physical Expressions

9.2 Column Expressions

9.3 Literal Expressions

9.4 Binary Expressions

9.5 Comparison Expressions

9.6 Math Expressions

9.7 Aggregate Expressions

9.8 Physical Plans

9.12 Hash Aggregate

10.1 Rule-Based Optimizations

10.2 Projection Push-Down

10.3 Cost-Based Optimizations

11.1 Apache Spark Example

11.2 KQuery Examples

11.3 Removing The Query Optimizer

12.2 Pratt Parser

12.3 Parsing SQL Expressions

12.4 Parsing a SELECT statement

12.5 SQL Query Planner

12.6 Translating SQL Expressions