You are on page 1of 424

Why?

• You don’t need deep knowledge of algorithms & data
structures to be a software engineer

• However, they are fundamental to a real
understanding of what is happening when you write
some code.
What is an algorithm?
What is an algorithm?

A list of actions, to be followed in order, to achieve some
goal
Agenda
1. Calculating Runtimes (Big O)
2. Algorithms
A. Searching (Linear/Binary)

B. Sorting (Selection/Insertion/Bubble/Merge/Quick/Counting)

3. Data Structures
C. Array

D. Lists(Single/Double)

E. ArrayList

F. Stack

G. Queue

H. Tree

I. Binary Search Tree

J. Hash Table

K. Set

L. Graph

M. Heap
Runtimes
Collection<String> animals = new ArrayList<>();
animals.add("cat");
animals.add("fish");
// ….
animals.contains("dog");

Vs

Collection<String> animals = new Set<>();
animals.add("cat");
animals.add("fish");
// ….
animals.contains("dog");
Runtimes
You’ll need some Algebra to fully understand Big
O and asymptotic analysis.

If your algebra is rusty, there are a lot of online
resources to review

Udacity College Algebra
Fibonacci Sequence

F20 = 6765
F50 = 12586269025
F100 = 354224848179261915075
F500 =
139423222456169788013972438287040728
395007025658769730726410896294832557
1622863290691557658876222521294125
Fibonacci Sequence
def FibRecurse(n):
if n <= 1:
return n
else:
return FibRecurse(n - 1) + FibRecurse(n - 2)

Vs

def FibList(n):
create an array F[0…n]
F[0] = 0
F[1] = 1
for i from 2 to n:
F[i] = F[i - 1] + F[i - 2]
return F[n]
How fast is this?

def FibList(n):
create an array F[0…n]
F[0] = 0
F[1] = 1
for i from 2 to n:
F[i] = F[i - 1] + F[i - 2]
return F[n]
T(n)

def FibList(n):
create an array F[0…n] 1
F[0] = 0 1
F[1] = 1 1
for i from 2 to n: n *
F[i] = F[i - 1] + F[i - 2] 1
return F[n] 1
Common runtimes
O(1) Constant

O(log n) Logarithmic

O(n) Linear

O(n log n) Linearithmic

(n2) Quadratic

O(n3) Cubic

O(2n) Exponential

https://www.desmos.com/calculator/ty01lmuivl
O(n)

def FibList(n):
create an array F[0…n] O(n)
F[0] = 0 O(1)
F[1] = 1 O(1)
for i from 2 to n: O(n) *
F[i] = F[i - 1] + F[i - 2] O(1)
return F[n] O(1)

O(n) + O(1) + O(1) + O(n) * O(1) + O(1) = ???
Recursive

def FibRecurse(n):
if n <= 1: O(1)
return n O(1)
else: O(1)
return FibRecurse(n - 1) + FibRecurse(n - 2) O(???)
Recursive
4

def FibRecurse(n):
if n <= 1:
return n
else:
return FibRecurse(n - 1) + FibRecurse(n - 2)
Recursive
4

3 2

def FibRecurse(n):
if n <= 1:
return n
else:
return FibRecurse(n - 1) + FibRecurse(n - 2)
Recursive
4

3 2

2 1

def FibRecurse(n):
if n <= 1:
return n
else:
return FibRecurse(n - 1) + FibRecurse(n - 2)
Recursive
4

3 2

2 1

1 0

def FibRecurse(n):
if n <= 1:
return n
else:
return FibRecurse(n - 1) + FibRecurse(n - 2)
Recursive
4

3 2

2 1 1 0

1 0

def FibRecurse(n):
if n <= 1:
return n
else:
return FibRecurse(n - 1) + FibRecurse(n - 2)
Recursive
4

3 2

2 1 1 0

1 0

def FibRecurse(n):
if n <= 1:
return n
else:
return FibRecurse(n - 1) + FibRecurse(n - 2)
Recursive
5

4 3

3 2 2 1

2 1 1 0 1 0

1 0

def FibRecurse(n):
if n <= 1:
return n
else:
return FibRecurse(n - 1) + FibRecurse(n - 2)
Recursive
5

4 3

3 2 2 1

2 1 1 0 1 0

1 0

O(2n)
def FibRecurse(n):
if n <= 1:
return n
else:
return FibRecurse(n - 1) + FibRecurse(n - 2)
Common runtimes
Common runtimes
Agenda
1. Calculating Runtimes (Big O)
2. Algorithms
A. Searching (Linear/Binary)

B. Sorting (Selection/Insertion/Bubble/Merge/Quick/Counting)

3. Data Structures
C. Array

D. Lists(Single/Double)

E. ArrayList

F. Stack

G. Queue

H. Tree

I. Binary Search Tree

J. Hash Table

K. Set

L. Graph

M. Heap
Searching

3 4 5 7 1 8 9 6 2
Searching

3 4 5 7 1 8 9 6 2

How many steps does it take to check if the
number 6 is in this collection?
Linear Search

6?

3 4 5 7 1 8 9 6 2
Linear Search

6?

3 4 5 7 1 8 9 6 2
Linear Search

6?

3 4 5 7 1 8 9 6 2
Linear Search

6?

3 4 5 7 1 8 9 6 2
Linear Search

6?

3 4 5 7 1 8 9 6 2
Linear Search

6?

3 4 5 7 1 8 9 6 2
Linear Search

6?

3 4 5 7 1 8 9 6 2
Linear Search

6?

3 4 5 7 1 8 9 6 2
Searching

1 2 3 4 5 6 7 8 9

What about if the collection is sorted?
How many steps to find if number 6 is in this
collection?
Binary Search

6?

1 2 3 4 5 6 7 8 9
Binary Search

6?

1 2 3 4 5 6 7 8 9
5 is less than 6, so everything to the left of 5 can be ignored
Binary Search

6?

1 2 3 4 5 6 7 8 9
Binary Search

6?

1 2 3 4 5 6 7 8 9
7 is greater than 6, so everything to the right of 7 can be ignored
Binary Search

6?

1 2 3 4 5 6 7 8 9
Binary Search

1 2 3 4 5 6 7 8 9

In this case, the search took 3 steps, rather than the 9 steps using linear
search.

In a collection of 1 million elements, linear search will take time
proportional to 1 million, binary search will take time proportional to 20
Sorting

3 4 5 7 1 8 9 6 2
Selection Sort

3 4 5 7 1 8 9 6 2
Selection Sort

3 4 5 7 1 8 9 6 2
Selection Sort

1 4 5 7 3 8 9 6 2
Selection Sort

1 4 5 7 3 8 9 6 2
Selection Sort

1 4 5 7 3 8 9 6 2
Selection Sort

1 2 5 7 3 8 9 6 4
Selection Sort

1 2 5 7 3 8 9 6 4
Selection Sort

1 2 3 7 5 8 9 6 4
Selection Sort

1 2 3 7 5 8 9 6 4
Selection Sort

1 2 3 4 5 8 9 6 7
Selection Sort

1 2 3 4 5 8 9 6 7
Selection Sort

1 2 3 4 5 8 9 6 7
Selection Sort

1 2 3 4 5 8 9 6 7
Selection Sort

1 2 3 4 5 6 9 8 7
Selection Sort

1 2 3 4 5 6 9 8 7
Selection Sort

1 2 3 4 5 6 7 8 9
Selection Sort

1 2 3 4 5 6 7 8 9
Selection Sort

1 2 3 4 5 6 7 8 9
Selection Sort

1 2 3 4 5 6 7 8 9
Selection Sort

1 2 3 4 5 6 7 8 9

What is the complexity of selection sort?
Insertion Sort

3 4 5 7 1 8 9 6 2
Insertion Sort

3 4 5 7 1 8 9 6 2
Insertion Sort

3 4 5 7 1 8 9 6 2
Insertion Sort

3 4 5 7 1 8 9 6 2
Insertion Sort

1
3 4 5 7 8 9 6 2
Insertion Sort

1
3 4 5 7 8 9 6 2
Insertion Sort

1 3 4 5 7 8 9 6 2
Insertion Sort

1 3 4 5 7 8 9 6 2
Insertion Sort

1 3 4 5 7 8 9 6 2
Insertion Sort

6
1 3 4 5 7 8 9 2
Insertion Sort

6
1 3 4 5 7 8 9 2
Insertion Sort

1 3 4 5 6 7 8 9 2
Insertion Sort

2
1 3 4 5 6 7 8 9
Insertion Sort

2
1 3 4 5 6 7 8 9
Insertion Sort

1 2 3 4 5 6 7 8 9
Insertion Sort

1 2 3 4 5 6 7 8 9

What is the complexity of insertion sort?
Bubble Sort

3 4 5 7 1 8 9 6 2
Bubble Sort

3 4 5 7 1 8 9 6 2
Bubble Sort

3 4 5 7 1 8 9 6 2
Bubble Sort

3 4 5 7 1 8 9 6 2
Bubble Sort

3 4 5 1 7 8 9 6 2
Bubble Sort

3 4 5 1 7 8 9 6 2
Bubble Sort

3 4 5 1 7 8 9 6 2
Bubble Sort

3 4 5 1 7 8 9 6 2
Bubble Sort

3 4 5 1 7 8 6 9 2
Bubble Sort

3 4 5 1 7 8 6 9 2
Bubble Sort

3 4 5 1 7 8 6 2 9
Bubble Sort

3 4 5 1 7 8 6 2 9
Bubble Sort

3 4 5 1 7 8 6 2 9
Bubble Sort

3 4 5 1 7 8 6 2 9
Bubble Sort

3 4 1 5 7 8 6 2 9
Bubble Sort

3 4 1 5 7 8 6 2 9
Bubble Sort

3 4 1 5 7 8 6 2 9
Bubble Sort

3 4 1 5 7 8 6 2 9
Bubble Sort

3 4 1 5 7 6 8 2 9
Bubble Sort

3 4 1 5 7 6 8 2 9
Bubble Sort

3 4 1 5 7 6 2 8 9
Bubble Sort

3 4 1 5 7 6 2 8 9
Bubble Sort

3 4 1 5 7 6 2 8 9
Bubble Sort

3 4 1 5 7 6 2 8 9
Bubble Sort

3 1 4 5 7 6 2 8 9
Bubble Sort

3 1 4 5 7 6 2 8 9
Bubble Sort

3 1 4 5 7 6 2 8 9
Bubble Sort

3 1 4 5 7 6 2 8 9
Bubble Sort

3 1 4 5 6 7 2 8 9
Bubble Sort

3 1 4 5 6 7 2 8 9
Bubble Sort

3 1 4 5 6 2 7 8 9
Bubble Sort

3 1 4 5 6 2 7 8 9
Bubble Sort

3 1 4 5 6 2 7 8 9
Bubble Sort

3 1 4 5 6 2 7 8 9
Bubble Sort

1 3 4 5 6 2 7 8 9
Bubble Sort

1 3 4 5 6 2 7 8 9
Bubble Sort

1 3 4 5 6 2 7 8 9
Bubble Sort

1 3 4 5 6 2 7 8 9
Bubble Sort

1 3 4 5 6 2 7 8 9
Bubble Sort

1 3 4 5 2 6 7 8 9
Bubble Sort

1 3 4 5 2 6 7 8 9
Bubble Sort

1 3 4 5 2 6 7 8 9
Bubble Sort

1 3 4 5 2 6 7 8 9
Bubble Sort

1 3 4 5 2 6 7 8 9
Bubble Sort

1 3 4 5 2 6 7 8 9
Bubble Sort

1 3 4 5 2 6 7 8 9
Bubble Sort

1 3 4 5 2 6 7 8 9
Bubble Sort

1 3 4 2 5 6 7 8 9
Bubble Sort

1 3 4 2 5 6 7 8 9
Bubble Sort

1 3 4 2 5 6 7 8 9
Bubble Sort

1 3 4 2 5 6 7 8 9
Bubble Sort

1 3 4 2 5 6 7 8 9
Bubble Sort

1 3 4 2 5 6 7 8 9
Bubble Sort

1 3 4 2 5 6 7 8 9
Bubble Sort

1 3 4 2 5 6 7 8 9
Bubble Sort

1 3 2 4 5 6 7 8 9
Bubble Sort

1 3 2 4 5 6 7 8 9
Bubble Sort

1 3 2 4 5 6 7 8 9
Bubble Sort

1 3 2 4 5 6 7 8 9
Bubble Sort

1 3 2 4 5 6 7 8 9
Bubble Sort

1 3 2 4 5 6 7 8 9
Bubble Sort

1 3 2 4 5 6 7 8 9
Bubble Sort

1 3 2 4 5 6 7 8 9
Bubble Sort

1 2 3 4 5 6 7 8 9
Bubble Sort

1 2 3 4 5 6 7 8 9
Bubble Sort

1 2 3 4 5 6 7 8 9
Bubble Sort

1 2 3 4 5 6 7 8 9
Bubble Sort

1 2 3 4 5 6 7 8 9
Bubble Sort

1 2 3 4 5 6 7 8 9
Bubble Sort

1 2 3 4 5 6 7 8 9
Bubble Sort

1 2 3 4 5 6 7 8 9
Bubble Sort

1 2 3 4 5 6 7 8 9
Bubble Sort

1 2 3 4 5 6 7 8 9
Bubble Sort

1 2 3 4 5 6 7 8 9
Bubble Sort

1 2 3 4 5 6 7 8 9
Bubble Sort

1 2 3 4 5 6 7 8 9
Bubble Sort

1 2 3 4 5 6 7 8 9
Bubble Sort

1 2 3 4 5 6 7 8 9
Bubble Sort

1 2 3 4 5 6 7 8 9

What is the complexity of bubble sort?
Merge Sort

3 4 5 7 1 8 9 6 2
Merge Sort
3 4 5 7 1 8 9 6 2
Merge Sort
3 4 5 7 (magic)
1 2 6 8 9
Merge Sort
3 4 5 7 1 2 6 8 9
Merge Sort
3 4 5 7 2 6 8 9

1
Merge Sort
3 4 5 7 6 8 9

1 2
Merge Sort
4 5 7 6 8 9

1 2 3
Merge Sort
5 7 6 8 9

1 2 3 4
Merge Sort
7 6 8 9

1 2 3 4 5
Merge Sort
7 8 9

1 2 3 4 5 6
Merge Sort
8 9

1 2 3 4 5 6 7
Merge Sort
9

1 2 3 4 5 6 7 8
Merge Sort

1 2 3 4 5 6 7 8 9
What is the complexity of the merge?
Merge Sort

How do we sort the sub lists?
Merge Sort

How do we sort the sub lists?

Merge Sort!
Merge Sort
1 8 9 6 2

3 4 5 7
Merge Sort
1 8 9 6 2

3 4 5 7
Merge Sort
1 8 9 6 2

5 7

3 4
Merge Sort
1 8 9 6 2

5 7

3 4
Merge Sort
1 8 9 6 2

5 7

3 4
Merge Sort
1 8 9 6 2

5 7

4

3
Merge Sort
1 8 9 6 2

5 7

3 4
Merge Sort
1 8 9 6 2

3 4 5 7
Merge Sort
1 8 9 6 2

3 4

5 7
Merge Sort
1 8 9 6 2

3 4

5 7
Merge Sort
1 8 9 6 2

3 4

5 7
Merge Sort
1 8 9 6 2

3 4

7

5
Merge Sort
1 8 9 6 2

3 4

5 7
Merge Sort
1 8 9 6 2

3 4 5 7
Merge Sort
1 8 9 6 2

3 4 5 7
Merge Sort
1 8 9 6 2

4 5 7

3
Merge Sort
1 8 9 6 2

5 7

3 4
Merge Sort
1 8 9 6 2

7

3 4 5
Merge Sort
1 8 9 6 2

3 4 5 7
Merge Sort
3 4 5 7 1 8 9 6 2
Merge Sort
3 4 5 7

1 8 9 6 2
Merge Sort
3 4 5 7

1 8 9 6 2
Merge Sort
3 4 5 7

6 2
1 8 9
Merge Sort
3 4 5 7

6 2
1 8 9
Merge Sort
3 4 5 7

6 2
9
1 8
Merge Sort
3 4 5 7

6 2
9
1 8
Merge Sort
3 4 5 7

6 2
9
1 8
Merge Sort
3 4 5 7

6 2
9
8

1
Merge Sort
3 4 5 7

6 2
9

1 8
Merge Sort
3 4 5 7

6 2
1 8 9
Merge Sort
3 4 5 7

6 2
1 8 9
Merge Sort
3 4 5 7

6 2
8 9

1
Merge Sort
3 4 5 7

6 2
9

1 8
Merge Sort
3 4 5 7

6 2

1 8 9
Merge Sort
3 4 5 7

1 8 9 6 2
Merge Sort
3 4 5 7

1 8 9

6 2
Merge Sort
3 4 5 7

1 8 9

6 2
Merge Sort
3 4 5 7

1 8 9

6 2
Merge Sort
3 4 5 7

1 8 9

6

2
Merge Sort
3 4 5 7

1 8 9

2 6
Merge Sort
3 4 5 7

1 8 9 2 6
Merge Sort
3 4 5 7

1 8 9 2 6
Merge Sort
3 4 5 7

8 9 2 6

1
Merge Sort
3 4 5 7

8 9 6

1 2
Merge Sort
3 4 5 7

8 9

1 2 6
Merge Sort
3 4 5 7

9

1 2 6 8
Merge Sort
3 4 5 7

1 2 6 8 9
Merge Sort
3 4 5 7 1 2 6 8 9
Merge Sort
3 4 5 7 1 2 6 8 9
Merge Sort
3 4 5 7 2 6 8 9

1
Merge Sort
3 4 5 7 6 8 9

1 2
Merge Sort
4 5 7 6 8 9

1 2 3
Merge Sort
5 7 6 8 9

1 2 3 4
Merge Sort
7 6 8 9

1 2 3 4 5
Merge Sort
7 8 9

1 2 3 4 5 6
Merge Sort
8 9

1 2 3 4 5 6 7
Merge Sort
9

1 2 3 4 5 6 7 8
Merge Sort

1 2 3 4 5 6 7 8 9
Merge Sort

1 2 3 4 5 6 7 8 9
What is the complexity of mergesort?
Merge Sort
(The numbers are cleaner with an even n)

How many layers - and how much work in each layer?

3 4 5 7 1 8 9 6
3 4 5 7 1 8 9 6
3 4 5 7 1 8 9 6
3 4 5 7 1 8 9 6
Merge Sort
(The numbers are cleaner with an even n)

How many layers - and how much work in each layer?

O(n)
3 4 5 7 1 8 9 6
O(n/2) O(n/2) O(n)
O(log n)

3 4 5 7 1 8 9 6
O(n/4) O(n/4) O(n/4) O(n/4) O(n)
3 4 5 7 1 8 9 6
O(n/16) O(n/16) O(n/16) O(n/16) O(n/16) O(n/16)O(n/16) O(n/16) O(n)
3 4 5 7 1 8 9 6
Quick Sort

3 4 5 7 1 8 9 6 2
Quick Sort

Pivot

3 4 5 7 1 8 9 6 2
Less Greater
Unknown
Than Than
Quick Sort

Pivot

3 4 5 7 1 8 9 6 2
Less Greater
Unknown
Than Than
Quick Sort

Pivot

3 4 5 7 1 8 9 6 2
Less Greater
Unknown
Than Than
Quick Sort

Pivot

3 4 5 7 1 8 9 6 2
Less Greater
Unknown
Than Than
Quick Sort

Pivot

3 4 5 7 1 8 9 6 2
Less Greater
Unknown
Than Than
Quick Sort

Pivot

3 1 5 7 4 8 9 6 2
Less Greater
Unknown
Than Than
Quick Sort

Pivot

3 1 5 7 4 8 9 6 2
Less Greater
Unknown
Than Than
Quick Sort

Pivot

1 3 5 7 4 8 9 6 2
Less Greater
Unknown
Than Than
Quick Sort

Pivot

1 3 5 7 4 8 9 6 2
Less Greater
Unknown
Than Than
Quick Sort

Pivot

1 3 5 7 4 8 9 6 2
Less Greater
Unknown
Than Than
Quick Sort

Pivot

1 3 5 7 4 8 9 6 2
Less Greater
Unknown
Than Than
Quick Sort

Pivot

1 3 5 7 4 8 9 6 2
Less Greater
Unknown
Than Than
Quick Sort

Pivot

1 3 5 7 4 8 9 6 2
Less Greater
Unknown
Than Than
Quick Sort

Pivot

1 3 2 7 4 8 9 6 5
Less Greater
Unknown
Than Than
Quick Sort

Pivot

1 3 2 7 4 8 9 6 5
Less Greater
Unknown
Than Than
Quick Sort

Pivot

1 2 3 7 4 8 9 6 5
Less Greater
Unknown
Than Than
Quick Sort

Pivot

1 2 3 7 4 8 9 6 5
Less Greater
Unknown
Than Than
Quick Sort

Correct Location

1 2 3 7 4 8 9 6 5

Can be sorted separately
Quick Sort

Pivot

1 2 3 7 4 8 9 6 5
Quick Sort

Correct Location

1 2 3 7 4 8 9 6 5
Quick Sort

Pivot

1 2 3 7 4 8 9 6 5
Quick Sort

Correct Location

1 2 3 7 4 8 9 6 5
Quick Sort

Pivot

1 2 3 7 4 8 9 6 5
Quick Sort

Pivot

1 2 3 4 7 8 9 6 5
Quick Sort

Pivot

1 2 3 4 7 8 9 6 5
Quick Sort

Pivot

1 2 3 4 7 8 9 6 5
Quick Sort

Pivot

1 2 3 4 6 7 9 8 5
Quick Sort

Pivot

1 2 3 4 6 5 7 8 9
Quick Sort

Correct Location

1 2 3 4 6 5 7 8 9
Quick Sort

Pivot

1 2 3 4 6 5 7 8 9
Quick Sort

Pivot

1 2 3 4 6 5 7 8 9
Quick Sort

Pivot

1 2 3 4 6 5 7 8 9
Quick Sort

Correct Location

1 2 3 4 6 5 7 8 9
Quick Sort

Pivot

1 2 3 4 6 5 7 8 9
Quick Sort

Pivot

1 2 3 4 5 6 7 8 9
Quick Sort

Correct Location

1 2 3 4 5 6 7 8 9
Quick Sort

Pivot

1 2 3 4 5 6 7 8 9
Quick Sort

Correct Location

1 2 3 4 5 6 7 8 9
Quick Sort

Pivot

1 2 3 4 5 6 7 8 9
Quick Sort

Pivot

1 2 3 4 5 6 7 8 9
Quick Sort

Correct Location

1 2 3 4 5 6 7 8 9
Quick Sort

Pivot

1 2 3 4 5 6 7 8 9
Quick Sort

Correct Location

1 2 3 4 5 6 7 8 9
Quick Sort

1 2 3 4 5 6 7 8 9
Quick Sort

1 2 3 4 5 6 7 8 9

What is the complexity of quicksort?
Quick Sort

1 2 3 4 5 6 7 8 9

What is the complexity of quicksort?
O(n2)?? Why did you teach me that?
Quick Sort

Worst case: already sorted input

1 2 3 4 5 6 7 8 9
Quick Sort

Expected on random input: O(n log n)

8 4 9 7 5 1 3 6 2
If our pivot is roughly in the middle
Quick Sort

Expected on random input: O(n log n)

8 4 9 7 5 1 3 6 2
If our pivot is roughly in the middle

We can shuffle an array in O(n) time
Quick Sort

Expected on random input: O(n log n)

8 4 9 7 5 1 3 6 2
If our pivot is roughly in the middle

We can shuffle an array in O(n) time


Comparison-based sorting

For comparison-based sorting O(n log n)
is the best possible time

If we know more about our data, we can
sometimes do better
Counting Sort
If we know that our collection only contains integers in a given
(small) range, counting sort can be applied.

In this case, we have numbers between 1 and 10.

3 4 5 2 2 5 9 6 2
Counting Sort

3 4 5 2 2 5 9 6 2

1: 0 6: 0
2: 0 7: 0
3: 0 8: 0
4: 0 9: 0
5: 0 10: 0
Counting Sort

3 4 5 2 2 5 9 6 2

1: 0 6: 0
2: 0 7: 0
3: 1 8: 0
4: 0 9: 0
5: 0 10: 0
Counting Sort

3 4 5 2 2 5 9 6 2

1: 0 6: 0
2: 0 7: 0
3: 1 8: 0
4: 1 9: 0
5: 0 10: 0
Counting Sort

3 4 5 2 2 5 9 6 2

1: 0 6: 0
2: 0 7: 0
3: 1 8: 0
4: 1 9: 0
5: 1 10: 0
Counting Sort

3 4 5 2 2 5 9 6 2

1: 0 6: 0
2: 1 7: 0
3: 1 8: 0
4: 1 9: 0
5: 1 10: 0
Counting Sort

3 4 5 2 2 5 9 6 2

1: 0 6: 0
2: 2 7: 0
3: 1 8: 0
4: 1 9: 0
5: 1 10: 0
Counting Sort

3 4 5 2 2 5 9 6 2

1: 0 6: 0
2: 2 7: 0
3: 1 8: 0
4: 1 9: 0
5: 2 10: 0
Counting Sort

3 4 5 2 2 5 9 6 2

1: 0 6: 0
2: 2 7: 0
3: 1 8: 0
4: 1 9: 1
5: 2 10: 0
Counting Sort

3 4 5 2 2 5 9 6 2

1: 0 6: 1
2: 2 7: 0
3: 1 8: 0
4: 1 9: 1
5: 2 10: 0
Counting Sort

3 4 5 2 2 5 9 6 2

1: 0 6: 1
2: 3 7: 0
3: 1 8: 0
4: 1 9: 1
5: 2 10: 0
Counting Sort
3 4 5 2 2 5 9 6 2

1: 0 6: 1
2: 3 7: 0
3: 1 8: 0
4: 1 9: 1
5: 2 10: 0
Counting Sort
3 4 5 2 2 5 9 6 2

1: 0 6: 1
2: 3 7: 0
3: 1 8: 0
4: 1 9: 1
5: 2 10: 0
Counting Sort
3 4 5 2 2 5 9 6 2

1: 0 6: 1
2: 3 7: 0
3: 1 8: 0
4: 1 9: 1
5: 2 10: 0
Counting Sort
3 4 5 2 2 5 9 6 2

2
1: 0 6: 1
2: 2 7: 0
3: 1 8: 0
4: 1 9: 1
5: 2 10: 0
Counting Sort
3 4 5 2 2 5 9 6 2

2 2
1: 0 6: 1
2: 1 7: 0
3: 1 8: 0
4: 1 9: 1
5: 2 10: 0
Counting Sort
3 4 5 2 2 5 9 6 2

2 2 2
1: 0 6: 1
2: 0 7: 0
3: 1 8: 0
4: 1 9: 1
5: 2 10: 0
Counting Sort
3 4 5 2 2 5 9 6 2

2 2 2
1: 0 6: 1
2: 0 7: 0
3: 1 8: 0
4: 1 9: 1
5: 2 10: 0
Counting Sort
3 4 5 2 2 5 9 6 2

2 2 2 3
1: 0 6: 1
2: 0 7: 0
3: 0 8: 0
4: 1 9: 1
5: 2 10: 0
Counting Sort
3 4 5 2 2 5 9 6 2

2 2 2 3
1: 0 6: 1
2: 0 7: 0
3: 0 8: 0
4: 1 9: 1
5: 2 10: 0
Counting Sort
3 4 5 2 2 5 9 6 2

2 2 2 3 4
1: 0 6: 1
2: 0 7: 0
3: 0 8: 0
4: 0 9: 1
5: 2 10: 0
Counting Sort
3 4 5 2 2 5 9 6 2

2 2 2 3 4
1: 0 6: 1
2: 0 7: 0
3: 0 8: 0
4: 0 9: 1
5: 2 10: 0
Counting Sort
3 4 5 2 2 5 9 6 2

2 2 2 3 4 5
1: 0 6: 1
2: 0 7: 0
3: 0 8: 0
4: 0 9: 1
5: 1 10: 0
Counting Sort
3 4 5 2 2 5 9 6 2

2 2 2 3 4 5 5
1: 0 6: 1
2: 0 7: 0
3: 0 8: 0
4: 0 9: 1
5: 0 10: 0
Counting Sort
3 4 5 2 2 5 9 6 2

2 2 2 3 4 5 5
1: 0 6: 1
2: 0 7: 0
3: 0 8: 0
4: 0 9: 1
5: 0 10: 0
Counting Sort
3 4 5 2 2 5 9 6 2

2 2 2 3 4 5 5 6
1: 0 6: 0
2: 0 7: 0
3: 0 8: 0
4: 0 9: 1
5: 0 10: 0
Counting Sort
3 4 5 2 2 5 9 6 2

2 2 2 3 4 5 5 6
1: 0 6: 0
2: 0 7: 0
3: 0 8: 0
4: 0 9: 1
5: 0 10: 0
Counting Sort
3 4 5 2 2 5 9 6 2

2 2 2 3 4 5 5 6
1: 0 6: 0
2: 0 7: 0
3: 0 8: 0
4: 0 9: 1
5: 0 10: 0
Counting Sort
3 4 5 2 2 5 9 6 2

2 2 2 3 4 5 5 6
1: 0 6: 0
2: 0 7: 0
3: 0 8: 0
4: 0 9: 1
5: 0 10: 0
Counting Sort
3 4 5 2 2 5 9 6 2

2 2 2 3 4 5 5 6 9
1: 0 6: 0
2: 0 7: 0
3: 0 8: 0
4: 0 9: 0
5: 0 10: 0
Counting Sort
3 4 5 2 2 5 9 6 2

2 2 2 3 4 5 5 6 9
1: 0 6: 0
2: 0 7: 0
3: 0 8: 0
4: 0 9: 0
5: 0 10: 0
Counting Sort
3 4 5 2 2 5 9 6 2

2 2 2 3 4 5 5 6 9
1: 0 6: 0
2: 0 7: 0
3: 0 8: 0
4: 0 9: 0
5: 0 10: 0
Counting Sort
3 4 5 2 2 5 9 6 2

2 2 2 3 4 5 5 6 9
1: 0 6: 0
2: 0 7: 0
3: 0 8: 0
4: 0 9: 0
5: 0 10: 0

What is the complexity of counting sort?
Agenda
1. Calculating Runtimes (Big O)
2. Algorithms
A. Searching (Linear/Binary)

B. Sorting (Selection/Insertion/Bubble/Merge/Quick/Counting)

3. Data Structures
C. Array

D. Lists(Single/Double)

E. ArrayList

F. Stack

G. Queue

H. Tree

I. Binary Search Tree

J. Hash Table

K. Set

L. Graph

M. Heap
Array
‘h' ‘e’ ‘l’ ‘l’ ‘o’

• Contiguous space in memory

• Size must be specified ahead-of-time
Array
‘h' ‘e’ ‘l’ ‘l’ ‘o’

• Contiguous space in memory

• Size must be specified ahead-of-time

• O(1) reading from a given index

• O(1) writing to a given index
Array
‘h' ‘e’ ‘l’ ‘l’ ‘o’

• Contiguous space in memory

• Size must be specified ahead-of-time

• O(1) reading from a given index

• O(1) writing to a given index

• O(n) removing an element

• O(n) “inserting” an element
Single Linked List
‘h' ‘e’ ‘l’ ‘l’ ‘o’

• Each entry holds a value and a pointer to the next entry

• Takes more memory than array

• Grows and shrinks as needed
Single Linked List
‘h' ‘e’ ‘l’ ‘l’ ‘o’

• Each entry holds a value and a pointer to the next entry

• Takes more memory than array

• Grows and shrinks as needed

• O(n) to reach a given index

• O(1) to reach the next index given an entry
Single Linked List
‘h' ‘e’ ‘l’ ‘l’ ‘o’

• Each entry holds a value and a pointer to the next entry

• Takes more memory than array

• Grows and shrinks as needed

• O(n) to reach a given index

• O(1) to reach the next index given an entry

• If we have an entry, O(1) to insert after

• If we have an entry O(1) to delete after
Single Linked List Manipulation

Head Tail

‘h' ‘e’ ‘l’ ‘l’ ‘o’
Single Linked List Manipulation
Deleting from head O(1)

Head Tail

‘h' ‘e’ ‘l’ ‘l’ ‘o’
Single Linked List Manipulation
Deleting from head O(1)

Head Tail

‘h' ‘e’ ‘l’ ‘l’ ‘o’
Single Linked List Manipulation
Deleting from head O(1)

Head Tail

‘e’ ‘l’ ‘l’ ‘o’
Single Linked List Manipulation
Insert to head O(1)

Head Tail

‘e’ ‘l’ ‘l’ ‘o’
Single Linked List Manipulation
Insert to head O(1)

Head Tail

‘m’ ‘e’ ‘l’ ‘l’ ‘o’
Single Linked List Manipulation
Insert to tail O(1)

Head Tail

‘m’ ‘e’ ‘l’ ‘l’ ‘o’
Single Linked List Manipulation
Insert to tail O(1)

Head Tail

‘m’ ‘e’ ‘l’ ‘l’ ‘o’ ‘w’
Single Linked List Manipulation
Insert to tail O(1)

Head Tail

‘m’ ‘e’ ‘l’ ‘l’ ‘o’ ‘w’
Double Linked List
‘h' ‘e’ ‘l’ ‘l’ ‘o’

• Each entry holds a value and a pointer to the next entry and the previous entry

• Same characteristics as a single linked list

• Takes more memory than single linked list

• O(1) to reach the previous index given an entry
Double Linked List Manipulation
Delete from tail O(1)

Head Tail

‘h' ‘e’ ‘l’ ‘l’ ‘o’
Double Linked List Manipulation
Delete from tail O(1)

Head Tail

‘h' ‘e’ ‘l’ ‘l’ ‘o’
Double Linked List Manipulation
Delete from tail O(1)

Head Tail

‘h' ‘e’ ‘l’ ‘l’
ArrayList
AKA Dynamic Array

‘h' ‘e’ ‘l’ ‘l’ ‘o’

• Contiguous space in memory

• Automatically resize

• O(1) reading from a given index

• O(1) writing to a given index

• O(n) removing an element
Manipulating Arraylists
Appending to an array list

‘h' ‘e’ ‘l’ ‘l’ ‘o’
Manipulating Arraylists
Appending to an array list

‘h' ‘e’ ‘l’ ‘l’ ‘o’

Create a new array, double the size of the previous one
This takes O(n) time

‘h' ‘e’ ‘l’ ‘l’ ‘o’
Manipulating Arraylists
Appending to an array list

‘h' ‘e’ ‘l’ ‘l’ ‘o’

‘h' ‘e’ ‘l’ ‘l’ ‘o’ ‘‘
Manipulating Arraylists
Appending to an array list

‘h' ‘e’ ‘l’ ‘l’ ‘o’ ‘‘

What is the complexity of appending to an
arraylist?
Manipulating Arraylists
Appending to an array list

‘h' ‘e’ ‘l’ ‘l’ ‘o’ ‘‘

If you start with an array size of 1, and you append
n times. How many times have you resized?
Manipulating Arraylists
If you start with an array size of 1, and you append
n times. How many times have you resized?

You resized at 1, 2, 4, 8, …, n/4, n/2, n

The cost of these resizes is 1 + 2 + 4 + 8 + … + n/4 + n/2 + n

Re-ordering this addition can give us n + n/2 + n/4 + … + 8 + 4 + 2+ 1

n/2
n n/8
= n n
n/4 n/16

n + n/2 + n/4 + … + 8 + 4 + 2+ 1 2n

The cost of appending n times is O(n). So the cost of appending once
is amortised as O(1)
Stack

• A collection of elements which works in Last In First
Top 6 Out (LIFO) order

• O(1) “push” which adds an element to the top

• O(1) “pop” which removes an element from the top and
returns it

4
• O(1) “peek” which returns the element at the top
without removing

5
Stack Manipulation
Push

Top 6

4

5
Stack Manipulation
Push

Top 7

6

4

5
Stack Manipulation
Pop

Top 7

6

4

5
Stack Manipulation
Pop

7

Top 6

4

5
Stack Manipulation
Pop

Top 6

4

5
Stack

• A collection of elements which works in Last In First
Top 6 Out (LIFO) order

• O(1) “push” which adds an element to the top

• O(1) “pop” which removes an element from the top and
returns it

4
• O(1) “peek” which returns the element at the top
without removing

5

Of the data structures you know so far, which
would be most appropriate to implement a stack?
Queue

• A collection of elements which works in
First In First Out (FIFO) order

• O(1) “enqueue” which adds an element to
5 4 6 the tail

• O(1) “dequeue” which removes an
element from the head and returns it

Tail Head
• O(1) “peek” which returns the element at
the head without removing
Queue Manipulation
Enqueue

5 4 6

Tail Head
Queue Manipulation
Enqueue

7 5 4 6

Tail Head
Queue Manipulation
Dequeue

7 5 4 6

Tail Head
Queue Manipulation
Dequeue

7 5 4 6

Tail Head
Queue Manipulation
Dequeue

7 5 4 6

Tail Head
Queue Manipulation
Dequeue

7 5 4

Tail Head
Queue

• A collection of elements which works in
First In First Out (FIFO) order

• O(1) “enqueue” which adds an element to
5 4 6 the tail

• O(1) “dequeue” which removes an
element from the head and returns it

Tail Head
• O(1) “peek” which returns the element at
the head without removing

Of the data structures you know so far, which
would be most appropriate to implement a queue?
Tree
Root 5

• A hierarchical collection of nodes

1 2 3 • A node has 0 or more “children”, each of
which is itself a tree

• A node has 0 or 1 “parent”
Binary Tree
Root 5

1 2 • A tree where each node can only have 0,
1, or 2 children.
Binary Tree Traversal
Root 1

In Order (Left, Root, Right)
2 3 4,6,2,5,1,3

Pre Order (Root, Left, Right)

1,2,4,6,5,3

Post Order (Left,Right,Root)

6,4,5,2,3,1

4 5

6
Binary Tree Traversal
Depth-first
Root 1

Explore the tree, going as deep as
2 3
possible down one branch before trying
other branches.

In this case, a depth first search would
visit:

1, 2, 4, 6, 5, 3
4 5

6
Binary Tree Traversal
Depth-first
Root 1

Explore the tree, going as deep as
2 3
possible down one branch before trying
other branches.

In this case, a depth first search would
visit:

1, 2, 4, 6, 5, 3
4 5

Of the data structures you know so
far, which would be most appropriate
6
to use during a depth-first search?
Binary Tree Traversal
Breadth-first
Root 1

Explore the tree, exploring all children of
2 3
the current node before moving to the
next level

In this case, a breadth first search would
visit:

1, 2, 3, 4, 5, 6
4 5

6
Binary Tree Traversal
Breadth-first
Root 1

Explore the tree, exploring all children of
2 3
the current node before moving to the
next level

In this case, a breadth first search would
visit:

1, 2, 3, 4, 5, 6
4 5

Of the data structures you know so
far, which would be most appropriate
6
to use during a breadth-first search?
Binary Search Tree
Root 5

• A binary tree, where every left descendent of
a node must have a value less than the
node, and every right descendent of a node
3 9 must have a value greater than the node

• Allows for O(log n) lookup for a value

• O(log n) insertion

• O(log n) find minimum/maximum

• From one value, O(1) find next higher/lower

1 6 22 • Allows for efficient searching of ranges (e.g.
give me all nodes with values between 5 and
9)

15
BST Traversal
Searching for “6”

Pointer 5

3 9

1 6 22

15
BST Traversal
Searching for “6”

Pointer 5

6 is greater than 5, so
3 9
we move right

1 6 22

15
BST Traversal
Searching for “6”

5

3 9 Pointer

6 is less than 9, so we
1 6 22
move left

15
BST Traversal
Searching for “6”

5

3 9

6 is equal to 6. Found it

1 6 22

Pointer
15
BST Balancing
1

2

3

This is a valid BST, what is the 4
complexity of searching in this
tree?
BST as a Map
If we associate both a key and a value with
“jon”:
29 each node, we can use a BST as a map.

If we want to see how old a given person is,
we can search the BST in O(log n) time
“liam”: (using alphabetical order).
“cat”:
20 30
Other features of the BST still apply. E.g. we
can query for everyone who’s name begins
with “j”
“amy”: “kate”: “pete”:
19 30 21

“mary”
: 18
Hashing
“Hello there” 4357752745

1886414098

Identical objects should always result in the same hash value

Different objects would ideally result in a different hash values (but can’t always)
Hash Table
Let’s say we can hash a string into a number between 0 and 7

(we could do this by using any hash function
and then using modulus to restrict it to 0-7)

And we create an 8 element array
Hash Table

Using our hash function and this array, we can create a map

Let’s say we want to store “jon”’s age (29)

We hash “jon” and it gives us 882751264

882751264 % 8 = 0
Hash Table

{key: “jon”, value: 29}

Using our hash function and this array, we can create a map

Let’s say we want to store “jon”’s age (29)

We hash “jon” and it gives us 882751264

882751264 % 8 = 0
Hash Table

{key: “jon”, value: 29}

We can insert the value in O(1) time
Hash Table

{key: “pete”, value: 21}
{key: “cat”, value: 20}
{key: “jon”, value: 29}

We can insert a few more values
Hash Table

{key: “pete”, value: 21}
{key: “cat”, value: 20}
{key: “jon”, value: 29}

We can insert a few more values

And because the same key will always hash to the same
index, we can retrieve our values in O(1) time
Hash Collisions

{key: “pete”, value: 21}
{key: “cat”, value: 20}
{key: “jon”, value: 29}

What if “liam" also hashes into index 0?
Hash Collisions

{key: “pete”, value: 21}
{key: “cat”, value: 20}
{key: “jon”, value: 29}

What if “liam" also hashes into index 0?

There are two common ways of dealing with
this. Linear probing and Separate chaining.
Linear Probing

{key: “liam”, value: {key: “pete”, value: 21}
30}
{key: “cat”, value: 20}
{key: “jon”, value: 29}

With linear probing, we’d just insert our new data in the next
empty index after the one we hashed to.

When fetching data, if we don’t find the key we want in the first
index, we’d have to search through the rest of the array until we find
an empty slot.

This means, if your array is quite full. Inserting and fetching could be
O(n). You must ensure your array doesn’t get too full.
Linear Probing

{key: “liam”, value: {key: “pete”, value: 21}
30}
{key: “cat”, value: 20}
{key: “jon”, value: 29}

We can resize the array the same as with a dynamic array/arraylist

However we will have to re-hash every element for the new array
size. This means that some inserts will be O(n). It is amortised
O(1).

Deletion needs to be handled carefully, as if you remove an
element in the middle of a sequence of items which all hash to the
same index, those which follow the deleted element may become
inaccessible
Separate Chaining

{key: “cat”,
{key: “jon”, {key: “pete”,
value: 20}
value: 29} value: 21}

In separate chaining, each index actually contains a linked list of
elements which hash to that index
Separate Chaining

{key: “cat”,
{key: “jon”, {key: “pete”,
value: 20}
value: 29} value: 21}

{key:
“liam”,
value: 30}

In case of a hash collision, we just append to the list
Separate Chaining

{key: “cat”,
{key: “jon”, {key: “pete”,
value: 20}
value: 29} value: 21}

{key:
“liam”,
value: 30}

Worst case is O(n) lookup and set (if we have a bad hash function
and everything is in a single bucket).

With a reasonable implementation, and a good hash function - this
gives O(1) set and lookup
HashMap vs TreeMap

We can implement a map using either hashing or a BST

By hashing, we get O(1) set and get, as compared to O(log n)
with a BST

With a BST we get additional functionality, including max and
min, range queries, and previous/next
Set
1 400
9
22
7

A set is an onordered collection of (unique) things

We can insert into the set in O(1) time

And we can check if something is in a set in
O(1) time
Set
1 400
9
22
7

A set is an onordered collection of (unique) things

We can insert into the set in O(1) time

And we can check if something is in a set in
O(1) time

Any thoughts on how you might implement a Set?
Set
1 400
9
22
7

Sets can be implemented in the same way as a HashMap

Or could even use a HashMap, ignoring the value

We could also create a TreeSet using a BST. The would
have O(log n) insert and check. It would also allow for
range queries, min/max, etc.
Graph
jon

pete
cat

liam amy

A graph consists of nodes and edges

Used to represents geographical maps, social
networks, circuits, etc.
Graph Usage
jon

pete
cat

liam amy

Graph traversal is similar to tree traversal (DFS or BFS)

How would you calculate the minimum number of links
between cat and amy? What is the complexity of this?
Graph
London 15

Belfast
55
Hangzhou 392

134
Beijing Paris
315

The edges may have weights, for example to indicate cost or distance

Common algorithms exist for search for the min/
max edge weights between two nodes (e.g.
Djikstra’s Algorithm, A*)
Graph Representation
Adjacency Matrix

cat jon liam amy pete
jon
cat 0 0 1 0 0
jon 0 0 1 1 1 pete
cat
liam 1 1 0 1 0
amy 0 1 1 0 0
liam amy
pete 0 1 0 0 0

• Removing an edge takes O(1) time
• Checking for an edge takes O(1) time
• Sparse graphs consume a lot of space
• Adding a vertex is O(V2) time
Graph Representation
Adjacency Matrix

cat jon liam amy pete
jon

pete
cat

liam liam cat jon jon
liam amy

amy jon liam
• A collection of linked lists. One linked list per
node.
• Saves space in a sparse graph
• Queries of edge existing between two nodes
Pete amy is O(V)
• Adding an edge is O(1)
• Adding a vertex is O(1)
Heap
100

19 36

17 3 1

2 7

A heap is a tree in which the value of each node is
greater than or equal to its descendants

The maximum value is at the root of the tree

• Find the maximum value in O(1) time
• Remove maximum value in O(log n)
• Insert in O(log n)

Can also be a min-heap in which the rule is reversed and
the minimum is at the root of the heap.
Heap Manipulation
Remove max
100

19 36

17 3 1

2 7
Heap Manipulation
Remove max
100

19 36

17 3 1

2 7

Copy a leaf to the root
Heap Manipulation
Remove max
7

19 36

17 3 1

2 7

Copy a leaf to the root
Heap Manipulation
Remove max
7

19 36

17 3 1

2 7

Remove the leaf
Heap Manipulation
Remove max
7

19 36

17 3 1

2

Remove the leaf
Heap Manipulation
Remove max
7

19 36

17 3 1

2

Compare to largest child. Is it larger than our value?
Heap Manipulation
Remove max
7

19 36

17 3 1

2

Swap them
Heap Manipulation
Remove max
36

19 7

17 3 1

2

Swap them
Heap Manipulation
Remove max
36

19 7

17 3 1

2

Compare to largest child. Is it larger than our value?
Heap Manipulation
Remove max
36

19 7

17 3 1

2

No? Finished
Heap Manipulation
Remove max
36

19 7

17 3 1

2

What is the complexity of remove max?
Heap Manipulation
Inserting
36

19 7

17 3 1

2
Heap Manipulation
Inserting
36

19 7

17 3 1

2 22

Add new element as leaf
Heap Manipulation
Inserting
36

19 7

17 3 1

2 22

Compare with parent, is parent smaller?
Heap Manipulation
Inserting
36

19 7

17 3 1

2 22

Yes? Swap with parent
Heap Manipulation
Inserting
36

19 7

22 3 1

2 17

Yes? Swap with parent
Heap Manipulation
Inserting
36

19 7

22 3 1

2 17

Compare with parent, is parent smaller?
Heap Manipulation
Inserting
36

19 7

22 3 1

2 17

Yes? Swap with parent
Heap Manipulation
Inserting
36

22 7

19 3 1

2 17

Yes? Swap with parent
Heap Manipulation
Inserting
36

22 7

19 3 1

2 17

Compare with parent, is parent smaller?
Heap Manipulation
Inserting
36

22 7

19 3 1

2 17

No? Finished
Heap Manipulation
Inserting
36

22 7

19 3 1

2 17

What is the complexity of insert?
Heap Implementation
36

22 7

19 3

36 22 7 19 3

A heap can be implemented using an array, with the top of the heap at the 0th index

The children of the node at A[n] are at A[2n + 1] and A[2n + 2)
Priority Queue

A heap can be used to implement a “priority queue”

This pairs each element with a “priority” and elements are returned in priority order
Agenda
1. Calculating Runtimes (Big O)
2. Algorithms
A. Searching (Linear/Binary)

B. Sorting (Selection/Insertion/Bubble/Merge/Quick/Counting)

3. Data Structures
C. Array

D. Lists(Single/Double)

E. ArrayList

F. Stack

G. Queue

H. Tree

I. Binary Search Tree

J. Hash Table

K. Set

L. Graph

M. Heap
Recommended Reading
Books
The Algorithm Design Manual - Steven Skiena
Introduction to Algorithms - CSRL

Courses
MIT Introduction to Computer Science and Programming
Harvard CS50
Coursera Data Structures and Algorithms Specialization

Videos

Quicksort Folk Dance
Steven Skiena Algorithm Lectures