2IL05 Data Structures

Fall 2007

Lecture 3: Heaps
Analyzing algorithms using Θ-notation

One more time …

Solving recurrences

 easiest: master theorem

 alternatively: guess the solution and use the substitution

method to prove that it is correct

 guess the solution with the help of a recursion-tree

The substitution method
Θ(1) if n = 1
T(n) =
2T( n/2 ) + n if n > 1

Claim: T(n) = O(n)

Proof: by induction on n
Base case: n = n0
T(2) = 2T(1) + 2 = 2c1 + 2 = O(2)
Inductive step: n > n0
T(n) = 2T( n/2 ) + n
= 2O( n/2 ) + n (ind. hyp.)
= O(n) ■

Never use O, Θ, or Ω in a proof by induction!

Analyzing algorithms using Θ-notation

Example
Example
Example (A)
► A is an array of length n
1. n ← length[A]
2. if n=1
3. then return A[1]
4. else begin
5. Copy A[1… n/2 ] to auxiliary array B[1... n/2 ]
6. Copy A[1… n/2 ] to auxiliary array C[1… n/2 ]
7. b ← Example(B); c ← Example(C)
8. for i ← 1 to n
9. do for j ← 1 to i
10. do A[i] ← A[j]
11. return 43
12. end
Example
Example (A) Let T(n) be the worst case running time of
Example on an array of length n.
► A is an array of length n
1. n ← length[A] Lines 1,2,3,4,11, and 12 take Θ(1) time.
2. if n=1 Lines 5 and 6 take Θ(n) time.
3. then return A[1] Line 7 takes Θ(1) + 2 T( n/2 ) time.
4. else begin Lines 8 until 10 take
n i n
5. 
Copy A[1… n/2 ] to auxiliary array B[1...
i1 j1
n/2 ]  Θ(i)  Θ(n2 ) time.
Θ(1)
i1
6. Copy A[1… n/2 ] to auxiliary array C[1… n/2 ]
7. If n=1 lines 1,2,3 are executed,
b ← Example(B); c ← Example(C)
else lines 1,2, and 4 until 12 are executed.
8. for i ← 1 to n
9. do for j ← 1 to i ➨ T(n): Θ(1) if n=1
2T(n/2) + Θ(n2) if n>1
10. do A[i] ← A[j]
11. return 43 ➨ use master theorem …
12. end
Quiz
Recurrence Master theorem?

1. T(n) = 4 T(n/2) + Θ(n3) yes T(n) = Θ(n3)

2. T(n) = 4 T(n/2) + Θ(n) yes T(n) = Θ(n2)

3. T(n) = T(n/2) + 1 yes T(n) = Θ(log n)

4. T(n) = T(n/3) + T(2n/3) + n no T(n) = Θ(n log n)

5. T(n) = 9 T(n/3) + Θ(n2) yes T(n) = Θ(n2 log n)

6. T(n) = √n T(√n) + n no T(n) = Θ(n log log n)

Tips
 Analysis of recursive algorithms:
find the recursion and solve with master theorem if
possible

 Analysis of loops: summations

 Some standard recurrences and sums:

 T(n) = 2T(n/2) + Θ(n) ➨ T(n) = Θ(n log n)
n
  i  ½ n(n+1) = Θ(n2)
i1
n
 i
i1
2
 Θ(n3)
Heaps
Event-driven simulation
 Stores a set of events, processes first event (highest
priority)

 Supporting data structure:

 insert event
 find (and extract) event with highest priority
 change the priority of an event
Priority queue
Max-priority queue
abstract data type (ADT) that stores a set S of elements, each with
an associated key (integer value).

Operations
Insert(S, x): inserts element x into S, that is, S ← S ⋃ {x}
Maximum(S): returns the element of S with the largest key
Extract-Max(S): removes and returns the element of S with the
largest key
Increase-Key(S, x, k): give key[x] the value k
condition: k is larger than the current value of key[x]

Min-priority queue …
Implementing a priority queue

Insert Maximum Extract-Max Increase-Key

sorted list Θ(n) Θ(1) Θ(1) Θ(n)
sorted array Θ(n) Θ(1) Θ(n) Θ(n)

Today

Insert Maximum Extract-Max Increase-Key

heap Θ(lg n) Θ(1) Θ(lg n) Θ(lg n)
Max-heap 35

30 19
35 30 8 21

30 19 35

30 19

30 8 12 11 30 8 12

17 5 2

Heap
nearly complete binary tree, filled on all levels except possibly the lowest.
(lowest level is filled from left to right)

Max-heap property: for every node i other than the root

key[Parent(i)] ≥ key[i]
Properties of a max-heap
Lemma
The largest element in a max-heap is stored at the root.

Proof:
Properties of a max-heap
Lemma
The largest element in a max-heap is stored at the root.

Proof: x root
y arbitrary node x
z1, z2, …, zk nodes on path z1
between x and y
z2

Invariant ➨ key[x] ≥ key[z1] ≥ … ≥ key[zk] ≥ key[y]

➨ the largest element is stored at x ■
Implementing a heap with an array
35

30 19

30 8 12 11

17 2 5 array A[1 … length(A)]

35 30 19 30 8 12 11 17 2 5
1 2 3 4
heap-size[A]

length[A] = length of array A

heap-size[A] =number of elements in the heap
Implementing a heap with an array
level 0

level 1

level 2, position 3

array A[1 … length(A)]

1 2 3 4
heap-size[A]
kth node on level j is stored at position A[2j-1+k]
left child of node at position i = Left(i) = 2i
right child of node at position i = Right(i) = 2i + 1
parent of node at position i = Parent(i) = i / 2
Priority queue
Max-priority queue
abstract data structure (ADS) that stores a set S of elements, each
with an associated key (integer value).

Operations
Insert(S, x): inserts element x into S, that is, S ← S ⋃ {x}
Maximum(S): returns the element of S with the largest key
Extract-Max(S): removes and returns the element of S with the
largest key
Increase-Key(S, x, k): give key[x] the value k
condition: k is larger than the current value of key[x]
Implementing a max-priority queue
Set S is stored as a heap in an array A.
Operations: Maximum, Extract-Max, Insert, Increase-Key.

35 30 19 30 8 12 11 17 2 5
1 2 3 4
heap-size[A]
Implementing a max-priority queue
Set S is stored as a heap in an array A.
Operations: Maximum, Extract-Max, Insert, Increase-Key.

35 30 19 30 8 12 11 17 2 5
1 2 3 4
heap-size[A]
Maximum (A)
1. if heap-size[A] < 1
2. then error
3. else return A[1]

running time: O(1)

Implementing a max-priority queue
Set S is stored as a heap in an array A.
Operations: Maximum, Extract-Max, Insert, Increase-Key.

35 30 19 30 8 12 11 17 2 5
1 2 3 4
heap-size[A]
Implementing a max-priority queue
Set S is stored as a heap in an array A.
Operations: Maximum, Extract-Max, Insert, Increase-Key.

35
30

30 19

30
17 8 12 11

17 2 5
Implementing a max-priority queue
Set S is stored as a heap in an array A.
Operations: Maximum, Extract-Max, Insert, Increase-Key.

35
5

30 19

Heap-Extract-Max (A) 30 8 12 11
1. if heap-size[A] < 1
2. then error 17 2 5
3. max ← A[1]
4. A[1] ← A [heap-size[A]] restore heap property
5. heap-size[A] ← heap-size[A]–1
6. Max-Heapify(A,1)
7. return max
Max-Heapify
Max-Heapify(A, i)
► ensures that the heap whose root is stored at position i has the
max-heap property
► assumes that the binary trees rooted at Left(i) and Right(i) are max-
heaps
Max-Heapify
Max-Heapify(A, i)
► ensures that the heap whose root is stored at position i has the
max-heap property
► assumes that the binary trees rooted at Left(i) and Right(i) are max-
heaps

 Max-Heapify(A,1) 10
40

 exchange A[1] with largest child 40

10
35 19
 Max-Heapify(A,2)
30 10
35 12 11
 exchange A[2] with largest child
17 2 5
 Max-Heapify(A,5)
 A[5] larger than its children ➨ done.
Max-Heapify
Max-Heapify(A, i)
► ensures that the heap whose root is stored at position i has the
max-heap property
► assumes that the binary trees rooted at Left(i) and Right(i) are
max-heaps
1. if Left(i) ≤ heap-size[A] and A[Left(i)] > A[i]
2. then largest ← Left(i)
3. else largest ← i
4. if Right(i) ≤ heap-size[A] and A[Right(i)] > A[largest]
5. then largest ← Right(i)
6. if largest ≠ i
7. then exchange A[i ] and A[largest]
8. Max-Heapify (A, largest)

running time? O(height of the subtree rooted at i) = O(log n)

Implementing a max-priority queue
Set S is stored as a heap in an array A.
Operations: Maximum, Extract-Max, Insert, Increase-Key.

Insert (A, key)

1. heap-size[A] ← heap-size[A] +1
2. A[heap-size[A]] ← - infinity
3. Increase-Key(A, heap-size[A], key)
Implementing a max-priority queue
Set S is stored as a heap in an array A.
Operations: Maximum, Extract-Max, Insert, Increase-Key.
Implementing a max-priority queue
Set S is stored as a heap in an array A.
Operations: Maximum, Extract-Max, Insert, Increase-Key.

35

30 19

Increase-Key(A, i, key) 30 8 12
42 11
1. if key < A[i]
2. then error 17 2 5
3. A[i] ← key
4. while i>1 and A[parent(i)] < A[i]
5. do exchange A[parent(i)] and A[i]
6. i ← parent(i)

running time: O(log n)

Building a heap
Build-Max-Heap(A)
► Input: array A[1..n] of numbers
► Output: array A[1..n] with the same numbers, but rearranged, such
that the max-heap property holds
1. heap-size ← length[A] starting at length[A]/2 is sufficient
2. for i ← length[A] downto 1
3. do Max-Heapify(A,i)

P(k): nodes k, k+1, …, n are each the root of a max-heap

Invariant
P(k+1) holds before line 3 is executed with i=k,
P(k) holds afterwards
Building a heap
Build-Max-Heap(A)
1. heap-size ← length[A]
2. for i ← length[A] downto 1
3. do Max-Heapify(A,i) O(height of node i)

height of node i
∑i O(1 + height of i) # edges longest simple downward
= ∑0≤h≤ log n (# nodes of height h) ∙ O(1+h) path from i to a leaf.

≤ ∑0≤h≤ log n (n / 2h+1) ∙ O(1+h)

= O(n) ∙ ∑0≤h≤ log n h / 2h+1
= O(n) h=2

h=1

h=0
The sorting problem

Input: a sequence of n numbers ‹a1, a2, …, an›

Output: a permutation of the input such that ‹ai1 ≤ … ≤ ain›

Important properties of sorting algorithms:

running time: how fast is the algorithm in the worst case

in place: only a constant number of input elements are
ever stored outside the input array
Sorting with a heap: Heapsort
HeapSort(A)
1. Build-Max-Heap(A)
2. for i ← length[A] downto 2
3. do exchange A[1] and A[i ]
4. heap-size[A] ← heap-size[A] – 1
5. Max-Heapify(A,1)

P(k): A[k+1 .. n] is sorted and contains the n-k largest elements,

A[1..k] is a max-heap on the remaining elements

Invariant
P(k) holds before lines 3-5 are executed with i=k,
P(k-1) holds afterwards
Sorting algorithms

worst case running time in place

InsertionSort Θ(n2) yes
MergeSort Θ(n log n) no
HeapSort Θ(n log n) yes