The Fundamentals: Algorithms, Integers, and Matrices: X G O X F

The Fundamentals: Algorithms,
Integers, and Matrices
CSC-2259 Discrete Structures
Konstantin Busch - LSU 1
The Growth of Functions
f :R R g:RR
Big-Oh: f ( x) is O( g ( x))
is no larger order than
Big-Omega: f ( x ) is ( g ( x ))
is no smaller order than
Big-Theta: f ( x) is ( g ( x))
is of same order as
1
Big-Oh: f ( x) is O( g ( x))
(Notation abuse: f ( x)  O( g ( x)) )
There are constants C, k (called witnesses)

such that for all x  k :
| f ( x) | C  | g ( x) |
f ( x)  x 2 g ( x)  x 2  2 x  1
f ( x)  O( g ( x))
x 2  O( x 2  2 x  1)
For x  0 : x2  x2  2x 1
f ( x)  g ( x)
Witnesses: C  1, k  0
2
f ( x)  x 2 g ( x)  x 2  2 x  1
g ( x)  O( f ( x))
x 2  2 x  1  O( x 2 )
For x  1 : x2  2x 1  x2  2x2  x2  4x2

g ( x)  4  f ( x)
f ( x)  O( g ( x)) and g ( x)  O( f ( x))
f and g are of the same order
and x  2 x  1
2
Example: x2
are of the same order
3
f ( x)  O( g ( x)) and | g ( x) || h( x) |
f ( x)  O (h( x))
Example: x 2  2 x  1  O( x 2 )
x 2  2 x  1  O( x 3 )
| x 2 || x 3 |
n 2  O ( n)
Suppose n 2  O ( n)
Then for all nk: | n 2 | C  | n |
| n | C
Impossible for n  max( C , k )

4
Theorem: If f ( x)  an x n  an 1 x n 1    a1 x  a0
then f ( x)  O( x n )
Proof: for x  1
| f ( x) |  | an x n  an 1 x n 1    a1 x  a0 |
 | an | x n  | an 1 | x n 1    | a1 | x  | a0 |
 | an | x n  | an 1 | x n    | a1 | x n  | a0 | x n
 x n (| an |  | an 1 |    | a1 |  | a0 |)
Witnesses: C | an |  | an 1 |    | a0 |, k  1
End of Proof
1  2    n  O(n 2 )
1  2    n  n  n    n  n2
5
n! 1  2    n  O(n n )
n! 1 2    n  n  n    n  n n
2 n  O(n!)
2 n  2  2 n1
 2  ( 2  2   2)
 2  (2  3  n)
 2  n!
6
log n! O(n  log n)
log n! log n n  n  log n
n  O(2 n )
log n  O(n)
For n 1 : n  2n
log n  log 2 n  n  log 2  n
7
log a n  O(log n)
log n
log a n 
log a
1
Witnesses: C  , k 1
log a
1
constant  O (1)
x
x 1 : 1
For 1
x
8
Interesting functions
1 log n n n log n n 2 2n n!
Higher growth
Theorem: If f1 ( x)  O( g1 ( x)) , f 2 ( x)  O( g 2 ( x))

then ( f1  f 2 )( x)  O(max(| g1 ( x) |, | g 2 ( x) |))
Proof: x  k1 | f1 ( x) | C1  | g1 ( x) |
x  k2 | f 2 ( x) | C2  | g 2 ( x) |
x  max( k1 , k 2 ) | ( f1  f 2 )( x) | | f1 ( x)  f 2 ( x) | | f1 ( x) |  | f 2 ( x) |
 C1 | g1 ( x) | C2 | g 2 ( x) |
 (C1  C2 )  max(| g1 ( x) |, | g 2 ( x) |)
Witnesses: C  C1  C2 , k  max( k1 , k 2 )
Konstantin Busch - LSU End of Proof 18
9
Corollary: If f1 ( x)  O( g ( x)) , f 2 ( x)  O( g ( x))
then ( f1  f 2 )( x)  O( g ( x))
Theorem: If f1 ( x)  O( g1 ( x)) , f 2 ( x)  O( g 2 ( x))

then ( f1 f 2 )( x)  O( g1 ( x) g 2 ( x))
3n log( n!)  (n 2  3) log n  O(n 2 log n)
Multiplication
3n  O(n) 3n log( n!)
 O (n  n log n) Addition
log( n!)  O(n log n)
 O (n log n)
2
3n log( n!)  (n 2  3) log n

 O(n 2 log n)
n 2  3  O(n 2 )
(n 2  3) log( n)
log n  O(log n)  O(n 2 log n)
10
Big-Omega: f ( x) is ( g ( x))
(Notation abuse: f ( x)  ( g ( x)) )
There are constants C, k (called witnesses)

such that for all x  k :
| f ( x) | C  | g ( x) |
8 x 3  5 x 2  7  ( x 3 )
x 1 8x3  5x 2  7  8x3
11
Same order
Big-Theta: f ( x) is ( g ( x))
(Notation abuse: f ( x)  ( g ( x)) )
f ( x)  O( g ( x)) and f ( x)  ( g ( x))
Alternative definition:
f ( x)  O( g ( x)) and g ( x)  O ( f ( x))
3 x 2  8 x log x  ( x 2 )
3x 2  8 x log x  3x 2  8 x 2  11x 2
3 x 2  8 x log x  O( x 2 ) Witnesses: C  11, k  1
3 x 2  8 x log x  3 x 2
3 x 2  8 x log x  ( x 2 ) Witnesses: C  3, k  1
12
Theorem: If f ( x)  an x n  an 1 x n 1    a1 x  a0
then f ( x)  ( x n )
Proof: We have shown: f ( x)  O( x n )

We only need to show f ( x )  ( x n )
Take x  1 and examine two cases

Case 1: an  0
Case 2: an  0
Case 1: an  0 ( x  1)
b  max(| an 1 |, | an  2 |,, | a0 |)
f ( x)  an x n  an 1 x n 1    a1 x  a0
 an x n  nbx n 1
 ax n
nb
For 0  a  an and x 
(an  a)
Case 2 is similar
End of Proof
13
Complexity of Algorithms
Time complexity
Number of operations performed
Space complexity
Size of memory used
Linear search algorithm
Linear-Search( x, a1 , a2 ,, an ) {
i 1
while( i  n and x  a i )
i
if ( i  n) return i //item found
else return 0 //item not found
}
14
Time complexity
Comparisons
Item not found in list: 2(n  1)  1
Item found in position i : 2i  1
Worst case performance: 2(n  1)  1  O(n)
Binary search algorithm
Binary-Search( x, a1 , a2 ,, an ) {
i 1 //left endpoint of search area
jn //right endpoint of search area
while(i  j ) {
m  (i  j ) / 2
if ( x  a m) i  m  1 //item is in right half
else j  m //item is in left half
}
if (x  a i ) return i //item found
else return 0 //item not found
}
15
Search 19
1 2 3 5 6 7 8 10 12 13 15 16 18 19 20 22
1 2 3 5 6 7 8 10 12 13 15 16 18 19 20 22
12 13 15 16 18 19 20 22
18 19 20 22
18 19
Time complexity
n
Size of search list at iteration 1: 20
n
Size of search list at iteration 2:
21

n
Size of search list at iteration k:
2 k 1
16
n
Size of search list at iteration k:
2 k 1
Smallest list size: 1

n
in last iteration m : m 1
1
2
m  1 log n
Total comparisons:
(1  log n)  2  1  (log n)
#iterations Last comparison
Comparisons
per iteration
17
Bubble sort algorithm
Bubble-Sort( a1 , a2 , , an ) {
for ( i  1 to n  1 ) {
for ( j  1 to n  i )
if ( a j  a j 1)
swap a j , a j 1
}
First iteration
2 2 2 2 5
3 3 3 5 2
4 4 5 3 3
1 5 4 4 4
5 1 1 1 1
Second iteration Last iteration
5 5 5 5 5
2 2 2 4 4
3 3 4 2 3
4 4 3 3 2
1 1 1 1 1
18
Time complexity
Comparisons in iteration 1: n 1
Comparisons in iteration 2: n2

Comparisons in iteration n  1 : 1
(n  1)n
Total: 1  2    (n  1)   (n 2 )
2
Tractable problems
Class P:
Problems with algorithms whose
time complexity is polynomial O ( n b )
Examples: Search, Sorting, Shortest path
19
Intractable problems
Class NP :
Solution can be verified in polynomial time
but no polynomial time algorithm is known
Examples: Satisfiability, TSP, Vertex coloring
Important computer science question
P  NP ?
Unsolvable problems
There exist unsolvable problems which

do not have any algorithm
Example: Halting problem in Turing Machines
20
Integers and Algorithms
Base b expansion of integer n :

n  ak b k  ak 1b k 1    a1b1  a0
(ak ak 1  a1a0 ) b
Integers: k 0 0  ai  b
Example: (276 )10  2 10 2  7 10  6
Binary expansion
Digits: 0,1
(1 01011111) 2
 1  28  0  2 7  1  2 6  0  2 5
 1  2 4  1  23  1  2 2  1  21  1  2 0
 351
21
Hexadecimal expansion
Digits: 0,1,2,  ,9, A, B, C , D, E , F
(2 AE 0 B)16
 2 16 4  10 163  14 16 2  0 16  11
 175627
Octal expansion
Digits: 0,1,2,  ,7
(245)8
 2  82  4  8  5
 165
22
Conversion between binary and hexadecimal
(1001 0101 1111) 2
half byte (9 5 F )16
Conversion between binary and octal

(100 101 011 111) 2
( 4 5 3 7)8
Base b expansion( n) {
qn
k 0
While ( q  0 ) {
ak  q mod b
q  q / b 
k  k 1
}
return ( ak 1ak  2  a1a0 ) b
}
23
Binary expansion of 241  (11110001 ) 2
241  2 120  1 a0
120  2  60  0 a1
60  2  30  0 a2
30  2 15  0
 27 
15
7  23 
1
1

3  2 1  1
1  20  1 a7
Octal expansion of 12345  (30071 )8
12345  8 1543  1 a0
1543  8 192  7 a1
192  8  24  0 a2
24  83  0 a3
3  80  3 a4
24
Binary_addition(a, b ) {
a  ( an 1an  2  a1a0 ) 2
b  (bn 1bn  2  b1b0 ) 2
c0 //carry bit
for j  0 to n  1 {
d  (a j  b j  c) / 2 //auxilliary
s j  a j  b j  c  2d //j sum bit
cd //carry bit
}
sn  c //last sum bit
return ( sn sn 1  s1s0 ) 2
}
Carry bit: 1 1 1
111 0 a
 1 0 11 b
11 0 0 1
Time complexity of binary addition: O (n)

(counting bit additions) O (log a )
25
Binary_multiplication( a, b) {
a  ( an 1an  2  a1a0 ) 2
b  (bn 1bn  2  b1b0 ) 2
for j  0 to n  1 {
if ( b j  1 )
c j  a  2 j //a shifted j positions
else
cj  0
}
p  c0  c1    cn 1
return binary expansion of p
}
1 1 0 a
 1 0 1 b
1 1 0 c0
0 0 0 c1
1 1 0 c2
1 1 1 1 0
Time complexity of multiplication: O (n)
(counting shifts and bit additions) O(log 2 a)
26
Integers and Division
Integers a, b ( a  0)
a divides b : a|b c, b  a  c

factor
Examples: 3 | 12 12  3  4
3| 7
a , b, c integers
if a|b then a | bc
a|b s b  a  s bc  a  (sc )
27
a , b, c integers
if a | b and a|c then a | (b  c)
a|b s b  a  s
b  c  a  (s  t )
a|c t c  a t
a , b, c integers
if a|b and b|c then a|c
a|b s b  a  s
c  a  st
b|c t c  b t
28
a, b, c, m, n integers
if a|b and a|c then a | mb  nc
a|b a | mb
a | mb  nc
a|c a | nc
The division “algorithm”
aZ d Z
There are unique q, r  Z such that:
a  d q  r
divisor quotient remainder

0r d
29
a  d q  r
q  a div d r  a mod d
a a
q  r  a   d
d  d 
Examples: 101  11  9  2
9  101 div 11 2  101 mod 11
 11  3( 4)  1
 4  11 div 3 1  11 mod 3
a
 2d d 0 d 2d 3d  a d a 
    1  d
d 
  d  
Number of positive integers divisible by d

and not exceeding a :
a
d 
 
30
Division_algorithm(a, d ) {
q0 r | a |
while ( r  d ) {
r rd
q  q 1
}
if ( a  0 and r  0 ) { //a is negative
r d r //adjust r
q  (q  1) //adjust q
}
else if ( a  0 ) { q   q }
return q (a div d ) , r ( a mod d )
}
a  15 q
d 4
r
15 0
15  4  11 1
11  4  7 2
743 3
r  15 mod 4  3 q  15 div 4  3
Time complexity of division alg.: O(q log a )
There is a better algorithm: O (log a  log d )
(based on binary search)
31

The Fundamentals: Algorithms, Integers, and Matrices: X G O X F

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

The Fundamentals: Algorithms, Integers, and Matrices: X G O X F

Uploaded by

Copyright:

Available Formats

The Fundamentals: Algorithms,

Integers, and Matrices

CSC-2259 Discrete Structures

Konstantin Busch - LSU 1

The Growth of Functions

There are constants C, k (called witnesses)

Konstantin Busch - LSU 3

Konstantin Busch - LSU 4

For x  1 : x2  2x 1  x2  2x2  x2  4x2

Konstantin Busch - LSU 5

f ( x)  O( g ( x)) and g ( x)  O( f ( x))

f and g are of the same order

Konstantin Busch - LSU 7

Then for all nk: | n 2 | C  | n |

Impossible for n  max( C , k )

Konstantin Busch - LSU 10

Konstantin Busch - LSU 11

Konstantin Busch - LSU 12

log n! log n n  n  log n

Konstantin Busch - LSU 13

log n  log 2 n  n  log 2  n

Konstantin Busch - LSU 14

Konstantin Busch - LSU 15

Konstantin Busch - LSU 16

Konstantin Busch - LSU 17

Theorem: If f1 ( x)  O( g1 ( x)) , f 2 ( x)  O( g 2 ( x))

Theorem: If f1 ( x)  O( g1 ( x)) , f 2 ( x)  O( g 2 ( x))

Konstantin Busch - LSU 19

3n log( n!)  (n 2  3) log n  O(n 2 log n)

3n log( n!)  (n 2  3) log n

Konstantin Busch - LSU 20

There are constants C, k (called witnesses)

Konstantin Busch - LSU 21

Konstantin Busch - LSU 22

f ( x)  O( g ( x)) and f ( x)  ( g ( x))

Konstantin Busch - LSU 24

Proof: We have shown: f ( x)  O( x n )

Take x  1 and examine two cases

Konstantin Busch - LSU 27

Linear search algorithm

Konstantin Busch - LSU 28

Item not found in list: 2(n  1)  1

Item found in position i : 2i  1

Worst case performance: 2(n  1)  1  O(n)

Konstantin Busch - LSU 29

Binary search algorithm

Konstantin Busch - LSU 32

Smallest list size: 1

Konstantin Busch - LSU 33

#iterations Last comparison

Konstantin Busch - LSU 34

Konstantin Busch - LSU 35

Comparisons in iteration 2: n2

Examples: Search, Sorting, Shortest path

Konstantin Busch - LSU 38

Examples: Satisfiability, TSP, Vertex coloring

Important computer science question

There exist unsolvable problems which

Example: Halting problem in Turing Machines

Konstantin Busch - LSU 40

Base b expansion of integer n :

Example: (276 )10  2 10 2  7 10  6