Advanced Algorithms
Time Complexity
Dr. Rubi Quiñones
Computer Science Department
Southern Illinois University Edwardsville
Course Timeline
• Introduction to algorithms
ALGORITH
ANALYSIS
MIC
• Median findings and order statistics
• Time complexity
STRATEGIES • Activity selection problems
GREEDY
• Water connection problem, Egyptian fraction
• Huffman (de)coding
• shelves, mice, and policemen problem
•
CONQUER
Max subarray sum and Nearest neighbor
DIVIDE
AND
• Newton’s and bisection algorithm
• Matrix multiplication, skyline and hanoi
• Fibonacci and path counting
PROGRAMMING
• Coin row and collecting problem
DYNAMIC
• Matrix chain multiplication and longest common subsequence
• Knapsack and optimal binary trees
• Floyd Warshall Algorithm and A*
Concep
•
Advanc
Algorithm intractability
ed
ts
• Randomized algorithms 2
Time Complexity
• Running Time
• Asymptotic Notation
• Big oh
• Big omega
• Big theta
• Performance classification
• Complexity classes
• Standard analysis techniques
• Constant time statements
• Analyzing loops
• Analyzing nested loops
• Analyzing sequence of statements
• Analyzing conditional statements
• Iteration method
3
Review
• Algorithm: a finite set of statements that guarantees an optimal solution in finite interval of time
• But what makes an algorithm good?
4
Review
• Algorithm: a finite set of statements that guarantees an optimal solution in finite interval of time
• But what makes an algorithm good?
• Runs in less time
• Consumes less memory
• Uses less computational resources (Time complexity)
5
Factors
• Hardware
• Operating system
• Compiler
• Size of input
• Nature of input
• algorithms
6
Running Time of an Algorithm
• Depends on
• Input size
• Nature of input
• Generally, time grows with size of input, so running time of an algorithm is usually measured as a function
of input size
7
Running Time of an Algorithm
• Depends on
• Input size
• Nature of input
• Generally, time grows with size of input, so running time of an algorithm is usually measured as a function
of input size
• Running time is measured in terms of number of steps/primitive operations performed
8
Finding Running Time of an Algorithm
• Running time is measured by number of steps/primitive operations performed
• Steps mean elementary operations like
• +, *, <, =, A[i], etc.,
• we will measure number of steps taken in terms of size of input
9
Example 1
How many steps does this algorithm take in relation to the input size? Ie, what is the complexity function?
Remember to take into consideration each operation. 10
Example 1
1 2 8
Once
3 4 5 6 7
Once per each iteration of for
loop, N iterations
1
Total: 5N+3
4
2 3
The complexity function of
5 6 7 the algorithm is f(N): 5N+3
8
How many steps does this algorithm take? 11
Example 1
Let’s estimate the running time for different N values:
N=10 => 53 steps
N=100 => 503 steps
N=1,000 => 5003 steps
N=1,000,000 => 5,000,003 steps
1
As N grows, the number of steps grow in linear
4 proportion to N for this function “Sum”
2 3
5 6 7
8
F(N)=5N+3 12
Example 1
What happens to the +3 and 5 in f(N)=5N+3 as N gets larger?
13
Example 1
What happens to the +3 and 5 in f(N)=5N+3 as N gets larger?
+3 becomes insignificant
5 becomes inaccurate, different operations require varying amounts of time
What is fundamental to know is that the time is linear in N
Asymptotic Complexity: As N gets large, concentrate on the highest order term:
drop lower order terms such as +3
Drop the constant coefficient of the highest order term
Example: 7N-3 would be N
Example: 8n2logn+5n2+n would be ______?
14
Example 1
What happens to the +3 and 5 in f(N)=5N+3 as N gets larger?
+3 becomes insignificant
5 becomes inaccurate, different operations require varying amounts of time
What is fundamental to know is that the time is linear in N
Asymptotic Complexity: As N gets large, concentrate on the highest order term:
drop lower order terms such as +3
Drop the constant coefficient of the highest order term
Example: 7n-3 would be N
Example: 8n2logn+5n2+n would be n2logn
15
Asymptotic Notation
Big Oh Notation: Upper bound
Omega Notation: Lower bound
Theta Notation: Tighter bound
16
Big Oh Notation
If f(N) and g(N) are two complexity functions, we say
f(N) = O(g(N))
“f(N) is order g(N)” or “f(N) is big-O of g(N)”
If there are constants c and N0 such that for N > N0,
f(N) <= c * g(N)
For all sufficiently large N
Function cg(n) always dominates
f(n) to the right of n0
17
Example 2
Which function is better?
19
Example 2
Which function is better?
As inputs get larger, any algorithm of a
smaller order will be more efficient than
an algorithm of a large order
20
Example 3
Think about these two function
comparisons and the value where
they’re equal
Which function would be better what
inputs?
21
Big Omega Notation
If we wanted to say “running time is at least…” we use omega
Big Omega notation, Ω, is used to express the lower bounds on a function
If f(n) and g(n) are two complexity functions then we say:
f(n) is Ω(g(n)) if there exist positive numbers c and n 0 such that 0<f(n)>=c (n) for all n>=n0
In this instance, cg(n) is dominated by function
f(n) to the right of n0
22
Big Theta Notation
If we wish to express tight bounds, we use the theta notation, 𝜃
f(n) = 𝜃(g(n)) means that f(n) = O(g(n)) and f(n) = Ω
Takes into consideration both big Oh and big Omega giving us the “average” complexity
23
What does it all mean?
If f(n)= 𝜃(g(n)) we say that f(n) and g(n) grow at the same rate, asymptotically
If f(n) = O(g(n)) and f(n) != Ω(g(n)), then we say that f(n) is asymptotically slower growing than g(n)
If f(n) = Ω(g(n)) and f(n) != O(g(n)), then we say that f(n) is asymptotically faster growing than g(n)
24
Which notation to use?
What is the common notation computer scientists use when designing algorithms?
25
Which notation to use?
What is the common notation computer scientists use when designing algorithms?
As computer scientists, we generally like to express our algorithms as big O since we would like to know the
upper bounds of our algorithms
Why?
26
Which notation to use?
What is the common notation computer scientists use when designing algorithms?
As computer scientists, we generally like to express our algorithms as big O since we would like to know the
upper bounds of our algorithms
Why?
If we know the worst case then we can aim to improve it and/or avoid it
27
Performance Classification
28
Complexity Classes
29
Size Does Matter
What happens if we double the input size of N?
This is especially important in high performance computing, parallelization computing, and supercomputing
30
Size Does Matter
Suppose a program has run time O(n!) and the run time for n=1 is 10 seconds.
For n=12, the run time is 2 minutes (in minutes)
For n=200, the run time is ? (in minutes)
For n=1,000, the run time is ? (in hours)
For n=50,000, the run time is ? (in hours)
For n=1,000,000, the run time is ? (in days)
31
Size Does Matter
Suppose a program has run time O(n!) and the run time for n=1 is 10 seconds.
For n=12, the run time is 2 minutes
For n=200, the run time is 33.3 minutes
For n=1,000, the run time is 2.77 hours
For n=50,000, the run time is 138.8 hours
For n=1,000,000, the run time is 115.74 days
32
Time Complexity
• Running Time
• Asymptotic Notation
• Big oh
• Big omega
• Big theta
• Performance classification
• Complexity classes
• Standard analysis techniques
• Constant time statements
• Analyzing loops
• Analyzing nested loops
• Analyzing sequence of statements
• Analyzing conditional statements
• Iteration method
33
Analyzing Loops - Linear Loops
Efficiency is proportional to the number of iterations.
Efficiency time function is:
f(n)= 1+ (n-1) +c*(n-1) + (n-1)
= (c+2)*(n-1)+1
= (c+2)n– (c+2) +1
Asymptotically, efficiency is: O(n)
34
Analyzing Nested Loops
Treat just like a single loop and evaluate each level of nesting as needed:
int j, k;
for (j=0; j<N; j++)
for (k=N; 0<k; k--)
sum += k+j;
Start with outer loop:
how many iterations? N
how much time per iteration? Need to evaluate the inner loop
35
Analyzing Nested Loops
Treat just like a single loop and evaluate each level of nesting as needed:
int j, k;
for (j=0; j<N; j++)
for (k=N; 0<k; k--)
sum += k+j;
Start with outer loop:
how many iterations? N
how much time per iteration? Need to evaluate the inner loop
Inner loop uses O(N) time
Total time is N* O(N) = O(N*N) = O(N^2)
36
Analyzing Nested Loops
What if the number of iterations of one loop depends on the counter of the other?
int j, k;
for (j=0; j<N; j++)
for (k=N; k<j; k++)
sum += k+j;
Analyze inner and outer loop together!
Number of iterations of the inner loop is:
0+1+2+…+(N-1) = O(N^2)
How did we get this answer?
37
Analyzing Nested Loops
What if the number of iterations of one loop depends on the counter of the other?
int j, k;
for (j=0; j<N; j++) When doing Big O analysis, we sometimes have to
for (k=N; k<j; k++) compute a series like:
sum += k+j;
i.e. sum of the first n numbers. What is the complexity of
Analyze inner and outer loop together! this?
Number of iterations of the inner loop is: Gauss figured out that the sum of the first n numbers is
0+1+2+…+(N-1) = O(N^2) always:
𝑛
How did we get this answer? 𝑛 ∗ (𝑛 + 1) 𝑛2 + 𝑛
𝑖 = = = 𝑂 𝑛2
2 2
𝑖=1
38
Sequence of Statements
For a sequence of statements, compute their complexity functions individually and add them up
Total cost is O(n^2) + O(n) + O(1) = O(n^2)
39
Conditional Statements
What about conditional statements such as
Lets say statement1 runs in O(n) time and statement2 runs in O(n^2) times
We use “worst case” complexity; among all inputs of size n, what is the maximum running time?
40
Time Complexity
• Running Time
• Asymptotic Notation
• Big oh
• Big omega
• Big theta
• Performance classification
• Complexity classes
• Standard analysis techniques
• Constant time statements
• Analyzing loops
• Analyzing nested loops
• Analyzing sequence of statements
• Analyzing conditional statements
• Iteration method
41
Deriving a Recurrence Equation
So far, all algorithms that we have been analyzing have been non recursive
Example: recursive power method
If N=0, then running time T(N) is 2
However, if N>=2, then running time T(N) is the cost of each step taken plus time required to compute
power(x, n-1). (i.e., T(N)=2+T(N-1) for N>=2)
How do we solve this? One way is to use the iteration method.
42
Iteration Method
This is sometimes known as ”Back Substituting”
Involves expanding the recurrence in order to see a pattern
Solving formula from previous example using the iteration method
Solution: Expand and apply to itself:
Let T(1) = n0 = 2
T(N) = 2+T(N-1)
= 2 + 2 + T(N-2)
= 2 + 2 + 2 + T(N-3)
= 2 + 2 + 2 + … + 2 + T(1)
= 2N + 2 remember that t(1) = n0 = 2 for N=1
So T(N) = 2N+2 is O(N) for last example
43
Summary
Algorithms can be classified according to their complexity (O notation)
only relevant for large input sizes
“Measurements” are machine dependent
worst-, average-, best-case analysis
44
Time Complexity
• Running Time
• Asymptotic Notation
• Big oh
• Big omega
• Big theta
• Performance classification
• Complexity classes
• Standard analysis techniques
• Constant time statements
• Analyzing loops
• Analyzing nested loops
• Analyzing sequence of statements
• Analyzing conditional statements
• Iteration method
45
Course Timeline
• Introduction to algorithms
ALGORITH
ANALYSIS
MIC
• Median findings and order statistics
• Time complexity
STRATEGIES • Activity selection problems
GREEDY
• Water connection problem, Egyptian fraction
• Huffman (de)coding
• shelves, mice, and policemen problem
•
CONQUER
Max subarray sum and Nearest neighbor
DIVIDE
AND
• Newton’s and bisection algorithm
• Matrix multiplication, skyline and hanoi
• Fibonacci and path counting
PROGRAMMING
• Coin row and collecting problem
DYNAMIC
• Matrix chain multiplication and longest common subsequence
• Knapsack and optimal binary trees
• Floyd Warshall Algorithm and A*
Concep
•
Advanc
Algorithm intractability
ed
ts
• Randomized algorithms 46
