1.2 Analysis I (Normal Code) (ORIG)

Algorithms Analysis & Design
Analysis I
(Normal Code: statements, conditions, loops)
PRE-TEST
• Two solutions to calculate the variance… Which is Faster?
ALGORITHM1 ALGORITHM2
Variance(A[], N) Variance(A[], N)
var = 0 var = 0
for i= 1 to N M = Mean(A,N)
var += (A[i] – Mean(A,N)) * for i= 1 to N
(A[i] – Mean(A,N)) var += Pow(A[i] – M, 2)
return var return var
• What’s the Worst- & Best-Case Complexity of Each?

Agenda
• Space & Time Trade-offs
• Running Time vs. Order PART I
• How to Calculate the Order?
• Worst, Best, Expected Cases
• Complexity Classes PART II
• Insertion Sort
• Warm-up Competition [bonus]
• Summary & Questions
IMP: Refer to this doc for a review on a set of important math. rules
Learning Outcomes
• Give examples that illustrate time-space trade-offs of algorithms.
• State the formal definition of order (big O).
• Explain what is meant by “best”, “expected”, and “worst” case

behavior of an algorithm.
• Determine informally the time complexity of simple algorithms.
• List and contrast standard complexity classes.

References
• Chapter 2
– Section 2.1
– Section 2.2
Algorithmic Performance
• Time Analysis
• Instructions take time.
• How fast does the algorithm perform?
• What affects its runtime?
• Space Analysis
• Data structures take space.
• What kind of data structures can be used?
• How does choice of data structure affect the runtime?
 We will focus on time:
1. How to estimate the required time of an algorithm  [ANALYSIS]
2. How to reduce the required time  [DESIGN]
6
Space & Time Trade-offs
• Sometimes we increase space to reduce time
• Ex.: Fibonacci 0, 1, 1, 2, 3, 5, 8, 13, 21, …
F(0)=0, F(1)=1 F(n)=F(n-1)+F(n-2) for n>1
F(n) F(n)
if n≤1 return n F[0] = 0; F[1] = 1
else for i = 2 to n do
return F(n-1)+F(n-2) F[i] = F[i-1] + F[i-2]
return F[n]
Time = O(2N) Time = O(N)
Space = O(1) Space = O(N)
Agenda
• Insertion Sort
Running Time vs. Order
FIRST: Running Time T(N)
– Counting # steps that the algorithm takes as a function in the input size
– Each operation in an algorithm has a cost.
Ex.1: Sequence of Operations

count = count + 1  take a certain amount of time, but it is constant
Cost: c1
sum = sum + count Cost: c2
 Total Cost = c1 + c2
FIRST: Running Time T(N)
Ex.2: Simple Loop
Cost Times
i = 1; c1 1
sum = 0; c2 1
while (i ≤ n) c3 n+1
{
i = i + 1; c4 n
sum = sum + i; c5 n
}
T(N) = c1 + c2 + (n+1)×c3 + n×c4 + n×c5
 The time required for this algorithm is proportional to n
SECOND: Order/Complexity O(…)
– Max dominant factor in the running time without any constants
– Examples:
1. T(N) = c1 + c2 + (n+1)×c3 + n×c4 + n×c5  O(N)
2. T(N) = 2xN2 + 106xN + 1000  O(N2)
3. T(N) = (20xN)7  O(N7)
4. T(N) = 108  O(1)
5. T(N) = log(N100) + 1010  O(log(N))
Note: log(NA) = A x log(N)
Refer to this doc for a review on a set of important math. rules

SECOND: Order/Complexity O(…)
Is it rationale to use order O() instead of T()?
In time analysis… What we care about?

(small or large input size)
SECOND: Order/Complexity O(…) use this online graph representation to compare different functions
103 log(N) vs N
At large input size:
– Constants are not important …
N T(N) = 106 Log2 N T(N) = 20 N Comparison
8 3 x 106 160 O(N) is better
106 20 x 106 20 x 106 Equals
109 30 x 106 20 x 109 O(Log2 N) is better
– Largest factor is dominating…

N T(N) = N2 + N Log N + 10 N O(N2) T(N) / O(N)
10 210 100 2.1
1000 106 + 1000x3 + 10x1000 = 106 + 13,000 106 1.013
Agenda
• Running Time vs. Order
• Complexity Classes
• Insertion Sort
How to Calculate the Order O()?
• Statements:
– O(1)
– Exception for the function calls!
• Conditions:
– Chose the max body order
– Take care of the condition(s)!
• Loops:
– Number of iterations × body order
Ex.2: Simple Loop [Revisited]
Type Order
i = 1; statement 1
sum = 0; statement 1
while (i ≤ n) loop #iterations x body
{
i = i + 1 statement 1
sum= sum + i statement 1
}
#iterations = N, Body O(1)  Final Order is O(N)

Ex.3: Does any of array A or array B contains element T?
for I = 1 to N
If A[I] == T return True
for I = 1 to N
If B[I] == T return True
return False
Ex.4: Do array A & array B have a number in common?
for I = 1 to N
for J = 1 to N
If A[I] == B[J]
return True
return False
Ex.5: Does array A have duplicate entries?
for I = 1 to N
for J = I + 1 to N
If A[I] == A[J]
return True
return False
Ex.6: Test yourself? (#1)
For the following code, answer the give questions
1: Fact = 1
2: For I = 1 to Max(1000, N)
3: Fact = Fact x I
4: End for
Ex.6: Test yourself? (#2) 2) What's the upper 3) What's the final order
For the following code, answer the give questions bound (i.e. max) of the entire code?
i = 1; number of iterations
sum = 0;
for the inner loop?
while (i <= n)
{
j=1;
while (j <= i)
{
sum = sum + i;
j = j + 1;
}
i = i + 1;
}
1) What's the number of iterations for outer loop?
Ex.6: Test yourself? (#3) 2) What's the lower 3) What's the final order
bound (i.e. min) numberof the entire code?
For the following code, answer the give questions
i=1;
of comparisons for the
sum = 0; inner loop?
j=1;
while (i <= n) {
while (j <= n) {
sum = sum + i;
j = j + 1;
}
i = i + 1;
}
1) What's the number of iterations for outer loop?
Agenda
• Insertion Sort
Worst, Best, Expected Cases
• Worst case:
– A situation (i.e. input) that leads the algorithm to behave at its worst time
• Best case:
– A situation (i.e. input) that leads the algorithm to behave at its best time
• Expected (Average) case:

– The expected time of the algorithm when it run over random
(unexpected) input
Worst, Best, Expected Cases
Ex.8: Linear search for an item in the array
– Best case: item is found at first place  O(1)
– Worst case: item is not found  O(N)
– Expected (Average) case: item is found at any place
• To find it at any location (i):
– Equally like probability = 1/N
– Number of comparisons = i
– Order:
Refer to this doc for a review on a set of important math. rules

Agenda
• Insertion Sort
Common Complexity Classes
Function Growth Rate Name
c Constant
log N Logarithmic
log2N Log-squared
N Linear
N log N
N2 Quadratic
N3 Cubic
2N Exponential
Running times for small inputs
29
Running times for moderate inputs
30
More Examples
Ex.9:
sum = 0;
for(i = 1 ; i < N; i*=2)
{
sum += i;
}
Agenda
• Insertion Sort
INSERTION-SORT
Sorting a hand of cards using insertion sort
33
Insertion Sort
• Iteration i: repeatedly swap element i with the one to its left if
smaller.
• Property: after ith iteration, a[0] through a[i] contain first i+1
elements in ascending order.
1 i j n
A:
key
sorted
Insertion Sort
Array index 0 1 2 3 4 5 6 7 8 9
Value 2.78 7.42 0.56 1.12 1.17 0.32 6.21 4.42 3.14 7.71
Iteration 1: step 0.
35
Insertion Sort
Array index 0 1 2 3 4 5 6 7 8 9
Value 2.78 0.56
7.42 7.42
0.56 1.12 1.17 0.32 6.21 4.42 3.14 7.71
36
Insertion Sort
Array index 0 1 2 3 4 5 6 7 8 9
Value 2.78 2.78
0.56 0.56 7.42 1.12 1.17 0.32 6.21 4.42 3.14 7.71
37
Insertion Sort
Array index 0 1 2 3 4 5 6 7 8 9
Value 0.56 2.78 7.42 1.12 1.17 0.32 6.21 4.42 3.14 7.71
38
Insertion Sort
Array index 0 1 2 3 4 5 6 7 8 9
Value 0.56 2.78 1.12
7.42 7.42
1.12 1.17 0.32 6.21 4.42 3.14 7.71
39
Insertion Sort
Array index 0 1 2 3 4 5 6 7 8 9
Value 0.56 1.12
2.78 2.78
1.12 7.42 1.17 0.32 6.21 4.42 3.14 7.71
40
Insertion Sort
Array index 0 1 2 3 4 5 6 7 8 9
Value 0.56 1.12 2.78 7.42 1.17 0.32 6.21 4.42 3.14 7.71
41
Insertion Sort
Array index 0 1 2 3 4 5 6 7 8 9
Value 0.56 1.12 2.78 1.17
7.42 7.42
1.17 0.32 6.21 4.42 3.14 7.71
42
Insertion Sort
Array index 0 1 2 3 4 5 6 7 8 9
Value 0.56 1.12 1.17
2.78 2.78
1.17 7.42 0.32 6.21 4.42 3.14 7.71
43
Insertion Sort
Array index 0 1 2 3 4 5 6 7 8 9
Value 0.56 1.12 1.17 2.78 7.42 0.32 6.21 4.42 3.14 7.71
44
Insertion Sort
Array index 0 1 2 3 4 5 6 7 8 9
Value 0.56 1.12 1.17 2.78 0.32
7.42 7.42
0.32 6.21 4.42 3.14 7.71
45
Insertion Sort
Array index 0 1 2 3 4 5 6 7 8 9
Value 0.56 1.12 1.17 0.32
2.78 2.78
0.32 7.42 6.21 4.42 3.14 7.71
46
Insertion Sort
Array index 0 1 2 3 4 5 6 7 8 9
Value 0.56 1.12 0.32
1.17 1.17
0.32 2.78 7.42 6.21 4.42 3.14 7.71
47
Insertion Sort
Array index 0 1 2 3 4 5 6 7 8 9
Value 0.56 0.32
1.12 1.12
0.32 1.17 2.78 7.42 6.21 4.42 3.14 7.71
48
Insertion Sort
Array index 0 1 2 3 4 5 6 7 8 9
Value 0.56 0.56
0.32 0.32 1.12 1.17 2.78 7.42 6.21 4.42 3.14 7.71
49
Insertion Sort
Array index 0 1 2 3 4 5 6 7 8 9
Value 0.32 0.56 1.12 1.17 2.78 7.42 6.21 4.42 3.14 7.71
50
Insertion Sort
Array index 0 1 2 3 4 5 6 7 8 9
Value 0.32 0.56 1.12 1.17 2.78 6.21
7.42 7.42
6.21 4.42 3.14 7.71
51
Insertion Sort
Array index 0 1 2 3 4 5 6 7 8 9
Value 0.32 0.56 1.12 1.17 2.78 6.21 7.42 4.42 3.14 7.71
52
Insertion Sort
Array index 0 1 2 3 4 5 6 7 8 9
Value 0.32 0.56 1.12 1.17 2.78 6.21 4.42
7.42 7.42
4.42 3.14 7.71
53
Insertion Sort
Array index 0 1 2 3 4 5 6 7 8 9
Value 0.32 0.56 1.12 1.17 2.78 4.42
6.21 6.21
4.42 7.42 3.14 7.71
54
Insertion Sort
Array index 0 1 2 3 4 5 6 7 8 9
Value 0.32 0.56 1.12 1.17 2.78 4.42 6.21 7.42 3.14 7.71
55
Insertion Sort
Array index 0 1 2 3 4 5 6 7 8 9
Value 0.32 0.56 1.12 1.17 2.78 4.42 6.21 3.14
7.42 7.42
3.14 7.71
56
Insertion Sort
Array index 0 1 2 3 4 5 6 7 8 9
Value 0.32 0.56 1.12 1.17 2.78 4.42 3.14
6.21 6.21
3.14 7.42 7.71
57
Insertion Sort
Array index 0 1 2 3 4 5 6 7 8 9
Value 0.32 0.56 1.12 1.17 2.78 3.14
4.42 4.42
3.14 6.21 7.42 7.71
58
Insertion Sort
Array index 0 1 2 3 4 5 6 7 8 9
Value 0.32 0.56 1.12 1.17 2.78 3.14 4.42 6.21 7.42 7.71
59
Insertion Sort
Array index 0 1 2 3 4 5 6 7 8 9
Value 0.32 0.56 1.12 1.17 2.78 3.14 4.42 6.21 7.42 7.71
60
Insertion Sort
Array index 0 1 2 3 4 5 6 7 8 9
Value 0.32 0.56 1.12 1.17 2.78 3.14 4.42 6.21 7.42 7.71
Iteration 10: DONE.
61
Insertion Sort
INSERTION-SORT (A, n) ⊳ A[1 . . n]
for j = 2 to n do
key = A[j]
i = j –1
while i > 0 and A[i] > key do
A[i+1] = A[i]
i = i –1
A[i+1] = key
• #iterations of outer for loop = N-1

• #iterations of inner while loop = ??
• All others are statements of O(1)
Insertion Sort
Best case
• already sorted numbers
• inner loop body never executed (i.e. O(1))
• while loop always break at conditon A[i] > key
• Final order: O(N)
Worst case
• reversely sorted numbers
• inner loop body executed for all previous elements (i.e. O(N) as upper
bound)
• while loop always break at 1st condition i > 0
• Final order: O(N2)
Agenda
• Insertion Sort
WARM-UP COMPETITION – BONUS
Document Distance:
• Given: two documents D1 & D2
• Req.: Calculate distance between them d(D1, D2)?
CASE INSENSITIVE
• Apps:
– Search for similar documents
– Detecting duplicates (e.g. Wikipedia mirrors)
– Plagiarism detection
• Definitions:
– Word = sequence of alpha-numeric characters ONLY
– Document = sequence of words (ignore space, punctuation, …etc.)
• Treat all upper-case letters as if they are lower-case, so “Cat" & “cat" are same
• Word end at a non-alphanumeric char, so "can't" contains 2 words: "can" & "t"
Document Distance:
• Idea: define distance in terms of shared words.
– Think of document D as a vector
– D[w] = # occurrences of word w
• Example: D1 = “the cat” D2 = “the dog”
– D1[“the”] = 1, D1[“cat”] = 1
– D2[“the”] = 1, D2[“dog”] = 1
• d(D1,D2) = angle between 2 vectors.
– 0◦: identical, 90◦: no common words
Document Distance:
• Algorithm:
1. Split each document into words
2. Count word frequencies (document vectors)
• NOTE: Max Freq. Value BOUNDED BY 100,000
3. Compute distance
Document Distance:
• Competition Rules:
– Startup Code: C# Template – Check LMS
– Criteria:
1. Correct O/P
2. Execution time
– Delivery:
• DUE TO: SUN 19 FEB 23:59
• Submit “DocDistance.cs” to the LMS
– Bonuses:
1. TOP 50
2. TOP 5
Agenda
• Insertion Sort
POST-TEST
• Compare Expected Time?
Variance(A[], N) Variance(A[], N)
var = 0 var = 0
for i= 1 to N M = Mean(A,N)
var += (A[i] – Mean(A,N)) * for i= 1 to N
(A[i] – Mean(A,N)) var += Pow(A[i] – M, 2)
return var return var
• What’s the Worst- & Best-Case Complexity of Each?

Summary
• Sometimes: increase space to reduce time
• This course about time!
– how to calculate? [Analysis]
– How to reduce? [Design]
• Guiding Principles:
1. At large input: use Order O()! Don’t pay attention to constants & lower orders
2. Worst case analysis  useful for “general-purpose” routines
• To calculate order:
– Statements  O(1) Conditions  O(Max Body) Loops  O(Body) x #iterations
• Insertion Sort:
– Fast at small input
– Worst & Average O(N2), Best O(N)
Questions
• Test yourself @this online quiz.
– Questions: 1, 2, 3
– Answers are available

1.2 Analysis I (Normal Code) (ORIG)

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

1.2 Analysis I (Normal Code) (ORIG)

Uploaded by

Copyright:

Available Formats

Algorithms Analysis & Design

• What’s the Worst- & Best-Case Complexity of Each?

• Give examples that illustrate time-space trade-offs of algorithms.

• State the formal definition of order (big O).

• Explain what is meant by “best”, “expected”, and “worst” case

• List and contrast standard complexity classes.

Ex.1: Sequence of Operations

Refer to this doc for a review on a set of important math. rules

Is it rationale to use order O() instead of T()?

In time analysis… What we care about?

– Largest factor is dominating…

#iterations = N, Body O(1)  Final Order is O(N)

• Expected (Average) case:

Refer to this doc for a review on a set of important math. rules

Sorting a hand of cards using insertion sort

Iteration 10: DONE.

• #iterations of outer for loop = N-1

• What’s the Worst- & Best-Case Complexity of Each?

You might also like