Analysis of Algorithm Week 2a

Adnan YAZICI Sinan KALKAN

Dept. of Computer Engineering Middle East Technical Univ. Ankara - TURKEY

Anaysis of Algorihms, A.Yazici & S. Kalkan, Fall 2011

CEng 315

Measure the height of a tall building with a barometer

What would be your answer? One student answered:
I would tie the barometer to the end of a rope and release the rope till it touches the ground. The length of the rope then gives me the height! Of course, the instructor rejects the answer since it doesnt include any physics
Check the following for two different versions of the `legend: http://www.snopes.com/college/exam/barometer.asp
Anaysis of Algorihms, A.Yazici & S. Kalkan, Fall 2011 CEng 315

Measure the height of a tall building with a barometer (contd)

There are other ways:
1. Drop the barometer from the top, and measure the time it takes to reach the ground. 2. Make a pendulum and time its frequency at the top and the bottom of the building. 3. Walk down the stairs marking "barometer units" on the wall. 4. Measure its shadow and the buildings shadow. Workout the height of the building from barometers height.
Anaysis of Algorihms, A.Yazici & S. Kalkan, Fall 2011 CEng 315

Analyze/Compare Algorithms = Pros and Cons of Algorithms =

What are the disadvantages of these? 1. Drop the barometer from the top, and measure the time it takes to reach the ground. 2. Make a pendulum and time its frequency at the top and the bottom of the building. 3. Walk down the stairs marking "barometer units" on the wall. 4. Measure its shadow and the buildings shadow. Workout the height of the building from barometers height.
Anaysis of Algorihms, A.Yazici & S. Kalkan, Fall 2011

Cons: 1. You lose the barometer; it breaks. 2. You need a loooong rope and how long that rope is going to be depends on the answer to the question. It takes too long. It is too tiring.

3.

4.

What if there is no sun?

CEng 315

Where does the word algorithm come from?

From a Persian mathematician, astronomer and geographer: Mohammed ibn-Musa al-Khwarizmi Algorithmi is the latin form of his name He contributed to science by Decimal positional number system (e.g., 32 = 101x3 + 100x2) Presented the first systematic solutions to linear and quadratic equations In fact, the word Algebra comes from one of his operators (al-jabr: subtracting a number from both sides of an equation) for solving equations
Anaysis of Algorihms, A.Yazici & S. Kalkan, Fall 2011 CEng 315

Mohammed ibn-Musa al-Khwarizmi (780-850)

Source: Wikipedia

Where does the word algorithm come from? (contd)

al-Khwarizmi reduced equations to one of the following six forms by using al-jabr (in Arabic: restoring, completion):
squares equal roots (ax2 = bx) squares equal number (ax2 = c) roots equal number (bx = c) squares and roots equal number (ax2 + bx = c) squares and number equal roots (ax2 + c = bx) roots and number equal squares (bx + c = ax2)

al-Khwarizmis Compendious Book on Calculation by 2 = 40x 4x2 is reduced to 5x2 = For example, x Completion and 40x. From this reduced form, it is easily deducable Balancing

that the variable is either 0 or 8.

Source: Wikipedia
Anaysis of Algorihms, A.Yazici & S. Kalkan, Fall 2011 CEng 315

What does algorithm mean?

A procedure or formula for solving a problem A set of instructions to be followed to solve a problem an effective method expressed as a finite list of well-defined instructions for calculating a function step-by-step procedure for calculations
Anaysis of Algorihms, A.Yazici & S. Kalkan, Fall 2011 CEng 315

A more formal definition Better suited to us

Starting from an initial state and initial input (perhaps empty), the instructions describe a computation that, when executed, will proceed through a finite number of well-defined successive states, eventually producing "output" and terminating at a final ending state.
8

Anaysis of Algorihms, A.Yazici & S. Kalkan, Fall 2011

CEng 315

A more formal definition Better suited to us

Starting from an initial state and initial input (perhaps empty), the instructions describe a computation that, when executed, will proceed through a finite number of well-defined successive states, eventually producing "output" and terminating at a final ending state.

There are three important aspects to this definition:

Computation Well-defined successive states Terminate at a final state

Anaysis of Algorihms, A.Yazici & S. Kalkan, Fall 2011

CEng 315

Computation (cont.d)

Problem: Find temperature of the water if A&B were mixed together. Any suggestions on how to solve it?
Anaysis of Algorihms, A.Yazici & S. Kalkan, Fall 2011 CEng 315

10

Computation
Digital vs. analog computation Sequential vs. parallel computation Batch vs. interactive computation Evolutionary, molecular, quantum computation Physical computation / Digital Physics
The whole universe is itself a computation
11

Anaysis of Algorihms, A.Yazici & S. Kalkan, Fall 2011

CEng 315

Well-defined successive states

Anaysis of Algorihms, A.Yazici & S. Kalkan, Fall 2011

CEng 315

12

Terminate at a final state

Anaysis of Algorihms, A.Yazici & S. Kalkan, Fall 2011

CEng 315

13

Algorithms
An algorithm is a set of instructions to be followed to solve a problem. In other words, an algorithm is a complete set of rules that transform the input into the output in a finite number of steps. There can be more than one solution (more than one algorithm) to solve a given problem. An algorithm can be implemented using different programming languages on different platforms. An algorithm should correctly solve the problem. e.g., for sorting, this means even if (1) the input is already sorted, or (2) it contains repeated elements. Once we have a correct algorithm for a problem, we have to determine the efficiency of that algorithm (computational complexity).
Anaysis of Algorihms, A.Yazici & S. Kalkan, Fall 2011 CEng 315

14

What properties/aspects algorithms should have other than performance?

In this course, we concentrate on performance.

Anaysis of Algorihms, A.Yazici & S. Kalkan, Fall 2011

CEng 315

15

Design and Analysis of Algorithms

Aspects of studying algorithms: 1. Designing algorithms: putting the pieces of the puzzles together, choosing data structures, selecting the basic approaches to the solution of the problem, The most popular design strategies are divide&conquer, greedy, dynamic prog., backtracking, and branch&bound. 2. Expressing and implementing the algorithm Concerns are: clearness conciseness effectiveness etc.
Anaysis of Algorihms, A.Yazici & S. Kalkan, Fall 2011 CEng 315

16

Design and Analysis of Algorithms

3. Analyzing the algorithm
Algorithm analysis is assessing the time and space resources required by an algorithm as a function of the size of the problem, without actually implementing the algorithm.

4. Compare UB and LB to see if your solution is good enough

Analyzing the algorithm gives us the upper and the lower bound to solve the problem

5. Validate the algorithm

We show that the algorithm computes the correct answer for all possible legal (or given) inputs

Anaysis of Algorihms, A.Yazici & S. Kalkan, Fall 2011

CEng 315

17

Design and Analysis of Algorithms

6. Verifying the algorithm (or program)
An algorithm is said to be correct (verified) if, for every input instance, it halts with the correct output.

7. Testing algorithms
There are two phases; Debugging: The process of executing programs on sample data sets to determine if faulty results occur, and if so, to correct them. Debugging can only point to the presence of errors, but not to their absence Edsger W. Dijkstra (1930 2002)

Profiling: the process of executing a correct program on various data sets and measuring the time (and space) it takes to compute the results.
Anaysis of Algorihms, A.Yazici & S. Kalkan, Fall 2011 CEng 315

18

Algorithmic Performance
There are two aspects of algorithmic performance: - Time & space Time
Instructions take time. How fast does the algorithm perform? What affects its runtime?

Space
Data structures take space What kind of data structures can be used? How does choice of data structure affect the runtime?

We will focus on time:

How to estimate the time required for an algorithm How to reduce the time required
Anaysis of Algorihms, A.Yazici & S. Kalkan, Fall 2011 CEng 315

19

ANALYSIS OF ALGORITHMS

Anaysis of Algorihms, A.Yazici & S. Kalkan, Fall 2011

CEng 315

20

Analysis of Algorithms
Analysis of Algorithms
analyze the efficiency of different methods of solutions.

How do we compare the time efficiency of two algorithms that solve the same problem? Nave Approach: implement these algorithms in a programming language (i.e., C++), and run them to compare their time requirements.
Anaysis of Algorihms, A.Yazici & S. Kalkan, Fall 2011 CEng 315

21

Analysis of Algorithms
Comparing the programs (instead of algorithms) has difficulties.
What data should the program use?
Any analysis must be independent of specific data. Execution time is sensitive to the amount of data manipulated, grows as the amount of data increases.

What computer should we use?

We should compare the efficiency of the algorithms independently of a particular computer. Because of the execution speed of the processors, the execution times for an algorithm on the same data set on two different computers may differ.

How are the algorithms coded?

Comparing running times means comparing the implementations. We should not compare implementations, because they are sensitive to programming style that may cloud the issue of which algorithm is inherently more efficient.

absolute measure for an algorithm is not appropriate.

Anaysis of Algorihms, A.Yazici & S. Kalkan, Fall 2011 CEng 315

22

Analysis of Algorithms
Analyze algorithms independently of
specific implementations, computers, or data.

To analyze algorithms:
First, we start to count the number of significant operations in a particular solution to assess its efficiency. Then, we will express the efficiency of algorithms using growth functions.

Anaysis of Algorihms, A.Yazici & S. Kalkan, Fall 2011

CEng 315

23

What is Important?
An array-based list retrieve operation is O(1), a linkedlist-based list retrieve operation is O(n). But insert and delete operations are much easier on a linked-list-based list implementation. When selecting the implementation of an Abstract Data Type (ADT), we have to consider how frequently particular ADT operations occur in a given application. If the problem size is always very small, we can probably ignore the algorithms efficiency. We have to weigh the trade-offs between an algorithms time requirement and its memory requirements.
Anaysis of Algorihms, A.Yazici & S. Kalkan, Fall 2011 CEng 315

24

Analysis of Algorithms
A running time function, T(n), yields the time required to execute the algorithm of a problem of size n.
T(n) may contain unspecified constants, which depend on the characteristics of the ideal machine. We cannot determine this function exactly. T(n) = an2 + bn + c,
where a, b and c are unspecified constants.

Anaysis of Algorihms, A.Yazici & S. Kalkan, Fall 2011

CEng 315

25

The Execution Time of Algorithms (cont.)

Example: Simple If-Statement if (n < 0) absval = -n; else absval = n; Cost c1 c2 Times 1 1

c3

Total Cost <= c1 + max(c2,c3)

Anaysis of Algorihms, A.Yazici & S. Kalkan, Fall 2011

CEng 315

26

General Rules for Estimation

Loops
The running time of a loop is at most the running time of the statements inside of that loop times the number of iterations.

Nested Loops
Running time of a nested loop containing a statement in the inner most loop is the running time of statement multiplied by the product of the size of all loops.

Consecutive Statements
Just add the running times of those consecutive statements.

If/Else
Never more than the running time of the test plus the larger of running times of c1 and c2.
Anaysis of Algorihms, A.Yazici & S. Kalkan, Fall 2011 CEng 315

27

The Execution Time of Algorithms

Example: Simple Loop i = 1; sum = 0; while (i <= n) { i = i + 1; sum = sum + i; } Cost c1 c2 c3 c4 c5 Times 1 1 n+1 n n

Total Cost = c1 + c2 + (n+1).c3 + n.c4 + n.c5 The time required for this algorithm is proportional to n
Anaysis of Algorihms, A.Yazici & S. Kalkan, Fall 2011 CEng 315

28

The Execution Time of Algorithms (cont.)

Example: Nested Loop
i=1; sum = 0; while (i <= n) { j=1; while (j <= n) { sum = sum + i; j = j + 1; } i = i +1; c8 n } Total Cost = c1 + c2 + (n+1)*c3 + n*c4 + n*(n+1)*c5+n*n*c6+n*n*c7+n*c8 The time required for this algorithm is proportional to n2
29

Cost c1 c2 c3 c4 c5 c6 c7

Times 1 1 n+1 n n*(n+1) n*n n*n

Anaysis of Algorihms, A.Yazici & S. Kalkan, Fall 2011

CEng 315

Analysis of Algorithms
We measure the complexity of an algorithm by identifying a basic operation and then counting how any times the algorithm performs that basic operation for an input size n.
problem searching a list sorting a list input of size n lists with n elements lists with n elements

multiplying two matrices

traversing a tree Towers of Hanoi

two n-by-n matrices

tree with n nodes n disks

Basic operation comparison comparison multiplication accessing a node moving a disk

Anaysis of Algorihms, A.Yazici & S. Kalkan, Fall 2011

CEng 315

30

Algorithm Growth Rates An algorithms proportional time requirement is known as growth rate.
The growth rate of T(n) is referred to the computational complexity of the algorithm.

The computational complexity gives a concise way of saying how the running time, T(n), varies with n and is independent of any particular implementation.

Anaysis of Algorihms, A.Yazici & S. Kalkan, Fall 2011

CEng 315

31

Definition of the Orders of an Algorithm

-Notation: Given two running time functions f(n) and g(n), we say that f(n) is (g(n)) if there exists a real constant c, and a positive integer n0, such that f(n) c.g(n) for all n n0 O-notation gives an upper bound for a function to within a constant factor. Example: We want to show that 1/2n2+ 3n is O(n2) f(n) = 1/2n2+ 3n g(n) = n2 to show desired result, need c and n0 such that 0 1/2n2 + 3n c.n2 try c =1 1/2n2 + 3n n2 3n 1/2n2 6 n i.e. n0 = 6.
cg(n) f(n)

f(n) O(g(n))
Anaysis of Algorihms, A.Yazici & S. Kalkan, Fall 2011 CEng 315

32

Algorithm Growth Rates

We can compare the efficiency of two algorithms by comparing their growth rates.
The lower the growth rate, the faster the algorithm, at least for large values of n.

For example; T(n) = an2 + bn + c, the growth rate is O(n2) The goal of the algorithm designer should be an algorithm with as low a growth rate of the running time function, T(n), of that algorithm as possible.

Anaysis of Algorihms, A.Yazici & S. Kalkan, Fall 2011

CEng 315

33