Introduction to Algorithms

CHAPTER ONE
Introduction to Theory of Algorithm

Lecturer Name:
Adem M

3/29/202 1
3
 Outline
In this chapter:
• Definition of concepts of algorithms
• System component model
• Application of Algorithm
• Why we need algorithm
• Classification of Algorithm
• By implementation
• By design paradigm
• Elementary Sorting Algorithm
• Insertion sort
• Selection Sort

3/29/202 • Bubble sort 2

3
 What is analgorithm?

• An algorithm is a clearly specified set of simple instructions to be followed to

solve a problem

• Any well-defined computational procedure that takes some value (or set of
values) as an input and produces some value (or set of values) as an output

• A sequence of computational steps that transforms the input into the output

• A set of well-defined, finite rules used for problem solving

• A finite set of instructions that, if followed, accomplishes a particular task

3/29/202 3
3
 Cont’d
• An algorithm is a sequence of unambiguous instructions for solving a problem, i.e., for
obtaining a required output for any legitimate input in a finite amount of time

• From the above definition, algorithm has the following five properties: Sequence,
Unambiguous, Input, Output, Finite

3/29/202 4
3
 System component model
• To apply algorithm first of all there should be aproblem
that needs solution
–Is there any problem for which we can’t design algorithm?

–Program = Algorithm + Data Structure

3/29/202 5
3
 Application of algorithm
❖ Search Algorithms:
o Algorithms such as binary search, linear search, and depth-first search are commonly used in data
analysis, information retrieval, and artificial intelligence systems.
❖ Sorting Algorithms:
o Sorting algorithms such as merge sort, quicksort, and heap sort are used to arrange data in a
specific order, making it easier to analyze, process, and store.
❖ Machine Learning:
o Algorithms such as decision trees, random forests, and neural networks are used in machine
learning to classify data, make predictions, and detect patterns.
❖ Cryptography:
o Algorithms such as RSA and AES are used in cryptography to encrypt and decrypt data, ensuring
secure communication and data transfer.
3/29/202 6
3
 Why we need algorithm
• Standardization:
o Algorithms provide a standardized approach to solving problems, ensuring that everyone
follows the same set of rules and procedures.
• Automation:
o Algorithms automate routine tasks, freeing up time for humans to focus on more complex tasks
that require critical thinking and decision-making.
• Optimization:
o Algorithms can optimize processes and workflows, making them more efficient and cost-
effective.
• Scalability:
o Algorithms can scale up or down depending on the size and complexity of the problem,
making them suitable for a wide range of applications.

3/29/202 7
3
 Properties of an algorithm:Sequence

▪ It is a step-by-step procedure for solving a given problem

▪ Every algorithm should have a beginning (start) and a halt (end) step

▪ The first step (start step) and last step (halt step) must be clearly noted

▪ Between the two every step should have preceding and succeeding steps

▪ That is, each step must have a uniquely defined preceding and succeedingstep

3/29/202 8
3
 Properties of an algorithm:Unambiguous
▪ Define rigorously the sequence of operations performed for transforming the inputs into
theoutputs
▪ No ambiguous statements are allowed: Each step of an algorithm must be clearly and
precisely defined,having one and only one interpretation.

▪ At each point in computation, one should be able totell exactly what will happen next

▪ Every detail of each step must be spelled out,including how to handle errors

▪ This ensures that if the algorithm is performed at different times or by different

systems using thesame data, the output will be thesame

3/29/202 9
3
 Properties of an algorithm: Input specified

• The inputs are the data that will be transformed during the computation to produce the output

• An input to an algorithm specifies an instance of the problem the algorithmsolves

• Every algorithm should have a specified number (zero or more) input values (or quantities)
which are externally supplied

– We must specify the type of data and the amount of data

• Note that, correct algorithm is not one that works most of the time but one that works
correctly for all legitimate inputs

3/29/202 10
3
 Properties of an algorithm: Outputspecified
• The output is the data resulting from the computation
– It is the intended result

• Every algorithm should have one or a sequence of output values

–There must be one or more result values

• A possible output for some computations is a statement that there can be no output, i.e., no
solution is possible

• The algorithm can be proved to produce the correct output given a validinput.

3/29/202 11
3
 Properties of an algorithm:Finiteness
▪ Every valid algorithm must complete or terminate after a finite number ofsteps.

▪ If you trace out the instructions of an algorithm, then for all cases the algorithm must
terminate after a finite number of steps

▪ It must eventually stop either with the right output or with a statement that no solution is
possible

▪ Finiteness is an issue for computer algorithms because

▪ Computer algorithms often repeat instructions
▪ If the algorithm doesn’t specify when to stop, the computer will continue to repeatthe
instructions forever

3/29/202 12
3
 Whyneed algorithm analysis ?

▪ There are many ways to solve a given problem

▪ So writing a working program to solve a problem is not good enough

▪ The program may be inefficient and/or incorrect!

▪ If the program is run on a large data set, then the running time becomes anissue

▪ Always we have to undertake algorithm analysis before implementation

▪ Example: Selection Problem

▪ Given a list of N numbers, determine the kth largest,where k N.

3/29/202 13
3
 Algorithm vs.Programs

▪ A computer program is one concrete implementation of an algorithm using

a particular computer language

▪ The design phase should produce an algorithm

▪ The implementation phase should produce a program

▪ The design phase is typically much longer than the programming phase

3/29/202 14
3
 Cont.…
A computer program is an instance, or concrete representation,
for an algorithm in some programming language

Set of instructions which the computer follows to solve a problem

Problem Algorithm: A sequence of

instructions describing how
to do a task

High Level
Language 15
Program
3/29/202 15
3
 Solving Problems (1)

When faced with a problem:

1. First clearly define the problem

2. Think of possible solutions

3. Select the one that seems the best under the prevailing
circumstances

4. And then apply that solution

5. If the solution works as desired, fine; else go back to step 2

16

3/29/202 16
3
 Solving Problems (2)

• It is quite common to first solve a problem for a particular case

• Then for another
• And, possibly another
• And watch for patterns and trends that emerge
• And to use the knowledge from these patterns and trends in
coming up with a general solution
• And this general solution is called …………….
“Algorithm”

17

3/29/202 17
3
 One Problem, Many Algorithms

Problem
The statement of the problem specifies, in general terms, the
desired input/output relationship.
Algorithm
The algorithm describes a specific computational procedure for
achieving input/output relationship.
Example
Sorting a sequence of numbers into non-decreasing order.
Algorithms
Various algorithms e.g. merge sort, quick sort, heap sorts etc. 18

3/29/20
23
 One Problem, Many Algorithms

Problem
The statement of the problem specifies, in general terms, the
desired input/output relationship.
Algorithm
The algorithm describes a specific computational procedure for
achieving input/output relationship.
Example
Sorting a sequence of numbers into non-decreasing order.
Algorithms
Various algorithms e.g. merge sort, quick sort, heap sorts etc. 19

3/29/20
23
 Classification of Algorithm

❖ There are many ways of classifying algorithms and a few of them are shown below:
o Implementation Method
✓ Recursion or Iteration
✓ Procedural or Declarative (non-Procedural)
✓ Serial or Parallel or Distributed
✓ Deterministic or Non-Deterministic
o Design Method
✓ Greedy Method
✓ Divide and Conquer
✓ Dynamic Programming
✓ Linear Programming

3/29/20 20
23
 By implementation Method
❖ Recursion or Iteration
✓ A recursive algorithm is one that calls itself repeatedly until a base condition is satisfied. It is a common
method used in functional programming languages like C, C++, etc.
✓ Iterative algorithms use constructs like loops and sometimes other data structures like stacks and queues
to solve the problems.

❖ Procedural or Declarative (non-Procedural)

✓ In declarative programming languages, we say what we want without having to say how to do it.
✓ With procedural programming, we have to specify the exact steps to get the result.
✓ For example, SQL is more declarative than procedural, because the queries don’t specify the steps to
produce the result.
✓ Examples of procedural languages include C, PHP, and PERL.

3/29/20 21
23
 Cont.…

❖ Serial or Parallel or Distributed

✓ In general, while discussing the algorithms we assume that computers execute one instruction at a time.
These are called serial algorithms.
✓ Parallel algorithms take advantage of computer architectures to process several instructions at a time.
✓ They divide the problem into subproblems and serve them to several processors or threads. Iterative
algorithms are generally parallelized.
✓ If the parallel algorithms are distributed to different machines then we call such algorithms distributed
algorithms.

❖ Deterministic or Non-Deterministic
✓ Deterministic algorithms solve the problem with a predefined process,
✓ whereas non-deterministic algorithms guess the best solution at each step through the use of heuristics.
3/29/20 22
23
 Design Method
❖ Greedy Method
o Greedy algorithms Work in stages.
o In each stage, a decision is made that is good at that point, without bothering about the future
consequences.
o Generally, this means that some local best is chosen.
o It assumes that the local best selection also makes for the global optimal solution.

❖ Divide and Conquer

o Divide: Breaking the problem into subproblems that are themselves smaller instances of the same typ
of problem.
o Recursion: Recursively solving these subproblems.
o Conquer: Appropriately combining their answers.
Examples: merge sort and binary search algorithms.
3/29/202 23
3
 Cont.…
❖ Linear Programming-
o In linear programming, there are inequalities in terms of inputs and maximizing (or minimizing)
some linear function of the inputs.

o Many problems (example: maximum flow for directed graphs) can be discussed using linear
programming.

3/29/202 24
3
 Elementary sorting algorithms

❖ Elementary sorting algorithms are very simple sorting algorithms.

❖ Their average case performance is O(n2).
❖ Hence they are not recommended for very large data sets,
❖ All elementary sorting algorithms work by dividing the array to be sorted into
two sub-arrays:

❖ Selection Sort
❖ Bubble Sort
3/29/202 25
3 ❖ Insertion Sort
 How does Selection Sort work ?

❖Idea:

o Find the smallest element in the array

o Exchange it with the element in the first position

o Find the second smallest element and exchange it with the element in the
second position

o Continue until the array is sorted

3/29/202 26
3
 Cont.…

3/29/202 27
3
 How does Bubble Sort work ?

❖ Given a list of N elements, repeat the following steps N-1 times:

❖ For each pair of adjacent numbers, if number on the left is greater than the
number on the right, swap them.
❖ “Bubble” the largest value to the end using pair-wise comparisons and
swapping

3/29/202 28
3
 Cont.…

3/29/202 29
3
 Insertion Sort work

Idea: like sorting a hand of playing cards

❖ Start with empty left hand and cards face down on the table.
❖ Remove one card at a time from the table, and insert it into the correct position
in the left hand
❖ compare it with each card already in the hand, from right to left
❖ The cards held in the left hand are sorted
❖ these cards were originally the top cards of the pile on the table

3/29/202 30
3
 How does Insertion Sort work ?

3/29/202 31
3
 Thank You !!!

3/29/202 32
3