220 03 PDF

Outline Complexity Basic tools Big-Oh Big Omega Big Theta Examples
Lecture 3: Analysing Complexity of Algorithms

Big Oh, Big Omega, and Big Theta Notation
Georgy Gimel’farb
COMPSCI 220 Algorithms and Data Structures
1 / 15
1 Complexity
2 Basic tools
3 Big-Oh
4 Big Omega
5 Big Theta
6 Examples
2 / 15
Typical Complexity Curves
Running time T (n)

is proportional to: Complexity:
T (n) ∝ log n logarithmic
T (n) ∝ n linear
T (n) ∝ n log n linearithmic
T (n) ∝ n2 quadratic
T (n) ∝ n3 cubic
T (n) ∝ nk polynomial
T (n) ∝ 2n exponential
T (n) ∝ k n ; k > 1 exponential
3 / 15
Separating an Algorithm Itself from Its Implementation
Two concepts to separate an algorithm from implementation:

• The input data size n, or the number of individual data items
in a single data instance to be processed.
• The number of elementary operations f (n) taken by an
algorithm, or its running time.
The running time of a program implementation: cf (n)

• The constant factor c can rarely be determined and depends
on a computer, operating system, language, compiler, etc.
When the input size increases from n = n1 to n = n2 , all other

factors being equal, the relative running time of the program
increases by a factor of TT (n2) cf (n1 ) f (n1 )
(n1 ) = cf (n2 ) = f (n2 ) .
4 / 15
f (n)
Relative Growth g(n) = f (5) of Running Time
The approximate running time for large input sizes gives enough
information to distinguish between a good and a bad algorithm.
Input size n
Function f (n) 5 52 = 25 53 = 125 54 = 625
Constant 1 1 1 1 1
Q
Logarithmic log5 n 1 2 3 4
Q
Linear n 1 5 25 125
Q
Linearithmic n log5 n 1 10 75 500
Q
Quadratic n2 1 52 = 25 54 = 625 56 = 15, 625
Q
Cubic n3 1 53 = 125 56 = 15, 625 59 = 1, 953, 125
Q
Exponential 2n 1 220 ≈ 106 2120 ≈ 1036 2620 ≈ 10187
5 / 15
Big-Oh, Big-Theta, and Big-Omega Tools
Let f (n) and g(n) be nonnegative-valued functions defined on

nonnegative integers n.
Math notation for “of the order of . . . ” or “roughly proportional to. . . ”:
• Big-Oh (actually: Big-Omicron) O(. . .) ⇒ g(n) = O(f (n))
• Big-Theta Θ(. . .) ⇒ g(n) = Θ(f (n))
• Big-Omega Ω(. . .) ⇒ g(n) = Ω(f (n))
Big Oh O(. . .) – Formal Definition

The function g(n) is O(f (n)) (read: g(n) is Big Oh of f (n)) iff
there exists a positive real constant c and a positive integer n0
such that g(n) ≤ cf (n) for all n > n0 .
The notation iff abbreviates “if and only if”.
6 / 15
Example 1.8; p.13: g(n) = 100 log10 n is O(n)
g(n) < n if n > 238 or g(n) < 0.3n if n > 1000.
By definition, g(n) is O(f (n)), or g(n) = O(f (n)) if a constant

c > 0 exists, such that cf (n) grows faster than g(n) for all n > n0 .
• To prove that some g(n) is O(f (n))

means to show that for g and f such
constants c and n0 exist.
• The constants c and n0 are
interdependent.
• g(n) is O(f (n)) iff the graph of g(n)
is always below or at the graph of
cf (n) after n0 .
7 / 15
Big-Oh O(. . .): Informal Meaning
O(f (n)) generalises an asymptotic upper bound.

• If g(n) is O(f (n)), an algorithm with running time g(n) runs
asymptotically, i.e. for large n, at most as fast, to within a
constant factor, as an algorithm with running time f (n).
• In other words, g(n) for large n may approach cf (n) closer and
closer while the relationship g(n) ≤ cf (n) holds for n > n0 .
• The scaling factor c and the threshold n0 are interdependent
and differ for different particular functions g(n) in O(f (n)).
• Notations g(n) = O(f (n)) or g(n) is O(f (n)) mean actually
g(n) ∈ O(f (n)).
• The notation g(n) ∈ O(f (n)) indicates that g(n) is a member
of the set O(f (n)) of functions.
• All the functions in the set O(f (n)) are increasing with the
same or the lesser rate as f (n) when n → ∞.
8 / 15
Big-Oh O(. . .): Informal Meaning
O(f (n)) generalises an asymptotic upper bound.

• If g(n) is O(f (n)), an algorithm with running time g(n) runs
asymptotically, i.e. for large n, at most as fast, to within a
constant factor, as an algorithm with running time f (n).
• In other words, g(n) for large n may approach cf (n) closer and
closer while the relationship g(n) ≤ cf (n) holds for n > n0 .
• The scaling factor c and the threshold n0 are interdependent
and differ for different particular functions g(n) in O(f (n)).
• Notations g(n) = O(f (n)) or g(n) is O(f (n)) mean actually
g(n) ∈ O(f (n)).
• The notation g(n) ∈ O(f (n)) indicates that g(n) is a member
of the set O(f (n)) of functions.
• All the functions in the set O(f (n)) are increasing with the
same or the lesser rate as f (n) when n → ∞.
8 / 15
Big Omega Ω(. . .)
The function g(n) is Ω(f (n))

iff there exists a positive real constant c and a positive integer n0
such that g(n) ≥ cf (n) for all n > n0 .
• Ω(. . .) is complementary to O(. . .).

• It generalises the concept of “lower bound” (≥) in the same
way as O(. . .) generalises the concept of “upper bound” (≤):
if g(n) is Ω(f (n)) then f (n) is O(g(n)).
• Example 1: 5n2 is Ω(n) because 5n2 ≥ 5n for n ≥ 1.
• Example 2: 0.01n is Ω(log n) because 0.01n ≥ 0.5 log10 n for
n ≥ 100.
9 / 15


n ≥ 100.
9 / 15


n ≥ 100.
9 / 15
Big Theta Θ(. . .)
The function g(n) is Θ(f (n))

iff there exists two positive real constants c1 and c2 and a positive
integer n0 such that c1 f (n) ≤ g(n) ≤ c2 f (n) for all n > n0 .
• If g(n) is Θ(f (n)) then g(n) is O(f (n)) and f (n) is O(g(n))
or, what is the same, g(n) is O(f (n)) and g(n) is Ω(f (n)):
• g(n) is O(f (n)) → g(n) ≤ c0 f (n) for n > n0 .
• g(n) is Ω(f (n)) → f (n) ≤ c00 g(n) for n > n00 .
• g(n) is Θ(f (n)) ← c00 = c1 ; c0 = c2 , and n0 = max{n0 , n00 }.
1
• Informally, if g(n) is Θ(f (n)) then both the functions have

the same rate of increase.
• Example: the same rate of increase for g(n) = n + 5n0.5 and
f (n) = n because n ≤ n + 5n0.5 ≤ 6n for n > 1.
10 / 15
Big Theta Θ(. . .)

1

10 / 15
Big Theta Θ(. . .)

1

10 / 15
Comparisons: Two Crucial Ideas
• The exact running time function g(n) is not very important

since it can be multiplied by an arbitrary positive constant, c.
• The relative behaviour of two functions is compared only
asymptotically, for large n, but not near the origin where it
may make no sense.
• If the constants c involved are very large, the asymptotical
behaviour loses practical interest!
• In most cases, however, the constants remain fairly small.
• To prove that g(n) is not O(f (n)), Ω(f (n)), or Θ(f (n)), one
has to show that the desired constants do not exist, i.e. lead
to a contradiction.
• g(n) and f (n) in the Big-Oh, -Omega, and -Theta definitions
mostly relate, respectively, to “exact” and rough approximate
(like log n, n, n2 , etc) running time on inputs of size n.
11 / 15
Example 1.11, p.14
Prove that linear function g(n) = an + b; a > 0, is O(n).
The proof: By the following chain of inequalities:

g(n) ≤ an + |b| ≤ (a + |b|)n for all n ≥ 1
Do not write O(2n) or O(an + b) as this means still O(n)!
T (n) = 108 + n

T (n) = 3n + 1
O(n)-time: −8
T (n) = 50 + 10 n T (n) = 106 n + 1
• Remember that “Big-Oh”, as well as “Big-Omega” and
“Big-Theta”, describes an asymptotic behaviour for
large problem sizes.
• Only the dominant terms as n → ∞ need to be shown as the
argument of “Big-Oh”, “Big-Omega”, and “Big-Theta”.
12 / 15
Example 1.12, p.15
The polynomial Pk (n) = ak nk + ak−1 nk−1 + . . . + a2 n2 + a1 n + a0 ;

ak > 0, is O(nk ).
The proof: Pk (n) ≤ (ak + |ak−1 | + . . . + |a0 |)nk for n ≥ 1.

• Do not write O(Pk (n)) as this means still O(nk )!
• O(n k
 )-time:
 T (n) = 3n2 + 5n + 1 is O(n2 ) Is it also O(n3 )?
−8
T (n) = 10 n + 10 n + 30 is O(n ) Is it also Ω(n2 )?
3 8 2 3
T (n) = 10−8 n8 + 1000n + 1 is O(n8 ) Is it also Θ(n8 )?


 O(nm ); m ≥ k

T (n) = Pk (n) is Θ(nk );

Ω(nm ); m ≤ k

13 / 15
Example 1.12, p.15
The polynomial Pk (n) = ak nk + ak−1 nk−1 + . . . + a2 n2 + a1 n + a0 ;

ak > 0, is O(nk ).
The proof: Pk (n) ≤ (ak + |ak−1 | + . . . + |a0 |)nk for n ≥ 1.

• Do not write O(Pk (n)) as this means still O(nk )!
• O(n k
 )-time:
 T (n) = 3n2 + 5n + 1 is O(n2 ) Is it also O(n3 )?
−8
T (n) = 10 n + 10 n + 30 is O(n ) Is it also Ω(n2 )?
3 8 2 3
T (n) = 10−8 n8 + 1000n + 1 is O(n8 ) Is it also Θ(n8 )?


 O(nm ); m ≥ k

T (n) = Pk (n) is Θ(nk );

Ω(nm ); m ≤ k

13 / 15
Example 1.13, p.15
• The exponential function g(n) = 2n+k , where k is a constant,

is O(2n ) because 2n+k = 2k 2n for all n.
• Generally, g(n) = mn+k is O(ln ); l ≥ m > 1, because
mn+k ≤ ln+k = lk ln for any constant k.
A “brute-force” search for the best combination of n binary
decisions by exhausting all the 2n possible combinations has
exponential time complexity!
• 230 ≈ 109 = 1, 000, 000, 000 and
240 ≈ 1012 = 1, 000, 000, 000, 000
• Therefore, try to find a more efficient way of solving the
decision problem if n ≥ 30 . . . 40.
14 / 15
Example 1.14, p.15
For each m > 1, the logarithmic function g(n) = logm (n) has the
same rate of increase as lg(n), i.e. log2 n, because
logm (n) = logm (2) lg(n) for all n > 0.
Omit the logarithm base when using “Big-Oh”, “Big-Omega”, and

“Big-Theta” notation: logm n is O(log n), Ω(log n), and Θ(log n).
You will find later that the most efficient search for data in an
ordered array has logarithmic time complexity.
15 / 15

220 03 PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

220 03 PDF

Uploaded by

Copyright:

Available Formats

Outline Complexity Basic tools Big-Oh Big Omega Big Theta Examples

Lecture 3: Analysing Complexity of Algorithms

COMPSCI 220 Algorithms and Data Structures

Typical Complexity Curves

Running time T (n)

Separating an Algorithm Itself from Its Implementation

Two concepts to separate an algorithm from implementation:

The running time of a program implementation: cf (n)

When the input size increases from n = n1 to n = n2 , all other

Big-Oh, Big-Theta, and Big-Omega Tools

Let f (n) and g(n) be nonnegative-valued functions defined on

Big Oh O(. . .) – Formal Definition

The notation iff abbreviates “if and only if”.

Example 1.8; p.13: g(n) = 100 log10 n is O(n)

g(n) < n if n > 238 or g(n) < 0.3n if n > 1000.

By definition, g(n) is O(f (n)), or g(n) = O(f (n)) if a constant

• To prove that some g(n) is O(f (n))

Big-Oh O(. . .): Informal Meaning

O(f (n)) generalises an asymptotic upper bound.

Big-Oh O(. . .): Informal Meaning

O(f (n)) generalises an asymptotic upper bound.

Big Omega Ω(. . .)

The function g(n) is Ω(f (n))

• Ω(. . .) is complementary to O(. . .).

Big Omega Ω(. . .)

The function g(n) is Ω(f (n))

• Ω(. . .) is complementary to O(. . .).

Big Omega Ω(. . .)

The function g(n) is Ω(f (n))

• Ω(. . .) is complementary to O(. . .).

Big Theta Θ(. . .)

The function g(n) is Θ(f (n))

• Informally, if g(n) is Θ(f (n)) then both the functions have

Big Theta Θ(. . .)

The function g(n) is Θ(f (n))

• Informally, if g(n) is Θ(f (n)) then both the functions have

Big Theta Θ(. . .)

The function g(n) is Θ(f (n))

• Informally, if g(n) is Θ(f (n)) then both the functions have

Comparisons: Two Crucial Ideas

• The exact running time function g(n) is not very important

Example 1.11, p.14

Prove that linear function g(n) = an + b; a > 0, is O(n).

The proof: By the following chain of inequalities:

Do not write O(2n) or O(an + b) as this means still O(n)!

Example 1.12, p.15

The polynomial Pk (n) = ak nk + ak−1 nk−1 + . . . + a2 n2 + a1 n + a0 ;

The proof: Pk (n) ≤ (ak + |ak−1 | + . . . + |a0 |)nk for n ≥ 1.

T (n) = 10−8 n8 + 1000n + 1 is O(n8 ) Is it also Θ(n8 )?

T (n) = Pk (n) is Θ(nk );

Example 1.12, p.15

The polynomial Pk (n) = ak nk + ak−1 nk−1 + . . . + a2 n2 + a1 n + a0 ;

The proof: Pk (n) ≤ (ak + |ak−1 | + . . . + |a0 |)nk for n ≥ 1.

T (n) = 10−8 n8 + 1000n + 1 is O(n8 ) Is it also Θ(n8 )?

T (n) = Pk (n) is Θ(nk );

Example 1.13, p.15

• The exponential function g(n) = 2n+k , where k is a constant,

Example 1.14, p.15

Omit the logarithm base when using “Big-Oh”, “Big-Omega”, and

You might also like