This action might not be possible to undo. Are you sure you want to continue?

Welcome to Scribd! Start your free trial and access books, documents and more.Find out more

‣ ‣ ‣ ‣ ‣

Reference: Algorithms in Java, Chapter 2 Intro to Programming in Java, Section 4.1

http://www.cs.princeton.edu/algs4

estimating running time mathematical analysis order-of-growth hypotheses input models measuring space

Except as otherwise noted, the content of this presentation is licensed under the Creative Commons Attribution 2.5 License.

Algorithms in Java, 4th Edition

· Robert Sedgewick and Kevin Wayne · Copyright © 2008

·

May 2, 2008 10:35:54 AM

Running time

“ As soon as an Analytic Engine exists, it will necessarily guide the future course of the science. Whenever any result is sought by its aid, the question

how many times do you have to turn the crank?

Charles Babbage (1864)

Analytic Engine

2

**Reasons to analyze algorithms Predict performance. Compare algorithms. Provide guarantees. Understand theoretical basis.
**

theory of algorithms (COS 423) this course (COS 226)

client gets poor performance because programmer did not understand performance characteristics

**Primary practical reason: avoid performance bugs.
**

3

Some algorithmic successes Discrete Fourier transform. Break down waveform of N samples into periodic components. Applications: DVD, JPEG, MRI, astrophysics, …. Brute force: N2 steps. FFT algorithm: N log N steps, enables new technology.

Freidrich Gauss 1805

• • • •

time

64T

quadratic

32T

16T 8T

linearithmic linear

1K 2K 4K 8K

4

size

Linear, linearithmic, and quadratic

Some algorithmic successes N-body Simulation. Simulate gravitational interactions among N bodies. Brute force: N2 steps. Barnes-Hut: N log N steps, enables new research.

Andrew Appel PU '81

• • •

time

64T

quadratic

32T

16T 8T

linearithmic linear

1K 2K 4K 8K

5

size

Linear, linearithmic, and quadratic

‣ ‣ ‣ ‣ ‣

estimating running time mathematical analysis order-of-growth hypotheses input models measuring space

6

Scientific analysis of algorithms A framework for predicting performance and comparing algorithms. Scientific method. Observe some feature of the universe. Hypothesize a model that is consistent with observation. Predict events using the hypothesis. Verify the predictions by making further observations. Validate by repeating until the hypothesis and observations agree.

• • • • •

• •

Principles. Experiments must be reproducible. Hypotheses must be falsifiable.

**Universe = computer itself.
**

7

Experimental algorithmics Every time you run a program you are doing an experiment!

Why is my program so slow ?

First step. Debug your program! Second step. Choose input model for experiments. Third step. Run and time the program for problems of increasing size.

8

Example: 3-sum 3-sum. Given N integers, find all triples that sum to exactly zero. Application. Deeply related to problems in computational geometry.

% more 8ints.txt 30 -30 -20 -10 40 0 10 5 % java ThreeSum < 8ints.txt 4 30 -30 0 30 -20 -10 -30 -10 40 -10 0 10

9

3-sum: brute-force algorithm

public class ThreeSum { public static int count(long[] a) { int N = a.length; int cnt = 0; for (int i = 0; i < N; i++) for (int j = i+1; j < N; j++) for (int k = j+1; k < N; k++) if (a[i] + a[j] + a[k] == 0) cnt++; return cnt; } public static void main(String[] args) { int[] a = StdArrayIO.readLong1D(); StdOut.println(count(a)); } }

check each triple

10

Measuring the running time Q. How to time a program? A. Manual.

11

**Measuring the running time Q. How to time a program? A. Automatic.
**

Stopwatch stopwatch = new Stopwatch(); ThreeSum.count(a); double time = stopwatch.elapsedTime(); StdOut.println("Running time: " + time + " seconds");

client code

public class Stopwatch { private final long start = System.currentTimeMillis(); public double elapsedTime() { long now = System.currentTimeMillis(); return (now - start) / 1000.0; } }

implementation

12

3-sum: initial observations Data analysis. Observe and plot running time as a function of input size N.

N 1024 2048 4096 8192

time (seconds) † 0.26 2.16 17.18 137.76

† Running Linux on Sun-Fire-X4100

13

Empirical analysis Log-log plot. Plot running time vs. input size N on log-log scale.

slope = 3

power law

Regression. Fit straight line through data points: c N a. Hypothesis. Running time grows cubically with input size: c N 3.

slope

14

Prediction and verification Hypothesis. 2.5 × 10 –10 × N 3 seconds for input of size N.

**Prediction. 17.18 seconds for N = 4,096. Observations.
**

N 4096 4096 4096 time (seconds) 17.18 17.15 17.17

agrees

Prediction. 1100 seconds for N = 16,384. Observation.

N 16384

time (seconds) 1118.86

agrees

15

Doubling hypothesis Q. What is effect on the running time of doubling the size of the input?

N 512 1024 2048 4096 8192

time (seconds) † 0.03 0.26 2.16 17.18 137.76

ratio 7.88 8.43 7.96 7.96

numbers increases by a factor of 2

running time increases by a factor of 8

lg of ratio is exponent in power law (lg 7.96 ≈ 3)

**Bottom line. Quick way to formulate a power law hypothesis.
**

16

Experimental algorithmics Many obvious factors affect running time: Machine. Compiler. Algorithm. Input data.

• • • • • • • •

More factors (not so obvious): Caching. Garbage collection. Just-in-time compilation. CPU use by other applications.

**Bad news. It is often difficult to get precise measurements. Good news. Easier than other sciences.
**

e.g., can run huge number of experiments

17

‣ ‣ ‣ ‣ ‣

estimating running time mathematical analysis order-of-growth hypotheses input models measuring space

18

Mathematical models for running time Total running time: sum of cost × frequency for all operations. Need to analyze program to determine set of operations. Cost depends on machine, compiler. Frequency depends on algorithm, input data.

• • •

Donald Knuth 1974 Turing Award

**In principle, accurate mathematical models are available.
**

19

Cost of basic operations

operation integer add integer multiply integer divide floating point add floating point multiply floating point divide sine arctangent ...

example a + b a * b a / b a + b a * b a / b Math.sin(theta) Math.atan2(y, x) ...

nanoseconds †

2.1 2.4 5.4 4.6 4.2 13.5 91.3 129.0 ...

† Running OS X on Macbook Pro 2.2GHz with 2GB RAM

20

Cost of basic operations

operation variable declaration assignment statement integer compare array element access array length 1D array allocation 2D array allocation string length substring extraction string concatenation

example int a a = b a < b a[i] a.length new int[N] new int[N][N] s.length() s.substring(N/2, N) s + t

nanoseconds † c1 c2 c3 c4 c5 c6 N c7 N 2 c8 c9 c10 N

**Novice mistake. Abusive string concatenation.
**

21

Example: 1-sum Q. How many instructions as a function of N?

int count = 0; for (int i = 0; i < N; i++) if (a[i] == 0) count++;

operation variable declaration assignment statement less than comparison equal to comparison array access increment

frequency

2 2 N+1 N N ≤ 2N

between N (no zeros) and 2N (all zeros)

22

**Example: 2-sum Q. How many instructions as a function of N?
**

int count = 0; for (int i = 0; i < N; i++) for (int j = i+1; j < N; j++) if (a[i] + a[j] == 0) count++;

operation variable declaration assignment statement less than comparison equal to comparison array access increment

frequency

0 + 1 + 2 + . . . + (N − 1)

= =

N+2 N+2 1/2 (N + 1) (N + 2) 1/2 N (N − 1) N (N − 1) ≤ N2

tedious to count exactly

1 N (N − 1) 2 N 2

23

Tilde notation

• •

Estimate running time (or memory) as a function of input size N. Ignore lower order terms.

**- when N is large, terms are negligible - when N is small, we don't care
**

6 N 3 + 20 N + 16
6 N 3 + 100 N 4/3 + 56
6 N 3 + 17 N

2

Ex 1. Ex 2. Ex 3.

~ 6N3 ~ 6N3 ~ 6N3

lg N + 7 N

discard lower-order terms (e.g., N = 1000 6 trillion vs. 169 million)

Technical definition. f(N) ~ g(N) means

N→

lim

f (N) = 1 ∞ g(N)

€

24

**Example: 2-sum Q. How long will it take as a function of N?
**

int count = 0; for (int i = 0; i < N; i++) for (int j = i+1; j < N; j++) if (a[i] + a[j] == 0) count++;

"inner loop"

operation variable declaration assignment statement less than comparison equal to comparison array access increment total

frequency

cost

total cost

~ N ~ N ~ 1/2 N 2

c1 c2 c3

~ c1 N ~ c2 N ~ c3 N 2 ~ c4 N 2 ≤ c5 N 2 ~ cN2

25

~ 1/2 N 2 ~ N2 ≤ N2 c4 c5

Example: 3-sum

478

Algorithms and Data Stru

**Q. How many instructions as a function of N?
**

public class ThreeSum { public static int count(int[] a) { int N = a.length; int cnt = 0;

the leading term of mathemati pressions by using a mathem device known as the tilde no 1 We write f (N) to represe for (int i = 0; i < N; i++) quantity that, when divided b for (int j = i+1; j < N; j++) N approaches 1 as N grows. W inner ~N / 2 for (int k = j+1; k < N; k++) loop write N g (N)N (Nf − 1)(N −to indicat (N) 2) = ~N / 6 if (a[i] + a[j] + a[k] == 0) 3 3! cnt++; g (N) f (N)1approaches 1 as N ∼ N 6 With this notation, we can return cnt; depends on input data } complicated parts of an exp public static void main(String[] args) that represent small values. F { ample, the if statement in Thr int[] a = StdArrayIO.readInt1D(); int cnt = count(a); is executed N 3/6 times b StdOut.println(cnt); N (N 1)(N 2)/6 N 3/6 N } } N/3, which certainly, when div Remark. Focus on instructions in inner loop; ignore everything else! N 3/6, approaches 1 as N grow Anatomy of a program’s statement execution frequencies notation is useful when the ter 26 ter the leading term are relativ signiﬁcant (for example, when N = 1000, this assumption amounts to sayi

2 3

3

2

Mathematical models for running time In principle, accurate mathematical models are available. In practice, Formulas can be complicated. Advanced mathematics might be required. Exact models best left for experts.

costs (depend on machine, compiler)

• • •

TN = c1 A + c2 B + c3 C + c4 D + c5 E A = variable declarations B = assignment statements C = compare D = array access E = increment

frequencies (depend on algorithm, input)

**Bottom line. We use approximate models in this course: TN ~ c N3.
**

27

‣ ‣ ‣ ‣ ‣

28

Common order-of-growth hypotheses To determine order-of-growth: Assume a power law TN ~ c N a. Estimate exponent a with doubling hypothesis. Validate with mathematical analysis.

N 1,000 2,000 4,000 8,000 16,000 32,000 64,000 time (seconds) † 0.43 0.53 1.01 2.87 11.00 44.64 177.48 observations

• • •

Ex. ThreeSumDeluxe.java Food for thought. How is it implemented?

**Caveat. Can't identify logarithmic factors with doubling hypothesis.
**

29

482

Algorithms and Data Structures

**Common order-of-growth hypotheses
**

Linearithmic. We use the term linearithmic to describe programs whose running

Good news. the smallrelevant. For example, CouponCollector (PROGRAM 1.4.2) is linlogarithm is not set of functions suffices

time for a problem of size N has order of growth N log N. Again, the base of the

time

1024T 512T

earithmic.N, prototypical example 2, mergesort (see PROGRAM 4.2.6). Several im1, log The N, N log N, N is N3, and 2N portant problems have natural solutions that are quadratic but clever algorithms that are linearithmic. Such algorithms (including mergesort) are critically to describe order-of-growth of typical algorithms. important in practice because they enable us to address problem sizes far larger than could be addressed with quadratic solutions. In the next section, we consider a general design technique for developing growth rate name linearithmic algorithms.

cubi c

dra tic

T(2N) / T(N) 1 ~1 2 ~2 4 8 T(N)

exponential

lin ea rit h lin mic ea r

constant Quadratic. A 1 typical program whose

64T

8T 4T 2T T

running time has order of growth N 2 log N logarithmic has two nested for loops, used for some calculation involving all pairs linear eleof N N ments. The force update double loop in NBody (PROGRAM 3.4.2) is a linearithmicof prototype N log N the programs in this classiﬁcation, as is quadratic the elementaryN2 sorting algorithm Insertion (PROGRAM 4.2.4). 3

N 2 cubic

qua

size

is cubic (its running time has constant order of growth N 3) because it has three nested for loops, to process all triples of 1K 2K 4K 8K 1024K N elements. The running time of matrix factor for Orders of growth (log-log plot) multiplication, as implemented in SEC- doubling hypothesis TION 1.4 has order of growth M 3 to multiply two M-by-M matrices, so the basic matrix multiplication algorithm is often considered to be cubic. However, the size of the input (the number of entries in the matrices) is proportional to N = M 2, so the algorithm is best classiﬁed as N 3/2, not cubic.

logarithmic

**Cubic. Our example for this section, N
**

ThreeSum,

exponential

30

**Common order-of-growth hypotheses
**

growth rate 1

name constant

typical code framework a = b + c;

description statement

example add two numbers

log N

logarithmic

{

while (N > 1) N = N / 2; ...

}

divide in half

binary search

N

linear

for (int i = 0; i < N; i++) { ... }

loop

find the maximum

N log N

linearithmic

[see lecture 5]

divide and conquer

mergesort

N

2

quadratic

for (int i = 0; i < N; i++) for (int j = 0; j < N; j++) { ... } for (int i = 0; i < N; i++) for (int j = 0; j < N; j++) for (int k = 0; k < N; k++) { ... } [see lecture 24]

double loop

check all pairs

N3

cubic

triple loop

check all triples

2N

exponential

exhaustive search

**check all possibilities
**

31

Practical implications of order-of-growth Q. How long to process millions of inputs? Ex. Population of NYC was "millions" in 1970s; still is. Q. How many inputs can be processed in minutes? Ex. Customers lost patience waiting "minutes" in 1970s; they still do.

seconds 1 10 102 103 104 equivalent 1 second 10 seconds 1.7 minutes 17 minutes 2.8 hours 1.1 days 1.6 weeks 3.8 months 3.1 years 3.1 decades 3.1 centuries forever age of universe

32

**For back-of-envelope calculations, assume:
**

processor speed 1 MHz 10 MHz 100 MHz 1 GHz instructions per second 106 107 108 109

105 106 107 108 109 1010 … 1017

decade 1970s 1980s 1990s 2000s

Practical implications of order-of-growth

growth rate

problem size solvable in minutes 1970s 1980s 1990s 2000s 1970s

time to process millions of inputs 1980s 1990s 2000s

1

any

any

any

any

instant

instant

instant

instant

log N

any

any tens of millions millions

any hundreds of millions millions

any

instant

instant

instant

instant

N

millions hundreds of thousands hundreds

billions hundreds of millions tens of thousands thousands

minutes

seconds

second tens of seconds months

instant

N log N

hour

minutes

seconds

N2

thousand

thousands

decades

years

weeks

N3

hundred

hundreds

thousand

never

never

never

millennia

33

Practical implications of order-of-growth

growth rate

name

description

effect on a program that runs for a few seconds time for 100x more data a few minutes a few minutes several hours several weeks forever size for 100x faster computer 100x 100x 10x 4-5x 1x

1 log N N N log N N2 N3 2N

constant logarithmic linear linearithmic quadratic cubic exponential

independent of input size nearly independent of input size optimal for N inputs nearly optimal for N inputs not practical for large problems not practical for medium problems useful only for tiny problems

34

‣ ‣ ‣ ‣ ‣

35

Types of analyses Best case. Running time determined by easiest inputs. Ex. N-1 compares to insertion sort N elements in ascending order. Worst case. Running time guarantee for all inputs. Ex. No more than ½N2 compares to insertion sort any N elements. Average case. Expected running time for "random" input. Ex. ~ ¼ N2 compares on average to insertion sort N random elements.

36

Commonly-used notations

notation

provides

example

shorthand for

used to

Tilde

leading term

~ 10 N 2

10 N 2 10 N 2 + 22 N log N 10 N 2 + 2 N +37 N2 9000 N 2 5 N 2 + 22 N log N + 3N N2 100 N 22 N log N + 3 N 9000 N 2 N5 N 3 + 22 N log N + 3 N

provide approximate model

Big Theta

asymptotic growth rate

Θ(N 2)

classify algorithms

Big Oh

Θ(N 2) and smaller

O(N 2)

develop upper bounds

Big Omega

Θ(N 2) and larger

Ω(N 2)

develop lower bounds

37

Commonly-used notations Ex 1. Our brute-force 3-sum algorithm takes Θ(N 3) time. Ex 2. Conjecture: worst-case running time for any 3-sum algorithm is Ω(N 2). Ex 3. Insertion sort uses O(N 2) compares to sort any array of N elements; it uses ~ N compares in best case (already sorted) and ~ ½N 2 compares in the worst case (reverse sorted). Ex 4. The worst-case height of a tree created with union find with path compression is Θ(N). Ex 5. The height of a tree created with weighted quick union is O(log N).

base of logarithm absorbed by big-Oh

loga N =

1 logb N logb a

38

Predictions and guarantees Theory of algorithms. Worst-case running time of an algorithm is O(f(N)). Advantages describes guaranteed performance. O-notation absorbs input model.

f(N) values represented by O(f(N)) time/memory

• • • •

Challenges Cannot use to predict performance. Cannot use to compare algorithms.

input size

39

Predictions and guarantees Experimental algorithmics. Given input model,average-case running time is ~ c f(N). Advantages. Can use to predict performance. Can use to compare algorithms.

values represented by ~ c f(N) c f(N) time/memory

• • • •

Challenges. Need to develop accurate input model. May not provide guarantees.

input size

40

‣ ‣ ‣ ‣ ‣

41

Typical memory requirements for primitive types in Java Bit. 0 or 1. Byte. 8 bits. Megabyte (MB). 210 bytes ~ 1 million bytes. Gigabyte (GB). 220 bytes ~ 1 billion bytes.

type boolean byte char int float long double bytes 1 1 2 4 4 8 8

42

Typical memory requirements for arrays in Java Array overhead. 16 bytes on a typical machine.

type char[] int[] double[]

bytes 2N + 16 4N + 16 8N + 16

type char[][] int[][] double[][]

bytes 2N2 + 20N + 16 4N2 + 20N + 16 8N2 + 20N + 16

one-dimensional arrays

two-dimensional arrays

**Q. What's the biggest double[] array you can store on your computer?
**

typical computer in 2008 has about 1GB memory

43

3.2.1) object uses 32 ee double instance variTypical memory requirements for objects in Java ny programs create milthe information needed Object overhead. 8 bytes on a typical machine. or transparency). A referReference. 4 bytes on a typical machine. n a data type contains a 4 bytes for theEach Complex object consumes 24 bytes of memory. reference Ex 1. y needed for the object’s

public class Complex 32 bytes { object private double re; overhead private double im; rx ... double ry values } q

8 bytes overhead for object 8 bytes 8 bytes

M

(Program 3.2.1)

class Charge

ate double rx; ate double ry; ate double q;

24 bytes

ct (Program 3.2.6)

24 bytes

object overhead re im

class Complex

ate double re; ate double im;

double

values

Java library)

12 bytes

object overhead

44

class Color

ate byte r;

... }

q

values

omplex object (Program 3.2.6)

public object { overhead private double re; private double im; double re Object overhead. im bytes on a 8 values ... }

class Typical memory requirements for objects in Java Complex

24 bytes

typical machine.

**Reference. 4 bytes on a typical machine.
**

12 bytes

olor object (Java library)

public class Color object { Ex 2. A String of length overhead private byte r; private byte g; r g b a private byte b; private byte a; public class String byte ... values } {

N consumes 2N + 40 bytes.

8 bytes overhead for object 4 bytes 4 bytes 4 bytes 4 bytes for reference (plus 2N + 16 bytes for array) 2N + 40 bytes

ocument object (Program 3.3.4)

private int offset; 16 bytes + string + vector private int count; public class Document object private int hash; { overhead private String id;private char[] value; id private Vector profile; references ...

... }

}

profile

**ring object (Java library)
**

public class String { private char[] val; private int offset; private int count; private int hash; ... }

**24 bytes + char array
**

object overhead value offset count hash

reference values

45

int

Typical object memory requirements

Example 1 Q. How much memory does this program use as a function of N ?

public class RandomWalk { public static void main(String[] args) { int N = Integer.parseInt(args[0]); int[][] count = new int[N][N]; int x = N/2; int y = N/2; for (int i = 0; i < N; i++) { // no new variable declared in loop ... count[x][y]++; } } }

46

Example 2 Q. How much memory does this code fragment use as a function of N ?

... int N = Integer.parseInt(args[0]); for (int i = 0; i < N; i++) { int[] a = new int[N]; ... }

**Remark. Java automatically reclaims memory when it is no longer in use.
**

47

Out of memory Q. What if I run out of memory?

% java RandomWalk 10000 Exception in thread "main" java.lang.OutOfMemoryError: Java heap space % java -Xmx 500m RandomWalk 10000 ... % java RandomWalk 30000 Exception in thread "main" java.lang.OutOfMemoryError: Java heap space % java -Xmx 4500m RandomWalk 30000 Invalid maximum heap size: -Xmx4500m The specified size exceeds the maximum representable size. Could not create the Java virtual machine.

48

Turning the crank: summary In principle, accurate mathematical models are available. In practice, approximate mathematical models are easily achieved. Timing may be flawed? Limits on experiments insignificant compared to other sciences. Mathematics might be difficult? Only a few functions seem to turn up. Doubling hypothesis cancels complicated constants.

• • • • • • •

Actual data might not match input model? Need to understand input to effectively process it. Approach 1: design for the worst case. Approach 2: randomize, depend on probabilistic guarantee.

49

- APC
- APC
- Оптимизация и нарезка графики для профессиональной вёрстки
- верстка в IntelliJIDEA
- Вёрстка для мобильных устройств
- API Яндекс.Карт
- rial Algorithms for Web Search Engines - Three Success Stories
- Algorithm 781
- To Catch a Predator - A Natural Language Approach for Eliciting Malicious Payloads
- Robust Music Identification, Detection, And Analysis
- All Your iFRAMEs Point to Us
- Learning Forgiving” Hash Functions
- Kernel Methods for Learning Languages
- Distributional Identification of Non-Referential Pronouns
- Intelligent Email
- Learning Bounds for Domain Adaptation
- Natural Justice and Procedural Fairness
- A Machine Learning Framework for Spoken-dialog
- Trading Capacity for Data Protection
- Image Segmentation by Branch-and-Mincut (extended)
- Formal semantics of language and the Richard-Berry paradox
- Commonsense Knowledge, Ontology and Ordinary Language
- Automatic Thumbnail Cropping and Its Effectiveness (Best Student Paper)
- A Joint Model of Text and Aspect Ratings for Sentiment Summarization
- Index wiki database - design and experiments

Princeton University: Algorithms Course

Princeton University: Algorithms Course

- Tech Report NWU-CS-99-02
- Fingerprint Indexing Based on Local Arrangements of Minutiae Neighborhoods
- Econometric Computing With HC an HAC
- Python Stat
- RWeka manual
- Main Program
- Gaussian Process Emulation of Dynamic Computer Codes (Conti, Gosling Et Al)
- orcl3
- fgh
- pmf
- Random Essay
- 9ib_SPAT
- DigitalController_a8_b15
- Raptor Codes
- 1-s2.0-S0165011405001144-main
- 04 Steganography
- wcnc
- 10.1.1.92.2423
- Aplicación Difusa
- Camera Calibration
- g 01434249
- Evaluation of MANET Performance in Presence of Obstacles
- 05475063
- Hyper Spectral Data Art
- kleinbauer.pdf
- the sequential clustered method for dynamic chemical plant simulation
- Ejemplo de Analisis Multivariado Con R.
- Asic for Dsp
- Project Report
- Reversible data hiding

Are you sure?

This action might not be possible to undo. Are you sure you want to continue?

We've moved you to where you read on your other device.

Get the full title to continue

Get the full title to continue listening from where you left off, or restart the preview.

scribd