E7 – Introduction To Programming for Engineers

Final Review

Çağlar Tamur, Johannes Yu, Shaofan Li

Fall 2021
Instructor: Shaofan Li
Agenda

Post-Midterm Topics

• Root Finding
• Linear Systems
• Regression
• Interpolation
• Gradient Descent
• Numerical Differentiation
• Numerical Integration
• Ordinary Differential Equations
Root Finding
 Bisection Method
Numerical Root Finding:

Given 𝑓: 𝑅 → 𝑅, 𝑓𝑖𝑛𝑑 𝑥 ∗ 𝑠𝑢𝑐ℎ 𝑡ℎ𝑎𝑡 𝑓 𝑥 ∗ = 0 ± 𝑡𝑜𝑙

Bisection Method Algorithm:

• Find an interval [a,b], where a root of the function exists
sign(f(a)) ≠ sign(f(b))
• Find the midpoint between a and b, call this m
m = (a+b)/2
• Check if f(m) is within the given tolerance around 0
f(m) = 0 ± tol ?
– If true, m is your solution
– If not, re-assign a (or b) and iterate these steps, until reaching the desired tolerance
 Bisection Method: Remarks

• Method is guaranteed to converge, given the function is continuous.

• Converges slowly!
– Error is halved at each step (linear convergence)

• If one of the initial guess (a or b) is close to the root,

method will take a large number of iterations to reach the root.
y

f(x)
• Doesn’t work if the function ‘touches’ the x- axis

x
Newton-Raphson Method
𝒇 𝒙𝒏
𝒙𝒏+𝟏 = 𝒙𝒏 − ′
𝒇 (𝒙𝒏 )

(Update rule for Newton-Raphson scheme)

Algorithm:

• Start with an initial guess xn

• Find the improved guess xn+1 by using the update rule

• Check if f(xn+1) is within the given tolerance around 0

• Iterate until reaching the desired tolerance…

Newton-Raphson Method: Remarks
Unlike Bisection, Newton’s method is not guaranteed to converge!

Converges much faster than Bisection (quadratic convergence)

Limitations:
• Need to start with a good initial guess
• If f’(x) happens to be close to zero,
next step would be far away from the root
• Can converge to a root that
we’re not interested in
Example
Example

𝒇 𝒙𝒏
𝒙𝒏+𝟏 = 𝒙𝒏 −
𝒇′ (𝒙𝒏 )
Example
Error is halved at each step (linear convergence)
Example
Linear Systems
Linear Systems
Linear Systems: Solutions

A linear system 𝑨𝒎×𝒏 𝒙𝒏×𝟏 = 𝒚𝒎×𝟏 can behave in three different ways:

𝑟𝑎𝑛𝑘 𝐴, 𝑦 ≠ 𝑟𝑎𝑛𝑘(𝐴) 1) No Solution

(Overconstrained!)

𝑟𝑎𝑛𝑘 𝐴 = 𝑛 2) Unique Solution

𝑟𝑎𝑛𝑘 𝐴, 𝑦 = 𝑟𝑎𝑛𝑘(𝐴)
3) Infinitely Many
𝑟𝑎𝑛𝑘 𝐴 < 𝑛
Solutions
(Underconstrained)
Linear Systems: Solutions
1) Unique Solution

𝒙=𝑨\𝒚

2) Infinitely Many Solutions (Under-constrained)

Can get a solution:
−𝟏
𝒙 = 𝒑𝒊𝒏𝒗 𝑨 𝒚 = 𝑨𝑻 𝑨𝑨𝑻 𝒚

3) No Solution (Over-constrained)

Can get an approximate (regression) solution:

−𝟏 𝑻
𝒙 = 𝑨𝑻 𝑨 𝑨 𝒚
Example
Example
Example
Regression
 Least Squares Regression Simple Linear Regression

Idea: Minimize the ‘error’ of the fit

Least Square Regression: Minimize the total squared error

where : individual errors (or residuals)

: estimation
: real data
 Least Squares Regression 𝑦(𝑥)
ො = 𝛽1 + 𝛽2 𝑥

Optimization problem:
Find parameters (𝜷) which will minimize TSE

Using some Calculus….

Where;
Least Squares Regression 𝐿𝑖𝑛𝑒𝑎𝑟 𝐵𝑎𝑠𝑖𝑠 𝐹𝑢𝑛𝑐𝑡𝑖𝑜𝑛𝑠:
𝑦(𝑥)
ො = 𝛽1 + 𝛽2 𝑥

In general:
where : basis functions
: model parameters

where matrix A is defined as:

𝐴𝑖𝑗 = 𝑓𝑗 𝑥𝑖

and
is called the pseudoinverse of A
Least Squares Regression
1 𝑥1
Simple Linear fit 1 𝑥2
1 𝑥3 𝐴𝑖𝑗 = 𝑓𝑗 𝑥𝑖
෣ = 𝜷𝟏 + 𝜷𝟐 𝒙
𝒚(𝒙) 𝑨= . .
. .
1 𝑥𝑛
Polynomial fit: 1 𝑥1 𝑥1 2 . . .
1 𝑥2 𝑥2 2 . . .
෣ = 𝜷𝟏 + 𝜷𝟐 𝒙 + 𝜷𝟑 𝒙𝟐 + … 1 𝑥3 𝑥3 2 . . .
𝒚(𝒙) 𝑨= . . . . . .
. . . . . .
1 𝑥𝑛 𝑥𝑛 2 . . .
In MATLAB:
Example
Example
Example
Example
Example
Example
Interpolation
Interpolation

Regression: Functions that best estimates a set of data points, by minimizing

the sum of the squared errors.

Interpolation: Functions that goes through the data points

 Linear Interpolation

Given two data points 𝑥𝑖 , 𝑦𝑖 and 𝑥𝑖+1 , 𝑦𝑖+1 ,

Linear interpolation at x, where 𝑥𝑖 < 𝑥 < 𝑥𝑖+1

Connect piecewise linear functions

 Lagrange Polynomials
A single polynomial that goes through all the data points.
For a data set of N points, Lagrange Polynomial has to be at most degree N-1

Lagrange Basis Functions (n-1 degree)

Lagrange Polynomial (n-points)

“Goes through all points in the data set”

Lagrange Polynomials: How to construct?

Example: Interpolate 3 points (x,y) = { (0,1), (1,3), (2,2) }

Example
Polynomial Interpolation and polynomial regression

Suppose that the interpolation polynomial is in the form

The statement that p interpolates the data points means that

If we substitute Eq. (1) to Eq. (2) , we get a system of linear equations in the coefficients.
 Polynomial interpolation:

Polynomial regression:

A isn’t a square matrix!

Example
Example

Lagrangian.m
Example
Gradient Descent
Gradient Descent
Task: We want to optimize (find the minimum) of the function 𝑓 𝑥

𝑓 𝑥 decreases fastest in the opposite direction of its gradient.

Start from an initial guess and move in the opposite direction of the gradient.

𝑥Ԧ𝑛+1 = 𝑥Ԧ𝑛 − 𝛾 ∇𝑓(𝑥𝑛 )

Update rule for Gradient Descent

∇𝑓 𝑥 = Grad of the objective function

𝛾 = Learning Rate (step size)

Remarks on Gradient Descent
Need to choose a suitable learning rate:

Convergence:
 Numerical Differentiation
(Finite Difference Methods)
Finite Difference Approximations
(Approximate Slope)
𝒇(𝒙)

Forward Difference ∆𝒙
𝑓(𝑥𝑖+1 )
𝒇(𝒙𝒊+𝟏 ) − 𝒇(𝒙𝒊 ) 𝒇(𝒙𝒊+𝟏 ) − 𝒇(𝒙𝒊 )
𝒇′(𝒙𝒊 ) ≈
∆𝒙
𝑓(𝑥𝑖 )
𝒇′(𝒙𝒊 )
𝐸𝑟𝑟𝑜𝑟 ~ 𝑂(∆𝑥) (True Slope)

𝑥𝑖−1 𝑥𝑖 𝑥𝑖+1
Finite Difference Approximations 𝒇(𝒙)

Backward Difference
(Approximate Slope)

𝒇(𝒙𝒊 ) − 𝒇(𝒙𝒊−𝟏 )
𝒇′(𝒙𝒊 ) ≈
∆𝒙 ∆𝒙
𝑓(𝑥𝑖 )
𝒇(𝒙𝒊 ) − 𝒇(𝒙𝒊−𝟏 ) 𝒇′(𝒙𝒊 )
𝑓(𝑥𝑖+1 ) (True Slope)
𝐸𝑟𝑟𝑜𝑟 ~ 𝑂(∆𝑥)
𝑥𝑖−1 𝑥𝑖 𝑥𝑖+1
Finite Difference Approximations 𝒇(𝒙)
(Approximate Slope)

Central Difference 𝟐∆𝒙

𝑓(𝑥𝑖+1 )

𝒇(𝒙𝒊+𝟏 ) − 𝒇(𝒙𝒊−𝟏 )
𝒇′(𝒙𝒊 ) ≈ 𝒇(𝒙𝒊+𝟏 ) − 𝒇(𝒙𝒊−𝟏 )
𝟐∆𝒙
𝒇′(𝒙𝒊 )
𝐸𝑟𝑟𝑜𝑟 ~ 𝑂(∆𝑥 2 ) 𝑓(𝑥𝑖−1 ) (True Slope)

𝑥𝑖−1 𝑥𝑖 𝑥𝑖+1
Example
Example
Example
Example
Numerical Integration
Riemann Sum
Approximate the area under a curve by summing a series of rectangles

𝑏−𝑎 𝑥𝑖 + 𝑥𝑖+1
where; 𝑁 = 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝐼𝑛𝑡𝑒𝑟𝑣𝑎𝑙𝑠, ℎ= is the 𝑺𝒕𝒆𝒑 𝑺𝒊𝒛𝒆 and 𝑦𝑖 =
𝑁 2
Trapezoidal Rule

Approximate the area under a curve as a trapezoid.

Divide the integration interval and apply the Trapezoidal Rule to each of the sub-intervals

where;
𝑏−𝑎
𝑁 = 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝐼𝑛𝑡𝑒𝑟𝑣𝑎𝑙𝑠, ℎ= is the 𝑆𝑡𝑒𝑝 𝑆𝑖𝑧𝑒
𝑁
𝑥𝑖+1 = 𝑥𝑖 + ℎ
Simpson’s Rule
Approximate the area by fitting quadratic polynomials through 3 consecutive data points

ℎ ℎ

𝑥0 𝑥1 𝑥2
Monte Carlo Integration
Use random sampling to approximate an integral:

𝑏 𝑁
𝑏−𝑎
න 𝑓 𝑥 𝑑𝑥 ≈ ෍ 𝑓(𝑥𝑖∗ )
𝑎 𝑁
𝑖=1

𝑥𝑖∗ = Random points, sampled uniformly from the interval a, b

𝑁 = Number of sampling points
Order of Accuracy
Order of accuracy of other integration schemes can be by using Taylor Series expansions

Sub-interval Whole Interval

Left & Right Riemann 𝑂(ℎ2 ) 𝑂(ℎ)

Midpoint Rule 𝑂(ℎ3 ) 𝑂(ℎ2 )
Trapezoidal Rule 𝑂(ℎ3 ) 𝑂(ℎ2 )
Simpson’s Rule 𝑂(ℎ5 ) 𝑂(ℎ4 )

𝐌𝐨𝐧𝐭𝐞 𝐂𝐚𝐫𝐥𝐨 = 𝑂(𝑁 −1/2 )

Example
Example
Example
Example
Example
Example
 Midpoint Rule

Trapezoidal Rule

Simpson’s Rule
Example

Left Riemann Mid-Point Rule

Right Riemann

Trapezoidal
Simpson’s Rule
Approximate the area by fitting quadratic polynomials through 3 consecutive data points

ℎ ℎ
𝑥0 𝑥1 𝑥2

To approximate the integral over (a,b), divide the interval into segments of equal
width
 Numerical Methods for
Ordinary Differential Equations
 Euler’s Method Explicit Euler Method

Consider the first order ordinary differential equation:

𝑑𝑦
= 𝑓(𝑡, 𝑦) 𝑡 ∈ [𝑡0 , 𝑡𝑛 ] 𝑦(𝑡0 ) = 𝑦0
𝑑𝑡

Explicit Euler Method: 𝑦𝑖+1 = 𝑦𝑖 + ℎ ∙ 𝑓 𝑡𝑖 , 𝑦𝑖

‘Explicit’ since it only requires information at 𝑡𝑖 to compute the state at 𝑡𝑖+1

𝑵𝒖𝒎𝒆𝒓𝒊𝒄𝒂𝒍 𝒔𝒐𝒍𝒖𝒕𝒊𝒐𝒏, 𝒚𝒊

Implicit Euler Method: 𝑦𝑖+1 = 𝑦𝑖 + ℎ ∙ 𝑓 𝑡𝑖+1 , 𝑦𝑖+1 𝑻𝒓𝒖𝒆 𝒔𝒐𝒍𝒖𝒕𝒊𝒐𝒏, 𝒚(𝒕)

 𝑑𝑦
Runge-Kutta (RK) Methods 𝑑𝑡
= 𝑓(𝑡, 𝑦)

Get a higher accuracy method by evaluating the derivative several times at different locations

Fourth Order Runge-Kutta Method (RK4):

ℎ
𝑦𝑖+1 = 𝑦𝑖 + (𝑘1 + 2𝑘2 + 2𝑘3 + 𝑘4 )
6

where

𝑘1 = 𝑓(𝑡𝑖 , 𝑦𝑖 )
𝑘2 = 𝑓(𝑡𝑖 + ℎ/2 , 𝑦𝑖 + ℎ/2 𝑘1 )
𝑘3 = 𝑓(𝑡𝑖 + ℎ/2 , 𝑦𝑖 + ℎ/2 𝑘2 )
𝑘4 = 𝑓(𝑡𝑖 + ℎ , 𝑦𝑖 + ℎ 𝑘3 )

“Slopes”
Other Methods
Second-Order Runge-Kutta Method (RK2)

𝑦𝑖+1 = 𝑦𝑖 + ℎ 𝑘2

𝑘1 = 𝑓(𝑡𝑖 , 𝑦𝑖 )

𝑘2 = 𝑓(𝑡𝑖 + ℎ/2 , 𝑦𝑖 + (ℎ/2) 𝑘1 )

Central Difference Method

𝑦𝑖+1 − 𝑦𝑖−1
𝑓(𝑡𝑖 , 𝑦𝑖 ) ≈ 𝑦𝑖+1 = 𝑦𝑖−1 + 2ℎ 𝑓 𝑡𝑖 , 𝑦𝑖
2∆𝑥
Order of Accuracy

Accuracy: How fast a scheme gets close to the exact solution, as a function of step size.

Euler’s Method = 𝑂(ℎ)

RK2 and Central Difference= 𝑂(ℎ2 )

RK4 Method = 𝑂(ℎ4 )

Example
Example
Example
Example
We have solved an ODE using three different methods, and

Hint: Midpoint method is 𝑂 ℎ2

Example
We have solved an ODE using three different methods, and

Hint: Midpoint method is 𝑂 ℎ2

Example
Example
Derivation of the Fourth-Order Runge-Kutta (RK4) Method
We use Simpson’s 1/3 rule to integrate this equation:

𝑘1 = 𝑓(𝑡𝑖 , 𝑦𝑖 )
ℎ 𝑘2 = 𝑓(𝑡𝑖 + ℎ/2 , 𝑦𝑖 + ℎ/2 𝑘1 )
𝑦𝑖+1 = 𝑦𝑖 + (𝑘1 + 2𝑘2 + 2𝑘3 + 𝑘4 )
6 𝑘3 = 𝑓(𝑡𝑖 + ℎ/2 , 𝑦𝑖 + ℎ/2 𝑘2 )
𝑘4 = 𝑓(𝑡𝑖 + ℎ , 𝑦𝑖 + ℎ 𝑘3 )
Example
(II): Runge-Kutta Method
Second-Order Runge-Kutta (RK2) Method Midpoint rule
We use midpoint rule to integrate this equation: 𝑑𝑦
= 𝑓(𝑡, 𝑦)
𝑑𝑡
Example

Meshgrid.m
These are the topics
covered before midterm,
which will be tested again
in final exam.
Good luck on your finals!