THE METHOD OF LEAST SQUARES Part 3

Chapter 4

Discrete Least-
Squares
At the end of this chapter, students should be able to:
 Describe curve fitting and distinguish the two approaches to curve fitting
 Describe and derive linear least-squares
 Describe polynomial least-squares
 Apply least-squares method in curve fitting problems

4.1 Introduction
There are many instances in engineering practice where a relationship between
two variables can be characterized experimentally. Often no underlying theory is
available to represent this relationship mathematically and analysis needs to be
done to see the behavior or the trend of the dependent variable given the
independent variable. The process of using the pattern of the data to make
prediction is called trend analysis. The procedure of finding a function to
represent this relationship is known as curve fitting. In this chapter, we shall
begin with a brief discussion on two general approaches to curve fitting
techniques namely least-squares and interpolation methods. The rest of the
chapter will focus on discrete least-squares method. We shall learn to fit a
polynomial to a given set of data using least-squares procedure. Interpolation
method will be discussed in depth in Chapter 4 (Interpolating Polynomials).

51
 THE METHOD OF LEAST SQUARES Part 3

4.2 Curve Fitting: Least-Squares or Interpolation

The curve fitting process fits equations of approximating curves to the raw field
data. Nevertheless, for a given set of data, the fitting curves of a given type are
generally NOT unique. There are two general approaches for curve fitting that
are distinguished from each other on the basis of the amount of error associated
with the data: least-squares and interpolation methods. Least-squares method is
used when the data exhibits a significant degree of error or “noise”. The strategy
is to derive a single curve that represents the general trend of the data. Since
any individual data may be incorrect, we make no effort to intersect at any point.
When the data available is known to be very precise or have considerably high
accuracy, the basic approach is to fit a curve that pass through directly to each of
the available points. The estimation of values between these well-known discrete
points is called interpolation.

f(x)
f(x)



  


 

x x
(a) Least-squares (b) Interpolation
Figure 4.1

4.3 Fitting a Line (Linear Regression)

Often in the real world one expects to find linear relationships between variables.
For example, the force of a spring linearly depends on the displacement of the
spring: y = kx (here y is the force, x is the displacement of the spring from rest,
and k is the spring constant). To test the proposed relationship, researchers go
to the lab and measure what the force is for various displacements.

52
 THE METHOD OF LEAST SQUARES Part 3

The objective in linear least-squares procedure is to determine the best fit

straight line that suit a set of data points.The best fit line is defined as the line
giving the smallest possible total error obtained by adding the squares of the
residuals (or squares of the difference between the true or observed values and
the calculated values).
y

f(x)=a0+a1x
(x2,y2) 


 E3

 E1 (x3,y3)
(x1,y1)
x

Figure 4.2

Suppose the equation of the line that we are looking for is f(x)  a0  a1 x .
Linear least-squares is a procedure in which the coefficients a0 and a1
(representing the y-intercept and slope) are determined such that the line best
fits the data points. For a given set of data points (x, y), x is assumed to be exact
and only the y values randomly have errors. The error (or called “residual”) for
each data set is the discrepancy between the observed value y and the
approximate value of y which is f(x)  a0  a1 x , predicted by the line (or called
the “calculated value”):
Ei  f(xi )  y i or Ei  (a0  a1 xi ) - yi
The strategy for fitting the “best” line through the data would be to minimize the
sum of the residual error squares for all available data, i.e.
n n
 E i2   (a0  a1xi )  y i 2 to be a minimum.
i 1 i 1
n
Let E (a0 , a1 )   (a
i 1
0  a1 x i )  y i  .
2
The objective is to minimize the function

E ao , a1  , which is a function of two variables, a0 and a1 . An elementary result

from multivariate calculus implies that, for a minimum to occur at ( a0 , a1 ) it is

necessary to differentiate partially with respect to each of these variables and
equate the derivatives to zero.

53
 THE METHOD OF LEAST SQUARES Part 3

Differentiating with respect to a0:


a0
 (a 0  a1 x i - yi )2  0 
 2(a 0  a1 xi - yi )  0

2a0 n  2a1  x - 2 y i i 0

a0 n  a x  y
1 i i (1)

Differentiating with respect to a1 :


a1
 (a 0  a1 x i - yi )2  0 
 2(a 0  a1 xi - yi ) xi  0

2a0 x i  2a1 x i
2
- 2 x y i i 0

a0 x i  a1 x i
2
 x y i i (2)

Hence we obtain a pair of simultaneous equations in a0 and a1 :

n n n
(1) a
i 1
0  a1 x
i 1
i  y
i 1
i

n n n
(2) a0  i 1
x i  a1 
i 1
xi
2
 x y
i 1
i i

By solving these two simultaneous equations (called the normal equations) we

obtain a0 and a1 as follows:
n n n
n 
i 1
xi y i   y
i 1
xi
i 1
i
a1  2
 n  n
n x i   x i   2

i 1  i 1 
a0  y  a1 x, where x and y are means of x and y .

The normal equations (1) and (2) above can also be written in matrix form and
the values of the coefficients a0 and a1 are obtained by solving the matrix:

 n
  n 
 n  xi   yi  
 i 1  a0  =  i 1 
 n  
2   a1   n 
n


 xi
 i 1
 i 1
xi 

 xi y i 
 i 1 


54
 THE METHOD OF LEAST SQUARES Part 3

Steps : Fitting a Line

 Write the linear function to be fitted

 Write the normal equations in matrix form (2x2).
 Find the values for all the elements of the matrix:
n (total no. of data points),
n n n n

 i 1
xi , 
i 1
2
xi , 
i 1
xi y i , y
i 1
i
2

 Solve for a0 and a1.

 Thus, the least-squares line is f(x) = a0 + a1x.

Example 1
Find the straight line that minimizes the least-squares error for the points (1,
4), (2, 5) and (3, 8).

Solution

 Write linear function to be fitted

f(x) = a0 + a1x
 Write the normal equations in matrix form (2x2).
 n
  n 
 n  xi   yi  
 i 1  a0  =  i 1 
 n  
2   a1   n 
n


 xi
 i 1
i 1
xi 

 xi y i 
 i 1 

 Find the values for all the elements of the matrix:
Total no. of points n = 3

i xi yi xi2 xiyi
1 1 4 1 4
2 2 5 4 10
3 3 8 9 24
n n n n


i 1
xi  6 
i 1
y i  17  i 1
x i  14
2
x y
i 1
i i  38

55
 THE METHOD OF LEAST SQUARES Part 3

Thus, the normal equations:

3 6  a0  17 
6 14   = 38
   a1   

 Solve for a0 and a1:

Apply Cramer’s rule:
3 6
A  6
6 14

17 6 3 17
A0   10 A1   12
38 14 6 38

A0 A1
a0   1.6667 a1  2
A A

 Thus, the least-squares line is f(x) = a0 + a1x = 1.6667 + 2x.

Example 2
Consider the data presented below.
xi 1.5 1.8 2.4 3.0 3.5 3.9 4.4 4.8 5.0
yi 4.8 5.7 7.0 8.3 10.9 12.4 13.1 13.6 15.3
Determine the least-squares linear function that best-fit the data and find the
value of y when x = 2.0.

Solution

 Write linear function to be fitted

f(x) = a0 + a1x
 Write the normal equations in matrix form (2x2).
 n
  n 
 n x i 
a0 
  yi 
 i 1   i 1 
 n   =  n
2 
n


 xi
 i 1

i 1
xi 

 a1 

 xi y i 
 i 1 
 Find the values for all the elements of the matrix:
Total no. of points: n = 9

56
 THE METHOD OF LEAST SQUARES Part 3

i xi yi xi2 xiyi
1 1.5 4.8 2.25 7.20
2 1.8 5.7 3.24 10.26
3 2.4 7.0 5.76 16.80
4 3.0 8.3 9.00 24.90
5 3.5 10.9 12.25 38.15
6 3.9 12.4 15.21 48.36
7 4.4 13.1 19.36 57.64
8 4.8 13.6 23.04 65.28
9 5.0 15.3 25.00 76.50
n n n n

i 1
x i  30.3 
i 1
y i  91.1 
i 1
x i  115.11
2
x y
i 1
i i  345.09

Thus, the normal equations:

 9 30.3  a0   91.1 
30.3 115.11   = 345.09
   a1   

 Solve for a0 and a1:

a0 = 0.2569
a1 = 2.9303
 The least-squares line is
f(x) = a0 + a1x = 0.2569 + 2.9303x.

Thus, when x = 2.0,

y(2.0) = f(2.0)
= a0 + a1(2.0)
= 0.2569 + 2.9303(2.0)
= 6.1175.

57
 THE METHOD OF LEAST SQUARES Part 3

Warm up exercise
An ecologist wished to know whether certain species of aquatic insects
have their ecological range limited by temperature. He collected the
following data, relating the average daily temperature at different
portions of a creek with the elevation of that portion of the creek (above
a sea level).

ELEVATION AVERAGE TEMPERATURE

(KILOMETERS) (DEGREE IN CELCIUS)
2.7 11.2
2.8 10
3.0 8.5
3.5 7.5

(i) Find the straight line that provides the best least-squares fit to this
data.
(ii) Use the linear function to estimate the average daily temperature
for this creek at altitude 3.2 kilometers.

4.3 Fitting a Polynomial (Degree two or more)

Linear least-squares or linear regression provides a powerful technique for fitting
a best line to a given set of data. However, this is not always the case. The ‘best’
fitting straight line may not fit the points satisfactorily, and in some instances a
quadratic or higher degree of polynomial may be preferable.
The first step in any regression analysis is to determine the most suitable model
to fit the data points and whether a linear function is the most appropriate. This
could be accomplished by doing a scatter plot of the data and visually inspect to
ascertain whether a linear model applies best. Otherwise, a polynomial of higher
degree could be analyzed and tested. Figure 4.3 illustrates an example whereby
a linear model may not be the best choice for a set of data points, but instead a
polynomial of degree two or parabola is more suitable.

58
 THE METHOD OF LEAST SQUARES Part 3

 
  
  
  
 

 

Linear fit Parabola fit

Figure 4.3

In least-squares polynomial of degree two or higher, the set of given data

(x1 , y1 ), (x2 , y 2 ), ....., (xn , y n ) , need to be fitted to that polynomial f(x) of
degree, say m.
i.e.
f(x)  a0  a1 x  a2 x 2  ...  am x m , where m < n

Likewise in this case, the objective is to solve for the coefficients

a0 , a1 , a2 , ...., am using least-squares procedure. Similarly, the assumption is
that x values are assumed to be exact and only the y values are in error. The

a0  a1 xi  a2 xi  ...  am xi
2 m
error at the ith point is Ei = - yi and we

seek to find the values of a0 , a1 , a2 , ...., am which give a minimum value of

n

E
i 1
i
2
. Here, the least squares procedure is handled in a similar manner as in

linear least-squares but requires choosing the constants a0 , a1 , a2 , ...., am to

n
minimize E
i 1
i
2
.

Using partial differentiation for each of the variables a0 , a1 , a2 , ...., am , (m + 1)

equations are obtained as follows:

59
 THE METHOD OF LEAST SQUARES Part 3

n n n n n

∑ a0  a1 ∑ x i  a2 ∑ x i  .....  am
2
∑ xi
m
 ∑y i (1)
i 1 i 1 i 1 i 1 i 1

n n n n n
a0 ∑ x i  a1 ∑ xi
2
 a2 ∑ x i  .....  am
3
∑ xi
m 1
 ∑y x i i (2)
i 1 i 1 i 1 i 1 i 1

n n n n n
a0 ∑x i
2
 a1 ∑x i
3
 a2 ∑x i
4
 .....  am ∑x i
m2
 ∑y x i i
2
(3)
i 1 i 1 i 1 i 1 i 1

n n n n n
a0 ∑x i
3
 a1 ∑x i
4
 a2 ∑x i
5
 .....  am ∑x i
m3
 ∑y x i i
3
(4)
i 1 i 1 i 1 i 1 i 1
. .
. .
. .
n n n n n
a0 ∑ xi
m
 a1 ∑ xi
m 1
 a2 ∑ xi
m2
 .....  am ∑ xi
2m
 ∑y x i i
m
(m+1)
i 1 i 1 i 1 i 1 i 1

The coefficients of these normal equations may be calculated by an extension of

the method used in the straight line case. The normal equations always have a
unique solution provided that the x i for i = 1, …., n are distinct.
The normal equations can be written in matrix form as shown below:


n x i x i
2
... x i
m


 a0  

y i


 
 x i x i
2
x i
3
... x i
m 1
 a
  1  x y i i 
 m2 
 x i
2
x i
3
x i
4
... x i
 a2  = 
   x y i
2
i


 ... ... ... ... ...   ...   
 2m   
  xi
m
 xi
m 1
 xi
m2
...  x i  am 
    m
xi y i 

The values for ai for i = 0, 1, 2, …, n can be solved by solving this system of

equations using any of the available technique such as Gauss Elimination,
Gauss-Jordan method and Cramer’s Rule.

Steps : Fitting a Polynomial of degree m

 Determine the degree m of the polynomial and write the

general expression of the polynomial:
Degree = m, f(x) = a0  a1 x  a2 x 2  ...  am x m
 Write the normal equations in matrix form (m+1)x(m+1).
 Find the values for all the elements of the matrix.
n (total no. of data points) and rest of elements
 Solve for the coefficients a0, a1…am.
 Thus, the least-squares function is:
60
f(x) = a0  a1 x  a2 x 2  ...  am x m

 THE METHOD OF LEAST SQUARES Part 3

Example 4
Fit the data in the given table with the discrete least squares polynomial of
degree two.
i 0 1 2 3 4
xi 0 0.25 0.5 0.75 1.00
yi 1.0 1.2840 1.6487 2.1170 2.7183

Solution

 Determine the degree m of the polynomial and write the general

expression of the polynomial:

Degree : m = 2 , f(x) = a0  a1x  a2 x 2

 Write the normal equations in matrix form (3x3).
 n
  x  x 2  a 0    y 
a   


 x  x 2  x 3   1   x y 
a 2  

 x
2
 x 3  x 4    x 2 y 
 Find the values for all the elements of the matrix.
Total no. of points: n = 5

i x y x2 x3 x4 xi yi xi 2 yi
1 0 1.0 0 0 0 0 0
2 0.25 1.2840 0.0625 0.0156 0.0039 0.3210 0.0803
3 0.5 1.6487 0.2500 0.1250 0.0625 0.8244 0.4122
4 0.75 2.1170 0.5625 0.4219 0.3164 1.5878 1.1908
5 1.00 2.7183 1.0000 1.0000 1.0000 2.7183 2.7183
 2.5000 8.7680 1.8750 1.5625 1.3828 5.4514 4.4015

 5 2.5000 1.8750 a0  8.7680

     
2.5000 1.8750 1.5625  a1   5.4514
1.8750 1.5625 1.3828 a 2  4.4015

 Solve for the coefficients a0, a1, a2:

Apply Gauss elimination method
 5 2.5 1.875 8.7680
 
 2.5 1.875 1.5625 5.4514 - (2.5/5)R1 R2
1.875 1.5625 1.3828 4.4015 - (1.875/5)R1  R3

61
 THE METHOD OF LEAST SQUARES Part 3

5 2.5 1.875 8.7680

 
0 0.625 0.625 1.0674
0 0.625 0.6797 1.1135 - (0.625/0.625)R2  R3

5 2.5 1.875 8.7680

 
0 0.625 0.625 1.0674
0 0 0.0547 0.0461

Using back substitutions :

Refer row 3: 0.0547a2  0.0461  a2  0.8432

Refer row 2: 0.625a1  1.0674  0.625a2  a1  0.8647

Refer row 1: 5a0  8.7680  2.5a1  1.875a2  a0  1.0051

Therefore, a0  1.0051, a1  0.8647, a2  0.8432

Hence, the least-squares function is:

f(x)  a0  a1 x  a2 x 2
 1.0051  0.8647x  0.8432x2

Example 5
Fit a quadratic curve to the following data.
x 2 4 6 8 10
y 3.07 12.85 31.47 57.38 91.29

Solution

Degree : m = 2 , f(x)  a0  a1 x  a2 x 2

 n

x x 2

3
a0  
a   
 y 
 x x x
2
  1  xy 

 x2
x x
3 4

a2  
  x y 
2

Total no. of points: n = 5

62
 THE METHOD OF LEAST SQUARES Part 3

i x y x2 x3 x4 xi yi xi 2 yi
1 2 3.07 4 8 16 6.14 12.28
2 4 12.85 16 64 256 51.4 205.6
3 6 31.47 36 216 1296 188.82 1132.92
4 8 57.38 64 512 4096 459.04 3672.32
5 10 91.29 100 1000 10000 912.9 9129
 30 196.06 220 1800 15664 1618.3 14152.1

 5 30 220  a0   196.06 

     
 30 220 1800   a1    1618.3 
220 1800 15664 a2  14152.1

Therefore, a0  0.6935 a1  0.8540, a2  0.9919

Hence,
f(x)  a0  a1 x  a2 x 2
 0.6935 - 0.8540x  0.9919x2

Warm up exercise

Given the data, use least-squares regression to fit

(i) a straight line
(ii) a parabola

x 5 10 15 20 25 30 35 40 45 50
y 16 25 32 33 38 36 39 40 42 42

Make a scatter plot of these points. Which of the function above is more
suitable?

63
 THE METHOD OF LEAST SQUARES Part 3

Exercise 4

1. Fit a straight line to the given data.

f(x) = 0.7x +2.6 f(x)=0.79x-1.26 f(x)=0.85x-4.15
a) x y b) x y c) x y
-2 1 -6 -5.3 -3 -6
-1 2 -2 -3.5 1 -4
0 3 0 -1.7 3 -2
1 3 2 0.2 5 0
2 4 6 4.0 9 4

2. Fit a parabola to the given data.

f(x)=7.88-9.481x+4.0476x2 f(x)=2.125-1.7x+0.875x2 f(x)=3.13+2.13x+x2

a) x y b) x y c) x y
2.0 5.1 -3 15 -2 2.80
2.3 7.5 -1 5 -1 2.10
2.6 10.6 1 1 0 3.25
2.9 14.4 3 5 1 6.00
3.2 19.0 2 11.50

3. The table below shows the time (in seconds) required for water to drain through a
hole in the bottom of a bottle as a function of depth (in meters) to which the bottle
has been filled.

Depth 0.05 0.10 0.15 0.20 0.30 0.35 0.40

Time 65.99 120.28 166.69 207.85 279.95 313.04 344.24

a) Find the regression line to fit the data

b) Estimate the time required for water to drain at depth of 0.25 meters.

4. Find the best-fit line for the given values.

x 5 6 10 14 16 20 22 28 28 36 38
y 30 22 28 14 22 16 8 8 14 0 4

5. Fit a least-squares quadratic function (parabola) to the given data.

x 0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0

p(x) 0.0 2.0 5.0 8.0 12.0 25.0 40.0 57.0

6. The following tabulated values are taken from an experiment. Complete the table
and calculate the sum of error squares (use linear regression).

x 0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0

Observed y 0.0 2.0 5.0 8.0 12.0 25.0 40.0 57.0
Calculated y
Error Square

64