Numerical Analysis

by
Dr. Anita Pal
Assistant Professor
Department of Mathematics
National Institute of Technology Durgapur
Durgapur-713209
email: anita.buie@gmail.com

1
.

Chapter 3

Approximation of Functions

Module No. 1

Least Squares Method

......................................................................................

Lots of experimental data are available in science and engineering applications and it
is very essential to construct a curve based on this data. From the fitted curve, it is easy
to determine the (approximate) values of unknown parameter(s) at non-experimental
points. So, curve fitting is a very useful and important topic. But, the question is which
curve is better for a given data set?
There are two broad methods are available for this problem, one is matching method
and another is approximation method. In Chapter 2, lots of interpolation methods
are discussed to construct a polynomial which passes through the given points. These
methods are called the matching methods. If n points (xi , yi ), i = 1, 2, . . . , n are given,
then in interpolation an nth degree polynomial is constructed. But, it is obvious that
the evaluation of large degree polynomial is a difficult task, though it gives exact values
at the given nodes x0 , x1 , . . . , xn . Again, the cubic spline generates n third degree
polynomials for n data points. By using approximation method one can construct
lower degree polynomials such as linear, quadratic etc. and other types of curve, viz.,
geometric, exponential etc. Mainly least squares method is used to construct such curve.
The least squares method does not give the guarantee that the curve must passes through
the given points, but it will minimizes the sum of square of the absolute errors. In this
method, the user can decide the degree of the polynomial irrespective of the data size,
but in interpolation the degree of the polynomial depends on the data set.

1.1 General least squares method

Let (xi , yi ), i = 1, 2, . . . , n be the give data set, i.e. a bivariate sample of size n. Our
problem is to construct a curve of the following form

y = g(x; a0 , a1 , . . . , ak ), (1.1)

where a0 , a1 , . . . , ak are the unknown parameters and their values are to be determined
in terms of given sample.
When x = xi , the ith argument, the value of y obtained from the equation (1.1) is
denoted by Yi . This value is called the predicted value or expected value or observed
value of y and in general, it is approximate value of y, i.e. erroneous value, where as yi
is the exact value of y.
1
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Least Squares Method

These points and the least squares curve are shown in Figure 1.1. The points are
drawn in blue colour and the curve is drawn in red colour. The dotted line represents
the actual value of y at x = xi . The predicted value Yi is the length between the (red)
curve and the x-axis.

y
6
r r
r r
r r r
r r r
r
r
6 6
yi -Yi r r
r r r6 r
?
r
r r
y i
Yi

? ? - x
O xi

Figure 1.1: The sample points and the least squares curve

The predicted value Yi is given by the equation

Yi = g(xi ; a0 , a1 , . . . , ak ). (1.2)

In general, Yi and yi are different as the predicted values does not necessarily satisfy
the curve (1.1). The difference (yi − Yi ) is called the residual corresponding to x = xi .
Now, the sum of the square of the residuals be denoted by S and it is given by the
following expression
n
X n
X
S= (yi − Yi )2 = [yi − g(xi ; a0 , a1 , . . . , ak )]2 . (1.3)
i=1 i=1

The values of the parameters a0 , a1 , . . . , ak are determined by least squares method in

such a way that S is minimum. Note that S contains (k + 1) parameters a0 , a1 , . . . , ak .
2
 ......................................................................................

The value of S will be minimum if

∂S ∂S ∂S
= 0, = 0, . . . , = 0. (1.4)
∂a0 ∂a1 ∂ak
This is a system of (k + 1) equations and these equations are called normal equa-
tions. Let the solution of these (k + 1) equations for the parameters a0 , a1 , . . . , ak be
a0 = a∗0 , a1 = a∗1 , . . . , ak = a∗k .
Then the required fitted curve is

y = g(x; a∗0 , a∗1 , . . . , a∗k ). (1.5)

The sum of the square of residuals is determined from the following equation
n
X n
X
S= 2
(yi − Yi ) = [yi − g(xi ; a∗0 , a∗1 , . . . , a∗k )]2 . (1.6)
i=1 i=1

This is the general method to fit a curve by using least squares method. Now, we
discuss this method to fit some special type of curves.

1.2 Fitting of a straight line

The straight line is a most simple curve and it is easy to handle. Let the equation of
a straight line be

y = a + bx, (1.7)

where a and b are two parameters and their values are to be determined by the least
squares method.
Let (xi , yi ), i = 1, 2, . . . , n, be a given sample of size n.
In this case, the sum of square of residuals S is given by
n
X n
X
S= (yi − Yi )2 = (yi − a − bxi )2 .
i=1 i=1

Here, the parameters are a and b. The normal equations are

∂S X
= −2 (yi − a − bxi ) = 0
∂a
∂S X
= −2 (yi − a − bxi )xi = 0.
∂b
3
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Least Squares Method

After simplification, these equations reduce to

X X
yi = na + b xi
X X X
x i yi = a xi + b x2i . (1.8)

The solution of these equations is

P P P
n x i yi − x i yi 1hX X i
b= and a= yi − b xi . (1.9)
n x2i − ( xi )2
P P
n

The expression of b can also be written as

n
P
(x − x)(y − y) n n
i=1 1X 1X
b= n , where x = xi , y = yi . (1.10)
n n
(xi − x)2
P
i=1 i=1
i=1

The above formula is generally used when the data set is large.
Let a∗ and b∗ be the values of a and b. Then the fitted straight line is

y = a∗ + b∗ x. (1.11)

Example 1.1 Let (2, 2), (−1.5, −2), (4, 4.5) and (−2.5, −3) be a sample. Use least
squares method to fit the line y = a + bx based on this sample and estimate the total
error.
Solution. The sample size is n = 4. The normal equations for straight line are
X X
yi = na + b xi
X X X
x i yi = a xi + b x2i
P P P
The values of xi , yi and xi yi are calculated in the following table.

xi yi x2i x i yi
2.00 2.00 4.00 4.00
−1.50 −2.00 2.25 3.00
4.00 4.50 16.00 18.00
−2.50 −3.00 6.25 7.50
Total 2.00 1.50 28.50 32.50
4
......................................................................................

Thus the normal equations are

1.5 = 4a + 2b and 32.5 = 2a + 28.5b.

The solution of these equations is

a = −0.2023, b = 1.1545.
Hence, the fitted straight line is

y = −0.2023 + 1.1545x.
Estimation of error.

x Given y y, obtained from (yi − Yi )2

(yi ) the curve (Yi )
2.0 2.0 2.1067 0.0114
−1.5 −2.0 −1.9340 0.0043
4.0 4.5 4.3657 0.0180
−2.5 −3.0 −3.0885 0.0078
Total 0.0415

Hence, the sum of the square of the residuals is 0.0415.

1.3 Fitting of polynomial of degree k

Here, we consider a general case of polynomial curve fitting.

Let (xi , yi ), i = 1, 2, . . . , n be the given bivariate sample of size n. The problem is to
fit the following polynomial curve based on the above data. Let

y = a0 + a1 x + a2 x2 + · · · + ak xk , (1.12)

be a polynomial curve of degree k, where a0 , a1 , . . . , ak are the (k + 1) parameters and

their values are to be determined by least squares method in terms of the sample values.
Let Yi be the value of y when x = xi obtained from the curve (1.12).
The sum of square of residuals is
n
X n h
X i2
S= (yi − Yi )2 = yi − (a0 + a1 xi + a2 x2i + · · · + ak xki ) . (1.13)
i=0 i=0

5
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Least Squares Method

As in previous cases, the normal equations are given by

∂S ∂S ∂S
= 0, = 0, . . . , = 0.
∂a0 ∂a1 ∂ak
After simplification, these equations become
X X X X
na0 + a1 xi + a2 x2i + · · · + ak xki = yi
X X X X X
a0 xi + a1 x2i + a2 x3i + · · · + ak xk+1
i = x i yi (1.14)
X X X X X
a0 x2i + a1 x3i + a2 x4i + · · · + ak xk+2
i = x2i yi
················································ ··· ······
X X X X X
a0 xki + a1 xk+1
i + a2 xk+2
i + · · · + ak x2k
i = xki yi .

This is a set of (k + 1) linear equations containing (k + 1) parameters a0 , a1 , . . . , ak .

Let a0 = a∗0 , a1 = a∗1 , . . . , ak = a∗k be the solution of this system of linear equations.
Then the fitted polynomial is

y = a∗0 + a∗1 x + a∗2 x2 + · · · + a∗k xk . (1.15)

1.3.1 Fitting of second degree parabolic curve

Let us consider a particular case of kth degree polynomial. Here, we consider k = 2.

Then the curve is known as parabolic curve or a polynomial curve of degree 2.
Let the equation of the parabolic curve be
y = a + bx + cx2 , (1.16)

where a, b, c are unknown parameters and their values are to be determined by least
squares method.
Let Yi be the value of y when x = xi obtained from the curve (1.16).
Therefore, Yi = a + bxi + cx2i .
For this case, the sum of square of residuals S is
n
X n
X
S= (yi − Yi )2 = (yi − a − bxi − cx2i )2 .
i=1 i=1

Hence, the normal equations are

∂S ∂S ∂S
= 0, = 0, and = 0.
∂a ∂b ∂c
6
......................................................................................

After simplification, these equations become

X X X
yi = na + b xi + c x2i
X X X X
xi yi = a xi + b x2i + c x3i
X X X X
x2i yi = a x2i + b x3i + c x4i .

Let a = a∗ , b = b∗ and c = c∗ be the solution of the above equations. Then the fitted
parabolic curve is

y = a∗ + b∗ x + c∗ x2 (1.17)

Example 1.2 Fit a quadratic polynomial curve for the following data

x : 1 2 3 4 5 6 7 8 9
y : 3 4 6 7 7 8 10 11 11

Solution. We transform the x values by u = 5−x to reduce the arithmetic computation.

Let the quadratic curve be y = a + bu + cu2 .
For this problem, the normal equations are

X
= na + b u + c u2
P P
y
X
uy = a u + b u2 + c u3
P P P
X
u2 y = a u2 + b u3 + c u4 .
P P P

The necessary calculations are shown below:

7
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Least Squares Method

x u y u2 uy u3 u2 y u4
1 4 3 16 12 64 48 256
2 3 4 9 12 27 36 81
3 2 6 4 12 8 24 16
4 1 7 1 7 1 7 1
5 0 7 0 0 0 0 0
6 −1 8 1 −8 −1 8 1
7 −2 10 4 −20 −8 40 16
8 −3 11 9 −33 −27 99 81
9 −4 11 16 −44 −64 176 256
Total 0 67 60 −62 0 438 708

Using this values the normal equations are

67 = 9a + 0b + 60c
−62 = 0a + 60b + 0c
438 = 60a + 0b + 708c.

Solution of these equations is b = −1.033, a = 7.632, c = −0.028.

Thus, the fitted quadratic curve is

y = 7.632 + (−1.033)u + (−0.028)u2 = 7.632 + (−1.033)(5 − x) + (−0.028)(5 − x)2

= 1.767 + 1.313x − 0.028x3 .

1.4 Fitting of other curves

Using least squares method one can fit many other curves such as geometric curve,
exponent curve, hyperbolic curve, etc. These type of curves occur in different applica-
tions and the nature of the curve depends on the problem and the data. Some of these
curves are described below.

8
 ......................................................................................

1.4.1 Geometric curve

Let us consider the geometric curve. Let

y = axb (1.18)

be the geometric curve. Here x and y are independent and dependent variables re-
spectively, and a and b are called the parameters of the curve. The values of these
parameters are to be determined by least squares method in terms of the sample
(xi , yi ), i = 1, 2, . . . , n.
Taking logarithm on both sides of the equation (1.18), we obtain log y = log a+b log x.
This equation can be written as Y = A + bX, where Y = log y, X = log x, A =
log a. Thus to fit the curve of equation (1.18), we have to determine the values of the
parameters A and b.
For this purpose, the given sample (xi , yi ) is to be converted to (Xi , Yi ), i = 1, 2, . . . , n.
Using this sample, and by using the procedure described in Section 1.2, one can deter-
mine the values of A and b. The value of a is then determine from the equation a = eA .

Example 1.3 Fit a geometric curve of the form y = axb for the following data.

x : 1 2 3 4 5
y : 2.7 5.6 10.2 15.3 21

Solution. Let y = axb . Here a and b are parameters. Taking logarithm both sides, we
get Y = A + bX, where log y = Y, log x = X, log a = A.
The normal equations for the curve Y = A + bX are

X X
Yi = nA + b Xi
X X X
Xi Yi = A Xi + b Xi2 .

The necessary values are calculated in the following table.

9
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Least Squares Method

x y X Y X2 XY
1 2.7 0 1.99325 0 0
2 5.6 0.69315 1.72276 0.48045 1.19413
3 10.2 1.09861 2.32238 1.20694 2.55139
4 15.3 1.38629 2.72785 1.92179 3.78159
5 21.0 1.60944 3.04452 2.59029 4.89997
Total 4.78749 10.81076 6.19947 12.42708
Thus the normal equations are

5A + 4.78749b = 10.81076,
4.78749A + 6.19947b = 12.42708.

The solution of these equations is A = 0.93180, b = 1.28496. Then a = eA = 2.53907.

Hence, the fitted curve is
y = 2.53907 x1.28496 .

1.4.2 Exponential curve

This is a particular case of geometric curve. Let the exponential curve be

y = aebx . (1.19)

Here the parameters are a and b and their values are to be determined by least squares
method.
Taking logarithm on both sides of the equation (1.19), we get log y = log a + bx. This
equation can be written as
Y = A + bx, where Y = log y, A = log a, i.e. a = eA .
The parameters A (and hence a) and b can be determined by the same process
discussed in Section 1.2.

1.4.3 Rectangular hyperbola

Let us consider another kind of curve called the rectangular hyperbola whose equation
is of the following form:
1
y= . (1.20)
a + bx
10
......................................................................................

This is a non-linear equation and it can be converted to linear form by substituting

Y = 1/y. Then the above equation is transformed to Y = a + bx.
By using same process discussed in Section 1.2, we can fit the rectangular hyperbola.

1.5 Weighted least squares method

This is an extension of the least squares method. Weighted least squares method has
two fold importance. If a particular sample point, say (xk , yk ) is repeated for wk (say)
times, then this method is useful. Here, wk is an integer. Again, if a particular point(s),
say (xk , yk ) is (are) very significant, then the amount of importance can be represented
by a weight say wk . In this case, wk need not be an integer. Some times it may required
that the curve must passes through some given points. For this case also the weighted
least squares method is useful. If all the sample points have same significance, then the
weights are taken as 1.
This method is similar to the least squares method, only the weights are incorporated
in this new method.

1.5.1 Fitting of weighted straight line

Let y = a + bx be the straight line to be fitted for the sample (xi , yi ), i = 1, 2, . . . , n.

Each sample point (xi , yi ) is associated with a weight wi , i = 1, 2, . . . , n, where wi is a
real number.
Let the predicted value of y at x = xi be Yi . Then the error for the ith sample point
is yi − Yi and the sum of square of residuals is

n
X h i2
S= wi yi − (a + bxi ) . (1.21)
i=1

Like previous cases, for the minimum S,

∂S ∂S
= 0, = 0.
∂a ∂b
11
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Least Squares Method

These give
n
X
−2 wi [yi − (a + bxi )] = 0
i=1
n
X
and −2 wi [yi − (a + bxi )]xi = 0
i=1

After simplification, the above equations reduce to

n
X n
X n
X n
X n
X n
X
a wi + b wi x i = wi yi and a wi x i + b wi x2i = wi x i yi .
i=1 i=1 i=1 i=1 i=1 i=1

These are the normal equations and solving these equations we obtain the values of
a and b. This method is illustrated in the following example.
The same method can be used to fit other type of curves in case of weighted data.
Example 1.4 Consider the following sample:

x : 0 3 6 9
y : 9 16 21 28

(a) Fit a straight line for this data.

(b) Fit a straight line by considering the weights of the sample points (3,16) and (6,21)
as 6 and 15 respectively.
(b) Again, fit the line by considering the modified weights 20 and 30 for the same sample
points.

Solution. Let the straight line be y = a + bx.

(a) As in previous case, the fitted straight line is y1 = 9.2000 + 2.0667x.
(b) In case of weighted least squares method, the normal equations to fit the straight
line are
X X X
a wi + b wi x i = wi yi
X X X
and a wi x i + b wi x2i = wi x i yi .

In this case, the weights are 1, 6, 15, 1.

The calculations are shown in the following table.
12
......................................................................................

x y w wx wx2 wy wxy
0 9 1 0 0 9 0
3 16 6 18 54 96 288
6 21 15 90 540 315 1890
9 28 1 9 81 28 252
Total 23 117 675 448 2430

Then the normal equations are 23a + 117b = 448 and 117a + 675b = 2430.
The solution is a = 9.8529, b = 1.8921.
Thus the fitted line is
y2 = 9.8529 + 1.8921x.
(c) In this case, the weights are 1, 20, 30, 1.
The calculations are shown in the following table.

x y w wx wx2 wy wxy
0 9 1 0 0 9 0
3 16 20 60 180 320 960
6 21 30 180 1080 630 3780
9 28 1 9 81 28 252
Total 52 249 1341 987 4992

Then the normal equations are 52a + 249b = 987 and 249a + 1341b = 4992 and the
solution is a = 10.4202, b = 1.7877.
Hence, the fitted line is
y3 = 10.4202 + 1.7877x.
Estimation of error.

x y y1 y2 y3 |y − y1 | |y − y2 | |y − y3 |
0 9 9.2000 9.8529 10.4202 0.2000 0.8529 1.4202
3 16 15.4001 15.5292 15.7833 0.5999 0.4708 0.2167
6 21 21.6002 21.2055 21.1464 0.6002 0.2055 0.1464
9 28 27.8003 26.8818 26.5095 0.1997 1.1182 1.4905
Sum of square of errors 1.5998 2.6474 3.2738

13
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Least Squares Method

Note that the sum of the square of errors is minimum for the line y1 , than the lines
for y2 and y3 . When the weights on second and third sample points are increased then
the absolute errors in y are reduced at these points, but, the sum of square of errors is
increased for the lines y2 and y3 .
The three lines are shown in Figure 1.2.

Figure 1.2: Comparison of three fitted lines

14