Department of Applied Mechanics

APL 103

Experimental Methods
Semester I, 2020-21

L9

Murali R Cholemari
 Least squares
• Relationship between two sets of data?

Which line is correct?

Line a, line b or line c?
Subjective!
 Least squares
• Regression methods: find the relationship between two sets of data, when one
set is known to have much higher accuracy than the other
-(when both sets have the same accuracy –’correlation methods’
• Least squares is one such method
• Linear regression analysis – when the relationship between the two sets of data
is known to be linear

The Problem
1. We have two sets of data -- 𝑥1 , 𝑦1 , 𝑥2 , 𝑦2 ⋯ 𝑥𝑛 , 𝑦𝑛 each 𝑥𝑖 is associated with only
one 𝑦𝑖 .
2. If the relationship between 𝑥𝑖 , 𝑦𝑖 is known to be linear, what is the best straight line
that can be drawn through the data?
Visual fit – subjective. An objective criterion is required.
 Linear least squares
• Objective criterion
• Geometric criterion
• The method:
• Take 𝑥𝑖 to be taken much more precise than 𝑦𝑖 -- assumed to be known exactly.
• Assume a linear relation between 𝑥𝑖 , 𝑦𝑖 of the form 𝑦 = 𝑎𝑥 + 𝑏
• Here 𝑎 and 𝑏 are unknown constants to be determined by the data set 𝑥𝑖 , 𝑦𝑖
• Since 𝑥𝑖 are without error, if we know the values of 𝑎 and 𝑏 the ‘true values’ of 𝑦𝑖 are
𝑦𝑖∗ = 𝑎𝑥𝑖 + 𝑏
• However, 𝑦𝑖 are imprecise and the deviations from their true values are,
𝜖𝑖 = 𝑦𝑖 − 𝑦𝑖∗ = 𝑦𝑖 −𝑎𝑥𝑖 − 𝑏
• Least squares principle: ‘best straight line’ minimises the sum of squared errors 𝐸 :
𝐸 = 𝜖12 + 𝜖22 + 𝜖32 + ⋯ 𝜖𝑛2
= 𝑦1 − 𝑎𝑥1 − 𝑏 2 + 𝑦2 − 𝑎𝑥2 − 𝑏 2 ⋯ + 𝑦𝑛 − 𝑎𝑥𝑛 − 𝑏 2 (1)
The parameters are 𝑎 and 𝑏 for the straight line. We need to minimise 𝐸 wrt 𝑎 and 𝑏 :
• For the best straight line:
𝜕𝐸 𝜕𝐸
= 0 and =0
𝜕𝑎 𝜕𝑏
The equations give
2
𝑋𝑌 𝑛 − 𝑎 𝑋 𝑛
− 𝑏𝑋𝑛 = 0 (2)
𝑌𝑛 − 𝑎𝑋𝑛 − 𝑏 = 0 (3)
Where,
𝑋𝑛 = 𝑥1 + 𝑥2 + ⋯ + 𝑥𝑛 Τ𝑛
𝑌𝑛 = 𝑦1 + 𝑦2 + ⋯ + 𝑦𝑛 Τ𝑛
𝑋 2 𝑛 = 𝑥12 + 𝑥22 + ⋯ + 𝑥𝑛2 Τ𝑛
𝑋𝑌 𝑛 = 𝑥1 𝑦1 + 𝑥2 𝑦2 + ⋯ + 𝑥𝑛 𝑦𝑛

We solve (2) and (3) for 𝑎 and 𝑏 :

 2 2
𝑎𝑛 = 𝑋𝑌 𝑛 − 𝑋 𝑌
𝑛 𝑛 ൗ 𝑋 𝑛
− 𝑋𝑛 (4)

𝑏𝑛 = 𝑋2 𝑛
𝑌𝑛 − 𝑋𝑛 𝑋𝑌 𝑛 ∕ 𝑋2 𝑛
− 𝑋𝑛
2
(5)

These are the best estimates of 𝑎 and 𝑏 for 𝑛 measurements

What is the precision of the values 𝑎𝑛 and 𝑏𝑛 ?

Note: errors in 𝑎𝑛 and 𝑏𝑛 arise because of errors in 𝑦𝑖 . No errors in 𝑋𝑛 , 𝑋 2 𝑛
etc. as 𝑥𝑖 are assumed without errors.
From (4),
𝑎𝑛 = 𝑥1 𝑦1 − 𝑥𝑛 𝑦1 + 𝑥2 𝑦2 − 𝑥𝑛 𝑦2 … /𝑛 𝑋2 𝑛
− 𝑋𝑛
2

= [ 𝑥1 − 𝑋𝑛 𝑦1 + 𝑥2 − 𝑋𝑛 𝑦2 … + 𝑥𝑛 − 𝑋𝑛 𝑦𝑛 ]Τ𝑛 𝑋2 𝑛
− 𝑋𝑛
2
• Hence, the std. deviation of 𝑎𝑛 can be related to the std. deviations
1/2
𝑥1 −𝑋𝑛 2 𝜎 2 𝑦1 + 𝑥2 −𝑋𝑛 2 𝜎 2 𝑦2 +⋯+ 𝑥𝑛 −𝑋𝑛 2 𝜎 2 𝑦𝑛
of 𝑦𝑖 : 𝜎 𝑎𝑛 =
𝑛 𝑋 2 𝑛 −𝑋𝑛2
• We need 𝜎 2 𝑦1 , 𝜎 2 𝑦2 … 𝜎 2 𝑦𝑛

The underlying assumption, all errors are equally important (hence unit weights
in 𝐸 = 𝜖12 + 𝜖22 + 𝜖32 + ⋯ 𝜖𝑛2 )
Also the measurements are done only once. This implies

𝜎 𝑦1 = 𝜎 𝑦2 =… 𝜎 𝑦𝑛 = 𝜎 𝑦 say. Then,
𝑥1 − 𝑋𝑛 2 + 𝑥2 − 𝑋𝑛 2 + ⋯ + 𝑥𝑛 − 𝑋𝑛 2 1/2 𝜎 𝑦
𝜎 𝑎𝑛 =
𝑛 𝑋 2 𝑛 − 𝑋𝑛2
2 2 2 2 1/2
𝑥1 + 𝑥2 + ⋯ + 𝑥𝑛 − 2 𝑥1 + 𝑥2 + ⋯ 𝑥𝑛 𝑋𝑛 + 𝑛𝑋𝑛 𝜎 𝑦
=
𝑛 𝑋 2 𝑛 − 𝑋𝑛2
 1/2
𝑋 2 𝑛−𝑋𝑛2 𝜎 𝑦 𝜎 𝑦
= = 1/2
𝑋 2 𝑛−𝑋𝑛2 𝑛1/2 𝑋 2 𝑛−𝑋𝑛2 𝑛1/2

Similarly, from (5)

[ 𝑋2 𝑛
𝑦1 + 𝑦2 − ⋯ + 𝑦𝑛 /𝑛 −𝑋𝑛 ൫𝑥1 𝑦1 + 𝑥2 𝑦2 … + 𝑥𝑛 𝑦𝑛 )Τ𝑛]
𝑏𝑛 =
𝑋2 𝑛 − 𝑋𝑛2

[ 𝑋2 𝑛
− 𝑋𝑛 𝑥1 𝑦1 + 𝑋2 𝑛
− 𝑋𝑛 𝑥2 𝑦2 + ⋯ ]
𝑏𝑛 =
𝑛 𝑋2 𝑛 − 𝑋𝑛2
Hence
2 2
[ 𝑋2 𝑛
− 𝑋𝑛 𝑥1 + 𝑋2 𝑛
− 𝑋𝑛 𝑥2 + ⋯ ]1/2 𝜎 𝑦
𝜎 𝑏𝑛 =
𝑛 𝑋2 𝑛 − 𝑋𝑛2

1/2
𝑋2 𝑛
𝜎 𝑦
𝜎 𝑏𝑛 = 1/2
2
𝑛1/2 𝑋 2 𝑛 − 𝑋𝑛
Note that we still do not know 𝜎 𝑦
To determine 𝜎 𝑦 , once 𝑎𝑛 and 𝑏𝑛 are calculated, deviations are calculated:
𝜖𝑖 = 𝛿𝑖 = 𝑦𝑖 − 𝑎𝑛 𝑥𝑖 − 𝑏𝑛 = 𝑦𝑖 − 𝑌𝑛 − 𝑎𝑛 𝑥𝑖 + 𝑎𝑛 𝑋𝑛 (the last using (3))
= 𝑦𝑖 − 𝑌𝑛 − 𝑎𝑛 (𝑥𝑖 −𝑋𝑛 )

Then mean squared deviations are determined:

෍ 𝛿𝑖2 = ෍ 𝑦𝑖 − 𝑌𝑛 2
− 2𝑎𝑛 𝑦𝑖 − 𝑌𝑛 𝑥𝑖 − 𝑋𝑛 +𝑎𝑛2 𝑥𝑖 − 𝑋𝑛 2

𝑖=1 𝑖
= 𝑛 𝑌2 𝑛
− 𝑛𝑌 2
𝑛 − 2𝑎𝑛 𝑛 𝑋𝑌 𝑛 − 𝑋 𝑌
𝑛 𝑛 + 𝑛𝑎 2
𝑛 𝑋𝑛
2
− 𝑋𝑛2

(note σ𝑖 𝑦𝑖 − 𝑌𝑛 = 0 etc.) using (4) for 𝑎𝑛 gives,

[ 𝑋𝑌 𝑛 −𝑋𝑛 𝑌𝑛 ]2
𝜎𝑛2 𝑦 = ෌ 𝛿𝑖2 Τ𝑛 = 𝑌 2
− 𝑌𝑛2 − (6)
𝑛 𝑋 2 𝑛 −𝑋𝑛2
• Adjusted std. deviation
1
2൘ 𝑛
𝑠𝑛 𝑦 = 𝛿12 + 𝛿22 + ⋯ 𝛿𝑛2 = 𝜎𝑛 𝑦
𝑛−2 𝑛−2
(similar to the case when 𝜎𝑛 was meaningless when 𝑛 was 1, here even 𝑛=2 is meaningless
since a straight line can pass exactly through any two points hence 𝑛 − 2 is required)
Since we make only one determination of 𝑎𝑛 and 𝑏𝑛 their adjusted std. deviations and std.
errors are same:
𝜎𝑛 𝑦
𝑠𝑛 𝑎 = 1 ൗ2
2
𝑛−2 𝑋2 𝑛 − 𝑋𝑛

2 1Τ2
𝜎𝑛 𝑦 𝑋 𝑛
𝑠𝑛 𝑏 = 1ൗ
2
2
𝑛−2 𝑋2 𝑛 − 𝑋𝑛
• Alternate expressions: