Example Simple Linear Regression Model: C œ "  "" B

3 B3 C3

" $' '

# )# "%
$ %& &
% %* "$
& #" &
' &) "%
( ($ ""
) )& ")
* &# '
"! #% )

Ô" $' × Ô ' ×

Ö" )# Ù Ö "% Ù
Ö Ù Ö Ù
Ö" %& Ù Ö & Ù
Ö Ù Ö Ù
Ö" %* Ù Ö "$ Ù
Ö Ù Ö Ù
Ö" #" Ù Ö & Ù
Ö Ù" œ Ö Ù
Ö" &) Ù Ö "% Ù
Ö Ù Ö Ù
Ö" ($ Ù Ö "" Ù
Ö Ù Ö Ù
Ö" )& Ù Ö ") Ù
Ö Ù Ö Ù
Õ" Ø Õ ) Ø
" &# '
#%
Å Å
\ (model matrix) C (observation vector)

The normal equations \ X \ " œ \ X C are

” &#& • "œ”
&*)! •
"! &#& "!!
$#!)&

\ X \ is invertible, so there's a unique least squares solution:

s œ ” "! • ” &*)! •
"
&#& "!!
"œ"
&#& $#!)&

%&##& ” •” • ¸” • œ– !—
$#!)&  &#& "!! "Þ&#&( s
"
"
œ
 &#& "! &*)! !Þ"'"% s
" "
(rounded to 4 decimal places)
The regression line is

s s "
Cœ" s B ¸ "Þ&#&(  !Þ"'"%B
! "

The data points and the regression line are pictured below.
Example Simple Linear Regression Model

3 B3 C3 best fitting line C œ "!  "" B ? (“predict C from B”)

" " "

# $ $
$ % #

Ô" "× Ô"×

Õ" %Ø Õ#Ø
" $ "œ $

Å Å
\ (model matrix) C (observation vector)

Normal equations: \ X \ " œ \ X C are

”) • "œ” •
$ ) '
#' ")

\ X \ is invertible (so unique least squares solution)

s œ ”$ • ” • œ – ($ — œ – ! —
" ' s
) ' "
"œ"
) #' ") (
s
""

The regression line for C on B Гfor predicting

C from B”) : s Cœ" s " s B œ '  $B
! " ( (
Alternate model for the same data:

3 B3 C3 best fitting line B œ α!  α" C ? (“predict B from C ”)

" " "

# $ $
$ % #

Ô" "× Ô"×

Õ" #Ø Õ%Ø
" $ αœ $

Å Å
] (model matrix) B (observation vector)

Normal Equation ] X] α œ ] XB

”' • αœ” •
$ ' 8
"% ")

αœ” • ” • ” • œ” !•
" #
$ ' ) $
s
α
αœs œ
' "% ") " s"
α

The regression line for B on C

Гfor predicting B from C ”)
#
s
Bœ s!  α
α s"C œ $  "C

This is a different line. It it the line that minimizes

 ÐB3  ÐÐ α
8
s " C3 ÑÑ#
s!  α
3œ"

œ “sum of squares of horizontal distances from data points to line"

For the first model, we found the line that minimizes  ÐC3  Ð "
8
s "s B3 ÑÑ#
! "
3œ"

œ “sums of squares of vertical distances from data points to line"

Here are the two regression lines.
 3 B3 C3

" #Þ"  !Þ)

# $Þ# "Þ'
$ %Þ& (Þ&
% 'Þ# #"Þ"
& (Þ" $!Þ"

Find best fit by: C œ "!  "" B  "# B#

”Exact fit” would require

"!  "" B"  "# B#" œ C"

"!  "" B#  "# B## œ C# or
ã
#
"!  "" B&  "# B& œ C&

Ô" B#" × Ô C" ×

# Ô "! ×
Ö" B# Ù ÖC Ù
B"
Ö Ù " œÖ #Ù
B#
ã Õ "Ø
Õ" # Ø "# Õ C& Ø
ã ã ã
B& B&
Å Å Å
\ " œ C

s so that ll C  \ "
Inconsistent. Find the " œ " s ll is as small as possible.
 \ X \ " œ \ X C is (approximately, after
rounding)

Ô & #$Þ" "#$Þ) × Ô &*Þ& ×

Õ "#$Þ) %&&Þ$# Ø Õ #%*$Þ# Ø
#$Þ" "#$Þ) (#*Þ% " œ $)"Þ(
(#*Þ%

so that (Using Matlab)

Ô "#$Þ) × Ô &*Þ& ×
"
& #$Þ"

Õ "#$Þ) Ø Õ #%*$Þ# Ø
s œ #$Þ"
"œ" "#$Þ) (#*Þ% $)"Þ(
(#*Þ% %&&Þ$#

"Þ%(&* × Ô "
Ô
s
!×
¸  $Þ#&*" œ Ö "
s ÙÞ
Õ "Þ!#*& Ø Õ s Ø
"
" #

The “best fitting quadratic” is

s s "
Cœ" s B"
s B#
! " #

¸ "Þ%(&*  $Þ#&*"B  "Þ!#*&B#

This is the “best fit” in the sense that

it minimizes the sum of the squared (vertical) residuals.