Probabilistic Reasoning:

ECE 368 Probabilistic Reasoning

Week 7
Topics:

• Gauss and Least Squares Method

• Regression to the Mean
• Linear Regression: Orthogonal Projection
• Linear Regression: Curve-Fitting
• Bayesian Regression
• Regularization and Ridge Regression
Gauss, Planetary Motion, and Least Squares

• Soon after the dwarf planet Ceres was discovered in 1801, it was lost.
• The massive object within the asteroid belt between Mars and Jupiter
traveled behind the sun, before astronomers could calculate its orbit.
• Gauss took 41 daily observations by Giuseppe Piazi to calculate the
orbit.
• French astronomers rediscovered Ceres in January 1802, looking
where Gauss predicted it would be.
• Gauss was 24 years old.
• More later…
Regression to the Mean

• “Regression towards mediocrity in

hereditary stature”
• Adult children’s heights tend to
deviate less from the mean height
than their parents
• Why?
Linear Regression
Linear Regression: Orthogonal Projection
Linear Regression: Curve Fitting
Selecting Model Size
Bayesian Regression
Summary:
• Linear regression modeling
• ML model leads to ordinary least squares
• Problems with overfitting and model selection
• Relation to covariance matrix estimation
• MAP estimator using Bayesian linear regression
• Ridge regression deals with overfitting and model selection
• Uncertainty in model weights
• A posteriori predictor