ACTL2002/ACTL5101 Probability and Statistics
c Katja Ignatieva
School of Risk and Actuarial Studies
Australian School of Business
University of New South Wales
k.ignatieva@unsw.edu.au
Week 11
Week 2
Week 3
Week 4
Probability:
Week 6
Review
Estimation: Week 5
Week
7
Week
8
Week 9
Hypothesis testing:
Week
10
Week
12
Linear regression:
Week 2 VL
Week 3 VL
Week 4 VL
Video lectures: Week 1 VL
Week 1
Week 5 VL
ACTL2002/ACTL5101 Probability and Statistics: Week 11
Last ten weeks
Introduction to probability;
Moments: (non)central moments, mean, variance (standard
deviation), skewness & kurtosis;
Special univariate (parametric) distributions (discrete &
continue);
Joint distributions;
Convergence; with applications LLN & CLT;
Estimators (MME, MLE, and Bayesian);
Evaluation of estimators;
Interval estimation.
3201/3252
ACTL2002/ACTL5101 Probability and Statistics: Week 11
Final two weeks
Simple linear regression:

Idea;
Estimating using LSE (& BLUE estimator & relation MLE);
Partition of variability of the variable;
Testing:
i) Slope;
ii) Intercept;
iii) Regression line;
iv) Correlation coefficient.
Multiple linear regression:
3202/3252
Matrix notation;
LSE estimates;
Tests;
Rsquared and adjusted Rsquared.
ACTL2002/ACTL5101 Probability and Statistics: Week 11 Matrix notation Linear Algebra and Matrix Approach Multiple Linear regression Matrix notation Linear Algebra and Matrix Approach The Model in Matrix Form Linear models Statistical Properties of the Least Squares Estimates Statistical Properties of the Least Squares Estimates CI and Tests for Individual Regression Parameters CI and Tests for functions of Regression Parameters Example: Multiple Linear Regression Example regression output Exercise: Multiple Linear Regression Example: Multiple Linear Regression Appendix Simple linear regression in matrix form .
. .p−1 3203/3252 . . . .p−1 y2 x21 x22 .ACTL2002/ACTL5101 Probability and Statistics: Week 11 Matrix notation Linear Algebra and Matrix Approach Linear Algebra and Matrix Approach In general we will consider multiple regression problem: y = β0 + β1 x 1 + β2 x 2 + . . .p−1 . . x2. . .. . . yn xn1 xn2 . + βp−1 x p−1 and data points: y1 x11 x12 . xn. . . . . .. x1. ...
. . . x2. . . . . 3204/3252 . Regression coefficients are the vector (p by 1) β = [β0 .. ..p−1 1 x21 x22 .. . . x1.p−1 X= . . . . xn. . . . .. . βp−1 ]> where > indicates transpose (β a column vector). The matrix X (size n by p) is: 1 x11 x12 .ACTL2002/ACTL5101 Probability and Statistics: Week 11 Matrix notation Linear Algebra and Matrix Approach Multiple Regression: Linear Algebra and Matrix Approach Observations yi are written in a vector y . β1 . 1 xn1 xn2 .p−1 Predicted values are: yb = Xβ. . .
X> Xβ If X> X is nonsingular then the parameter estimates are: −1 b = X> X β X> y . 3205/3252 . The residuals are: b = y − yb = y − Xβ. Proof: see next slides.ACTL2002/ACTL5101 Probability and Statistics: Week 11 Matrix notation Linear Algebra and Matrix Approach Multiple Regression: Linear Algebra and Matrix Approach Least squares problem is to select β to minimize: > S β = y − Xβ y − Xβ . Differentiate with respect to each of the β 0 s and the normal equations become: b = X> y .
. Derivation of least squares estimator: > > ∂ ∂ > > > > y − Xβ y − Xβ = 0= y y −2 X y β + β X Xβ ∂β ∂β > = − 2X> y + X> Xβ + X> X β = − 2X> y + 2X> Xβ −1 ⇒ X> y =X> Xβ ⇒ β = X> X X> y . 3206/3252 . . − βp−1 xip−1 )2 i=1 = y − Xβ > y − Xβ .ACTL2002/ACTL5101 Probability and Statistics: Week 11 Matrix notation Linear Algebra and Matrix Approach The least squares problem is to find the vector β that minimizes: S β = n X 2i = i=1 = n X n X (yi − ybi )2 i=1 (yi − β0 − β1 xi1 − .
If X> X exists. −1 i.e. The corresponding vector of fitted (or predicted) values of y is: b = Xβ b Y and the vector of residuals: b = Y − Xβ b b = Y − Y gives the differences between the observed and fitted values. β and equating equal to zero leads: X> Xβ = X> Y ..ACTL2002/ACTL5101 Probability and Statistics: Week 11 Matrix notation Linear Algebra and Matrix Approach The Least Squares Estimates Differentiating this matrix w. 3207/3252 .r. the normal equations. the solution is: −1 b = X> X β X> Y .t.
ACTL2002/ACTL5101 Probability and Statistics: Week 11 Matrix notation The Model in Matrix Form Multiple Linear regression Matrix notation Linear Algebra and Matrix Approach The Model in Matrix Form Linear models Statistical Properties of the Least Squares Estimates Statistical Properties of the Least Squares Estimates CI and Tests for Individual Regression Parameters CI and Tests for functions of Regression Parameters Example: Multiple Linear Regression Example regression output Exercise: Multiple Linear Regression Example: Multiple Linear Regression Appendix Simple linear regression in matrix form .
.. . . [n×1] [p×1] yn βp−1 3208/3252 for i = 1. Fitted to data. n. . n . + βp−1 xip−1 + i . + βp−1 xp−1 + . . . . the model becomes: yi = β0 + β1 xi1 + . .ACTL2002/ACTL5101 Probability and Statistics: Week 11 Matrix notation The Model in Matrix Form The Model in Matrix Form Consider the regression model of the form: y = β0 + β1 x1 + . . . . β = . . . . 2.. and = [n×1] 1 2 . Define the vectors: y1 β0 y2 β1 Y = . .
[n×p] . .. . x1.p−1 1 x21 .p−1 X = . xn. .ACTL2002/ACTL5101 Probability and Statistics: Week 11 Matrix notation The Model in Matrix Form The Model in Matrix Form Together with the matrix: 1 x11 . . . 1 xn1 . β [p×1] . . . . .. . Y = X n×1 [n×p] [p×1] [n×1] The fitted value is: b = X Y [n×1] 3209/3252 [n×p] b . .p−1 Write the model in matrix form as follows: β + . . .. x2. .
ACTL2002/ACTL5101 Probability and Statistics: Week 11 Matrix notation Linear models Multiple Linear regression Matrix notation Linear Algebra and Matrix Approach The Model in Matrix Form Linear models Statistical Properties of the Least Squares Estimates Statistical Properties of the Least Squares Estimates CI and Tests for Individual Regression Parameters CI and Tests for functions of Regression Parameters Example: Multiple Linear Regression Example regression output Exercise: Multiple Linear Regression Example: Multiple Linear Regression Appendix Simple linear regression in matrix form .
from Robert Engle.. Homoskedastic (constant) variance (otherwise use AutoRegressive Conditional Heteroscedasticity model (ARCH) model.e. 3210/3252 . symmetric (only in case of testing.ACTL2002/ACTL5101 Probability and Statistics: Week 11 Matrix notation Linear models Introduction To apply linear regression properly: Effects of the covariates (explanatory variables) must be additive. Errors must be independent of the explanatory variables with mean zero (weak assumptions). 2003 Nobel prize for Economics). and hence. strong assumptions). i. Errors must be Normally distributed.
(Yi X = x). 3211/3252 . where E[Yi X = x] ≡ µi . .Affine form with interaction terms: µi = β0 + xi β1 + zi β2 + (xi zi )β3 . yi . the values of which are unknown and need to be estimated using data.Affine form: µi = β0 + xi β1 . and the βi ’s are model parameters.ACTL2002/ACTL5101 Probability and Statistics: Week 11 Matrix notation Linear models Linear models in general A linear model involves a response variable datum. treated as an observation on a random variable. For all linear forms we have: Yi = µi + i .Polynomial (cubic) form: µi = β0 + xi β1 + xi2 β2 + xi3 β3 . . the i ’s are zero mean random variables independent of X . The following are examples of linear models: .
= . . the expected value vector µ is given by a model matrix (or design matrix)... 1 xn µn  {z } X So model has general form µ = Xβ. i. All linear models can be written in this general form.. 3212/3252 . . multiplied by a parameter vector. β.. X. . β1 .ACTL2002/ACTL5101 Probability and Statistics: Week 11 Matrix notation Linear models Linear models The first model µ1 µ2 µ3 .e. can be rewritten in matrixvector form as: 1 x1 1 x2 1 x3 β0 = [1n X ] β.
. 2 µn 1 xn xn xn3 {z }  X 3213/3252 be written in β0 β1 . .. . ... . . .ACTL2002/ACTL5101 Probability and Statistics: Week 11 Matrix notation Linear models Linear models The second model (the cubic) given above can matrixvector form as: µ1 1 x1 x12 x13 µ2 1 x 2 x 2 x 3 2 2 µ3 1 x 3 x 2 x 3 3 3 = .. .. . β2 β3 .
Then the model can be rewritten: y1 1 0 0 y2 1 0 0 y3 0 1 0 β0 y4 = 0 1 0 β1 + . β2 y5 0 0 1 y6  3214/3252 0 0 1 {z } X .ACTL2002/ACTL5101 Probability and Statistics: Week 11 Matrix notation Linear models Models in which data are divided into different groups. Consider the model: yi = βj + i if observation i is in group j. are less obviously of the form µ = Xβ. each with two data. but they can be written like this using dummy variables. and suppose there are three groups. each of which are assumed to have a different mean.
is therefore the expected increase (or decrease) in the value of y whenever you increase the value of xk by one unit. Assume that xk is a continuous variable so that if we increase it by one unit while holding the values of the other variables fixed. Since E [] = 0. . . the value of y becomes: y new = β0 + β1 x1 + . + βk (xk +1) + .ACTL2002/ACTL5101 Probability and Statistics: Week 11 Matrix notation Linear models Marginal effects Assume that we have the multiple regression model of the form: y = β0 + β1 x1 + . + βp−1 xp−1 + . then the marginal effect of xk is: βk = E [y new ] − E [y ] . . + βp−1 xp−1 + . . . . 3215/3252 .
ACTL2002/ACTL5101 Probability and Statistics: Week 11 Statistical Properties of the Least Squares Estimates Statistical Properties of the Least Squares Estimates Multiple Linear regression Matrix notation Linear Algebra and Matrix Approach The Model in Matrix Form Linear models Statistical Properties of the Least Squares Estimates Statistical Properties of the Least Squares Estimates CI and Tests for Individual Regression Parameters CI and Tests for functions of Regression Parameters Example: Multiple Linear Regression Example regression output Exercise: Multiple Linear Regression Example: Multiple Linear Regression Appendix Simple linear regression in matrix form .
. . the residuals have zero means. n. for i = 1. . . common variance. Cov (i . . for all i = 6 j. we have: E [] = 0.ACTL2002/ACTL5101 Probability and Statistics: Week 11 Statistical Properties of the Least Squares Estimates Statistical Properties of the Least Squares Estimates Assumptions The residuals terms εi satisfy the following: E [i X = x] = 0. Cov () = σ 2 In . Var (i X = x) = σ 2 . In words. 2. εj X = x) = 0. In matrix form. n. . are uncorrelated with explanatory variables and are independent of other residuals. 2. for i = 1. . 3216/3252 where In is a matrix of size n × n with ones on the diagonal and zeros on the offdiagonal elements. . .
n−p Note that: (n − p) · S 2 ∼ χ2 (n − p). σ2 3217/3252 and βb and S 2 are independent. 3. 1. The least squares estimates are unbiased: E β 2. . An unbiased estimate of σ 2 is: > 1 s2 = y − yb y − yb .ACTL2002/ACTL5101 Probability and Statistics: Week 11 Statistical Properties of the Least Squares Estimates Statistical Properties of the Least Squares Estimates Statistical Properties of the Least Squares Estimates The following properties of the least squares estimates can be verified: h i b = β. The variancecovariance matrix of the least squares estimates −1 2 > b =σ is: Var β b · X X .
Var βbk = σ where ckk is the (k + 1)th diagonal entry of the matrix −1 C = X> X (because c11 corresponds to the constant) and covariance between βbk and βbl : Cov βbk . Each component βbk is normally distributed with mean: h i E βbk = βk . 3218/3252 . and variance: b2 · ckk .ACTL2002/ACTL5101 Probability and Statistics: Week 11 Statistical Properties of the Least Squares Estimates Statistical Properties of the Least Squares Estimates Statistical Properties of the Least Squares Estimates 4. βbl = σ b2 · ckl .
ACTL2002/ACTL5101 Probability and Statistics: Week 11 Statistical Properties of the Least Squares Estimates CI and Tests for Individual Regression Parameters Multiple Linear regression Matrix notation Linear Algebra and Matrix Approach The Model in Matrix Form Linear models Statistical Properties of the Least Squares Estimates Statistical Properties of the Least Squares Estimates CI and Tests for Individual Regression Parameters CI and Tests for functions of Regression Parameters Example: Multiple Linear Regression Example regression output Exercise: Multiple Linear Regression Example: Multiple Linear Regression Appendix Simple linear regression in matrix form .
n−p · se βbk . we have: βbk − βk ∼ t (n − p) . se βbk A 100 (1 − α) % confidence interval for βk is given by: βbk ± t1−α/2. Under the normality (strong) assumption.ACTL2002/ACTL5101 Probability and Statistics: Week 11 Statistical Properties of the Least Squares Estimates CI and Tests for Individual Regression Parameters CI and Tests for Individual Regression Parameters The standard error of βbk is estimated using: √ se βbk = s ckk . 3219/3252 .
in which case the test statistic simply becomes: βbk T = . The common test is to test the significance of the presence of the variable xk . it has a tdistribution with n − p degrees of freedom. se βbk because we test H0 : βk = 0 against H1 : βk 6= 0 when we test for the significance/importance of the variable. 3220/3252 . we use the test statistic: T = βbk − βk0 se βbk which under the null hypothesis.ACTL2002/ACTL5101 Probability and Statistics: Week 11 Statistical Properties of the Least Squares Estimates CI and Tests for Individual Regression Parameters CI and Tests for Individual Regression Parameters In testing the null hypothesis H0 : βk = βk0 for some fixed constant βk0 .
Test the null hypothesis: H0 : βk = βk0 against the alternative: H1 : βk 6= βk0 . we can always have more general tests for the regression coefficients as demonstrated in the three cases below: 1. Use the decision rule (using generalized LRT. week 7): .ACTL2002/ACTL5101 Probability and Statistics: Week 11 Statistical Properties of the Least Squares Estimates CI and Tests for Individual Regression Parameters CI and Tests for Individual Regression Parameters However.
.
.
.
b .
βk − βk0 .
.
Reject H0 if: T  = .
.
.
> t1−α/2.n−p . .
se βbk .
3221/3252 .
se βbk 3. Test the hypothesis: H0 : βk = βk0 v.ACTL2002/ACTL5101 Probability and Statistics: Week 11 Statistical Properties of the Least Squares Estimates CI and Tests for Individual Regression Parameters CI and Tests for Individual Regression Parameters 2. Use the decision rule (using UMP.s. Use the decision rule (using UMP. H1 : βk < βk0 . H1 : βk > βk0 . se βbk . week 7): Reject H0 if: T = βbk − βk0 > t1−α.n−p .n−p .s. week 7): Reject H0 if: 3222/3252 T = βbk − βk0 < −t1−α. Test the hypothesis: H0 : βk = βk0 v.
ACTL2002/ACTL5101 Probability and Statistics: Week 11 Statistical Properties of the Least Squares Estimates CI and Tests for functions of Regression Parameters Multiple Linear regression Matrix notation Linear Algebra and Matrix Approach The Model in Matrix Form Linear models Statistical Properties of the Least Squares Estimates Statistical Properties of the Least Squares Estimates CI and Tests for Individual Regression Parameters CI and Tests for functions of Regression Parameters Example: Multiple Linear Regression Example regression output Exercise: Multiple Linear Regression Example: Multiple Linear Regression Appendix Simple linear regression in matrix form .
Then we have that: h i E Dβb =Dβ Var Dβb =DVar βb D> = σ 2 D(X> X)−1 D> Under the normality (strong) assumption. s 2 D(X> X)−1 D>  {z } =se (Dβb) A 100 (1 − α) % confidence interval for Dβ is given by: Dβb ± t1−α/2. 3223/3252 .n−p · se Dβb . we have: D(βb − β) p ∼ t (n − p) .ACTL2002/ACTL5101 Probability and Statistics: Week 11 Statistical Properties of the Least Squares Estimates CI and Tests for functions of Regression Parameters CI and Tests for functions of Regression Parameters Let D be a matrix (size m × p) of m linear combinations of the explanatory variables.
An alternative is to correct it for the number of regressor variables present. the Rsquared provides a descriptive measure of the success of the regressor variables in explaining the variation in the dependent variable. SST/ (n − 1) MST n−p . Thus. we define adjusted Rsquared: Ra2 = 1 − 3224/3252 SSE/ (n − p) s2 n−1 =1− =1− 1 − R2 . The Rsquared will always increase when adding additional regressor variables increase even if regressor variables added do not strongly influence the dependent variable. R2 = SST SST In the simple linear regression model.ACTL2002/ACTL5101 Probability and Statistics: Week 11 Statistical Properties of the Least Squares Estimates CI and Tests for functions of Regression Parameters Adjusted RSquared The coefficient of determination may is: SSE SST − SSE =1− .
3225/3252 Reject H0 if F > Fp−1. . H1 : at least one βk 6= 0. then Xβ2 should be large and 2 should be small. F = 2  /(n − p) SSE/(n − p) e β b 2 /σ 2 ∼ χ2 and Under the strong assumptions X p−1 2 2 2 e is the  /σ ∼ χn−p are chisquared distributed (note: X matrix X without the constant).n−p (1 − α). Hence. Interpretation: If the regression model explains a large proportion of the variability in y . . test H0 : β = 0 v.ACTL2002/ACTL5101 Probability and Statistics: Week 11 Statistical Properties of the Least Squares Estimates CI and Tests for functions of Regression Parameters Can we test wether the regression explains anything significant? E.n−p . .s.g. can we jointly test wether [β1 . βp−1 ]> = 0 (note: excluding β0 )? Use the Fstatistic: e β b 2 /(p − 1) SSM/(p − 1) X = ∼ Fp−1. . .
. . Xp−1 .SST is the total variability in the absence of knowledge of the variables X1 . . . Xp−1 . .ACTL2002/ACTL5101 Probability and Statistics: Week 11 Statistical Properties of the Least Squares Estimates CI and Tests for functions of Regression Parameters ANOVA table and sum of squares: . . Xp−1 . This partitioning of the variability is used in ANOVA tables: Sum of squares Degrees Mean F pvalue of freedom square n P 2 SSM MSM Regression SSM= (b yi − y ) DFM=p − 1 MSM= DFM MSE 1− Source i=1 Error Total 3226/3252 SSE= n P SST= i=1 n P i=1 FDFM.DFE (F ) (yi − ybi ) 2 DFE=n − p SSE MSE= DFE SST (yi − y )2 DFT=n − 1 MST= DFT . . . . . . . .SSE is the total variability remaining after introducing the effect of X1 . .SSM is the total variability “explained” because of knowledge of X1 .
ACTL2002/ACTL5101 Probability and Statistics: Week 11 Example: Multiple Linear Regression Example regression output Multiple Linear regression Matrix notation Linear Algebra and Matrix Approach The Model in Matrix Form Linear models Statistical Properties of the Least Squares Estimates Statistical Properties of the Least Squares Estimates CI and Tests for Individual Regression Parameters CI and Tests for functions of Regression Parameters Example: Multiple Linear Regression Example regression output Exercise: Multiple Linear Regression Example: Multiple Linear Regression Appendix Simple linear regression in matrix form .
DFE (F ) SSE (yi − ybi )2 DFE=n − p MSE= DFE SST (yi − y )2 DFT=n − 1 MST= DFT .ACTL2002/ACTL5101 Probability and Statistics: Week 11 Example: Multiple Linear Regression Example regression output Example regression output (=summary) Error variance and standard deviation s 2: s: MSE= √ s2 Pn 2 i=1 i n−p CI s 2 : CI s: SSE χ21−α/2 (n−p) SSE χ2α/2 (n−p) r r SSE χ21−α/2 (n−p) SSE χ2α/2 (n−p) ANOVA Source Sum of squares Regression SSM= n P i=1 Error Total 3227/3252 SSE= n P SST= i=1 n P i=1 Degrees of freedom Mean square SSM (b yi − y )2 DFM=p − 1 MSM= DFM F MSM MSE pvalue 1− FDFM.
ACTL2002/ACTL5101 Probability and Statistics: Week 11
Example: Multiple Linear Regression
Example regression output
Example regression output (cont.) (=summary)
R 2: 1 −
Ra2 : 1 −
Coefficients:
βb
>
X X
−1
>
X y
√
R2
R:
p
Ra2
Ra :
SSE
SST
SSE/(n−p)
SST/(n−1)
q se(β)
b kk
Cov (β)
b
t
βb
b
se(β)
pvalue
CI(β)
b
1 − tn−p (t) βb − t1−α/2 (n − p) · se(β)
b
βb + t1−α/2 (n − p) · se(β)
Covariance matrix:
b = s 2 · X> X −1
Cov(β)
3228/3252
ACTL2002/ACTL5101 Probability and Statistics: Week 11
Example: Multiple Linear Regression
Exercise: Multiple Linear Regression
Multiple Linear regression
Matrix notation
Linear Algebra and Matrix Approach
The Model in Matrix Form
Linear models
Statistical Properties of the Least Squares Estimates
Statistical Properties of the Least Squares Estimates
CI and Tests for Individual Regression Parameters
CI and Tests for functions of Regression Parameters
Example: Multiple Linear Regression
Example regression output
Exercise: Multiple Linear Regression
Example: Multiple Linear Regression
Appendix
Simple linear regression in matrix form
ACTL2002/ACTL5101 Probability and Statistics: Week 11
Example: Multiple Linear Regression
Exercise: Multiple Linear Regression
Exercise regression
Given is the following linear regression:
Yi = β0 + β1 · x1i + β2 · x2i + εi
For
with 20 observations we have
P20our sample
2
i=1 (yi − y ) = 53.82:
0.19 −0.08 −0.04
0.2
(X> X)−1 = −0.08 0.11 −0.03 βb = 0.93
−0.04 −0.03 0.05
0.95
20
X
ε2i = 11.67
i=1
a. Question: What is the estimate of variance of the residual?
b. Question: What is the 95% CI for β1 ?
c. Question: What is the 95% CI for β1 − β2 ?
3229/3252
d. Question: Are X1 and X2 jointly significant?
04 0 b =0.14 − 0.975 (17) · se(βb1 )) = (0.08] · 1 = 0.69 · [0 1 − 1] · −0.151.22 = 0.076 = 0.69 · [−0. (βb1 − t0.08 −0.11 = 0.35.975 (17) = 2.69 · 0.276.076 ⇒ se(βb1 ) = 0.04 0.110. b. Solution: s 2 = P20 2 i=1 εi /(n − p) = 11.69.04 −0.51) b = s 2 · D(X> X)−1 · D> is: c.67/17 = 0.08 0.975 (17) · se(βb1 ).03 · 1 Var (Dβ) −0. Solution:√Var (βb1 ) = s 2 · c11 = 0. thus 95% CI for β1 is. Var (Dβ) 0. βb1 + t0. 1. −1 3230/3252 . Solution: D = [0 1 − 1].69 · 0. F&T page 163: t0.11 −0.19 −0.ACTL2002/ACTL5101 Probability and Statistics: Week 11 Example: Multiple Linear Regression Exercise: Multiple Linear Regression Exercise regression a.05 −1 0 =0.03 0.
67/17=0.80) d. MSM=42.07/0.0. MSE=11.151 = 0. SSE=11.82.687=30.110. F0.07. 3231/3252 .01 (2.68. thus 95% CI for β1 − β2 is. b βb1 − βb2 + t0. 17) = 6.389. Solution (cont.112. Var (D β) F&T page 163: t0.67. thus X1 and X2 are jointly significant even for α = 0.): se(Dβ) q √ b = 0.01.975 (17) · se(D β)) b (βb1 − βb2 − t0.ACTL2002/ACTL5101 Probability and Statistics: Week 11 Example: Multiple Linear Regression Exercise: Multiple Linear Regression Exercise regression b = c.975 (17) = 2.975 (17) · se(D β).14/2=21. SSM=42. Solution: SST=53.687. F=21.84.14. = (−0.
ACTL2002/ACTL5101 Probability and Statistics: Week 11 Example: Multiple Linear Regression Example: Multiple Linear Regression Multiple Linear regression Matrix notation Linear Algebra and Matrix Approach The Model in Matrix Form Linear models Statistical Properties of the Least Squares Estimates Statistical Properties of the Least Squares Estimates CI and Tests for Individual Regression Parameters CI and Tests for functions of Regression Parameters Example: Multiple Linear Regression Example regression output Exercise: Multiple Linear Regression Example: Multiple Linear Regression Appendix Simple linear regression in matrix form .
4. YRSEXP = “the player’s experience in years”. PLAYED = “the number of games played in the previous year”. SALARY = “player’s salary”. DRAFT = “the round in which player was originally drafted”. 2. 3. 3232/3252 .ACTL2002/ACTL5101 Probability and Statistics: Week 11 Example: Multiple Linear Regression Example: Multiple Linear Regression Example: Multiple Linear Regression We use a dataset consisting of salaries of football players and some regressor variables that may influence their salaries: 1.
i.ACTL2002/ACTL5101 Probability and Statistics: Week 11 Example: Multiple Linear Regression Example: Multiple Linear Regression Example: Multiple Linear Regression Regressor variables (cont.): 5. 3233/3252 . 6. CITYPOP = “the population of the city in which the player is domiciled”. it is a dummy variable. 7. STARTED = “the number of games started in the previous year”. 0 = others). OFFBACK = “an indicator of player’s position in the game” (takes value 1 = offback defensive.e..
237 14 6.2367 0 0.ACTL2002/ACTL5101 Probability and Statistics: Week 11 Example: Multiple Linear Regression Example: Multiple Linear Regression Example: Multiple Linear Regression Summary Statistics of Variables in the Football Players Salary Data Count Mean Median Std Dev Minimum Maximum SALARY 169 336809 265000 255118 75000 1500000 DRAFT 169 6.97 1 6.859 0 16 CITYPOP 169 4980435 2421000 5098109 1176000 18120000 OFFBACK 169 0.4263 0 1 3234/3252 .077 4 3.352 0 17 PLAYED 169 10.999 0 16 STARTED 169 5.473 5 4.61 1 13 YRSEXP 169 4.
077 0.126 0.633 0.454 0.209 The Correlation Matrix YRSEXP PLAYED STARTED 0.178 0.129 0.557 0.253 −0.108 0.646 0.067 OFFBACK .440 0.179 0.ACTL2002/ACTL5101 Probability and Statistics: Week 11 Example: Multiple Linear Regression Example: Multiple Linear Regression Example: Multiple Linear Regression SALARY DRAFT YRSEXP PLAYED STARTED CITYPOP OFFBACK 3235/3252 SALARY DRAFT 0.193 0.345 0.212 0.050 0.059 0.043 0.081 CITYPOP 0.
ACTL2002/ACTL5101 Probability and Statistics: Week 11 Example: Multiple Linear Regression Example: Multiple Linear Regression Example: Multiple Linear Regression 3236/3252 .
ACTL2002/ACTL5101 Probability and Statistics: Week 11 Example: Multiple Linear Regression Example: Multiple Linear Regression Example: Multiple Linear Regression 3237/3252 .
ACTL2002/ACTL5101 Probability and Statistics: Week 11 Example: Multiple Linear Regression Example: Multiple Linear Regression Example: Multiple Linear Regression Source Regression Error Total 3238/3252 Degree of freedom p−1 n−p n−1 ANOVA Table Sum of Mean Squares Squares SSM MSM=SSM/p − 1 SSE MSE=SSE/n − p SST MST=SST/n − 1 FRatio Prob(> F) MSM/MSE pvalue .
we can derive several statistics that can be used to summarise the quality of the regression model.ACTL2002/ACTL5101 Probability and Statistics: Week 11 Example: Multiple Linear Regression Example: Multiple Linear Regression Example: Multiple Linear Regression From this ANOVA table.The coefficient of determination is defined by: R2 = SSM SST and has the interpretation that it gives the proportion of the total variability that is explained by the regression equation. 3239/3252 . For example: .
the Rsquared increases as the number of variables increases. In multiple regression. but not necessarily so for adjusted Rsquared.The adjusted coefficient of is defined by: Ra2 = 1 − SSE/ (n − p) s2 =1− 2 SST/ (n − 1) Sy and has the same interpretation as the Rsquared.ACTL2002/ACTL5101 Probability and Statistics: Week 11 Example: Multiple Linear Regression Example: Multiple Linear Regression Example: Multiple Linear Regression . It increases only if an influential variable is added. except that this is adjusted for the number of regressor variables. 3240/3252 .
The size of a typical error. denoted by s.ACTL2002/ACTL5101 Probability and Statistics: Week 11 Example: Multiple Linear Regression Example: Multiple Linear Regression Example: Multiple Linear Regression . 3241/3252 . s = s 2 = MSE = n−p It gives the average deviation of the actual y against that predicted by the regression equation. is the square root of s 2 and is also the square root of the error mean square: s √ √ SSE .
Its corresponding pvalue should be as small as possible. 3242/3252 . It provides another indication of how good the model is.The F ratio defined by: F ratio = MSM . MSE is the test statistic used for model adequacy.ACTL2002/ACTL5101 Probability and Statistics: Week 11 Example: Multiple Linear Regression Example: Multiple Linear Regression Example: Multiple Linear Regression .
0.032 S = 203817 Rsq = 38.2% 3243/3252 .34 2.17 5.003176 38241 T 8.21 3.19139 DRAFT + 21301 YRSEXP .5% Rsq(adj) = 36.ACTL2002/ACTL5101 Probability and Statistics: Week 11 Example: Multiple Linear Regression Example: Multiple Linear Regression Example: Multiple Linear Regression Summary of the results of the regression of the players’ salaries against the regressor variables: Regression Analysis The regression equation is SALARY = 361663 .017 0.17 p 0.42 4.000 0.000699 82941 SE Coef 43734 3674 6370 3281 3189 0.07 0.001 0.00070 CITYPOP + 82941 OFFBACK Predictor Constant DRAFT YRSEXP PLAYED STARTED CITYPOP OFFBACK Coef 361663 19139 21301 7948 12965 0.000 0.826 0.22 2.7948 PLAYED + 12965 STARTED .000 0.
000 .ACTL2002/ACTL5101 Probability and Statistics: Week 11 Example: Multiple Linear Regression Example: Multiple Linear Regression Example: Multiple Linear Regression ANOVA Table: Analysis of SOURCE Regression Error Total 3244/3252 Variance DF SS 6 4.87 p 0.00772E+11 41541379329 F 16.09343E+13 MS 7.72970E+12 168 1.20463E+12 162 6.
ACTL2002/ACTL5101 Probability and Statistics: Week 11 Example: Multiple Linear Regression Example: Multiple Linear Regression Example: Multiple Linear Regression 3245/3252 .
ACTL2002/ACTL5101 Probability and Statistics: Week 11 Example: Multiple Linear Regression Example: Multiple Linear Regression Example: Multiple Linear Regression 3246/3252 .
000000 CITYPOP + 0.8 + 0.009815 0.000 0.199 0.07332 0.6% Rsq(adj) = 52.00000001 0.52 S = 0.00981 PLAYED + 0.033 0.187 OFFBACK + 0.0.000 0.70 2.0264 STARTED + 0.9% 3247/3252 p 0.18741 0.ACTL2002/ACTL5101 Probability and Statistics: Week 11 Example: Multiple Linear Regression Example: Multiple Linear Regression Example: Multiple Linear Regression Improving the Regression Model Here we give you summary of the results of the improved regression model: Regression Analysis The regression equation is LOGSAL = 11.1242 T 144.000 .9334 SE Coef 0.08691 0.007607 0.001 0.42 4.4713 Rsq = 54.98 1.00000001 0.29 3.47 0.16 7.01471 0.007596 0.482 0.933 1/DRAFT Predictor Constant YRSEXP PLAYED STARTED CITYPOP OFFBACK 1/DRAFT Coef 11.7509 0.0733 YRSEXP .0814 0.026380 0.
3145 162 35.000 .ACTL2002/ACTL5101 Probability and Statistics: Week 11 Example: Multiple Linear Regression Example: Multiple Linear Regression Example: Multiple Linear Regression New ANOVA Table: Analysis of SOURCE Regression Error Total 3248/3252 Variance DF SS 6 43.2191 0.50 p 0.3035 MS 7.9891 168 79.2222 F 32.
ACTL2002/ACTL5101 Probability and Statistics: Week 11 Example: Multiple Linear Regression Example: Multiple Linear Regression Example: Multiple Linear Regression 3249/3252 .
ACTL2002/ACTL5101 Probability and Statistics: Week 11 Example: Multiple Linear Regression Example: Multiple Linear Regression Example: Multiple Linear Regression 3250/3252 .
ACTL2002/ACTL5101 Probability and Statistics: Week 11 Appendix Simple linear regression in matrix form Multiple Linear regression Matrix notation Linear Algebra and Matrix Approach The Model in Matrix Form Linear models Statistical Properties of the Least Squares Estimates Statistical Properties of the Least Squares Estimates CI and Tests for Individual Regression Parameters CI and Tests for functions of Regression Parameters Example: Multiple Linear Regression Example regression output Exercise: Multiple Linear Regression Example: Multiple Linear Regression Appendix Simple linear regression in matrix form .
. . ... . in matrix form we have: 1 x1 1 x2 X = .ACTL2002/ACTL5101 Probability and Statistics: Week 11 Appendix Simple linear regression in matrix form For simple linear regression y1 y2 y = . yn 1 xn Hence > X X= and −1 X> X = i=1 xi Pn Pni=1 x2i i=1 xi Pn 2 i=1 xi P Pn P n n · i=1 xi2 − ( ni=1 xi )2 − i=1 xi  {z } P 1 =n· 3251/3252 Pnn n 2 i=1 (xi −x) − Pn i=1 xi n . .. .
ACTL2002/ACTL5101 Probability and Statistics: Week 11 Appendix Simple linear regression in matrix form Thus: Pn xi i=1 P X y= . n i=1 xi yi > Hence " βb = βb0 βb1 # −1 = X> X X> y 1 = Pn n · i=1 (xi − x)2 3252/3252 Pn P P Pn xi2 ni=1 yi −P ni=1 xP i i=1P i=1 xi yi n n n n i=1 xi yi − i=1 xi i=1 yi .