You are on page 1of 90

ACTL2002/ACTL5101 Probability and Statistics: Week 10

ACTL2002/ACTL5101 Probability and Statistics
c Katja Ignatieva

School of Risk and Actuarial Studies
Australian School of Business
University of New South Wales
k.ignatieva@unsw.edu.au

Week 10
Week 2
Week 3
Week 4
Probability:
Week 6
Review
Estimation: Week 5
Week
7
Week
8
Week 9
Hypothesis testing:
Week
11
Week
12
Linear regression:
Week 2 VL
Week 3 VL
Week 4 VL
Video lectures: Week 1 VL
Week 1

Week 5 VL

ACTL2002/ACTL5101 Probability and Statistics: Week 10

3101/3173

ACTL2002/ACTL5101 Probability and Statistics: Week 10

Last nine weeks
Introduction to probability;
Moments: (non)-central moments, mean, variance (standard
deviation), skewness & kurtosis;
Special univariate (parametric) distributions (discrete &
continue);
Joint distributions;
Convergence; with applications LLN & CLT;
Estimators (MME, MLE, and Bayesian);
Evaluation of estimators;
Interval estimation & Hypothesis tests.
3102/3173

Testing: i) Slope. . ii) Intercept.Estimating using LSE (& BLUE estimator & relation MLE). .Idea.ACTL2002/ACTL5101 Probability and Statistics: Week 10 This week Simple linear regression: . . iii) Regression line.Partition of variability of the variable. iv) Correlation coefficient. 3103/3173 .

ACTL2002/ACTL5101 Probability and Statistics: Week 10 Simple linear regression Basic idea Simple Linear Regression Simple linear regression Basic idea Correlation Coefficient Assumptions Relation MLE and LSE Partitioning the variability Exercise Testing in simple linear regression Overview Inference on the slope Inference on the intercept Confidence Intervals for the Population Regression Line Prediction Intervals for the Actual Value of the Dependent Variable Hypothesis Test for Population Correlation Exercise Appendix Algebra: parameter estimates Properties of parameter estimates Proof: Partitioning the variability .

. . to the data: (xi . . x is called exogenous variable (or predictor/independent variable). 2. yi ) for i = 1.ACTL2002/ACTL5101 Probability and Statistics: Week 10 Simple linear regression Basic idea Basic idea Suppose observe data y = [y1 . . xn ]> . What can we say about the relationship between X and Y ? To do so we fit: y = β0 + β1 x + . y is called the endogenous variable (or response/dependent variable). . . . n. yn ]> . . . Assume that Y is affected by X with x = [x1 . 3104/3173 Question: How to determine β0 and β1 ? . . . .

ACTL2002/ACTL5101 Probability and Statistics: Week 10 Simple linear regression Basic idea Basic idea Regression. We determine β0 and β1 by minimizing S (β0 . .β1 β0 .β1 i=1 β0 . β1 ) = Pn 2 i=1 i . with E[i ] = 0: yi = β0 + β1 xi + i . β1 ) = −2 · ∂β1 3105/3173 n X i=1 xi (yi − (β0 + β1 xi )) . β1 )} = min 2i = min (yi − (β0 + β1 xi )) β0 .β1 i=1 The minimum is obtained by setting FOC equal to zero: n X ∂S (β0 . we use least squares estimates (LSE) for β0 and β1 : ( n ) ( n ) X X 2 min {S (β0 . Hence. β1 ) yi − (β0 + β1 xi ) = −2 · ∂β0 i=1 ∂S (β0 .

β1 = P . i=1 xi . See F&T page 24. i=1 Pn i=1 yi .ACTL2002/ACTL5101 Probability and Statistics: Week 10 Simple linear regression Basic idea The LSE βb0 and βb1 are given by setting the FOC equal to zero: n n X X b b yi =nβ0 + β1 xi i=1 n X i=1 xi yi =βb0 i=1 n X xi + βb1 i=1 Next step: βb0 P and βb1 as functions of P n n 2 i=1 xi yi . Pn i=1 xi . n 2 2 i=1 xi − n · x ! Pn P n X bi )2 ( ni=1 xi · yi − n · x · y )2 1 2 2 2 i=1 (yi − y Pn σ b = = · yi − n · y − 2 2 n−2 n−2 i=1 xi − n · x i=1 3106/3173 . and n X xi2 . slides 3161-3165 (MLE on slide 3123): Pn i=1 xi · yi − n · xy b b b β0 =y − β1 · x.

ACTL2002/ACTL5101 Probability and Statistics: Week 10 Simple linear regression Correlation Coefficient Simple Linear Regression Simple linear regression Basic idea Correlation Coefficient Assumptions Relation MLE and LSE Partitioning the variability Exercise Testing in simple linear regression Overview Inference on the slope Inference on the intercept Confidence Intervals for the Population Regression Line Prediction Intervals for the Actual Value of the Dependent Variable Hypothesis Test for Population Correlation Exercise Appendix Algebra: parameter estimates Properties of parameter estimates Proof: Partitioning the variability .

e. ρXY measures the strength of the dependence. −1 ≤ ρXY ≤ 1.e.. 3107/3173 . Recall from week 3: the correlation coefficient between a pair of random variables X and Y .ACTL2002/ACTL5101 Probability and Statistics: Week 10 Simple linear regression Correlation Coefficient Correlation Coefficient Regression: find the dependency of Y and X . σX σY E [(X − µX )2 ] · E [(Y − µY )2 ] The value of the correlation coefficient lays between -1 and 1. β1 give the marginal effect of a change in X . i. i. or simply ρ is: ρXY = E [(X − µX ) · (Y − µY )] Cov (X . they have a joint distribution. denoted by ρXY .. Y ) =p .

.A correlation of 0 implies no linear relationship. the stronger the (positive/negative) linear relationship.A correlation of −1 indicates a perfect negative linear relationship.A correlation of 1 indicates a perfect positive linear relationship. . .The larger the correlation in absolute value. 3108/3173 .ACTL2002/ACTL5101 Probability and Statistics: Week 10 Simple linear regression Correlation Coefficient Correlation Coefficient We know that the correlation coefficient has the following interpretations: .

ACTL2002/ACTL5101 Probability and Statistics: Week 10 Simple linear regression Correlation Coefficient Correlation Coefficient Correlations are indications of linear relationships . (xn . yn ) Estimate the population correlation ρXY using (week 3): v v u u n n u 1 X u 1 X 2 t t (xi − x) and sy = (yi − y )2 sx = n−1 n−1 i=1 3109/3173 i=1 to estimate the population standard deviations σX and σY . y2 ) . . (x2 . but are strongly dependent (non-linear). . The correlation coefficient is a population parameter that can be estimated from data. . . Suppose we have n pairs of observations denoted by: (x1 .it is possible that two variables have zero correlation. y1 ) . . respectively.

Y 1 X (xi − x) (yi − y ) . the sample covariance is given by: n sX . n 2 − nx 2 · 2 − ny 2 y x i=1 i i=1 i .ACTL2002/ACTL5101 Probability and Statistics: Week 10 Simple linear regression Correlation Coefficient Correlation Coefficient Similarly. = n−1 i=1 Thus the sample correlation coefficient is: r = = = 3110/3173 1 Pn (xi − x) (yi − y ) n − 1 i=1 sx sy Pn (x − x) (yi − y ) qP i=1 i n 2 Pn 2 i=1 (xi − x) · i=1 (yi − y ) Pn i=1 xi yi − n (x · y ) q P  Pn .

5 y 0 −0.8 ρ=−0.5 −2 −2 3111/3173 −1.ACTL2002/ACTL5101 Probability and Statistics: Week 10 Simple linear regression Correlation Coefficient Effect correlation ρ=0 ρ=0.3 2 1.5 1 1.5 −1 −0.5 0 x 0.5 −1 −1.5 1 0.5 2 .

ACTL2002/ACTL5101 Probability and Statistics: Week 10 Simple linear regression Correlation Coefficient Effect variance σx=4. σy=4 6 4 y 2 0 −2 −4 −6 −8 3112/3173 −6 −4 −2 0 x 2 4 6 8 . σy=1 σx=1. σy=1 σx=1.

ACTL2002/ACTL5101 Probability and Statistics: Week 10 Simple linear regression Correlation Coefficient Effect mean µx=3. µy=0 µx=0. µy=0 µx=0. µy=3 4 3 2 y 1 0 −1 −2 −3 −4 −5 3113/3173 0 x 5 .

ACTL2002/ACTL5101 Probability and Statistics: Week 10 Simple linear regression Assumptions Simple Linear Regression Simple linear regression Basic idea Correlation Coefficient Assumptions Relation MLE and LSE Partitioning the variability Exercise Testing in simple linear regression Overview Inference on the slope Inference on the intercept Confidence Intervals for the Population Regression Line Prediction Intervals for the Actual Value of the Dependent Variable Hypothesis Test for Population Correlation Exercise Appendix Algebra: parameter estimates Properties of parameter estimates Proof: Partitioning the variability .

i..ACTL2002/ACTL5101 Probability and Statistics: Week 10 Simple linear regression Assumptions In order to preform linear regression we require: . . . det(X > X ) 6= 0. σ 2 In ). Weak assumptions imply:     E y |X = x =E X β + |X = x = X β   Var y |X = x =Var X β + |X = x = σ 2 In . not required for LSE-estimates): L {|X = x} =Nn (0.Weak assumptions: E [|X = x] =0 Var (|X = x) =σ 2 In . .e. .Non-collinearity: x is linear independent of 1n or rank(X )=2 or X is not singular. 3114/3173 ⇒ conditional covariance matrix y |X = x is independent of X .Strong assumption (for tests/CI.Sometimes additional assumption X and  are independent (not required for LSE-estimates). where In is a matrix of size n × n with 1 on the diagonal.

ACTL2002/ACTL5101 Probability and Statistics: Week 10 Simple linear regression Assumptions Effect non-linear function exponential linear quadratic 5 4 3 y 2 1 0 −1 −2 −3 −3 3115/3173 −2 −1 0 1 x 2 3 4 5 .

ACTL2002/ACTL5101 Probability and Statistics: Week 10 Simple linear regression Assumptions Effect changing variance 8 6 y 4 2 0 −2 −4 −1.5 0 0.5 x 1 1.5 2 .5 3116/3173 −1 −0.

2 3117/3173 −1.8 y 0.5 0 0.4 0.5 2 .6 0.2 0 −0.5 x 1 1.5 −1 −0.ACTL2002/ACTL5101 Probability and Statistics: Week 10 Simple linear regression Assumptions Effect binary choice variable 1 0.

ACTL2002/ACTL5101 Probability and Statistics: Week 10 Simple linear regression Assumptions Unbiased regression parameters Statistical Properties of Least Squares Estimators. for i = 1. Recall the statistical model: yi = β0 + β1 xi + i . n. . . Under the weak assumptions we have unbiased estimates (see slide 3166): h i h i E βb0 =β0 and E βb1 = β1 . s2 = n−2 n−2 Proof: we use that βb0 and βb1 are unbiased and E[i ] = 0: h i h i ybi = βb0 + βb1 xi + i ⇒ E [b yi ] = E βb0 + E βb1 · xi . . 3118/3173 An (unbiased) estimate of σ 2 is given by:  2 Pn  Pn 2 b0 + βb1 xi y − β i i=1 i=1 i = . . .

σy=1 σx=1.ACTL2002/ACTL5101 Probability and Statistics: Week 10 Simple linear regression Assumptions Interpretation uncertainty slope σx=4. σy=4 10 y 5 0 −5 −10 −10 3119/3173 −5 0 x 5 10 . σy=1 σx=4.

σy=1 σx=1.ACTL2002/ACTL5101 Probability and Statistics: Week 10 Simple linear regression Assumptions Interpretation uncertainty slope σx=4. σy=1 σx=4. σy=4 6 4 ε 2 0 −2 −4 −6 −10 3120/3173 −5 0 x 5 10 .

µy=0 3 2 y 1 0 −1 −2 −3 −5 3121/3173 0 x 5 . µy=0 µx=−3. µy=−1 µx=1.ACTL2002/ACTL5101 Probability and Statistics: Week 10 Simple linear regression Assumptions Interpretation uncertainty intercept µx=3.

ACTL2002/ACTL5101 Probability and Statistics: Week 10 Simple linear regression Relation MLE and LSE Simple Linear Regression Simple linear regression Basic idea Correlation Coefficient Assumptions Relation MLE and LSE Partitioning the variability Exercise Testing in simple linear regression Overview Inference on the slope Inference on the intercept Confidence Intervals for the Population Regression Line Prediction Intervals for the Actual Value of the Dependent Variable Hypothesis Test for Population Correlation Exercise Appendix Algebra: parameter estimates Properties of parameter estimates Proof: Partitioning the variability .

Y2 . β0 .is the product of their marginals (independent by assumption) so that the likelihood is: ! n  Y 1 (yi − (β0 + β1 xi ))2 √ exp − L y . and σ 2 . . σ = 2σ 2 2πσ i=1 ! n 1 X 2 = exp − 2 (yi − (β0 + β1 xi )) 2σ (2π)n/2 σ n i=1 n √   1 X ` y . σ = − n log 2πσ − 2 (yi − (β0 + β1 xi ))2 . β1 . . . 2σ 1 i=1 3122/3173 . Joint density of Y1 .ACTL2002/ACTL5101 Probability and Statistics: Week 10 Simple linear regression Relation MLE and LSE Maximum Likelihood Estimates In the regression model there are three parameters to estimate: β0 . Yn -under the (strong) normality assumptions. β1 . β1 . β0 . .

n n i=1 Note: the parameters βb0 and βb1 are the same as in case of LS (see slide 3106). Pn (xi − x) (yi − y ) b β1 = i=1 . Pn 2 i=1 (xi − x) and n σ b2 =  2 n−2 1 X yi − βb0 + βb1 xi = s2 · .ACTL2002/ACTL5101 Probability and Statistics: Week 10 Simple linear regression Relation MLE and LSE Relation MLE and LSE Partial derivatives set to zero give the following MLEs: βb0 =y − βb1 x. which is the unbiased variant of the MLE. However. the MLE σ b2 is a biased estimator of σ 2 . . 3123/3173 Thus we use LSE.

One can show that the LS-estimator βb0 + X βb1 is BLUE for µ = β0 + X β1 under the weak assumptions (prove not required in this course). minimum variance. One can show that the LS-estimator βb0 + X βb1 is UMVUE for µ = β0 + X β1 under the strong assumptions (prove not required in this course). for any linear unbiased estimator τ ? .ACTL2002/ACTL5101 Probability and Statistics: Week 10 Simple linear regression Relation MLE and LSE BLUE estimator A point estimator τb(θ) is called linear if τb(θ) = Ax + b. A point estimator τb(θ) is called Best Linear Unbiased Estimator (BLUE): unbiased. Eθ [b τ (θ)] = τ (θ). Var (b τ (θ)) ≤ Var (τ ? (θ)) . 3124/3173 .

ACTL2002/ACTL5101 Probability and Statistics: Week 10 Simple linear regression Partitioning the variability Simple Linear Regression Simple linear regression Basic idea Correlation Coefficient Assumptions Relation MLE and LSE Partitioning the variability Exercise Testing in simple linear regression Overview Inference on the slope Inference on the intercept Confidence Intervals for the Population Regression Line Prediction Intervals for the Actual Value of the Dependent Variable Hypothesis Test for Population Correlation Exercise Appendix Algebra: parameter estimates Properties of parameter estimates Proof: Partitioning the variability .

ACTL2002/ACTL5101 Probability and Statistics: Week 10 Simple linear regression Partitioning the variability Partitioning the variability Partitioning the variability is used for economic significance. Using the estimated regression line. we can compute the fitted value: ybi = βb0 + βb1 xi . Define: SST = n X (yi − y )2 i=1 to be the total sum of squares. The squared deviations (yi − y )2 provide us with a measure of the spread of the data. 3125/3173 .

SSM: sum of squares model (sometime called regression).ACTL2002/ACTL5101 Probability and Statistics: Week 10 Simple linear regression Partitioning the variability Partitioning the variability Partition the total deviation as: yi − y | {z } (y − yb ) | i {z i } = total deviation + unexplained deviation (b y − y) . see slides 3171-3172 3126/3173 . Proof: similar to week 8 k-sample tests.SSE: sum of squares error (sometime called residual). SSM } where . . | i {z } explained deviation We then obtain: n X 2 n X } |i=1 {z (yi − y ) = |i=1 {z SST 2 n X } |i=1 {z (yi − ybi ) + SSE (b yi − y )2 .

ACTL2002/ACTL5101 Probability and Statistics: Week 10
Simple linear regression
Partitioning the variability

Interpret these sums of squares as follows:
- SST is the total variability in the absence of knowledge of the
variable X ;
- SSE is the total variability remaining after introducing the
effect of X ;
- SSM is the total variability “explained” because of knowledge
of X .

This partitioning of the variability is used in ANOVA tables:
Source
Model

Sum of squares
SSM=

Degrees
of freedom

Mean
square

F

(b
yi − y )2

DFM=1

SSM
MSM= DFM

MSM
MSE

(yi − ybi )2

DFE=n − 2

SSE
MSE= DFE

(yi − y )2

DFT=n − 1

SST
MST= DFT

n
P
i=1

Error
Total
3127/3173

SSE=

n
P

SST=

i=1
n
P
i=1

ACTL2002/ACTL5101 Probability and Statistics: Week 10
Simple linear regression
Partitioning the variability

Coefficient of Determination
Notice that the square of the correlation coefficient occurs in
the denominator of the t statistic used to test hypotheses
concerning the population correlation coefficient. The statistic
R 2 is called the coefficient of determination and provides
useful information.
Noting (prove: slide 3173, notation: slide3163):
SSE = Syy − βb1 Sxy ,
|{z} | {z }
=SST

=SSM

then the coefficient of determination may be written as:
!2
Sxy
SSE
2
R = p
=1−
.
SST
Sxx Syy
3128/3173

Thus, R 2 can be seen as the proportion of total variation of y
explained by the variable x in a simple linear regression model.

ACTL2002/ACTL5101 Probability and Statistics: Week 10
Simple linear regression
Exercise

Simple Linear Regression
Simple linear regression
Basic idea
Correlation Coefficient
Assumptions
Relation MLE and LSE
Partitioning the variability
Exercise
Testing in simple linear regression
Overview
Inference on the slope
Inference on the intercept
Confidence Intervals for the Population Regression Line
Prediction Intervals for the Actual Value of the Dependent Variable
Hypothesis Test for Population Correlation
Exercise
Appendix
Algebra: parameter estimates
Properties of parameter estimates
Proof: Partitioning the variability

ACTL2002/ACTL5101 Probability and Statistics: Week 10
Simple linear regression
Exercise

Exercise
A car insurance company is interested in how large adverse
selection effect is in his sample, i.e. how large the difference in
claim size relative to the premium is for different groups.
The insurance premium depends on the coverage (Gold
Comprehensive Car Insurance, Standard Comprehensive Car
Insurance and Third Party Property Car Insurance) and the
price of the insured vehicle (five categories).
a. Explain why there might be difference in the claim sizes for
the different groups.
Solution: high coverage ⇒ reckless behavior (example:
airbags). Expensive car ⇒ more wealthy drivers ⇒ better
drivers? Other explanation is also possible.
3129/3173

ACTL2002/ACTL5101 Probability and Statistics: Week 10
Simple linear regression
Exercise

Exercise
Each of the 15 categories has a different premiums and
number of contracts for the insurance contract.
The insurance company has the total claim sizes in the groups.
b. Give the linear regression model.
Solution: Let:
yi be the average claim size for group i = 1, . . . , 15;
xi be the average MVI premium for group i = 1, . . . , 15;
Then the regression is:
yi = β0 + β1 x1 + εi ,

3130/3173

where β0 and β1 are regression constants, εi for i = 1, . . . , 15
the residual independently distributed with mean zero,
variance σ 2 , independent of X .

have to check using data. have to check using data. . Note: assumed linear relation! Variance independent of explanatory variable: debatable (increasing?). Are the weak assumptions and strong assumptions reasonable in this regression model? Solution: Weak assumptions: Residual has a mean of zero: yes. Residuals are independent: yes. the mean is captured in β0 and β1 .ACTL2002/ACTL5101 Probability and Statistics: Week 10 Simple linear regression Exercise Exercise c. Additional strong assumption: 3131/3173 Residuals are normally distributed: debatable.

ACTL2002/ACTL5101 Probability and Statistics: Week 10 Simple linear regression Exercise Exercise data 1000 900 Average claim size 800 700 600 500 400 300 200 100 200 3132/3173 300 400 500 600 700 Premium (x) 800 900 1000 .

15 i=1 i=1 yi = 6288. 2 = 3. Find the LS estimates of the regression model. and x y i=1 i i Pi=1 15 i=1 xi yi = 3. 190. . 660. 3133/3173 Solution: See next slide.ACTL2002/ACTL5101 Probability and Statistics: Week 10 Simple linear regression Exercise Exercise The observed values for the 15 groups are: i xi yi i xi yi 1 210 189 9 380 323 2 230 267 10 410 313 3 235 234 11 460 456 4 250 142 12 540 528 5 260 302 13 720 768 6 280 149 14 880 963 7 320 308 15 910 954 8 360 392 P P Summary statistics: 15 xi = 6445. 325. 529. P P 15 15 2 = 3. 566. 000. d.

566.137 · = −69.329.137. 566.57. 325 − 15 √ =3200 ⇒σ b = 3200 = 56. 190 − − 64452 15 3. 000 − 6445·6288 62882 15 3. 000 − 6445·6288 15 = 64452 3. βb0 =y − βb1 · x 6288 6445 = − 1. 529. 529.ACTL2002/ACTL5101 Probability and Statistics: Week 10 Simple linear regression Exercise P15 i=1 xi · yi − n · x · y P15 2 2 i=1 xi − n · x 3. 1 = · 13 3134/3173 2 ! . 15 15  P 2  15 15 x · y x · y − n · i i=1 i 1 X 2  σ b2 = · yi − n · y 2 − P15 2  2 n−2 i=1 xi − n · x i=1 3. 325 − 15 βb1 = = 1. 660.

566. 000 − 6445·6228 15   64552 3. Solution: Pn − n (x · y )   Pn 2 · 2 2 2 i=1 yi − ny i=1 xi − nx r =q P n = r i=1 xi yi 3. 325 − 15 · 3. 660. 3135/3173  . Positive sample correlation (r > 0) ⇒ βb1 is positive.9795. 529. Relate the (sign) of the correlation coefficient to the estimates. Find the Correlation coefficient.ACTL2002/ACTL5101 Probability and Statistics: Week 10 Simple linear regression Exercise Exercise e. 190 − 62282 15 =0.

Partition the variability. 654. 024. 190 − 62882 = 1. 3136/3173 . 024. 606 = 982. Solution: n X SST = yi2 − n · y 2 i=1 =3. 660. 260 − 41. 260 15 SSE =(n − 2) · σ b2 =13 · 3200 = 41. 606 SSM =SST − SSE = 1.ACTL2002/ACTL5101 Probability and Statistics: Week 10 Simple linear regression Exercise Exercise f.

ACTL2002/ACTL5101 Probability and Statistics: Week 10 Simple linear regression Exercise Comment on residual plot 80 60 40 Residual 20 0 −20 −40 −60 −80 −100 −120 200 3137/3173 300 400 500 600 700 Premium (x) 800 900 1000 .

ACTL2002/ACTL5101 Probability and Statistics: Week 10 Testing in simple linear regression Overview Simple Linear Regression Simple linear regression Basic idea Correlation Coefficient Assumptions Relation MLE and LSE Partitioning the variability Exercise Testing in simple linear regression Overview Inference on the slope Inference on the intercept Confidence Intervals for the Population Regression Line Prediction Intervals for the Actual Value of the Dependent Variable Hypothesis Test for Population Correlation Exercise Appendix Algebra: parameter estimates Properties of parameter estimates Proof: Partitioning the variability .

Inference on a function of both parameters (β0 + xi β1 ) .Inference on white noise (i ) . Thus (using two estimated parameters βb0 and βb1 ): n (n − 2) · s 2 X = ( i /σ )2 = |{z} σ2 i=1 3138/3173 N(0.Inference on a function of both parameters and white noise (β0 + xi β1 + i ) Note: under strong assumptions: i ∼ N(0. σ 2 ).Inference on individual parameters (β0 or β1 ) .ACTL2002/ACTL5101 Probability and Statistics: Week 10 Testing in simple linear regression Overview Overview of tests and CI .1) Pn i=1 (yi − βb0 − βb1 · xi )2 ∼ χ2 (n−2) σ2 .

ACTL2002/ACTL5101 Probability and Statistics: Week 10 Testing in simple linear regression Inference on the slope Simple Linear Regression Simple linear regression Basic idea Correlation Coefficient Assumptions Relation MLE and LSE Partitioning the variability Exercise Testing in simple linear regression Overview Inference on the slope Inference on the intercept Confidence Intervals for the Population Regression Line Prediction Intervals for the Actual Value of the Dependent Variable Hypothesis Test for Population Correlation Exercise Appendix Algebra: parameter estimates Properties of parameter estimates Proof: Partitioning the variability .

σ is usually unknown. > e x σ x e P Notation: x =x −x ⇒ e x >e x = ni=1 (xi − x)2 and   e P Var βb1 = σ 2 / ni=1 (xi − x)2 (see slide 3168).ACTL2002/ACTL5101 Probability and Statistics: Week 10 Testing in simple linear regression Inference on the slope Inference on the slope: Often we want to test whether the exogenous variable has an influence on the endogenous variable or if the influence is larger/smaller than some value. 1) .1) 3139/3173 . The distribution of βb1 under the strong assumptions: βb1 − β1 q ∼ N (0. s (n−2)·s 2 σ2 | √ 2 n−2 {z } χ (n−2)/(n−2) ∼ t (n − 2) . and estimated by s so: βb − β βb1 − β1  1 q 1  = q > e σ · x >e x e s x e x | {z } N(0.

For testing the null hypothesis H0 : β1 = βe1 for some constant βe1 . > se βb1 e s x e x 3140/3173 . use the test statistic:   βb − βe βb1 − βe1 1 1 t βb1 =   = q . where   s se βb1 = q e x >e x is the standard error of the estimated slope coefficient.ACTL2002/ACTL5101 Probability and Statistics: Week 10 Testing in simple linear regression Inference on the slope Inference on the slope A 100 (1 − α) % confidence interval for β1 is given by:     b b b β1 − t1−α/2.n−2 · se β1 < β1 < β1 + t1−α/2.n−2 · se βb1 .

Decision Making Procedures for Testing H0 : β1 = βe1 Alternative H1 Reject 0 in favor of H1 if .ACTL2002/ACTL5101 Probability and Statistics: Week 10 Testing in simple linear regression Inference on the slope Inference on the slope The decision rules under various alternative hypotheses are summarized below.

 H.

.

b .

e β1 6= β1 .

t β1 .

n−2 β1 < βe1   t βb1 < −t1−α. 3141/3173 .n−2 To test whether the regressor variable is significant or not.n−2 β1 > βe1   t βb1 > t1−α. it is equivalent to testing whether the slope is zero or not. test H0 : β1 = 0 against H1 : β1 6= 0. > t1−α/2. Thus.

ACTL2002/ACTL5101 Probability and Statistics: Week 10 Testing in simple linear regression Inference on the intercept Simple Linear Regression Simple linear regression Basic idea Correlation Coefficient Assumptions Relation MLE and LSE Partitioning the variability Exercise Testing in simple linear regression Overview Inference on the slope Inference on the intercept Confidence Intervals for the Population Regression Line Prediction Intervals for the Actual Value of the Dependent Variable Hypothesis Test for Population Correlation Exercise Appendix Algebra: parameter estimates Properties of parameter estimates Proof: Partitioning the variability .

s (n−2)·s 2 βb0 − β0 βb0 − β0 σ2 q  q = ∼ t (n − 2) .1) . σ is usually unknown. and estimated by s thus: 3142/3173 . one can test the value of the intercept.ACTL2002/ACTL5101 Probability and Statistics: Week 10 Testing in simple linear regression Inference on the intercept Inference on the intercept Similar. using testing procedure / confidence interval. The distribution of βb0 under the strong assumptions: βb − β0 r0 ∼ N (0. Again. 1) . 2 2 |x| |x| n −2 s n1 + n2ex >ex σ · n1 + n2ex >ex | {z } | {z } √χ2 (n−2)/(n−2) N(0. |x|2 1 σ · n + 2 e> e n x x    Note (see slide 3169): Var βb0 = σ 2 · n1 + 2 P x (xi −x)2  .

s   s 1 x2 where se βb0 = r =s· + Pn 2 n |x|2 1 i=1 (xi − x) n + 2 e> e n x x is the standard error of the estimated slope coefficient. see slide 3141):   βb − βe βb − βe0 0 0 r0 t βb0 =   = .ACTL2002/ACTL5101 Probability and Statistics: Week 10 Testing in simple linear regression Inference on the intercept Inference on the intercept A 100 (1 − α) % confidence interval for β0 is given by:     βb0 − t1−α/2.n−2 · se βb0 < β0 < βb0 + t1−α/2.n−2 · se βb0 . use the test statistic (with similar decision rules as for the slope. For testing the null hypothesis H0 : β0 = βe0 for some constant βe0 . |x|2 1 se βb0 s · n + 2 e> e n x x 3143/3173 .

ACTL2002/ACTL5101 Probability and Statistics: Week 10 Testing in simple linear regression Confidence Intervals for the Population Regression Line Simple Linear Regression Simple linear regression Basic idea Correlation Coefficient Assumptions Relation MLE and LSE Partitioning the variability Exercise Testing in simple linear regression Overview Inference on the slope Inference on the intercept Confidence Intervals for the Population Regression Line Prediction Intervals for the Actual Value of the Dependent Variable Hypothesis Test for Population Correlation Exercise Appendix Algebra: parameter estimates Properties of parameter estimates Proof: Partitioning the variability .

3144/3173 .ACTL2002/ACTL5101 Probability and Statistics: Week 10 Testing in simple linear regression Confidence Intervals for the Population Regression Line Confidence Intervals for the Population Regression Line Suppose x = x0 is a specified value of the -out of sampleregressor variable and we want to predict the corresponding Y value associated with it. Thus the best predicted value (also unbiased) will be: yb0 = βb0 + βb1 x0 . Consider estimating the mean of this which will be: E [Y |x = x0 ] = E [β0 + β1 x +  |x = x0 ] = β0 + β1 x0 + 0.

= n (n − 1) sx2     * see slide 3169 for Var βb0 slide 3168 for Var βb0 and slide   3170 for Cov βb0 . βb1 . 3145/3173 . βb1     1 x2 σ2 −x · σ 2 ∗ 2 2 + · σ + x · + 2x · = 0 0 n (n − 1) sx2 (n − 1) sx2 (n − 1) sx2   1 x 2 − 2x0 x + x02 = + · σ2 n (n − 1) sx2 ! 1 (x − x0 )2 + · σ2.ACTL2002/ACTL5101 Probability and Statistics: Week 10 Testing in simple linear regression Confidence Intervals for the Population Regression Line Confidence Intervals for the Population Regression Line The variance of the prediction is:   Var (b y0 ) =Var βb0 + βb1 x0       =Var βb0 + x02 · Var βb1 + 2x0 · Cov βb0 .

n (n − 1) · sx2 3146/3173 Note: Regression line (mean response) does not include uncertainty due to i .n−2 × s · + . .     1 (x−x0 )2 2 Therefore. for that see slide 3150. . (x − x0 )2 1 + s· n (n − 1) · sx2 This pivot can therefore be used to construct the 100 (1 − α) % confidence interval for y0 : s   (x − x0 )2 1 βb0 + βb1 x0 ± t1−α/2. . so is βb0 + βb1 x0 . . we have: yb0 ∼ N β0 + β1 x0 .ACTL2002/ACTL5101 Probability and Statistics: Week 10 Testing in simple linear regression Confidence Intervals for the Population Regression Line Confidence Intervals for the Population Regression Line Since both βb0 and βb1 are linear functions of Y1 . . + (n−1)·s · σ 2 x n and we have: yb0 − (β0 + β1 x0 ) s ∼ t (n − 2) . Yn .

ACTL2002/ACTL5101 Probability and Statistics: Week 10 Testing in simple linear regression Confidence Intervals for the Population Regression Line Example σx=4. σy=1 σx=4. σy=4 10 y 5 0 −5 −10 −10 3147/3173 −5 0 x 5 10 . σy=1 σx=1.

ACTL2002/ACTL5101 Probability and Statistics: Week 10 Testing in simple linear regression Prediction Intervals for the Actual Value of the Dependent Variable Simple Linear Regression Simple linear regression Basic idea Correlation Coefficient Assumptions Relation MLE and LSE Partitioning the variability Exercise Testing in simple linear regression Overview Inference on the slope Inference on the intercept Confidence Intervals for the Population Regression Line Prediction Intervals for the Actual Value of the Dependent Variable Hypothesis Test for Population Correlation Exercise Appendix Algebra: parameter estimates Properties of parameter estimates Proof: Partitioning the variability .

p. this implies that for each observation the probability that it is between the upper and lower bound is α. Note. We base our prediction of Yi (given X = x) when X = xi on: ybi = βb0 + βb1 xi . all values of yi are between upper and lower bound w. . much smaller than α. X = xi ] = E βb0 + βb1 xi |X = x 3148/3173 = β0 + β1 xi . i.ACTL2002/ACTL5101 Probability and Statistics: Week 10 Testing in simple linear regression Prediction Intervals for the Actual Value of the Dependent Variable CI for the Actual Value of the Dependent Variable In the next slides we will find pointwise CI and prediction intervals for the value of yi .. The error in our prediction is: Yi − ybi = β0 + β1 xi + i − ybi = E[Y |X = xi ] − ybi + i .e. where we have: h i E [b yi |X = x.

ACTL2002/ACTL5101 Probability and Statistics: Week 10 Testing in simple linear regression Prediction Intervals for the Actual Value of the Dependent Variable CI for the Actual Value of the Dependent Variable Thus we have: E [Yi − ybi |X = x. Var (Yi − ybi |X = x. n Sxx Notation: Sxx = 3149/3173 Pn 2 i=1 xi . ybi |X = x. X = xi )   (xi − x)2 2 2 1 =σ + σ + −0 n Sxx   1 (xi − x)2 2 =σ 1 + + . X = xi ) =Var (Yi |X = xi ) + Var (b yi |X = x) −2Cov (Yi . Further (using 3145). . X = xi ] = E [Y |X = xi ] − (β0 + β1 · xi ) = 0.

ACTL2002/ACTL5101 Probability and Statistics: Week 10 Testing in simple linear regression Prediction Intervals for the Actual Value of the Dependent Variable Prediction Intervals It then follows that:   1 (xi − x)2 (Yi − ybi |X = x. 1 (xi − + n Sxx Thus: for the variance for the predicted individual response an additional σ 2 must be added to the variance for the predicted mean response (see slide 3146) 3150/3173 . X = xi ) ∼ N 0. σ · 1 + + n Sxx  2 and thus the test statistic is: Yi − ybi T = s s· 1+ x)2 ∼ tn−2 .

we have a 100(1 − α)% prediction interval for Yi .n−2 · s · 1 + + . n Sxx 3151/3173 . the value of Y at X = xi given by: s 1 (xi − x)2 ybi ± t1−α/2.ACTL2002/ACTL5101 Probability and Statistics: Week 10 Testing in simple linear regression Prediction Intervals for the Actual Value of the Dependent Variable Prediction Intervals Thus.n−2 · s · 1 + + n Sxx s 1 (xi − x)2 = βb0 + βb1 xi ± t1−α/2.

σy=1 σx=4.ACTL2002/ACTL5101 Probability and Statistics: Week 10 Testing in simple linear regression Prediction Intervals for the Actual Value of the Dependent Variable Example σx=4. σy=4 10 y 5 0 −5 −10 −10 3152/3173 −5 0 x 5 10 . σy=1 σx=1.

ACTL2002/ACTL5101 Probability and Statistics: Week 10 Testing in simple linear regression Hypothesis Test for Population Correlation Simple Linear Regression Simple linear regression Basic idea Correlation Coefficient Assumptions Relation MLE and LSE Partitioning the variability Exercise Testing in simple linear regression Overview Inference on the slope Inference on the intercept Confidence Intervals for the Population Regression Line Prediction Intervals for the Actual Value of the Dependent Variable Hypothesis Test for Population Correlation Exercise Appendix Algebra: parameter estimates Properties of parameter estimates Proof: Partitioning the variability .

The test statistic is based on (prove not required): √ R R · n−2 T =p = √ .ACTL2002/ACTL5101 Probability and Statistics: Week 10 Testing in simple linear regression Hypothesis Test for Population Correlation Testing the correlation coefficient See F&T page 25. but with the x and y replaced by X and Y . 3153/3173 . H1 : ρXY 6= 0 (or > or <). 1 − R2 (1 − R 2 ) /(n − 2) where R is the random variable denoting correlation coefficient.s. Using LRT test H0 : ρXY = 0 v. Let a pair of random variables come from a common bivariate normal distribution. it is possible to find a test statistic based on the likelihood ratio test to conduct a test of independence.

(x2 . (xn . y1 ) . . yn ). We can summarize a procedure for testing the independence between two sets of random variables as follow: Suppose we obtain observed pairs of variables (x1 .n−2 . y2 ) . .ACTL2002/ACTL5101 Probability and Statistics: Week 10 Testing in simple linear regression Hypothesis Test for Population Correlation Testing the correlation coefficient It can be shown that T ∼ t (n − 2) has a t-distribution with (n − 2) degrees of freedom. 1. To test H0 : ρXY = 0 against the alternative H1 : ρXY > 0. . the decision rule is: √ r · n−2 Reject H0 if the observed t = √ > t1−α. . 1 − r2 3154/3173 .

To test H0 : ρXY = 0 against the alternative H1 : ρXY < 0. Reject H0 if the observed t = √ 1 − r2 3. the decision rule is: .n−2 .ACTL2002/ACTL5101 Probability and Statistics: Week 10 Testing in simple linear regression Hypothesis Test for Population Correlation Testing the correlation coefficient 2. the decision rule is: √ r · n−2 < −t1−α. And to test H0 : ρXY = 0 against the alternative H1 : ρXY 6= 0.

√ .

.

r · n − 2.

.

Reject H0 if the observed |t| = . > t1−α/2.n−2 .

.

√ 1 − r2 .

3155/3173 .

Our test statistic is based on r . y1 ). . (xn . 2 √ and is the form: T = (z − ζ0 ) · n − 3. We wish to test: H0 : ρ = ρ0 3156/3173 against H1 : ρ 6= ρ0 . (x2 . yn ). the sample correlation coefficient. drawn from bivariate normal distribution with correlation ρ. . together with: 1 log ((1 + r )/(1 − r )) and z = 2 1 ζ0 = log (1 + ρ0 )/(1 − ρ0 )) .ACTL2002/ACTL5101 Probability and Statistics: Week 10 Testing in simple linear regression Hypothesis Test for Population Correlation Fisher’s z-transformation for testing correlation hypotheses We have n independent data points of the form: (x1 . . y2 ). . Each point is random. which is approximately normally distributed (prove not required). .

ACTL2002/ACTL5101 Probability and Statistics: Week 10 Testing in simple linear regression Exercise Simple Linear Regression Simple linear regression Basic idea Correlation Coefficient Assumptions Relation MLE and LSE Partitioning the variability Exercise Testing in simple linear regression Overview Inference on the slope Inference on the intercept Confidence Intervals for the Population Regression Line Prediction Intervals for the Actual Value of the Dependent Variable Hypothesis Test for Population Correlation Exercise Appendix Algebra: parameter estimates Properties of parameter estimates Proof: Partitioning the variability .

91 Thus p-value equals zero.55 for p = 0.  .9795· = 17. Test whether the correlation coefficient is positive.. 3157/3173 .2845 1−r √ and ζ0 = 0. √ √ n−2 13 Method 1: T = r√· 1−r = √0. T = 2.2845 · 12 = 7. Using F&T 2 1−0.97952 page 163: t1−p (13) = 17.ACTL2002/ACTL5101 Probability and Statistics: Week 10 Testing in simple linear regression Exercise Exercise Consider the previous exercise on slides 3129-3137 with the data on slide 3133.  1. reject null hypothesis. i. p-value is almost 0. i.e. Hence.55.9795 Method 2: z = log 1+r 2 = log 0. Solution: r = 0.9795 (see slide 3135).0205 2 = 2.

s.137 − 1 q 56.11. Solution: test H0 : β1 = 1 v.57/ 3. .ACTL2002/ACTL5101 Probability and Statistics: Week 10 Testing in simple linear regression Exercise Exercise ii.57 and βb1 = 1. Test whether the slope parameter is larger than one.137 = 2. H1 : β1 6= 1.137.5% or lower (note two-sided test). = 3158/3173 t1−0. Accept null for level of significance of 5. see slide 3134.11 0.027 (13) = 2.06488809 * using σ b = 56. 529. Test statistic: T = ∗ = βb − 1 qP 1 n 2 2 s/ i=1 xi − n · x 1. 325 − 64452 15 0.

529.475 * using σ b = 56.s. 31. Test statistic: βb0 − 0 T = s· q 1 n + i=1 x2 xi2 −n·x 2 −69. 3159/3173 t0.0231 (13) = −2. Solution: test H0 : β0 ≤ 0 v.20.329 = −2. see slide 3134.57 · = Pn 2 1 15 + ( 6445 15 ) 2 3.3% or lower (note one-sided test).57 and βb0 = −69. Test whether the intercept parameter is non-negative.329 ∗ = r 56. H1 : β0 > 0. Accept null only for level of significance of 2.325− 6445 15 −69.329.20.ACTL2002/ACTL5101 Probability and Statistics: Week 10 Testing in simple linear regression Exercise Exercise iii. .

449 t0. 529.57. 459.975 (13) = 2.329 + 1.61 − 2.57 · 1 + + 2 15 3. s p 1 (x − x0 )2 + Pn 2 2 n i=1 xi − n · x v u 2 6445 u 1 t 15 − 350 =56. 328. Thus Pr (yi ∈ (328.137 · 350 =328.160 · 60.619.160 · 60. Solution: We have: ybi |xi = 350 = −69.160.61 + 2.95 .04.449.18)) =0.95 3160/3173 Pr (yi ∈ (198. Calculate the 95% Confidence interval of Y given that X = 350. s = 56.449)) =0. 325 − 6445 15 Var (yi ) =s · 1+ =60.ACTL2002/ACTL5101 Probability and Statistics: Week 10 Testing in simple linear regression Exercise Exercise iv.

ACTL2002/ACTL5101 Probability and Statistics: Week 10 Appendix Algebra: parameter estimates Simple Linear Regression Simple linear regression Basic idea Correlation Coefficient Assumptions Relation MLE and LSE Partitioning the variability Exercise Testing in simple linear regression Overview Inference on the slope Inference on the intercept Confidence Intervals for the Population Regression Line Prediction Intervals for the Actual Value of the Dependent Variable Hypothesis Test for Population Correlation Exercise Appendix Algebra: parameter estimates Properties of parameter estimates Proof: Partitioning the variability .

3161/3173 . Pn Pn 2 2 n i=1 xi − ( i=1 xi ) i=1 yi *: (1 − a/b)c = d/b ⇒ (bc − ac)/b = d/b ⇒ c = d/(b − a).ACTL2002/ACTL5101 Probability and Statistics: Week 10 Appendix Algebra: parameter estimates Basic idea. used on slide 3106 Pn βb0 βb1 βb0 ! P ( ni=1 xi )2 b β0 1 − Pn n i=1 xi2 βb0 P − βb1 ni=1 xi = n Pn P yi − βb0 ni=1 xi i=1 xiP = n 2 i=1 xi P  n Pn Pn b Pn xi i yi −β0 i=1 i=1 xP y − n 2 i=1 i i=1 xi i=1 xi = n P Pn Pn Pn Pn n 2 b x y − x y − β x 0 i=1 i i=1 i i=1 i i i=1 i i=1 xi Pn = n i=1 xi2  Pn Pn Pn Pn 2 ∗ i=1 yi i=1 xi − i=1 xi yi i=1 xi = .

n 2 i=1 xi thus parameter β1 is estimated by (used on slide 3106): P  P n b1 Pn xi Pn xi n ni=1 xi yi − y − β i i=1 i=1 i=1 Pn βb1 = 2 n i=1 xi ! Pn Pn Pn P 2 ( i=1 xi ) n i=1 xi yi − i=1 yi ni=1 xi b P 1 − Pn β1 = n i=1 xi2 n ni=1 xi2 Pn Pn Pn ∗ n i=1 xi yi − i=1 yi i=1 xi b β1 = . 3162/3173 .ACTL2002/ACTL5101 Probability and Statistics: Week 10 Appendix Algebra: parameter estimates From the previous slide we have: Pn b Pn xi i=1 yi − β1 i=1 b β0 = n Pn P yi − βb0 ni=1 xi i=1 xiP b β1 = . Pn Pn 2 n i=1 xi − ( i=1 xi )2 *: (1 − a/b)c = d/b ⇒ (bc − ac)/b = d/b ⇒ c = d/(b − a).

we express the parameter estimates in terms of (squared) errors. we denote: sx. Moreover.y . sx · sy sxx · syy 3163/3173 . We have P the following sum of squares (see F&T P page 24-25): Sx = Pni=1 xi and Sy = Pni=1 yi Sxx = Pni=1 xi2 and Syy = ni=1 yi2 n Sxy = Pi=1 xi · yi sxx = Pni=1 (xi − x)2 = (n − 1) · sx2 n 2 syy = i=1 (yi − y ) = (n − 1) · sy2 Pn Pn sxy = i=1 (xi − x) · i=1 (yi − y ) = (n − 1) · sx.y ) denotes sample (co-)variance. where sx2 (sx.y sxy r= =√ .ACTL2002/ACTL5101 Probability and Statistics: Week 10 Appendix Algebra: parameter estimates Parameter estimates II: Notation More commonly.

ACTL2002/ACTL5101 Probability and Statistics: Week 10 Appendix Algebra: parameter estimates Parameter estimates II (used on slide 3106) We have: n βb1 = = n Pn Pn Pn nSxy − Sx Sy i=1 xi yi − i=1 yi i=1 xi = Pn Pn 2 2 nSxx − Sx2 n x −( xi )  Pn i=1 i Pn i=1 Pn n i=1 xi yi − i=1 yi i=1 xi · n2  P P n 2 − ( n x )2 · 1 n x i=1 i i=1 i n2 Pn xi yi − nxy = Pi=1 n x 2 − nx 2 Pni=1 i P P xi yi − ni=1 xi y − ni=1 xi y + nxy i=1 Pn Pn Pn = 2 2 i=1 xi + i=1 x − 2 i=1 xi x Pn (xi − x) · (yi − y ) = i=1Pn 2 i=1 (xi − x) √ syy e sxy sxy sy = =√ ·√ =r . √ e sxx sxx · sxx syy sx 3164/3173 .

We refer to βb1 as the slope of the regression line.ACTL2002/ACTL5101 Probability and Statistics: Week 10 Appendix Algebra: parameter estimates Parameter estimates II Thus we have that βb1 is the sample correlation coefficient times the quotient of the sample standard deviation of Y and X. . 3165/3173 For βb0 we have (used on slide 3106): Pn Pn Pn Pn 2 i=1 yi ( i=1 xi ) − i=1 xi yi i=1 xi b β0 = Pn Pn 2 2 n i=1 xi − ( i=1 xi ) Sy Sxx − Sxy Sx = nSxx − Sx2 Pn b Pn xi Sy Sx i=1 yi − β1 i=1 = = − βb1 . n n n We refer to βb0 as the intercept of the regression line.

ACTL2002/ACTL5101 Probability and Statistics: Week 10 Appendix Properties of parameter estimates Simple Linear Regression Simple linear regression Basic idea Correlation Coefficient Assumptions Relation MLE and LSE Partitioning the variability Exercise Testing in simple linear regression Overview Inference on the slope Inference on the intercept Confidence Intervals for the Population Regression Line Prediction Intervals for the Actual Value of the Dependent Variable Hypothesis Test for Population Correlation Exercise Appendix Algebra: parameter estimates Properties of parameter estimates Proof: Partitioning the variability .

h i Home exercise: show the same for E βb1 = β1 .ACTL2002/ACTL5101 Probability and Statistics: Week 10 Appendix Properties of parameter estimates Unbiased regression parameters For βb0 we have (used on slide 3118): "P #  Pn Pn Pn n h i 2 − y x x y x i i i i i=1 i=1 i=1 i=1 i E βb0 = E P P n ni=1 xi2 − ( ni=1 xi )2  Pn Pn Pn Pn 2 i=1 E [yi ] i=1 xi − i=1 xi E [yi ] i=1 xi = Pn Pn 2 2 n i=1 xi − ( i=1 xi )  Pn Pn Pn Pn 2 i=1 xi (β0 + β1 xi ) i=1 xi i=1 (β0 + β1 xi ) i=1 xi − = Pn P 2 n n i=1 xi2 − ( i=1 xi )  Pn P Pn n 2 nβ0 i=1 (β0 xi ) i=1 xi i=1 xi − = Pn Pn 2 2 n i=1 xi − ( i=1 xi ) = β0 . 3166/3173 .

3167/3173 .ACTL2002/ACTL5101 Probability and Statistics: Week 10 Appendix Properties of parameter estimates Regression parameters uncertainty Under the weak assumptions we have that the (co-)variance of the parameters is given by: P   σ 2 ni=1 xi2 Sxx 2 b sβ0 = Var β0 = Pn = sβ21 · Pn 2 2 n n i=1 xi − ( i=1 xi )   2 nσ ns2 sβ21 = Var βb1 = Pn = P nSxx − Sx2 n i=1 xi2 − ( ni=1 xi )2 P   −σ 2 ni=1 xi Sx = − · sβ21 Cov βb0 . βb1 = Pn Pn 2 2 n n i=1 xi − ( i=1 xi ) nSyy − Sy2 − βb12 (nSxx − Sx2 ) s2 = Var (b ) = n(n − 2) Proof: See next slides.

= 3168/3173 Pn .ACTL2002/ACTL5101 Probability and Statistics: Week 10 Appendix Properties of parameter estimates Proof sample variance slope (used on slide 3139 and 3145) Note that we have: Pn Pn (xi − x) (yi − y ) (xi − x) yi βb1 = i=1 = . Pn Pi=1 n 2 2 (x − (x − x) x) i i i=1 i=1 Then we have that:   Var βb1 = Var Pn (xi − x) yi Pi=1 2 n i=1 (xi − x) ! 2 i=1 (xi − x) Var (yi ) P 2 n 2 (x − x) i=1 i P n σ 2 i=1 (xi − x)2 = P 2 n 2 (x − x) i i=1 .X 2 n = σ2 i=1 (xi − x) .

ACTL2002/ACTL5101 Probability and Statistics: Week 10 Appendix Properties of parameter estimates Proof sample variance intercept Using that: βb0 = y − βb1 x.X n i=1 (xi − x)2 . we have (used on slide 3142 and 3145):     Var βb0 =Var y − βb1 x ! n   X =Var yi /n + x 2 Var βb1 i=1 = X n i=1 Var (yi ) /n2 + x 2 σ 2 P n n (xi − x)2 + nx 2 =σ 2 Pni=1 P n i=1 xi2 − ( ni=1 xi )2 P σ 2 ni=1 xi2 = Pn . P n i=1 xi2 − ( ni=1 xi )2 3169/3173 .

ACTL2002/ACTL5101 Probability and Statistics: Week 10 Appendix Properties of parameter estimates Proof sample covariance intercept and slope (used on slide 3145) Using that: βb0 = y − βb1 x. βb1   = − x · Cov βb1 . βb1 =Cov y − βb1 x. we have:     Cov βb0 . n 3170/3173 . βb1 Pn   xi = − i=1 · Var βb1 . βb1   =Cov −βb1 x.

ACTL2002/ACTL5101 Probability and Statistics: Week 10 Appendix Proof: Partitioning the variability Simple Linear Regression Simple linear regression Basic idea Correlation Coefficient Assumptions Relation MLE and LSE Partitioning the variability Exercise Testing in simple linear regression Overview Inference on the slope Inference on the intercept Confidence Intervals for the Population Regression Line Prediction Intervals for the Actual Value of the Dependent Variable Hypothesis Test for Population Correlation Exercise Appendix Algebra: parameter estimates Properties of parameter estimates Proof: Partitioning the variability .

Cont. 3171/3173 . next slide.ACTL2002/ACTL5101 Probability and Statistics: Week 10 Appendix Proof: Partitioning the variability Proof: Partitioning the variability SST = SSE + SSM = = n n X X (yi − y )2 = yi2 + y 2 − 2y yi i=1 n X i=1 n X i=1 i=1 (yi − ybi )2 + n X (b yi − y )2 yi2 + ybi2 − 2yi ybi + ybi2 + y 2 − 2y ybi i=1 = n X yi2 − 2yi ybi + y 2 − 2y ybi yi2 + 2b i=1 ∗ = n X yi2 + 2b yi2 − 2(b yi + i )b yi + y 2 − 2y (yi − i ) i=1 * using yi = ybi + i .

using *** i=1 2b independence. Used on slide 3126.ACTL2002/ACTL5101 Probability and Statistics: Week 10 Appendix Proof: Partitioning the variability Proof cont.: SST = n X i=1 SSE + SSM = n X (yi − y )2 = n X yi2 + y 2 − 2y yi  i=1 yi2 + 2b yi2 − 2(b yi + i )b yi + y 2 − 2y (yi − i ) i=1 = n X yi i + y 2 − 2y yi + 2y i yi2 − 2b i=1 n X ∗∗ = yi2 + y 2 − 2y yi = SST i=1 Pn 3172/3173 P ** using i=1 2y i = 2y ni=1 i = 0 and Pn ∗∗∗ yi i = 2nE[b yi i ] = 2nE[b yi ] · E[i ] = 0. .

2 .ACTL2002/ACTL5101 Probability and Statistics: Week 10 Appendix Proof: Partitioning the variability SSM = = n X (b yi − y )2 i=1 n  X βb0 + βb1 · xi − y 2 i=1 = n  X (y − βb1 · x) + βb1 · xi − y i=1 = n X βb12 · (xi − x)2 i=1 =βb12 · sxx ∗ =βb1 · sxy using βb1 = 3173/3173 sxy sxx from slide 3164.