Questions What are predictors and criteria? Write an equation for the linear regression. Describe each term. How do changes in the slope and intercept affect (move) the regression line? What does it mean to test the significance of the regression sum of squares? R-square? What is R-square? What does it mean to choose a regression line to satisfy the loss function of least squares? How do we find the slope and intercept for the regression line with a single independent variable? (Either formula for the slope is acceptable.) Why does testing for the regression sum of squares turn out to have the same result as testing for R- square?
Basic Ideas Jargon IV = X = Predictor (pl. predictors) DV = Y = Criterion (pl. criteria) Regression of Y on X e.g., GPA on SAT Linear Model = relations between IV and DV represented by straight line.
A score on Y has 2 parts (1) linear function of X and (2) error. Y X i i i = + + o | c (population values) Basic Ideas (2) Sample value: Intercept place where X=0 Slope change in Y if X changes 1 unit. Rise over run. If error is removed, we have a predicted value for each person at X (the line):
Y a bX e i i i = + + ' = + Y a bX Suppose on average houses are worth about $75.00 a square foot. Then the equation relating price to size would be Y=0+75X. The predicted price for a 2000 square foot house would be $150,000. Linear Transformation 1 to 1 mapping of variables via line Permissible operations are addition and multiplication (interval data) 1 0 8 6 4 2 0 X 4 0 3 5 3 0 2 5 2 0 1 5 1 0 5 0 Y C h a n g in g t h e Y I n t e r c e p t
Y = 5 + 2 X Y = 1 0 + 2 X Y = 1 5 + 2 X Add a constant 1 0 8 6 4 2 0 X 3 0 2 0 1 0 0 Y C h a n g in g t h e S lo p e
Y = 5 + . 5 X Y = 5 + X Y = 5 + 2 X Multiply by a constant ' = + Y a bX Linear Transformation (2) Centigrade to Fahrenheit Note 1 to 1 map Intercept? Slope? 120 90 60 30 0 Degrees C 240 200 160 120 80 40 0 D e g r e e s
F
32 degrees F, 0 degrees C 212 degrees F, 100 degrees C Intercept is 32. When X (Cent) is 0, Y (Fahr) is 32. Slope is 1.8. When Cent goes from 0 to 100 (run), Fahr goes from 32 to 212 (rise), and 212-32 = 180. Then 180/100 =1.8 is rise over run is the slope. Y = 32+1.8X. F=32+1.8C. ' = + Y a bX Review What are predictors and criteria? Write an equation for the linear regression with 1 IV. Describe each term. How do changes in the slope and intercept affect (move) the regression line? Regression of Weight on Height Ht Wt 61 105 62 120 63 120 65 160 65 120 68 145 69 175 70 160 72 185 75 210 N=10 N=10 M=67 M=150 SD=4.57 SD= 33.99 7 6 7 4 7 2 7 0 6 8 6 6 6 4 6 2 6 0 H e ig h t in I n c h e s 2 4 0 2 1 0 1 8 0 1 5 0 1 2 0 9 0 6 0 W e i g h t
i n
L b s R e g r e s s io n o f W e ig h t o n H e ig h t
R e g r e s s io n o f W e ig h t o n H e ig h t
R e g r e s s io n o f W e ig h t o n H e ig h t
R is e R u n Y = - 3 1 6 . 8 6 + 6 . 9 7 X X Correlation (r) = .94. Regression equation: Y=-316.86+6.97X ' = + Y a bX Illustration of the Linear Model. This concept is vital! 7 2 7 0 6 8 6 6 6 4 6 2 H e ig h t 2 0 0 1 8 0 1 6 0 1 4 0 1 2 0 1 0 0 W e i g h t R e g r e s s io n o f W e ig h t o n H e ig h t
7 2 7 0 6 8 6 6 6 4 6 2 H e ig h t R e g r e s s io n o f W e ig h t o n H e ig h t
R e g r e s s io n o f W e ig h t o n H e ig h t
( 6 5 , 1 2 0 ) M e a n o f X M e a n o f Y D e v ia t io n f r o m X D e v ia t io n f r o m Y L in e a r P a r t E r r o r P a r t y Y ' e Y X i i i = + + o | c Y a bX e i i i = + + Consider Y as a deviation from the mean. Part of that deviation can be associated with X (the linear part) and part cannot (the error).
' = + Y a bX ' i i i Y Y e = Predicted Values & Residuals N
Ht
Wt
Y'
Resid 1
61
105
108.19
-3.19 2
62
120
115.16
4.84 3
63
120
122.13
-2.13 4
65
160
136.06
23.94 5
65
120
136.06
-16.06 6
68
145
156.97
-11.97 7
69
175
163.94
11.06 8
70
160
170.91
-10.91 9
72
185
184.84
0.16 10
75
210
205.75
4.25 M 67
150
150.00
0.00 SD
4.57
33.99
31.85
11.89
V 20.89
1155.56
1014.37
141.32
7 2 7 0 6 8 6 6 6 4 6 2 H e ig h t 2 0 0 1 8 0 1 6 0 1 4 0 1 2 0 1 0 0 W e i g h t R e g r e s s io n o f W e ig h t o n H e ig h t
7 2 7 0 6 8 6 6 6 4 6 2 H e ig h t R e g r e s s io n o f W e ig h t o n H e ig h t
R e g r e s s io n o f W e ig h t o n H e ig h t
( 6 5 , 1 2 0 ) M e a n o f X M e a n o f Y D e v ia t io n f r o m X D e v ia t io n f r o m Y L in e a r P a r t E r r o r P a r t y Y ' e Numbers for linear part and error. Note M of Y and Residuals. Note variance of Y is V(Y) + V(res). ' = + Y a bX Finding the Regression Line Need to know the correlation, SDs and means of X and Y. The correlation is the slope when both X and Y are expressed as z scores. To translate to raw scores, just bring back original SDs for both. N z z r Y X XY
= X Y XY SD SD r b = To find the intercept, use: X b Y a = (rise over run) Suppose r = .50, SD X = .5, M X = 10, SD Y = 2, M Y = 5. 2 5 . 2 5 . = = b 15 ) 10 ( 2 5 = = a X Y 2 15 ' + = Slope Intercept Equation Line of Least Squares 7 2 7 0 6 8 6 6 6 4 6 2 H e ig h t 2 0 0 1 8 0 1 6 0 1 4 0 1 2 0 1 0 0 W e i g h t R e g r e s s io n o f W e ig h t o n H e ig h t
7 2 7 0 6 8 6 6 6 4 6 2 H e ig h t R e g r e s s io n o f W e ig h t o n H e ig h t
R e g r e s s io n o f W e ig h t o n H e ig h t
( 6 5 , 1 2 0 ) M e a n o f X M e a n o f Y D e v ia t io n f r o m X D e v ia t io n f r o m Y L in e a r P a r t E r r o r P a r t y Y ' e We have some points. Assume linear relations is reasonable, so the 2 vbls can be represented by a line. Where should the line go? Place the line so errors (residuals) are small. The line we calculate has a sum of errors = 0. It has a sum of squared errors that are as small as possible; the line provides the smallest sum of squared errors or least squares. Least Squares (2) Review What does it mean to choose a regression line to satisfy the loss function of least squares? What are predicted values and residuals? Suppose r = .25, SD X = 1, M X = 10, SD Y = 2, M Y = 5. What is the regression equation (line)? Partitioning the Sum of Squares e bX a Y + + = bX a Y + = ' e Y Y + = ' ' Y Y e = Definitions ) ' ( ) ' ( Y Y Y Y Y Y + = = y, deviation from mean
+ = 2 2 )] ' ( ) ' [( ) ( Y Y Y Y Y Y Sum of squares
+ = 2 2 2 ) ' ( ) ' ( ) ( Y Y Y Y y (cross products drop out) Sum of squared deviations from the mean = Sum of squares due to regression + Sum of squared residuals reg error Analog: SS tot =SS B +SS W Partitioning SS (2) SS Y =SS Reg + SS Res Total SS is regression SS plus residual SS. Can also get proportions of each. Can get variance by dividing SS by N if you want. Proportion of total SS due to regression = proportion of total variance due to regression = R 2
(R-square). Y s Y g Y Y SS SS SS SS SS SS Re Re + = ) 1 ( 1 2 2 R R + = Partitioning SS (3) YYY Wt (Y) M=150
Y'
Resid (Y-Y')
Resid 2
105
2025
108.19
-41.81
1748.076
-3.19
10.1761
120
900
115.16
-34.84
1213.826
4.84
23.4256
120
900
122.13
-27.87
776.7369
-2.13
4.5369
160
100
136.06
-13.94
194.3236
23.94
573.1236
120
900
136.06
-13.94
194.3236
-16.06
257.9236
145
25
156.97
6.97
48.5809
-11.97
143.2809
175
625
163.94
13.94
194.3236
11.06
122.3236
160
100
170.91
20.91
437.2281
-10.91
119.0281
185
1225
184.84
34.84
1213.826
0.16
0.0256
210
3600
205.75
55.75
3108.063
4.25
18.0625
Sum = 1500
10400
1500.01
0.01
9129.307
-0.01
1271.907
Variance
1155.56
1014.37
141.32
2 ) ( Y Y Y Y ' 2 ) ' ( Y Y Partitioning SS (4) Total Regress Residual SS 10400 9129.31 1271.91 Variance 1155.56 1014.37 141.32 12 . 88 . 1 10400 91 . 1271 10400 31 . 9129 10400 10400 + = + = Proportion of SS 12 . 88 . 1 56 . 1155 32 . 141 56 . 1155 37 . 1014 56 . 1155 56 . 1155 + = + = Proportion of Variance R 2 = .88 Note Y is linear function of X, so . XY YY r r = = 94 . ' 1 ' = X Y r 0 12 . 35 . . 88 . ' 2 2 2 ' = = = = = E Y YE YE YY r r r R r Significance Testing Testing for the SS due to regression = testing for the variance due to regression = testing the significance of R 2 . All are the same. 0 : 2 0 = population R H F SS df SS df SS k SS N k reg res reg res = =
/ / / / ( ) 1 2 1 k=number of IVs (here its 1) and N is the sample size (# people). F with k and (N-k-1) df. F SS df SS df reg res = =
= k N R k R F Equivalent test using R-square instead of SS. F =
= . / ( . ) / ( ) . 88 1 1 88 10 1 1 58 67 Results will be same within rounding error. Review What does it mean to test the significance of the regression sum of squares? R-square? What is R-square? Why does testing for the regression sum of squares turn out to have the same result as testing for R-square?