Type author name/s here

Dougherty

Introduction to Econometrics,
5th edition
Chapter heading
Chapter 4: Nonlinear Models and
Transformations of Variables

© Christopher Dougherty, 2016. All rights reserved.

 NONLINEAR REGRESSION

Y  1   2 X 3  u

Suppose you believe that a variable Y depends on a variable X according to the relationship
shown and you wish to obtain estimates of b1, b2, and b3 given data on Y and X.

1
 NONLINEAR REGRESSION

Y  1   2 X 3  u

There is no way of transforming the relationship to obtain a linear relationship, and so it is

not possible to apply the usual regression procedure.

2
 NONLINEAR REGRESSION

Y  1   2 X 3  u

Nevertheless, one can still use the principle of minimizing the sum of the squares of the
residuals to obtain estimates of the parameters. We will describe a simple nonlinear
regression algorithm that uses the principle. It consists of a series of repeated steps.
3
 NONLINEAR REGRESSION

Y  1   2 X 3  u

Nonlinear regression algorithm

1. Guess b1, b2, and b3. ̂ 1 , ˆ2 , and ̂ 3 are the guesses.

You start by guessing plausible values for the parameters.

4
 NONLINEAR REGRESSION

Y  1   2 X 3  u

Nonlinear regression algorithm

1. Guess b1, b2, and b3. ̂ 1 , ˆ2 , and ̂ 3 are the guesses.
ˆi  ˆ1  ˆ2 X iˆ3
Y
2. Calculate for each observation.

You calculate the corresponding fitted values of Y from the data on X, conditional on these
values of the parameters.

5
 NONLINEAR REGRESSION

Y  1   2 X 3  u

Nonlinear regression algorithm

1. Guess b1, b2, and b3. ̂ 1 , ˆ2 , and ̂ 3 are the guesses.
Yˆi  ˆ1  ˆ2 X iˆ3
2. Calculate for each observation.
ˆ
3. Calculate uˆ i  Yi  Yi for each observation.
4. Calculate RSS   uˆ i .
2

You calculate the residual for each observation in the sample, and hence RSS, the sum of
the squares of the residuals.

6
 NONLINEAR REGRESSION

Y  1   2 X 3  u

Nonlinear regression algorithm

1. Guess b1, b2, and b3. ̂ 1 , ˆ2 , and ̂ 3 are the guesses.
Yˆi  ˆ1  ˆ2 X iˆ3
2. Calculate for each observation.
ˆ
3. Calculate uˆ i  Yi  Yi for each observation.
4. Calculate RSS   uˆ i .
2

ˆ
5. Adjust ̂ ,  , and  .
ˆ
1 2 3

You then make small changes in one or more of your estimates of the parameters.

7
 NONLINEAR REGRESSION

Y  1   2 X 3  u

Nonlinear regression algorithm

Nonlinear regression algorithm
1. Guess b1, b2, and b3. ̂ 1 , ˆ2 , and ̂ 3 are the guesses.
1. GuessYˆb1,b
2ˆ  ˆb3X. b31, b2, and b3 are the guesses.
, and ˆ
2. Calculate i 1 2 i for each observation.
b1Yˆ+ b2Xi for each observation.
3. Calculate uˆ i  Y
2. Calculate Yi = 
i i for each observation.
RSSei 
3. Calculate
4. Calculate  2
ˆYi i . for each observation.
= Yi –u
4. Calculate
5. Adjust
ˆ
̂ 1 ,  2 ,RSS ˆ
and=∑e .
3 .i

6. Re-calculate ˆand
5. Adjust b1,Yb,2,u b3. .
i , RSS
i
6. Re-calculate Yi, ei, RSS.

Using the new estimates of b1, b2, and b3, you re-calculate the fitted values of Y. Then you
re-calculate the residuals and RSS.

8
 NONLINEAR REGRESSION

Y  1   2 X 3  u

Nonlinear regression algorithm

1. Guess b1, b2, and b3. ̂ 1 , ˆ2 , and ̂ 3 are the guesses.
Yˆi  ˆ1  ˆ2 X iˆ3
2. Calculate for each observation.
ˆ
3. Calculate uˆ i  Yi  Yi for each observation.
4. Calculate RSS   uˆ i .
2

ˆ
5. Adjust ̂ ,  , and  .
ˆ
1 2 3

6. Re-calculate Yi, uˆ i , RSS.

7. If new RSS < old RSS, continue adjustment.
Otherwise try different adjustment.

If RSS is smaller than before, your new estimates of the parameters are better than the old
ones and you continue adjusting your estimates in the same direction. Otherwise, you
would try different adjustments.
9
 NONLINEAR REGRESSION

Y  1   2 X 3  u

Nonlinear regression algorithm

1. Guess b1, b2, and b3. ̂ 1 , ˆ2 , and ̂ 3 are the guesses.
Yˆi  ˆ1  ˆ2 X iˆ3
2. Calculate for each observation.
ˆ
3. Calculate uˆ i  Yi  Yi for each observation.
4. Calculate RSS   uˆ i .
2

ˆ
5. Adjust ̂ ,  , and  .
ˆ
1 2 3

6. Re-calculate Yi, uˆ i , RSS.

7. If new RSS < old RSS, continue adjustment.
Otherwise try different adjustment.
8. Repeat steps 5, 6, and 7 to convergence.

You repeat steps 5, 6, and 7 again and again until you are unable to make any changes in
the estimates of the parameters that would reduce RSS.

10
 NONLINEAR REGRESSION

Y  1   2 X 3  u

Nonlinear regression algorithm

1. Guess b1, b2, and b3. ̂ 1 , ˆ2 , and ̂ 3 are the guesses.
Yˆi  ˆ1  ˆ2 X iˆ3
2. Calculate for each observation.
ˆ
3. Calculate uˆ i  Yi  Yi for each observation.
4. Calculate RSS   uˆ i .
2

ˆ
5. Adjust ̂ ,  , and  .
ˆ
1 2 3

6. Re-calculate Yi, uˆ i , RSS.

7. If new RSS < old RSS, continue adjustment.
Otherwise try different adjustment.
8. Repeat steps 5, 6, and 7 to convergence.

You conclude that you have minimized RSS, and you can describe the final estimates of the
parameters as the least squares estimates.

11
 NONLINEAR REGRESSION

Y  1   2 X 3  u

Nonlinear regression algorithm

1. Guess b1, b2, and b3. ̂ 1 , ˆ2 , and ̂ 3 are the guesses.
Yˆi  ˆ1  ˆ2 X iˆ3
2. Calculate for each observation.
ˆ
3. Calculate uˆ i  Yi  Yi for each observation.
4. Calculate RSS   uˆ i .
2

ˆ
5. Adjust ̂ ,  , and  .
ˆ
1 2 3

6. Re-calculate Yi, uˆ i , RSS.

7. If new RSS < old RSS, continue adjustment.
Otherwise try different adjustment.
8. Repeat steps 5, 6, and 7 to convergence.

It should be emphasized that, long ago, mathematicians devised sophisticated techniques

to minimize the number of steps required by algorithms of this type.

12
 NONLINEAR REGRESSION

2
e  1  u
g
3
Employment growth rate

0
0 1 2 3 4 5 6 7 8 9
GDP growth rate

We will return to the relationship between employment growth rate, e, and GDP growth rate,
g, in the first slideshow for this chapter. e and g are hypothesized to be related as shown.

13
 NONLINEAR REGRESSION

2
e  1  u
g
3
Employment growth rate

0
0 1 2 3 4 5 6 7 8 9
GDP growth rate

According to this specification, as g becomes large, e will tend to a limit of b1. Looking at
the figure, we see that the maximum value of e is about 3. So we will take this as our initial
value for b1. We then hunt for the optimal value of b2, conditional on this guess for b1.
14
 NONLINEAR REGRESSION

400

Conditional on ˆ1  3
RSS

300

200
-7 -6 -5 –4.22 -4 -3 -2 -1 0
estimate of b 2

The figure shows RSS plotted as a function of ˆ2 , conditional on ˆ1 = 3. From this we see
that the optimal value of ˆ2 , conditional on ̂ 1 = 3, is –4.22.

15
 NONLINEAR REGRESSION

200

Conditional on ˆ2  4.22

RSS

100

0
1 2 2.82 3 4
estimate of b1
b1

Next, holding ˆ2 at –4.22, we look to improve on our guess for ̂ 1 . The figure shows RSS as
a function of ̂ 1 , conditional on ̂ 2 = –4.22. We see that the optimal value of b1 is 2.82.

16
 NONLINEAR REGRESSION

200

Conditional on ˆ2  4.22

RSS

100

0
1 2 2.82 3 4
estimate of b1
b1

We continue to do this both parameter estimates converge on limits and cease to change.
We will then have reached the values that yield minimum RSS.

17
 NONLINEAR REGRESSION

200

Conditional on ˆ2  4.22

RSS

100

0
1 2 2.82 3 4
estimate of b1
b1

The limits must be the values from the transformed linear regression shown in the first
slideshow for this chapter: ̂ 1 = 2.18 and ̂ 2 = –2.36. They have been determined by the
same criterion, the minimization of RSS. All that we have done is to use a different method.
18
 NONLINEAR REGRESSION

2
. nl (e = {beta1} + {beta2}/g)
(obs = 31) e  1  u
Iteration 0: residual SS = 12.30411
g
Iteration 1: residual SS = 12.30411
----------------------------------------------------------------------------
Source | SS df MS
-----------+------------------------------ Number of obs = 31
Model | 5.80515805 1 5.80515805 R-squared = 0.3206
Residual | 12.304107 29 .42427955 Adj R-squared = 0.2971
-----------+------------------------------ Root MSE = .6513674
Total | 18.109265 30 .603642167 Res. dev. = 59.32851
----------------------------------------------------------------------------
e | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-----------+----------------------------------------------------------------
/beta1 | 2.17537 .249479 8.72 0.000 1.665128 2.685612
/beta2 | -2.356136 .6369707 -3.70 0.001 -3.658888 -1.053385
----------------------------------------------------------------------------

Here is the output for the present hyperbolic regression of e on g using nonlinear
regression. It is, as usual, Stata output, but output from other regression applications will
look similar.
19
 NONLINEAR REGRESSION

2
. nl (e = {beta1} + {beta2}/g)
(obs = 31) e  1  u
Iteration 0: residual SS = 12.30411
g
Iteration 1: residual SS = 12.30411
----------------------------------------------------------------------------
Source | SS df MS
-----------+------------------------------ Number of obs = 31
Model | 5.80515805 1 5.80515805 R-squared = 0.3206
Residual | 12.304107 29 .42427955 Adj R-squared = 0.2971
-----------+------------------------------ Root MSE = .6513674
Total | 18.109265 30 .603642167 Res. dev. = 59.32851
----------------------------------------------------------------------------
e | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-----------+----------------------------------------------------------------
/beta1 | 2.17537 .249479 8.72 0.000 1.665128 2.685612
/beta2 | -2.356136 .6369707 -3.70 0.001 -3.658888 -1.053385
----------------------------------------------------------------------------

The Stata command for a nonlinear regression is ‘nl’. This is followed by the hypothesized
mathematical relationship within parentheses. The parameters must be given names
placed within braces. Here b1 is {beta1} and b2 is {beta2}.
20
 NONLINEAR REGRESSION

2
. gen z = 1/g
e  1  u
. reg e z g
----------------------------------------------------------------------------
Source | SS df MS Number of obs = 31
-----------+------------------------------ F( 1, 29) = 13.68
Model | 5.80515811 1 5.80515811 Prob > F = 0.0009
Residual | 12.3041069 29 .424279548 R-squared = 0.3206
-----------+------------------------------ Adj R-squared = 0.2971
Total | 18.109265 30 .603642167 Root MSE = .65137
----------------------------------------------------------------------------
e | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-----------+----------------------------------------------------------------
z | -2.356137 .6369707 -3.70 0.001 -3.658888 -1.053385
_cons | 2.17537 .249479 8.72 0.000 1.665128 2.685612
----------------------------------------------------------------------------
2.36
eˆ  2.18  2.36 z  2.18 
g

The output is effectively the same as the linear regression output in the first slideshow for
this chapter.

21
 NONLINEAR REGRESSION

2
4 e  1  u
g
3
Employment growth rate

0
0 1 2 3 4 5 6 7 8 9

-1

-2

-3
GDP growth rate

The hyperbolic function imposes the constraint that the function plunges to minus infinity
for positive g as g approaches zero.

22
 NONLINEAR REGRESSION

2
. nl (e = {beta1} + {beta2}/({beta3} + g))
e  1  u
(obs = 31)
Iteration 0: residual SS = 12.30411
3  g
Iteration 1: residual SS = 12.27327
.....................................
Iteration 8: residual SS = 11.98063
----------------------------------------------------------------------------
Source | SS df MS
-----------+------------------------------ Number of obs = 31
Model | 6.12863996 2 3.06431998 R-squared = 0.3384
Residual | 11.9806251 28 .427879466 Adj R-squared = 0.2912
-----------+------------------------------ Root MSE = .654125
Total | 18.109265 30 .603642167 Res. dev. = 58.5026
----------------------------------------------------------------------------
e | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-----------+----------------------------------------------------------------
/beta1 | 2.714411 1.017058 2.67 0.013 .6310616 4.79776
/beta2 | -6.140415 8.770209 -0.70 0.490 -24.10537 11.82454
/beta3 | 1.404714 2.889556 0.49 0.631 -4.514274 7.323702
----------------------------------------------------------------------------

This feature can be relaxed by using the variation shown. Unlike the previous function, this
cannot be linearized by any kind of transformation. Here, nonlinear regression must be
used.
23
 NONLINEAR REGRESSION

2
. nl (e = {beta1} + {beta2}/({beta3} + g))
e  1  u
(obs = 31)
Iteration 0: residual SS = 12.30411
3  g
Iteration 1: residual SS = 12.27327
.....................................
Iteration 8: residual SS = 11.98063
----------------------------------------------------------------------------
Source | SS df MS
-----------+------------------------------ Number of obs = 31
Model | 6.12863996 2 3.06431998 R-squared = 0.3384
Residual | 11.9806251 28 .427879466 Adj R-squared = 0.2912
-----------+------------------------------ Root MSE = .654125
Total | 18.109265 30 .603642167 Res. dev. = 58.5026
----------------------------------------------------------------------------
e | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-----------+----------------------------------------------------------------
/beta1 | 2.714411 1.017058 2.67 0.013 .6310616 4.79776
/beta2 | -6.140415 8.770209 -0.70 0.490 -24.10537 11.82454
/beta3 | 1.404714 2.889556 0.49 0.631 -4.514274 7.323702
----------------------------------------------------------------------------

The output for this specification is shown, with most of the iteration messages deleted.

24
 NONLINEAR REGRESSION

2
4 e  1  u
3  g
3
(4.47)
Employment growth rate

(4.46)
1

0
0 1 2 3 4 5 6 7 8 9

-1

-2

-3
GDP growth rate

The figure compares the original (black) and new (red) hyperbolic functions. The overall fit
is not greatly improved, but the specification does seem more satisfactory.

25
Copyright Christopher Dougherty 2016.

These slideshows may be downloaded by anyone, anywhere for personal use.

Subject to respect for copyright and, where appropriate, attribution, they may be
used as a resource for teaching an econometrics course. There is no need to
refer to the author.

The content of this slideshow comes from Section 4.4 of C. Dougherty,

Introduction to Econometrics, fifth edition 2016, Oxford University Press.
Additional (free) resources for both students and instructors may be
downloaded from the OUP Online Resource Centre
www.oxfordtextbooks.co.uk/orc/dougherty5e/.

Individuals studying econometrics on their own who feel that they might benefit
from participation in a formal course should consider the London School of
Economics summer school course
EC212 Introduction to Econometrics
http://www2.lse.ac.uk/study/summerSchools/summerSchool/Home.aspx
or the University of London International Programmes distance learning course
EC2020 Elements of Econometrics
www.londoninternational.ac.uk/lse.

2016_05_03