Generalized Linear

Models

Chapter 7
Generalized Linear Models
EEE 485/585 Statistical Learning and Data Analytics The classification
problem

Simple logistic
regression

Example 1

Multiple logistic
regression

Example 2

Multiple logistic
regression - extensions

Generalized linear
models

Poisson regression

Cem Tekin
Bilkent University

Cannot be distributed outside this class without the permission of the

instructor. 7.1
 Generalized Linear
The classification problem Models

Response Y is categorical.
Y 2 {influenza, common cold, Covid-19} (3 classes)
Predictors: X1 , . . . , Xp
Can we use linear regression?
The classification
problem

Simple logistic
regression

Example 1

Multiple logistic
regression

Example 2

Multiple logistic
regression - extensions

Generalized linear
models

Poisson regression

7.2
 Generalized Linear
The classification problem Models

Response Y is categorical.
Y 2 {influenza, common cold, Covid-19} (3 classes)
Predictors: X1 , . . . , Xp
Can we use linear regression?
The classification
problem
Map classes to real values? Simple logistic
regression
8
<1 if influenza
> Example 1

Multiple logistic
Y = 2 if common cold regression
>
:
3 if Covid-19 Example 2

Multiple logistic
regression - extensions
Use least squares, ridge or lasso Generalized linear
models

Poisson regression

7.2
 Generalized Linear
The classification problem Models

Response Y is categorical.
Y 2 {influenza, common cold, Covid-19} (3 classes)
Predictors: X1 , . . . , Xp
Can we use linear regression?
The classification
problem
Map classes to real values? Simple logistic
regression
8
<1 if influenza
> Example 1

Multiple logistic
Y = 2 if common cold regression
>
:
3 if Covid-19 Example 2

Multiple logistic
regression - extensions
Use least squares, ridge or lasso Generalized linear
models

What if Poisson regression

8
<1 if common cold
>
Y = 2 if Covid-19
>
:
3 if influenza
7.2
 Generalized Linear
Problem with linear regression Models

Points on real line are ordered

Sometimes there is no natural way to order the classes
C = {influenza, common cold, Covid-19}
Linear regression coefficients will depend on our mapping
C ! {1, 2, 3} ⇢ R The classification
problem

Give some examples of classes which have a natural order Simple logistic
regression

Example 1

movie ratings
1,2191
Multiple logistic
regression

Example 2

Multiple logistic
regression - extensions

Generalized linear
models

Poisson regression

7.3
 Generalized Linear
Should we use linear regression for classification Models

Yes, if C is binary, then ordering does not matter

Example: C = {influenza, Covid-19}, then
(
0 if influenza
Y =
1 if Covid-19 The classification
problem

Simple logistic
and regression

( Example 1

0 if Covid-19 Multiple logistic

Y = regression

1 if influenza Example 2

Multiple logistic
regression - extensions
will produce the same result

IF 1
Generalized linear

Oi
5
models

Poisson regression

predictions are not proboblis

7.4
 Generalized Linear
Simple logistic regression Models

Y 2 {0, 1}. Ex: 1 = Covid-19, 0 = no Covid-19

X 2R

e 0+ 1X
⇡(X ) = (logistic function)
1 + e 0+ 1X The classification
problem
limX ! 1 ⇡(X )? limX !+1 ⇡(X )? Simple logistic
regression
⇡(X ) models Pr(Y = 1|X ) Example 1

Multiple logistic

L
regression

II Fat fist
0 o
Ri
Example 2

Multiple logistic
regression - extensions

Generalized linear

not
gig
models

Poisson regression

or 7.5
 Generalized Linear
Simple logistic regression Models

Odds:

P.MS ⇡(X )
=e 0+ 1X
1 ⇡(X )
PrCUoHI
Log-odds (logit): The classification
problem
✓ ◆ Simple logistic
⇡(X ) regression
logit(⇡(X )) = log = 0+ 1X
1 ⇡(X ) Example 1

Multiple logistic
regression
logit is linear in X ! Example 2

Multiple logistic
regression - extensions

Generalized linear
models

Poisson regression

7.6
 Generalized Linear
MLE for simple logistic regression Models

Data: D = {(xi , yi )}ni=1

lottiti
Model: Y1 , . . . , Yn are independent. Yi ⇠ Bernoulli(⇡(xi ))
log I MGill log It e
II BenomiCeGill
Likelihood:

The classification
Y Y problem
L( 0, 1) = p(D| 1, 0) = ⇡(xi ) (1 ⇡(xi 0 )) Simple logistic
regression
i:yi =1 i 0 :yi 0 =0
Example 1

j
Log-likelihood: Multiple logistic
regression

Example 2

Multiple logistic
n
X regression - extensions
0 + 1 xi
l( 0, 1) = log p(D| 1, 0) = [yi ( 0 + 1 xi ) log(1 + e )] Generalized linear
models
i=1
Poisson regression

MLE:

( ˆ0MLE , ˆ1MLE ) = arg max L( 0, 1) = arg max l( 0, 1)

0, 1 0, 1

7.7
 Generalized Linear
MLE for simple logistic regression Models

No closed form solution for ( ˆ0MLE , ˆ1MLE )

MLE can be found by Newton-Raphson method
or
gradient descent The classification
problem

Simple logistic
regression

Example 1

Multiple logistic
regression

Example 2

Multiple logistic
regression - extensions

Generalized linear
models

Poisson regression

7.8
 Generalized Linear
Simple logistic regression - Example Models

Credit card default prediction

X : credit card balance
Y : 1 if default on credit card payment, 0 else
⇡(X ) = Pr(default = 1|balance = X )

dearie The classification

problem
1.0

thresher
|||||||||||||||
|| ||||||| ||||||||||||||||||||||||| | |||| | | | | | | | | | | || | ||||| || ||| ||||||||||| ||||| |||||||||||||||||| |||||||| ||||||||||| |||||||||||||||||||||||||||| ||||||||||||||||||||||||| | |||| | | | | | |
Probability of Default

Simple logistic
regression
0.8

7 I
Example 1

0.5
0.6

Multiple logistic
regression

7 O
0.4

Example 2

pros eye
Multiple logistic
0.2

regression - extensions

Generalized linear

I
0.0

|||||||||||||||||| |||||||| || || ||||| | | || ||||||||||||||||||||||||||||||||||||||||||||||||||||||

||||||||||||||
|||||||||||||||||||||||||||||
||||||||||||||||||||||||||||||||||||||||||||
||||
|||||||
||||
||||||||
||||||||||||
||||||||||||
||||||
||||||||||
||||||||||||
||||||||||||||||||||
|||||
|||
||||||||||||||||||||||
||||
||||||||||||||
|||||||||||||||||
|||
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|||||||||||||||||||||||
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| |||||| || || ||||| | | || models

Poisson regression

2000 2500 0 500 1000 1500 2000 2500

Balance
ˆ MLE + ˆ MLE x
1 e 0
Orange: data points. Blue: ⇡(x) = ˆ MLE + ˆ MLE x
(logistic function)
1+e 0 1

Figure 4.2 from “An introduction to statistical learning" by James et al. 7.9
 Generalized Linear
Simple logistic regression - Example Models

ˆMLE = 10.65
0
ˆMLE = 0.0055
1
Log-odds:
✓ ◆
Pr(default = 1|balance = X ) The classification
log = 10.65 + 0.0055X problem
Pr(default = 0|balance = X ) Simple logistic
regression

Example 1

Multiple logistic
regression

Example 2

Multiple logistic
regression - extensions

Generalized linear
models

Poisson regression

7.10
 Generalized Linear
Simple logistic regression - Example Models

ˆMLE = 10.65
0
ˆMLE = 0.0055
1
Log-odds:
✓ ◆
Pr(default = 1|balance = X ) The classification
log = 10.65 + 0.0055X problem
Pr(default = 0|balance = X ) Simple logistic
regression

Making predictions: Example 1

Multiple logistic
1 What is the probability that an individual with a balance of regression

$1000 defaults? Example 2

Multiple logistic
2 What is the probability that an individual with a balance of regression - extensions

$2000 defaults? Generalized linear

models

Poisson regression

7.10
 Generalized Linear
Simple logistic regression - Example Models

ˆMLE = 10.65
0
ˆMLE = 0.0055
1
Log-odds:
✓ ◆
Pr(default = 1|balance = X ) The classification
log = 10.65 + 0.0055X problem
Pr(default = 0|balance = X ) Simple logistic
regression

Making predictions: Example 1

Multiple logistic
1 What is the probability that an individual with a balance of regression

$1000 defaults? Example 2

Multiple logistic
2 What is the probability that an individual with a balance of regression - extensions

$2000 defaults? Generalized linear

models

Poisson regression
e 10.65+0.0055⇥1000
1 ⇡(1000) = 1+e 10.65+0.0055⇥1000
⇡ 0.0058 = 0.58%
e 10.65+0.0055⇥2000
2 ⇡(2000) = 1+e 10.65+0.0055⇥2000
⇡ 0.586 = 58.6%

7.10
 Generalized Linear
Simple logistic regression - Example Models

Predicting Y directly given X = x:

ghettos positive
Risk neutral:

ŷ =
( i
0 if ⇡(x) < 0.5
n too
o N96
1 if ⇡(x) 0.5 The classification
problem s P 4
96
g 0 for all data points aceracy
Risk averse: Simple logistic
regression
( Example 1
0 if ⇡(x) < 0.1
ŷ = Multiple logistic
regression
Conform matrix
1 if ⇡(x) 0.1
Example 2
predictions Twtreenegative
4 Multiple logistic
regression - extensions Fp false positive
carefullytuned

IIE
Generalized linear
models

Poisson regression
TP true positive

FPOFAY
pro Far S Faizfalsenegativerole
7.11
 Generalized Linear
Simple logistic regression - Example Models

Predictor can also be categorical LRwith ocees

X = 1 student, X = 0 no student
predictor
Same MLE formula holds
b FPR
MLE solution:
Faiz For zig
I gÉ
The classification
problem
ˆMLE = 3.5041

y
0 Simple logistic
regression
ˆMLE = 0.4049
1 Example 1

3.5041+0.4049 Multiple logistic

e
1 ⇡(student) = 1+e 3.5041+0.4049 ⇡ 0.04 regression

Example 2
e 3.5041
2 ⇡(no student) = 1+e 3.5041
⇡ 0.03 Multiple logistic
regression - extensions

Generalized linear
models

Poisson regression

7.12
 Generalized Linear
Multiple logistic regression Models

Response: Y 2 {0, 1}
Predictors: X = [1, X1 , X2 , . . . , Xp ]T
T
Parameters: =[ 0, 1, . . . , p]
Logistic function:
The classification
problem
T
e 0 + 1 X1 +...+ p Xp e X Simple logistic
⇡(X ; ) = = regression
1 + e 0 + 1 X1 +...+ p Xp T
1+e X Example 1

Multiple logistic
⇡(X ; ) models Pr(Y = 1|X1 , X2 , . . . , Xp ; ) regression

Example 2

Multiple logistic
regression - extensions

Generalized linear
models

Poisson regression

7.13
 Generalized Linear
Multiple logistic regression Models

Odds:
⇡(X ; ) T
X
=e
1 ⇡(X ; )

Log-odds (logit): The classification

problem
✓ ◆ Simple logistic
⇡(X ; ) T regression
logit(⇡(X ; )) = log = X
1 ⇡(X ; ) Example 1

Multiple logistic
regression
logit is linear in X ! Example 2

Multiple logistic
regression - extensions

Generalized linear
models

Poisson regression

7.14
 Generalized Linear
MLE for multiple logistic regression

itt.gg
FYFEI
Models

Data: D = {(x i , yi )}ni=1 , x i = [1, xi1 , . . . , xip ]T

Model: Y1 , . . . , Yn are independent.
godfind minor maxoff
Yi ⇠ Bernoulli(⇡(X i ; ))
The classification ft o at extremr
find rootsof flako
Log-likelihood: problem

Simple logistic

Let w be set Fail o

n regression
X T
T xi
l( ) = log p(D| ) = [yi xi log(1 + e )] Example 1

i=1
Multiple logistic
let we be or initid at no
winterguytay
regression
expansion
MLE: flints Example 2

Multiple logistic Ann w

ud
of Cw f lastDulaf'CultBuf
regression - extensions

ˆ MLE = arg max l( ) Generalized linear

models
2Rp+1

f
ar
I
Poisson regression

DILI Q solve for Ime

weneed amerid
methods
i
etfl.tt
optimization

7.15
eotirae
of
 flail f lust
Generalized Linear
Newton-Raphson method for multiple logistic regression
2 3 2 3 2 3
Models
we w
1 x11 . . . x1p ⇡(x1 ; old ) y1
61 x21 . . . x2p 7 6⇡(x2 ; old )7 6 y2 7
6 7 6 7 6 7
cent
anti un f un f
X = 6. .. .. .. 7 ⇡ = 6 .. 7 y = 6 .. 7
4 .. . . . 5 4 . 5 4.5 Neetaupdate
1 xn1 ... xnp old yn
⇡(xn ; )
W=
2
⇡(x1 ; old
)(1
0
⇡(x1 ; old ))
⇡(x2 ; old )(1
0
old
... 0
0
3 The classification
problem Wnw anti a sty
6 ⇡(x2 ; )) . . . 7 Simple logistic
6
4 .. .. ..
.
.. 7 regression
5 Example 1 Newton update with legningrah
. . .
old old

anti un e f Califcul CE10,1

0 0 . . . ⇡(xn ; )(1 ⇡(xn ; )) Multiple logistic
regression

Example 2

for f RDR
Multiple logistic
regression - extensions Newton update
Generalized linear
models

en get
offing't
cent
Poisson regression
Intl
intern
of
Heroics
II t of second
derivatives
order parts
f f
7.16
e
 Hi Dif dfw
Iffy
Generalized Linear
Newton-Raphson method for multiple logistic regression Models
2 3 2 3 2 3
1 x11 . . . x1p ⇡(x1 ; old ) y1
61 x21 . . . x2p 7 6⇡(x2 ; old )7 6 y2 7
6 7 6 7 6 7
X = 6. .. .. .. 7 ⇡ = 6 .. 7 y = 6 .. 7
4 .. . . . 5 4 . 5 4.5
1 xn1 ... xnp old yn

iii i
⇡(xn ; )
W=
2 old
pied 3 The classification converges in a few iterations
⇡(x1 ; old ))
corpse cost it large
⇡(x1 ; )(1 0 ... 0 problem
iteration
6 0 ⇡(x2 ; old )(1 ⇡(x2 ; old
)) . . . 0 7 Simple logistic per
lexpecilywhen plarge
6 .. .. .. .. 7 regression
4 . . . .
5 Example 1
old old
0 0 . . . ⇡(xn ; )(1 ⇡(xn ; )) Multiple logistic
regression

Example 2

Newton
new
= old
+ (XT WX) 1
XT (y ⇡) Multiple logistic
regression - extensions
T 1 T
= (X WX) X W z Generalized linear
| {z } models
weighted least squares converged to a loadmin
Poisson regression
Offenko
old 1
where z = X +W (y ⇡) is the adjusted response.
cheaper periteration
convergence usually takes a larger of
number iterations

7.16 I I f l
 Generalized Linear
Newton-Raphson method for multiple logistic regression Models

new
= (XT WX) 1
XT Wz is the solution to
new
arg min(z X )T W(z X )

This is called iteratively reweighted least squares.

The classification
How to start? Ex: take = [0, 0, . . . , 0]T initially problem

Simple logistic
Update until convergence regression

withorigindleostsquo Example 1

Multiple logistic
regression
Corpor

ly Xfl
Example 2

angry CyXII Multiple logistic

regression - extensions

Generalized linear
models

Poisson regression

7.17
 Generalized Linear
Multiple logistic regression example Models

X1 : credit card balance

X2 : student status dent
Oinotot Littdent
X3 : income
Y : 1 if default on credit card payment, 0 else
The classification
logit(⇡(X )) = 0 + 1 X1 + 2 X2 + 3 X3
problem

Simple logistic
MLE estimate:
man
regression

than.gs9E
i
Example 1

Input varibles ˆMLE ˆMLE ˆMLE ˆMLE Multiple logistic

0 1 2 3 regression
X1 , X2 and X3 -10.8690 0.0057 -0.6468 0.0030 Example 2
X1 only -10.6513 0.0055 N/A N/A Multiple logistic

X2 only -3.5041 N/A 0.4049 N/A regression - extensions

Generalized linear
students on my likely

6
models

Poisson regression
Anything wrong?
On average students are more likely to default than
non-students
in
Given a certain balance and income students are less
likely to default
7.18
 Generalized Linear
Multiple logistic regression example Models

2500
0.8

Credit Card Balance

2000
0.6
Default Rate

1500
The classification
0.4

problem

1000
Simple logistic
regression
0.2

500
Example 1
0.0

Multiple logistic

0
regression
500 1000 1500 2000 No Yes
Example 2
Credit Card Balance Student Status
Multiple logistic
regression - extensions

Orange: student. Blue: non-student Generalized linear

models

Dashed lines: average default rates Poisson regression

Solid lines: default rates based on balance

Conclusion:
Student & balance are correlated

Figure 4.3 from “An introduction to statistical learning" by James et al. 7.19
 Generalized Linear
Boxplot example for a Gaussian distribution Models

The classification
problem

Simple logistic
regression

Example 1

Multiple logistic
regression

Example 2

Multiple logistic
regression - extensions

Generalized linear
models

Poisson regression

7.20
 Generalized Linear
l1 regularized logistic regression

IIE
Models

Optimization problem:

0 1
BX p C
MLE ˆ MLE
B n T T
xi
X C
ˆ
( 0 , ) = arg max B
B [yi xi log(1 + e )] | j |C
C The classification
0, @ i=1 j=1 A problem
| {z } Simple logistic
regression
log-likelihood
Example 1

Multiple logistic
What happens as ! 0?, ! 1? regression

I
Example 2

a Multiple logistic
regression - extensions

no regulara Generalized linear

models

Poisson regression

7.21
 Generalized Linear
l2 regularized logistic regression Models

Optimization problem:

0 1
BX p C
MLE ˆ MLE
B n T T
xi
X C
2C
ˆ
( 0 , ) = arg max B
B [yi xi log(1 + e )] j C The classification
0, @ i=1 j=1 A problem
| {z } Simple logistic
regression
log-likelihood
Example 1

Multiple logistic
What happens as ! 0?, ! 1? regression

Example 2

Multiple logistic
regression - extensions

Generalized linear
models

Poisson regression

7.22
 Generalized Linear
Logistic regression for K > 2 classses Models

Response: Y 2 C, C = {1, 2, . . . , K }.
Predictors: X = [1, X1 , X2 , . . . , Xp ]T O
T
Parameters: j =[ j0 , j1 , . . . , jp ] , j = 1, . . . , K

preach 1 oneut all classification The classification

problem

canbe used with any binary classifier Simple logistic

regression

Example 1

Need K classifiers Multiple logistic

1 or 12 1kt
regression

terrifies I Example 2

T T Multiple logistic

t
regression - extensions

label I label o strange Generalized linear

models

11 1k j
Poisson regression

lossifier j j or
i
label I label o trainstgH
7.23
Alternative
 J
lil
Generalized Linear
Logistic regression for K > 2 classses Models

Binary logistic regression PLY

Bernoulli
HE 1l
Response: Y 2 C, C = {1, 2, . . . , K }. Multinomial k 21 login regression
irk 011
Predictors: X = [1, X1 , X2 , . . . , Xp ]T
Parameters: j =[ j0 , j1 , . . . , jp ]
T
, j = 1, . . . , K PCIElMultinoll RICEQIN Ql
Model:
The classification
problem
PLEIN PEKI
III rge.ei ettfy.GE
Simple logistic

Éf
Pr(Y = 1|X = x) regression

log
T
log = 1x Example 1
Pr(Y = K |X = x) Multiple logistic
Pr(Y = 1|X = x) T
regression

2x
by ME
log = Example 2
Pr(Y = K |X = x)
..
.
Multiple logistic
regression - extensions

Generalized linear
Q Ifj a learned
models
k
Eyes are rise c
Pr(Y = 1|X = x) T Poisson regression
log = K 1x
Pr(Y = K |X = x)

T T T
Parameter vector ✓ = { 1, 2,..., K 1}

7.23
like D
Logistic regression for K > 2 classses
Generalized Linear
Models
I
Above equations can be solved for each
⇡k (x; ✓) = Pr(Y = k |X = x)

exp( Tj x)
⇡j (x; ✓) = PK 1 T
, j = 1, . . . , K 1
1 + l=1 exp( l x)
The classification
problem

1 Simple logistic
regression
⇡K (x; ✓) = PK 1 T
1 + l=1 exp( l x)
Example 1

Multiple logistic
regression

Is ⇡j (x; ✓) 2 [0, 1] for all j = 1, . . . , K ? Example 2

PK Multiple logistic
What is j=1 ⇡j (x; ✓)? regression - extensions

Generalized linear
models

Poisson regression

7.24
 Generalized Linear
How to solve logistic regression for K > 2 classses Models

Similar but not exactly the same as the case K = 2

Apply Newton method.

The classification
problem

Simple logistic
regression

Example 1

Multiple logistic
regression

Example 2

Multiple logistic
regression - extensions

Generalized linear
models

Poisson regression

7.25
 Generalized Linear
Generalized linear models (GLM) Models

Unifying framework for linear regression, logistic regression

and others.
Linear regression:
Y = 0 + 1 X1 + ... + p Xp +✏
T 2
p(Y |x1 , . . . , xp ) ⇠ N ( x, ) The classification
problem
T
E[Y |x1 , . . . , xp ] = x Simple logistic
regression

Logistic regression: Example 1

✓ ◆ Multiple logistic
Tx regression
e
p(Y |x1 , . . . , xp ) ⇠ Bernoulli Tx
1+e Example 2

Multiple logistic
T
logit(E[Y |x1 , . . . , xp ]) = x regression - extensions

Generalized linear
models

Poisson regression

7.26
 Generalized Linear
Exponential family distributions Models

✓ ◆
y✓ b(✓)
p(y |✓, ) = exp + c(y , )
a( )

✓: canonical parameter The classification

problem

: dispersion parameter Simple logistic

regression
a, b, c: functions Example 1

Facts: Multiple logistic

regression

Gaussian distribution is in the exponential family Example 2

Multiple logistic
Bernoulli and Binomial distributions are in the exponential regression - extensions

family Generalized linear

models

Poisson distribution is in the exponential family Poisson regression

7.27
 Generalized Linear
Link function Models

T
Let ⌘ = x be the linear predictor
Let µ = E[Y |x] be the mean response
Let g be a function such that ⌘ = g(µ). g is called a link
function.
The classification
What is the link function in linear regression? problem

Simple logistic
What is the link function in logistic regression? regression

Example 1

Multiple logistic
regression

Example 2

Multiple logistic
regression - extensions

Generalized linear
models

Poisson regression

7.28
 Generalized Linear
Linear, logistic and Poisson regression Models

Type p(Y |x) link g

Linear regression Gaussian g(µ) = µ
Logistic regression Bernoulli g(µ) = log(µ/(1 µ))
Possion regression Poisson g(µ) = log µ
The classification

All of these models can be solved using the Newton-Raphson problem

Simple logistic
method! regression

Example 1

Multiple logistic
regression

Example 2

Multiple logistic
regression - extensions

Generalized linear
models

Poisson regression

7.29
 Generalized Linear
Poisson regression Models

Response Y ⇠ Poisson(µ):

µy
Pr(Y = y ) = e µ
, y = 0, 1, 2, . . .
y!

Predictors: X = [1, X1 , . . . , Xp ]T The classification

problem
T
Parameters: =[ 0, 1, . . . , p]
Simple logistic
regression

Example 1

Multiple logistic
regression

Example 2

Multiple logistic
regression - extensions

Generalized linear
models

Poisson regression

7.30
 Generalized Linear
Poisson regression Models

Response Y ⇠ Poisson(µ):

µy
Pr(Y = y ) = e µ
, y = 0, 1, 2, . . .
y!

Predictors: X = [1, X1 , . . . , Xp ]T The classification

problem
T
Parameters: =[ 0, 1, . . . , p]
Simple logistic
regression

Model: Example 1

ELY p
Multiple logistic
Y1 , . . . , Yn are independent regression

Example 2
T
log(E[Y |x]) = x Multiple logistic

To
T regression - extensions
x
E[Y |x] = e Generalized linear
models

Poisson regression

log (E[Y |x]) = T x

|{z} | {z } |{z}
g: link µ: mean ⌘: linear predictor

7.30
 ett
Generalized Linear
MLE for Poisson regression
ELYIIB
Models

countvotes
Data: {(x i , yi )}ni=1 1t
PY4.11 141
2 3 2 3 2 T 3
1 x11 . . . x1p y1 ex 1
61 x21 . . . x2p 7 6 y2 7 6 T 7
6 7 6 7 6e x 2 7
X = 6. .. .. 6
.. 7 y = 6 .. 7 µ̂ = 6 .. 7
4 .. . . . 5 4.5 4 .
7
5 The classification
1 xn1 . . . xnp yn T
ex n problem

Log-likelihood:
Simple logistic
regression

Example 1
LCI ply Yi Ki In Il
n
X Multiple logistic
regression

Left Isplyilei.tl
l( ) = (yi x Ti exp(xi T
) log(yi !))
og
Example 2
i=1 Multiple logistic
regression - extensions

MLE:

É
Generalized linear
models

i E
Poisson regression
@l( )/@ =0
n
X
) (yi exp(xi T ))x i = 0
i=1

) XT y = XT µ̂ (no explicit formula for ˆ

MLE
gradient descent
)
7.31
Newt Raptor
 Generalized Linear
Applications of Possion regression Models

BILE Is predict j
ELYD ERILEE
y The classification
problem

or Simple logistic
regression

Ince
Example 1

y argma KY Multiple logistic

regression

Example 2

YE20,42
Multiple logistic
regression - extensions

Generalized linear
models

Poisson regression

7.32
 Generalized Linear
Applications of Possion regression Models

Predict:
Number of flight ticket purchases
Number of people that will attend to a concert
Number of phone calls to a call centre
The classification
problem

Simple logistic
regression

Example 1

Multiple logistic
regression

Example 2

Multiple logistic
regression - extensions

Generalized linear
models

Poisson regression

7.32
 Generalized Linear
Predicting number of cavalrymen deaths from horse-kicks
S men value
Models

Fatality by kicking data 1898

Ladislaus von Bortkiewicz

Pay
eÉI
Twenty years, ten army corps = 200 observations
Total number of fatalities = 122
The classification
= Mean deaths per corps-year = 122/200 = 0.61
PC07
GI
problem

Y = deaths per corps-year Simple logistic

regression é
Example 1

PHI et
éI
Multiple logistic
(y ) deaths actual num. P(Y = y ) expected num. regression

per corps-year observations observations Example 2

0 109 0.543 108.67 Multiple logistic

egg
regression - extensions

1 65 0.334 66.28
eggs
Generalized linear
models
2 22 0.101 20.21 Poisson regression
3 3 0.20 4.11
4
5
1
0
0.03
0.00 I 0.62
0.07 lawof small nubs
Ben plzPoisonCIT
7.33
l