Learning from Data

Solutions to Selected Exercises

1.1 The error function is given by

2
M
N
X
X
1
t n
wj xjn
E(w) =
2 n=1
j=0
To obtain the normal equations we take the partial derivatives with
respect to the weights and equate to zero. Application of the chain rule
gives

N
M
X
X
E
t n
=
wj xjn xin
wi
n=1
j=0
Equating this expression to zero gives

N
N
M X
N
M
X
X
X
X
t n

wj xjn xin =
tn xin +
xjn xin wj = 0
n=1

n=1

j=0

j=0 n=1

So we get
M X
N
X

(xn )i+j wj =

j=0 n=1

N
X

tn (xn )i

i = 0, . . . , M

n=1

1.2 The error function is given by

E(w) =

N
M
X 2
1X
w
(y(xn , w) tn )2 +
2 n=1
2 j=0 j

To obtain the normal equations we take the partial derivatives with

respect to the weights and equate to zero. Application of the chain rule
gives


N
X
E
y(xn , w)
=
(y(xn , w) tn )
+ wi
wi
wi
n=1

This gives the normal equations

N
X




N
X
y(xn , w)
y(xn , w)
y(xn , w)
+wi =
tn
wi
wi
n=1
n=1


Plugging in y(xn , w) =
M X
N
X

PM

j=0

i = 0, . . . , M

wj xjn we get

(xn )i+j wj + wi =

j=0 n=1

N
X

tn (xn )i

i = 0, . . . , M

n=1

1.3 Use the sum and product rules of probability.

Probability of drawing an apple:
X
p(a) =
p(a, box)
box
X
=
p(a|box)p(box)
box
= p(a|r)p(r) + p(a|b)p(b) + p(a|g)p(g)
= 0.3 0.2 + 0.5 0.2 + 0.3 0.6 = 0.34
Probability of green box given orange
p(o|g)p(g)
p(g, o)
=P
p(o)
box p(o|box)p(box)
0.18
=
= 0.5
0.36

p(g|o) =

1.5
var[f (x)] = E[(f (x) E[f (x)])2 ]
2

(1.38;expand)
2

= E[f (x) 2f (x)E[f (x)] + E[f (x)] ]

(push expectation inward)
= E[f (x)2 ] 2E[f (x)]E[f (x)] + E[f (x)]2
= E[f (x)2 ] E[f (x)]2

(1.39)

The important thing to notice is that E[f (x)] is a constant, and so

E[2f (x)E[f (x)]] = 2E[f (x)]E[f (x)]

and

E[E[f (x)]2 ] = E[f (x)]2

1.6
cov[x, y] = E[xy] E[x]E[y]

(1.41)

Well show that E[xy] = E[x]E[y] if x and y are independent.

XX
E[xy]
xy p(x, y)
x

XX
x

xy p(x)p(y)

x p(x)

y p(y) = E[x]E[y]

1.10 We dont need independence for the first one:

XX
E[x + z]
(x + z) p(x, z)
x

XX
x

x p(x, z) +

XX
z

z p(x, z)

X X
X X
p(x, z)
z
p(x, z) +
x

x p(x) +

z p(z) = E[x] + E[z]

In exercise 1.5 we have shown that var[x + z] = E[(x + z)2 ] E[x + z]2 .
Since E[x + z] = E[x] + E[z] as we have shown above, we get
var[x + z] = E[(x + z)2 ] E[x]2 2E[x]E[z] E[z]2
Left to determine E[(x + z)2 ]:
XX
(x + z)2 p(x, z)
E[(x + z)2 ]
x

XX

XX
x

(independence/expand)

(x2 + 2xz + z 2 ) p(x)p(z)

x2 p(x)p(z) +

x p(x)

XX
x

p(z) + 2

2xz p(x)p(z) +

X
x

x p(x)

XX
z

X
z

z p(z) +

z 2 p(x)p(z)

z 2 p(z)

= E[x2 ] + 2E[x]E[z] + E[z 2 ]

So finally we get
var[x + z] = E[x2 ] + 2E[x]E[z] + E[z 2 ] E[x]2 2E[x]E[z] E[z]2
= (E[x2 ] E[x]2 ) + (E[z 2 ] E[z]2 ) = var[x] + var[z]

X
x

p(x)

1.13
#
N
N
1 X
1 X
2
E
(xn ) =
E[(xn )2 ]
N n=1
N n=1
"

N
1 X
1
var(xn ) = N 2 = 2
N n=1
N