Facultad de Ingeniera

Departamento de Sistemas e Industrial

Machine Learning
2do Semestre de 2011

Assignment 1:
Linear Algebra and Probability.
Anibal Montero L. 162253
Delivered to: Prof. Fabio A. Gonzalez.
1.

Let D = {d1dn} be a set of documents and T = {t1tm} a set of terms (words). Let TD =
(TDij) i=1...m, j=1...n be a matrix such that TDij corresponds to the number of times the term
ti appears in the document dj. Assume a process where a document dj is randomly choosen with
uniform probability and then a term ti, present in dj, is randomly choosen with a probability
proportional to the frequency of ti in dj.
(a) How do you transform the matrix TD to obtain a matrix P(T,D), such that P(T,D)ij= P(ti,
dj) (the joint probability of term ti and document dj)?.
(b) How do you obtain the P(T |D) matrix?
According that was mentioned:

Then:

Where:

(c) How do you obtain the P(D |T) matrix?

According to the total probability theorem and the last results:

Besides that:

2.

Let li be the length, number of characters, of term ti, and let L = (li...lm) be a column vector.

(a) Calculate E[l], the expected value of the random variable l corresponding to the length of a
randomly choosen term.
Let:

Then:

And thus:

(b) Calculate Var(l), the variance of l.

(c) Show that Var(al + b) = a2Var(l), where a and b are arbitrary constants.

...
3.

Find an expression for P(T |D) that exclusively uses P(D |T), P(T, D) and, optionally, constant
matrices/vectors.
Thats simple, because according the last results:

4.

Find an expression for the matrix Cov = ( Cov(ti,ti) ) i,i=1...m, the covariance of the random
variables corresponding to terms.
Given the definition of covariance between random vectors:

That matrix product leads the covariance between pairs of terms(rows) , it gives the generic
value of a covariance matrix:

5.

(optional) What is the meaning of the Eigenvector of Cov corresponding to the largest
eigenvalue?.
The largest eigenvalue measures the variance of the first principal component, i.e., measures the
variance of the axis that maximizes the projections of each element (row) of the matrix.