Practice Question 3: Vector Spaces, Orthogonality, and Least Squares

1. [10 points] Consider the following 3 points P (1, 2, 4), Q (2, 2, 2), and R (1, 0, -4).
a. Find the equation of the line L that goes through the P and Q.
b. Find a vector that is perpendicular to L.
c. Find the projection of the vector from P to R on this line L.
d. Write a numpy code to find the projection vector in question (c) and the distance from R to
the line.
e. Find the equation of the plane that contains all three points.

{[ ] [ ] [ ]}
1 −2 1
−1 3 0
2. [10 points] Consider the set of vectors B= , , .
2 1 7
3 −2 7
a. Show that the vectors in B are linearly dependent using Gaussian Elimination. In class we
talked about two ways to doing this.
1. Put the vectors as columns of a matrix A and find the solution to Ax = 0. If only the
trivial solution exists then the vectors are linearly independent, otherwise dependent.
2. Put the vectors as rows of a matrix A and compute the Row Echelon Form (REF) (that
is, convert the matrix to an upper triangular matrix using elementary row operations) of
the matrix A. If the REF contains one or more rows of all zeros, then the vectors are
dependent. Why? Express the dependent vector as a linear combination of the
independent ones.
b. Write a numpy code to determine whether the vectors are linearly independent or not. (you
will simply need to code the Gaussian elimination algorithm in numpy)

[] []
1 2
3. [20 points] Consider the following two vectors: a1 = −2 and a2 = 1 .
3 0
a. Show that the vectors are orthogonal.
b. Convert the vectors a1 , a 2 to unit vectors v1 , v2 .

[]
10
c. Consider W =span(a1 , a 2 ) . Does the vector y= 5 belong to W? Justify. (hint: show
3
whether y=c 1 a1 +c 2 a2 has a solution or not)
d. If y does not belong to W, then find ^y = projection of y on W.
e. Find the coordinates of ^y using the basis set B={v 1 , v 2 } for W.
f. Find a vector z orthogonal to W.

4. [10 points] Consider the linear regression problem with the data matrix A with columns a1 and
a2 and the target output y (as defined in question 2). That is, we have 3 input datapoints (1, 2), (-
2, 1), and (3, 0) with target output for these datapoints are 10, 5 and 3 respectively.
a. Write down the linear regression equation, using the solution for 2(d, e). What sort of
equation is this (a line, plane, hyperplane?). Explain.
b. Find the MSE for this (training) data.
c. What will be the predicted out for the input (2, 1)?
5. [20 point] Consider the following 3 input datapoints given in the data matrix X and the
corresponding output values given in the vector y.

[ ] []
12 4 9
X= 3 1 2 , y= 11
24 3 14

a. How many features do we have?

b. Write the linear regression function you will be using to predict the output.
c. Find the normal equation ( X T X β =X T y ) for this problem.
d. Solve the normal equation using Gaussian elimination to find β.
e. Write down the linear regression equation.

6. [10 points]
a. Find the gradient of the loss/cost function of the regression problem in question 5 at

[]
0.1
β = 0.1 .
0

0.2
0.5
b. Find the next estimate of β using the gradient descent algorithm.