Linear Algebra

& Matrix
By iTutor.com
T- 1-855-694-8886
Email- info@iTutor.com
Linear Algebra
History:
 The beginnings of matrices and determinants goes back to
the second century BC although traces can be seen back to
the fourth century BC. But, the ideas did not make it to
mainstream math until the late 16th century.
 The Babylonians around 300 BC studied problems which
lead to simultaneous linear equations.
 The Chinese, between 200 BC and 100 BC, came much closer
to matrices than the Babylonians. Indeed, the text Nine
Chapters on the Mathematical Art written during the Han
Dynasty gives the first known example of matrix methods.
 In Europe, two-by-two determinants were considered by
Cardano at the end of the 16th century and larger ones by
Leibniz and, in Japan, by Seki about 100 years later.
What is a Matrix?
A matrix is a set of elements, organized into rows and columns
Rows

a b 
Columns
c d 
 
a and d are the diagonal elements.
 b and c are the off-diagonal elements.
 Matrices are like plain numbers in many ways: they can be
added, subtracted, and, in some cases, multiplied and
inverted (divided).
 Examples:

1 b 
A=  ; b = [α β δ]
1 − d 
Dimensions of a matrix:
Numbers of rows by numbers of columns. The Matrix A is
a 2x2 matrix, b is a 1x3 matrix.

A matrix with only one column or only one row is called a

vector.

If a matrix has an equal numbers of rows and columns, it is

called a square matrix. Matrix A, above, is a square matrix.

Usual Notation: Upper case letters => matrices

Lower case => vectors
 Matrix multiplication
 Multiplication of matrices requires a conformability condition
The conformability condition for multiplication is that the
column dimensions of the lead matrix A must be equal to the
row dimension of the lag matrix B.
If A is an (m x n) and B an (n x p) matrix (A has the same
number of columns a B has rows), then we define thenproduct
of AB. AB is (m x p) matrix with its ij-th element is ∑ j =1 aij b jk
What are the dimensions of the vector, matrix, and result?

b11 b12 b13 

aB = [ a11a12 ]   = c = [ c11 c12 c13 ]
b21 22 b23 

= [ a11b11 + a12b21 a11b12 + a12b22 a11b13 + a12b23 ]

 Dimensions: a(1 x 2), B(2 x 3) => c (1 x 3)

Transpose Matrix
 The transpose of a matrix A is another matrix AT (also written A
′) created by any one of the following equivalent actions:
- Write the rows (columns) of A as the columns (rows) of AT
- Reflect A by its main diagonal to obtain AT
 Formally, the (i, j) element of AT is the (j, i) element of A:
[AT]ij = [A]ji
 If A is a m × n matrix => AT is a n × m matrix.
 (A')' = A
 Conformability changes unless the matrix is square.

 3 1
3 8 − 9  8 0
Example : A =   => A′ =
1 0 4  
  − 9 4
Special Matrices: Identity and Null
 Identity
Matrix is a square matrix with ones
along the diagonal and zeros everywhere else.
Similar to scalar “1.” 1 0 0
0 1 0
 
 Null matrix is one in which all elements are 0 0 1
zero. Similar to scalar “0.”
0 0 0
0 0 0
 Both are diagonal matrices => off-diagonal  
elements are zero. 0 0 0

 Both are symmetric and examples of idempotent

matrices. That is,
A = AT and A = A2 = A3 = …
Basic Operations
 Addition, Subtraction, Multiplication

a b  e f   a + e b + f 
 c d  +  g h  = c + g d + h Just add elements
     

a b  e f   a − e b − f 
 c d  −  g h  = c − g d − h Just subtract elements
     

a b   e f  ae + bg af + bh Multiply each row by

c d   g  =
  h  ce + dg cf + dh  each column and add

a b  ka kb 
k  =  Multiply each
 c d   kc kd  element by the scalar
Basic Matrix Operations: Examples
 2 1  3 1   5 2 
 Matrix addition 7 9 + 0 2 = 7 11
     
A2 x 2 +B2 x 2 =C2 x 2

 Matrix subtraction 2 1 1 0 1 1
7 9 − 2 3 = 5 6
     

 Matrix  2 1  1 0   4 3 
multiplication 7 9 x 2 3 = 26 27 
     
A 2 x 2 xB2 x 2 = C 2 x 2

 Scalar multiplication 1 2 4 1 4 1 2
  = 
8 6 1  3 4 1 8
Laws of Matrix Addition & Multiplication
 Commutative law of Matrix Addition: A + B = B + A

 a11 a12   b11 b12   a11 + b11 b12 + a12 

A+ B =   +  = 
 a 21 a 22  b21 b22   a 21 + a 21 b22 + a 22 
 b11 b12   a11 a12   b11 + a11 b12 + a12 
B+ A=   +  = 
b b b b
 21 22   21 22   21 b + a 21 b22 + a 22 

 Matrix Multiplication is distributive across Additions:

A (B+ C) = AB + AC (assuming comformability applies).
 Matrix Multiplication
 Matrix multiplication is generally not commutative. That is,
AB ≠ BA even if BA is conformable
(because diff. dot product of rows or col. of A&B)

1 2 0 − 1
A=  ,B= 
3 4   6 7 

1( 0 ) + 2( 6 ) 1( − 1) + 2( 7 )  12 13 
AB =  =
3( 0 ) + 4( 6 ) 3( − 1) + 4( 7 )  24 25


0(1) + ( − 1)( 3) 0( 2) + ( − 1) 4  − 3 − 4
BA =   = 
 6 ( 1) + 7 ( 3) 6 ( 2 ) + ( 7 ) 4   27 40 
 Exceptions to non-commutative law:
AB=BA iff
B = a scalar,
B = identity matrix I, or
B = the inverse of A -i.e., A-1

 Theorem: It is not true that AB = AC => B=C

Proof:

1 2 1  1 −1 1 2 1 2
A =
1 0 −1
; B = 0
 1 0
; C = −1
 −1 −1


1 −2 −3
 
1 1 0
 
2 3 1

Note: If AB = AC for all matrices A, then B=C.

Inverse of a Matrix
 Identity matrix:
AI = A
1 0 0
I 3 = 0 1 0
0 0 1

Notation: Ij is a jxj identity matrix.

 GivenA (mxn), the matrix B (nxm) is a right-inverse for A iff

AB = Im
 Given A (mxn), the matrix C (mxn) is a left-inverse for A iff
CA = In
 Theorem: If A (mxn), has both a right-inverse B and a left-inverse C,
thenC = B.
Proof:
We have AB=Im and CA=In.
Thus,
C(AB)= C Im = C and C(AB)=(CA)B= InB = B
=> C(nxm)=B(mxn)

Note:
- This matrix is unique. (Suppose there is another left-inverse D,
then D=B by the theorem, so D=C.)
- If A has both a right and a left inverse, it is a square matrix. It
is usually called invertible.
 Inversionis tricky:
(ABC)-1 = C-1B-1A-1

 Theorem: If A (mxn) and B (nxp) have inverses, then AB is

invertible and (AB)-1 = B-1A-1
Proof:
We have AA-1=In and A-1A=Im
BB-1=In and B-1B=Ip
Thus,
B-1A-1(AB) = B-1 (A-1A) B= B-1 InB = B-1 B = Ip
(AB) B-1A-1 = A (BB-1) A-1 = A In A-1 = A A-1 = Im
=> AB is invertible and (AB)-1 = B-1A-1

 More on this topic later

Transpose and Inverse Matrix
 (A + B)' = A' + B'
 If A' = A, then A is called a symmetric matrix.

 Theorems:

- Given two conformable matrices A and B, then (AB)' = B'A'

- If A is invertible, then (A-1)' = (A')-1 (and A' is also invertible).

16
Properties of Symmetric Matrices
 Definition:

If A' = A, then A is called a symmetric matrix.

 Theorems:

- If A and B are n x n symmetric matrices, then (AB)' = BA

- If A and B are n x n symmetric matrices, then (A+B)' = B+A
- If C is any n x n matrix, then B = C'C is symmetric.

 Usefulsymmetric matrices:
V = X’X
P = X(X’X)-1X’
M = I – P = I - X(X’X)-1X’
4.1 Application I: System of equations
 Assume an economic model as system of linear equations with:
aij parameters, where i = 1.. n rows, j = 1.. m columns, and n=m
xi endogenous variables,
di exogenous variables and constants

a11 x1 + a12 x2 +  a1m xn = d1

a21 x1 + a22 x2 +  a 2 m xn = d 2
   
an1 x1 + a n 2 x2 +  anm xn = d n
Application: System of equations
A general form matrix of a system of linear equations
Ax = d where
A = matrix of parameters
x = column vector of endogenous variables
d = column vector of exogenous variables and constants
 Solve for x*

 a11 a12  a1m   x1   d1 

a a22  a2 m   x2  d 2 
 21 =
      
    
an1 an 2  anm   xn  d n 
Ax = d
 Questions:For what combinations of A and d there will zero,
one, many or an infinite number of solutions? How do we
compute (characterize) those sets of solutions?
Solution of a General-equation System
 Theorem: Given A (mxn). If A has a right-inverse, then the
equation Ax = d has at least one solution for every d (mx1).

Proof:
Pick an arbitrary d. Let H be a right-inverse (so AH=Im).
Define x*=Hd.
Thus,
Ax* = A Hd = Imd = d => x* is a solution.

20
 Theorem: Given A (mxn). If A has a left-inverse, then the
equation Ax=d has at most one solution for every d (mx1). That
is, if Ax=d has a solution x* for a particular d, then x* is unique.

Proof:
Suppose x* is a solution and z* is another solution. Thus, Ax*=d
and Az*=d. Let G be a left-inverse for A (so GA=In).

Ax*=d => GA x*= Gd

=> Inx* = x* = Gd.

Az*= d => GA z* = Gd
=> Inz* = z* = Gd.
Thus,
x*=z*=Gd. ■
 Problem with the previous proof? We’re assuming the left-
inverse exists (and there’s always a solution).

 Assume the 2x2 model  Why?

2x + y = 12 4x + 2y =24
4x + 2y = 24 2(2x + y) = 2(12)
Find x*, y*:  One equation with two
y = 12 – 2x unknowns
4x + 2(12 – 2x) = 24 2x + y = 12
4x +24 – 4x = 24 x, y
0 = 0 ? indeterminate! Conclusion:
not all simultaneous
equation models have
solutions
(not all matrices have inverses).
Linear dependence
A set of vectors is linearly dependent if any one of them can
be expressed as a linear combination of the remaining
vectors; otherwise, it is linearly independent.

 Formal definition: Linear independence (LI)

The set {u1,...,uk} is called a linearly independent set of vectors
iff
c1 u1+....+ ckuk = θ => c1= c2=...=ck,=0.

 Notes:
- Dependence prevents solving a system of equations. More
unknowns than independent equations.
- The number of linearly independent rows or columns in a
matrix is the rank of a matrix (rank(A)).
 Examples:

v1' = [5 12]
v2' = [10 24]
 5 10  v1' 
12 24 =  ' 
  v2 
2v1/ − v2/ = 0 /

2  1 4 
v1 =  ; v2 =  ; v3 =  
7  8 5
3v1 − 2v2
= [ 6 21] − [ 2 16]
= [ 4 5] = v3
3v1 − 2v2 − v3 = 0
Upper and Lower Triangular Matrices
A square (n x n) matrix C is: 1 2 5
-Upper Triangular (UT) iff Cij=0 for i>j 0 0 6 UT
(if the diagonal elements are all equal to 1, 
 0 0 1
we have a upper-unit triangular (UUT)matrix)
0 0 0
-Lower Triangular (LT) iff Cij=0 for i<j 2 1 0 LT

(if the diagonal elements are all equal to 1,  1 2 0
we have a lower-unit triangular (LUT) matrix)

-Diagonal (D) iff Cij=0 for i ≠j

 Theorem:
The product of the two UT (UUT) matrices is UT (UUT).
The product of the two LT (LUT) matrices is LT (LUT).
The product of the two D matrices is D.
 Q: Why are we interested in these matrices?
Suppose Ax=d, where A is LT (with non-zero diagonal
terms). Then, the solutions are recursive (forward substitution).
Example:
x1 = d1
a21 x1 + a22 x2 = d2
a31 x1 +a32 x2 + a33 x3 = d3

Similarly, suppose Ax=d, where A is UT (with non-zero diagonal

terms). Then, the solutions are recursive (backward substitution).

Example:
a11 x1 +a12 x2 + a13 x3 = d1
a22 x2 + a23 x3 = d2
a31 x3 = d3
 Finding a solution to Ax=d
Given A (n x n). Suppose we can decompose A into A=LU,
where L is LUT and U is UUT (with non-zero diagonal).

Then
Ax=d => LUx = d.
Suppose L is invertible=> Ux = L-1d = c (or d = Lc)
=>Solve by forward substitution for c.

Then, Ux = c(Gaussian elimination) => solve by backward

substitution for x.

 Theorem:
If A (n x n) can be decomposed A=LU, where L is LUT
and U is UUT (with non-zero diagonal), then Ax=d has a
unique solution for every d.
 We can write a “symmetric” decomposition. Since U has non-
zero diagonal terms, we can write U=DU*, where U* is UUT.

Example:
2 4 8 2 0 0 1 2 4
U = 0 3 6; D = 0 3 0; => U * = 0 1 2
0 0 5 0 0 5 0 0 1

Theorems:
If we can write A (n x n) as A=LDU, where L is LUT, D is
diagonal with non zero diagonal elements, and U is UUT, then L,
D, and U are unique.

If we can write A (n x n) as A=LDU, and A is symmetric,

then we can write A=LDL’.
Theorem: Cholesky decomposition
If A is symmetric, A=LDL’, and all diagonal elements of D are positive,
then A = HH’.
Proof:
Since A is symmetric, then A=LDL.
The product of a LUT matrix and a D matrix is a LUT
matrix.
Let D*=D1/2 and L be a LT matrix.
Then H=LD* is matrix is LT.
=> A=HH’. ■

The Cholesky decomposition is unique. It is used in the numerical

solution systems of equations, non-linear optimization, Kalman
filter algorithms, etc..

André-Louis Cholesky (1875–1918)

Cholesky decomposition: Application
System of Equations
If A is symmetric and A=LDL’, and all elements of D are
positive (A is a positive definite matrix), then we can solve

Ax = b by
(1) Compute the Cholesky decomposition A=HH’.
(2) Solve Hy = b for y,
(3) With y known, solve H’x = y for x.

Note: A-1 is never computed.

Ordinary Least squares

Systems of the form Ax = b with A symmetric and positive
definite are common in economics. For example, the normal
equations in ordinary –i.e., linear- least squares problems are of this
form. A-1 is never computed. 30
Call us for more
Information:
1-855-694-8886
Visit
www.iTutor.com

The End