Methodology

Let’s say you have a linear system Ax=b, where A has more rows than columns. The system
is what we call over-determined.

An example of this would be searching for intersection of 3 lines in a plane. There may not
be an intersection, but there mat be points that are close to being an intersection.

One common “close” intersection point is what we call the least squares solution to Ax=b. the
way we find the least squares solution is to solve this system:

( AT A )x= AT b

AT b is a symmetric matrix. If it has an inverse, then x will be the least squares solution. A
good linear algebra book will explain the logic involved.

Another example is what we call a covariance matrix. Let’s say we have three random
variables. Each variable has it own variance, and also a covariance with the other two
variables. We can arrange all of the data like this:

Var 1 Cov 12 Cov 13

|
Cov 21 Var 2 Cov 23
Cov 31 Cov 32 Var 3 |
Since Cov XY = Cov YX for variables X and Y, the matrix is symmetric.