BASIC Linear Algebra Tools in Pure Python

without Numpy or Scipy

Published by Thom Ives on December 11, 2018

Please �nd the code for this post on GitHub. As always, I hope you’ll clone it and make it your own. The main
module in the repo that holds all the modules that we’ll cover is named LinearAlgebraPurePython.py. There’s
a simple python �le named BasicToolsPractice.py that imports that main module and illustrates the modules
functions. It’d be great if you could clone or download that �rst to have handy as we go through this post.

Why This Post?

REMINDER: Our goal is to better understand principles of machine learning tools by exploring how to code
them ourselves …

Meaning, we are seeking to code these tools without using the AWESOME python modules available for
machine learning.
These efforts will provide insights and better understanding, but those insights won’t likely �y out at us every
post. Rather, we are building a foundation that will support those insights in the future.

To streamline some upcoming posts, I wanted to cover some basic functions that will make those future posts
easier. Obviously, if we are avoiding using numpy and scipy, we’ll have to create our own convenience functions
/ tools. This post covers those convenience tools. 

At one end of the spectrum, if you are new to linear algebra or python or both, I believe that you will �nd this
post helpful among, I hope, a good group of saved links.

At the other end of the spectrum, if you have background with python and linear algebra, your reason to read
this post would be to compare how I did it to how you’d do it. The review may give you some new ideas, or it
may con�rm that you still like your way better. 

The Tools
You’ll �nd documentation and comments in all of these functions. When more description is warranted, I will
give it or provide directions to other resource to describe it in more detail. If there is a speci�c part you don’t
understand, I am eager for you to understand it better. What’s the best way to do that? Rebuild these functions
from the inner most operations yourself and experiment with them at that level until you understand them, and
then add the next layer of looping, or code that repeats that inner most operation, and understand that, etc.

First up is zeros_matrix. When we just need a new matrix, let’s make one and �ll it with zeros.

01
02 def zeros_matrix(rows, cols):
03     """
04     Creates a matrix filled with zeros.
05         :param rows: the number of rows the matrix should have
06         :param cols: the number of columns the matrix should have
07  
08         :return: list of lists that form the matrix
09     """
10     M = []
11     while len(M) < rows:
12         M.append([])
13         while len(M[-1]) < cols:
14             M[-1].append(0.0)
15  
    return M

As you’ve seen from the previous posts, matrices and vectors are both being handled in Python as two
dimensional arrays. Thus, the array of rows contains an array of the column values, and each column value is
initialized to 0. Notice the -1 index to the matrix row in the second while loop. This is a simple way to reference
the last element of an array, and in this case, it’s the last array (row) that’s been appended to the array.

Our Second helper function is identity_matrix used to create an identity matrix. And, as a good constructively
lazy programmer should do, I have leveraged heavily on an initial call to zeros_matrix. All that’s left once we
have an identity matrix is to replace the diagonal elements with 1.
 01
02 def identity_matrix(n):
03     """
04     Creates and returns an identity matrix.
05         :param n: the square size of the matrix
06  
07         :return: a square identity matrix
08     """
09     IdM = zeros_matrix(n, n)
10     for i in range(n):
11         IdM[i][i] = 1.0
12  
    return IdM

Third is copy_matrix also relying heavily on zeros_matrix. We want this for those times where we need to work
on a copy and preserve the original matrix. Here, we are simply getting the dimensions of the original matrix
and using those dimensions to create a zeros matrix and then copying the elements of the original matrix to
the new matrix element by element.

01
02 def copy_matrix(M):
03     """
04     Creates and returns a copy of a matrix.
05         :param M: The matrix to be copied
06  
07         :return: A copy of the given matrix
08     """
09     # Section 1: Get matrix dimensions
10     rows = len(M)
11     cols = len(M[0])
12  
13     # Section 2: Create a new matrix of zeros
14     MC = zeros_matrix(rows, cols)
15  
16     # Section 3: Copy values of M into the copy
17     for i in range(rows):
18         for j in range(cols):
19             MC[i][j] = M[i][j]
20  
    return MC

Fourth is print_matrix so that we can see if we’ve messed up or not in our linear algebra operations! Here, we
are just printing the matrix, or vector, one row at a time. The “+0” in the list comprehension was mentioned in a
previous post. Try the list comprehension with and without that “+0” and see what happens. 

1
2 def print_matrix(M, decimals=3):
3     """
4     Print a matrix one row at a time
5         :param M: The matrix to be printed
6     """
7     for row in M:
print([round(x,decimals)+0 for x in row])

In case you don’t yet know python list comprehension techniques, they are worth learning. There are tons of
good blogs and sites that teach it. As I always, I recommend that you refer to at least three sources when
picking up any new skill but especially when learning a new Python skill. Some brief examples would be …

1
2 some_new_list = [<method(x)> for x in list if <condition> else <other>]
3 # or
4 another_list = [s.method() for s in string_list if <condition>]
5 # or
one_more_list = [<method(x)> for x in list]

The point of showing one_more_list is to make it abundantly clear that you don’t actually need to have any
conditionals in the list comprehension, and the method you apply can be one that you write.

Fifth is transpose. Transposing a matrix is simply the act of moving the elements from a given original row
and column to a  row = original column and a column = original row. That is, if a given element of M is mi,j , it
will move to mj,i in the transposed matrix, which is shown as

1
MT[j][i] = M[i][j]

in the code. In relation to this principle, notice that the zeros matrix is created with the original matrix’s number
of columns for the transposed matrix’s number of rows and the original matrix’s number of rows for the
transposed matrix’s number of columns. What a mouthful!

Notice that in section 1 below, we �rst make sure that M is a two dimensional Python array. Then we store the
dimensions of M in section 2. Next, in section 3, we use those dimensions to create a zeros matrix that has
the transposed matrix’s dimensions and call it MT. Finally, in section 4, we transfer the values from M to MT in
a transposed manner as described previously.

01
02 def transpose(M):
03     """
04     Returns a transpose of a matrix.
05         :param M: The matrix to be transposed
06  
07         :return: The transpose of the given matrix
08     """
09     # Section 1: if a 1D array, convert to a 2D array = matrix
10     if not isinstance(M[0],list):
11         M = [M]
12  
13     # Section 2: Get dimensions
14     rows = len(M)
15     cols = len(M[0])
16  
17     # Section 3: MT is zeros matrix with transposed dimensions
18     MT = zeros_matrix(cols, rows)
19  
20     # Section 4: Copy values from M to it's transpose MT
21     for i in range(rows):
22         for j in range(cols):
23             MT[j][i] = M[i][j]
24  
    return MT

Sixth and Seventh are matrix_addition and matrix_subtraction. I am explaining them at the same time,
because they are essentially identical with the exception of the single line of code where the element by
element additions or subtractions take place. In section 1 of each function, you see that we check that each
matrix has identical dimensions, otherwise, we cannot add them. Section 2 of each function creates a zeros
matrix to hold the resulting matrix. Section 3 of each function performs the element by element operation of
addition or subtraction, respectively. 

01
02 def matrix_addition(A, B):
03     """
04     Adds two matrices and returns the sum
05         :param A: The first matrix
06         :param B: The second matrix
07  
08         :return: Matrix sum
09     """
10     # Section 1: Ensure dimensions are valid for matrix addition
11     rowsA = len(A)
12     colsA = len(A[0])
13     rowsB = len(B)
14     colsB = len(B[0])
15     if rowsA != rowsB or colsA != colsB:
16         raise ArithmeticError('Matrices are NOT the same size.')
17  
18     # Section 2: Create a new matrix for the matrix sum
19     C = zeros_matrix(rowsA, colsB)
20  
21     # Section 3: Perform element by element sum
22     for i in range(rowsA):
23         for j in range(colsB):
24             C[i][j] = A[i][j] + B[i][j]
25  
26     return C
27  
28 def matrix_subtraction(A, B):
29     """
30     Subtracts matrix B from matrix A and returns difference
31         :param A: The first matrix
32         :param B: The second matrix
33  
34         :return: Matrix difference
35     """
36     # Section 1: Ensure dimensions are valid for matrix subtraction
37     rowsA = len(A)
38     colsA = len(A[0])
39     rowsB = len(B)
40     colsB = len(B[0])
41     if rowsA != rowsB or colsA != colsB:
42         raise ArithmeticError('Matrices are NOT the same size.')
43  
44     # Section 2: Create a new matrix for the matrix difference
45     C = zeros_matrix(rowsA, colsB)
46  
47     # Section 3: Perform element by element subtraction
48     for i in range(rowsA):
49         for j in range(colsB):
50             C[i][j] = A[i][j] - B[i][j]
51  
    return C
Eighth is matrix_multiply. The �rst rule in matrix multiplication is that if you want to multiply matrix A times
matrix B , the number of columns of A MUST equal the number of rows of B . Thus, if A has dimensions of m
rows and n columns (m x n for short) B must have n rows and it can have 1 or more columns. Let’s say it
has k columns. Thus, the resulting product of the two matrices will be an m x k matrix, or the resulting matrix
has the number of rows of A and the number of columns of B .  Hence, we create a zeros matrix to hold the
resulting product of the two matrices that has dimensions of rowsA x colsB in the code. Also, IF A and B
have the same dimensions of n rows and n columns, that is they are square matrices, A ⋅ B does NOT equal
B ⋅ A. Remember that the order of multiplication matters when multiplying matrices. Finally, the result for
each new element ci,j in C , which will be the result of A ⋅ B , is found as follows using a 3 x 3 matrix as an
example:

ci,j = ai,0 ⋅ b0,j + ai,1 ⋅ b1,j + ai,2 ⋅ b2,j

That is, to get ci,j we are multiplying each column element in each row i of A times each row element in each
column j of B and adding up those products. Phew!

01
02 def matrix_multiply(A, B):
03     """
04     Returns the product of the matrix A * B
05         :param A: The first matrix - ORDER MATTERS!
06         :param B: The second matrix
07  
08         :return: The product of the two matrices
09     """
10     # Section 1: Ensure A & B dimensions are correct for multiplication
11     rowsA = len(A)
12     colsA = len(A[0])
13     rowsB = len(B)
14     colsB = len(B[0])
15     if colsA != rowsB:
16         raise ArithmeticError(
17             'Number of A columns must equal number of B rows.')
18  
19     # Section 2: Store matrix multiplication in a new matrix
20     C = zeros_matrix(rowsA, colsB)
21     for i in range(rowsA):
22         for j in range(colsB):
23             total = 0
24             for ii in range(colsA):
25                 total += A[i][ii] * B[ii][j]
26             C[i][j] = total
27  
    return C

Ninth is a function, multiply_matrices, to multiply out a list of matrices using matrix_multiply. Note that we
simply establish the running product as the �rst matrix in the list, and then the for loop starts at the second
element (of the list of matrices) to loop through the matrices and create the running product, matrix_product,
times the next matrix in the list. 
01
02 def multiply_matrices(list):
03     """
Find the product of a list of matrices from first to last
 04         :param list: The list of matrices IN ORDER
05  
06         :return: The product of the matrices
07     """
08     # Section 1: Start matrix product using 1st matrix in list
09     matrix_product = list[0]
10  
11     # Section 2: Loop thru list to create product
12     for matrix in list[1:]:
13         matrix_product = matrix_multiply(matrix_product, matrix)
14  
15     return matrix_product

Tenth, and I confess I wasn’t sure when it was best to present this one, is check_matrix_equality. There will be
times where checking the equality between two matrices is the best way to verify our results. However, those
operations will have some amount of round off error to where the matrices won’t be exactly equal, but they will
be essentially equal. Thus, note that there is a tol (tolerance parameter), that can be set. If the default is used,
the two matrices are expected to be exactly equal. If a tolerance is set, the value of tol is the number of
decimal places the element values are rounded off to to check for an essentially equal state.

01
02 def check_matrix_equality(A, B, tol=None):
03     """
04     Checks the equality of two matrices.
05         :param A: The first matrix
06         :param B: The second matrix
07         :param tol: The decimal place tolerance of the check
08  
09         :return: The boolean result of the equality check
10     """
11     # Section 1: First ensure matrices have same dimensions
12     if len(A) != len(B) or len(A[0]) != len(B[0]):
13         return False
14  
15     # Section 2: Check element by element equality
16     #            use tolerance if given
17     for i in range(len(A)):
18         for j in range(len(A[0])):
19             if tol == None:
20                 if A[i][j] != B[i][j]:
21                     return False
22             else:
23                 if round(A[i][j],tol) != round(B[i][j],tol):
24                     return False
25  
    return True

The dot product between two vectors or matrices is essentially matrix multiplication and must follow the same
rules. It’s important to note that our matrix multiplication routine could be used to multiply two vectors that
could result in a single value matrix. In such cases, that result is considered to not be a vector or matrix, but it
is single value, or scaler. However, using our routines, it would still be an array with a one valued array inside of
it. To read another reference, check HERE, and I would save that link as a bookmark – it’s a great resource.

The Eleventh function is the unitize_vector function. Let’s step through its sections. Section 1 ensures that a
vector was input meaning that one of the dimensions should be 1. Also, it makes sure that the array is 2
dimensional. This tool kit wants all matrices and vectors to be 2 dimensional for consistency. Section 2 uses
the Pythagorean theorem to �nd the magnitude of the vector.  Section 3 makes a copy of the original vector
(the copy_matrix function works �ne, because it still works on 2D arrays), and Section 4 divides each element
by the determined magnitude of the vector to create a unit vector. 

01
02 def unitize_vector(vector):
03     """
04     Find the unit vector for a vector
05         :param vector: The vector to find a unit vector for
06  
07         :return: A unit-vector of vector
08     """
09     # Section 1: Ensure that a vector was given
10     if len(vector) > 1 and len(vector[0]) > 1:
11         raise ArithmeticError(
12             'Vector must be a row or column vector.')
13  
14     # Section 2: Determine vector magnitude
15     rows = len(vector); cols = len(vector[0])
16     mag = 0
17     for row in vector:
18         for value in row:
19             mag += value ** 2
20     mag = mag ** 0.5
21  
22     # Section 3: Make a copy of vector
23     new = copy_matrix(vector)
24  
25     # Section 4: Unitize the copied vector
26     for i in range(rows):
27         for j in range(cols):
28             new[i][j] = new[i][j] / mag
29  
    return new

Using Numpy For The Above Operations

How would we do all of these actions with numpy? It’s pretty simple and elegant. The code below follows the
same order of functions we just covered above but shows how to do each one in numpy. The code below is in
the �le NumpyToolsPractice.py in the repo. Copy the code below or get it from the repo, but I strongly
encourage you to run it and play with it.

01
02 import numpy as np
03  
04  
05 # Zeros Matrix
06 print(np.zeros((3, 3)), '\n')
07  
08 # Identity Matrix
09 print(np.identity(3), '\n')
10  
11 A = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
12 C = np.array([[2, 4, 6], [8, 10, 12], [14, 16, 18]])
 
13 # Matrix Copy
14 AC = A.copy()
15 print(AC, '\n')
16  
17 # Transpose a matrix
18 AT = np.transpose(A)
19 print(AT, '\n')
20  
21 # Add and Subtract
22 SumAC = A + C
23 print(SumAC, '\n')
24  
25 DifCA = C - A
26 print(DifCA, '\n')
27  
28 # Matrix Multiply
29 ProdAC = np.matmul(A, C)
30 print(ProdAC, '\n')
31  
32 # Multiply a List of Matrices
33 arr = [A, C, A, C, A, C]
34 Prod = np.matmul(A, C)
35 num = len(arr)
36 for i in range(2, num):
37     Prod = np.matmul(Prod, arr[i])
38 print(Prod, '\n')
39  
40 ChkP = np.matmul(
41             np.matmul(
42                 np.matmul(
43                     np.matmul(
44                         np.matmul(arr[0], arr[1]),
45                         arr[2]), arr[3]), arr[4]), arr[5])
46 print(ChkP, '\n')
47  
48 # Check Equality of Matrices
49 print(Prod == ChkP, '\n')
50 # https://docs.scipy.org/doc/numpy/reference/generated/numpy.allclose.html
51 print(np.allclose(Prod, ChkP), '\n')
52  
53 # Dot Product (follows the same rules as matrix multiplication)
54 # https://docs.scipy.org/doc/numpy/reference/generated/numpy.dot.html
55 v1 = np.array([[2, 4, 6]])
56 v2 = np.array([[1], [2], [3]])
57 ans1 = np.dot(v1, v2)
58 ans2 = np.dot(v1, v2)[0, 0]
59 print(f'ans1 = {ans1}, ans2 = {ans2}\n')
60  
61 # Unitize an array
62 mag1 = (1*1 + 2*2 + 3*3) ** 0.5
63 mag2 = np.linalg.norm(v2)
64 norm1 = v2 / mag1
65 norm2 = v2 / mag2
66 print(f'mag1 = {mag1}, mag2 = {mag2}, they are equal: {mag1 == mag2}\n')
67 print(norm1, '\n')
68 print(norm2, '\n')
69 print(norm1 == norm2)
Closing
That’s it for now. This library will grow of course with each new post. I’ll introduce new helper functions if and
when they are needed in future posts, and have separate posts for those additions that require more
explanation. But these functions are the most basic ones. Some of these also support the work for the inverse
matrix post and for the solving a system of equations post. 

Thom Ives

Data Scientist, PhD multi-physics engineer, and python loving geek living in the United States.