theta = theta - (alpha/m) * (X' * (X * theta - y))

Assume that the following values of X, y and θ are given:

 m = number of training examples

 n = number of features + 1

Here

 m = 5 (training examples)
 n = 4 (features+1)
 X = m x n matrix
 y = m x 1 vector matrix
 θ = n x 1 vector matrix
 xi is the ith training example
 xj is the jth feature in a given training example

Further,

 h(x) = ([X] * [θ]) (m x 1 matrix of predicted values for our training set)
 h(x)-y = ([X] * [θ] - [y]) (m x 1 matrix of Errors in our predictions)

whole objective of machine learning is to minimize Errors in predictions. Based on the above
corollary, our Errors matrix is m x 1 vector matrix as follows:
To calculate new value of θj, we have to get a summation of all errors (m rows) multiplied
by jth feature value of the training set X. That is, take all the values in E, individually multiply
them with jth feature of the corresponding training example, and add them all together. This will
help us in getting the new (and hopefully better) value of θj. Repeat this process for all j or the
number of features. In matrix form, this can be written as:

This can be simplified as:

 [E]' x [X] will give us a row vector matrix, since E' is 1 x m matrix and X is m x n
matrix. But we are interested in getting a column matrix, hence we transpose the resultant
matrix.

More succinctly, it can be written as:

Since (A * B)' = (B' * A'), and A'' = A, we can also write the above as