Image classification in PHP

using neural networks

Karol Przybyszewski
kprzybyszewski@gmail.com
About me

My first PHP source code - around 1999

Live and work in

Białystok / Poland

CTO VP of Engineering
Presentation goals
1. To show how neural network work at the lowest
possible level.
2. To demystify neural networks, show how simple it is.
3. To show that you can train neural networks in pure PHP.
What is a neural
network ? (Artificial) Neural network is a
simplified brain model.
You have one in your head -
Natural Neural Network ;)
It consists of a number of neurons
that process information.
Aka:
● artificial neural network
(ANN)
● neural network (NN)
 Input layer Hidden layer Output layer

X1

X2

Output 1
X3

Output 2
X4

X5
Output m

Xn
Who knows how neural
networks work in detail?

That's what we'll find out today :)

Which year is considered to be
the beginning of NN?
 1943
McCulloch and Pitts

The first mathematical description of the neuron's operation

and data processing.
 It’s a basic building block of NN

What is a
A system that processes the values
neuron ? of input signals into a single output
value:

X1
w1
● x - input signals
w2 ● w - input weights
X2
output
∑ f(∑)
● ∑ - weighted sum (x,w)
● Bias - threshold value
(additional weight)
wn ● f(∑) - activation function
Xn
Bias
 the 70’s
In 1969 Minsky and Papert in their research paper proved limitations of single
layer neural networks

Effect: significant reduction of NN research funding

 the 80’s
1986 - Significant advances in multilayered neural network training methods -
backpropagation as error correction method

Effect: to this day, the unstoppable success story of NN

continues
Handwritten digits recognition
The MNIST dataset

● dataset of handwritten digits from 0 to 9

● training dataset - 60,000 images
● test dataset - 10,000 images
● available on: http://yann.lecun.com/exdb/mnist/
● one digit - 28x28 pixels

Yann LeCun (1960)

Director of AI Research, Facebook

MNIST - Modified National Institute of Standards and Technology

The MNIST classification

0
4

5
9

2
7

1
Input data (1)
Input data - simplified MNIST (2)

00000000000000110000000000000000
00000000000000111100000000000000
00000000000001111110000000000000
00000000000011111111000000000000
Grayscale values transformed to black (1) and white (0)
00000000000111111100000000000000
00000000000111111110000000000000
00000000000111111110000000000000
00000000011111110000000000000000
00000000001111110000000000000000
00000000011111100000000000000000
00000000111111100000000000000000
<?php
00000000111111000000000000000000
00000000111110000000000000000000
00000000111111000000000000000000
00000000111110000000000000000000
$input = array();
00000000111110011111110000000000
00000000111111111111111000000000
00000000111111111111111100000000
00000000111111111111111111000000
//read characters line by line and add to $input
00000000111111111101111111000000
00000000111111000000000011110000
for each pixel
00000000111110000000000011110000
00000000111110000000000011110000
$input[] = $pixel;
00000000011111000000000001111000
00000000011111000000000001111000
00000000001111100000000001111100
00000000001111110000001111111000
00000000001111110000111111111000
00000000000011111111111111110000
print count($input);
00000000000001111111111111100000
00000000000000111111111111100000
00000000000000001111111100000000
//Result is 28x28 = 784

The input layer is 784 numbers (0 or 1)

Output data (1) 6

0 0
One hot encoding

1 0

2 0
<?php
3 0
//for the number 6 it’s a 10 elements array
with 1 on the sixth index 4 0

1,0,0,0);
$output = array(0,0,0,0,0,0, 5 0

6 1
7 0

8 0

9 0
 0 1 2 3 4 5 6 7 8 9

0 1 0 0 0 0 0 0 0 0 0

1 0 1 0 0 0 0 0 0 0 0

2 0 0 1 0 0 0 0 0 0 0

3 0 0 0 1 0 0 0 0 0 0

4 0 0 0 0 1 0 0 0 0 0

5 0 0 0 0 0 1 0 0 0 0
Output data (2)

6 0 0 0 0 0 0 1 0 0 0

7 0 0 0 0 0 0 0 1 0 0

8 0 0 0 0 0 0 0 0 1 0

9 0 0 0 0 0 0 0 0 0 1
 One hot encoding
Pixels of digit 6 The hidden layer Output 1
0
Dragons ?

X1 1 0 Output 2

? Output 3
X2 0
0

? Output 4
0
X3
0

0 Output 5
?
X4
0
1 Output 6
?
X5
0 0 Output 7

Output 8
0
?

X784 1 Output 9
0

0 Output 10
LINEAR ALGEBRA
Scalar / Vector / Matrix
 Matrix in PHP - 2 dim array
$a = array(array());
Columns Rows
Matrix addition <?php

public function add($matrix)

{
$result = [[]];
for($i = 0; $i<$this->height; $i++)
{
for($j = 0; $j<$this->width; $j++)
{
$result[$i][$j] =
$this->array[$i][$j]

+
$matrix->array[$i][$j];
}
}
return new Matrix($result);
}
 Matrix
subtraction <?php

public function sub($matrix)

{
$result = [[]];
for($i = 0; $i<$this->height; $i++)
{
for($j = 0; $j<$this->width; $j++)
{
$result[$i][$j] =
$this->array[$i][$j]

-
$matrix->array[$i][$j];
}
Same as addition but }
return new Matrix($result);
with the minus sign :) }
 Multiplying
matrix by a <?php

public function multiply($val)

{

single number $result = [[]];

for($i = 0; $i<$this->height; $i++)
{
for($j = 0; $j<$this->width; $j++)
scalar multiplication {
$result[$i][$j] =
$this->array[$i][$j]

*
$val;
}
}
return new Matrix($result);
}
Element-wise
matrix <?php

public function multiplyByMatrix($matrix)

{

multiplication $result = [[]];

for($i = 0; $i<$this->height; $i++)
{
for($j = 0; $j<$this->width; $j++)
Hadamard product {
$result[$i][$j] =
$this->array[$i][$j]

*
$matrix->array[$i][$j];
}
}
return new Matrix($result);
}
 Proper matrix <?php

multiplication public function dot($matrix)

{
$w = 0;
$result=[[]];
dot product for($i = 0; $i<$this->height; $i++)
{
for($j = 0; $j<$matrix->width; $j++)
{
for($h = 0; $h<$this->width; $h++)
{
$w +=
$this->array[$i][$h]

*
$matrix->array[$h][$j];
}
$result[$i][$j] = $w;
1*7 + 2*9 + 3*11 = 7+18+33 = 58 $w=0;
}
}
return new Matrix($result);
}
Transposed
matrix <?php

public function transpose()

{
$result = [[]];
for($i = 0; $i<$this->width; $i++)
{
for($j = 0; $j<$this->height; $j++)
{
$result[$i][$j] =
$this->array[$j][$i];
}
}
return new Matrix($result);
}
 Summary class Matrix
{
public function add($matrix)
public function sub($matrix)
1. Matrix class in ‘pure’ public function multiply($val)
public function multiplyByMatrix($matrix)
PHP
public function dot($matrix)
public function transpose()
2. 6 methods to perform
}
operations on matrices
3. about 100 lines of code
class Matrix

Constructor

$m = new Matrix($width, $height);

$m = new Matrix(array(array()));

public function applyFunction($function)

{
$result = [[]];
for($i = 0; $i<$this->height; $i++)
{
for($j = 0; $j<$this->width; $j++)
{
$result[$i][$j] =
$function($this->array[$i][$j]);
}
}
return new Matrix($result);
}
Reminder: neuron

X1 Activation function
w1

w2
X2
output
∑ f(∑)

wn
Xn
Bias
Sigmoid
function
<?php

$__SIGMOID = function($x)
{
return 1/(1+exp(-$x));
};
 Input layer
Hidden layer Output layer
$X = new Matrix(1,784);

$H = new Matrix(1,15); $Y = new Matrix(1,10);

X1

X2

Output 1
X3

Output 2
X4

X5
Output 10

$B2 = new Matrix(1,10);

Output layer bias

X784 $B1 = new Matrix(1,15);

$W2 = new Matrix(15,10);
$W1 = new Matrix(784,15); Hidden layer bias Weights hidden -> output
Weights input -> hidden
Summary of data structures
Description Code

1 Input layer $X = new Matrix(1,784);

2 Weights input-> hidden $W1 = new Matrix(784,15);

3 Hidden layer bias $B1 = new Matrix(1,15);

4 Hidden layer $H = new Matrix(1,15);

5 Weights hidden -> output $W2 = new Matrix(15,10);

6 Output layer bias $B2 = new Matrix(1,10);

7 Output layer $Y = new Matrix(1,10);

Morpheus was right ….
Propagating data forward through the NN

X1
w11

SIGMOID

SIGMOID
W2
w12
X2
H Y

function computeOutput($input) w1784

X784
1 $X = new Matrix([$input]); B1 B2

2 $H = $X->dot($W1)->add($B1)->applyFunction($__SIGMOID);

3 $Y = $H->dot($W2)->add($B2)->applyFunction($__SIGMOID);
Network training

Process of selection of the neural network hyperparameters:

● $W1, $W2 - weights

● $B1, $B2 - bias

that input data (vector of pixels of digit 6) after propagating through the network
gives the output vector:

0 0 0 0 0 0 0 0 0
1
Finding values of the hyperparameters

● You can do it manually by trial and error :)

○ not recommended !
● It can be done automatically
○ the backpropagation method
The backpropagation process

1. Initialize the network and the algorithm

2. Computing the output based on the input data
3. Calculating output error
4. Weights and biases correction
5. Is the network trained?
a. YES – go to 6

b. NO – go back to point 2

6. The end
Calculating the network error (1)

● a function that calculates mean squared error of the

network
● Y* - expected output for a given input
● Y - actual output calculated by the network
Calculating the network error (2)

● We are looking for the minimum of

the error function
● When Y* = Y the J = 0 - no error

Warning !
Faint-hearted folks are asked to
close their eyes for the next 5
slides.
DEAMONS FROM THE DEEPEST
BOTTOM OF HELL

DERIVATIVES
Calculating network error (1) - derivatives
 Calculating network error (2) - derivatives

1 $dJdB2 =
$Y->
sub($Y2)->
multiply($H->dot($W2)->add($B2)->applyFunction($__SIGMOIDPRIME));

2 $dJdB1 =
$dJdB2->
dot($W2->transpose())->
multiply($X->dot($W1)->add($B1)->applyFunction($__SIGMOIDPRIME));

3 $dJdW2 = $H->transpose()->dot($dJdB2);
4 $dJdW1 = $X->transpose()->dot($dJdB1);
Calculating network error (3) - weights/biases
correction
Calculating network error (4) - weights/biases
correction

1 $W1 = $W1->sub($dJdW1->multiply($learningRate));

2 $W2 = $W2->sub($dJdW2->multiply($learningRate));

3 $B1 = $B1->sub($dJdB1->multiply($learningRate));

4 $B2 = $B2->sub($dJdB2->multiply($learningRate));
Network training - summary
● essential source code contains only 8 lines
● it’s all about correcting weights and biases
● it’s all based on matrix operations !

function learn($expectedOutput)
$Y2 = new Matrix([$expectedOutput]); // expected network output
1. $dJdB2 = $Y->sub($Y2)->multiply($H->dot($W2)->add($B2)->applyFunction($__SIGMOIDPRIME));
2. $dJdB1 = $dJdB2->dot($W2->transpose())->multiply($X->dot($W1)->add($B1)->applyFunction($__SIGMOIDPRIME));
3. $dJdW2 = $H->transpose()->dot($dJdB2);
4. $dJdW1 = $X->transpose()->dot($dJdB1);
5. $W1 = $W1->sub($dJdW1->multiply($learningRate));
6. $W2 = $W2->sub($dJdW2->multiply($learningRate));
7. $B1 = $B1->sub($dJdB1->multiply($learningRate));
8. $B2 = $B2->sub($dJdB2->multiply($learningRate));
Training network in PHP !

● $numberOfEpochs - how many times we want to repeat the training

process for the whole training set
● $numberOfRecords - number of training records

for($j = 0; $j<$numberOfEpochs; $j++)

{
for ($i=0; $i<$numberOfRecords; $i++)
{
computeOutput($data[0][$i]);
learn($data[1][$i]);
}
}
Training takes time ...

Training time depends on:

● amount of observations (here number

of images)
● size of the input layer (and other layers
too)
● number of epochs - how many times we
want to train the network, or
● maximum level of network error that we
are willing to accept
Testing the trained network

$testRecords - number of test records

Our hyperparameters are already trained: weights $W1, $W2, and biases $B1, $B2

for ($i = 0; $i<$testRecords; $i++)

{
print expectedOutput($data[$i]);
print computeOutput($data[$i]);
}
Summary

1. Input data is an array of numbers

2. Output data is an array of numbers
3. Training is a process of calculating
weights and biases in order to get the
correct output
4. Network training is based on matrix
operations
 I KNOW
NEURAL NETWORKS !

Neo, The Matrix

Simple experiment

● 926 images in the training dataset

● 20 images in the test dataset
● 30 epochs

Laptop used for the experiment

● SSD hard drive

● processor intel i7, 2.60 GHz
● RAM 8 GB
● processor utilization during the experiment 10%-15%
Example script output / accuracy

Expected value | Actual value

0 0 0 0 0 0 0 0 1 0 |0|0|0|0|0|0|0|0|0|0|
0 0 0 1 0 0 0 0 0 0 |0|0|0|1|0|0|0|0|0|0|
0 0 0 0 0 0 0 0 1 0 |0|0|0|0|0|0|0|0|0|0|
0 0 0 0 0 0 0 1 0 0 |0|0|0|0|0|0|0|1|0|0|
0 1 0 0 0 0 0 0 0 0 |0|1|0|0|0|0|0|0|0|0|
0 0 0 0 0 0 0 1 0 0 |0|0|0|0|0|0|0|1|0|0|
0 0 0 0 0 0 0 0 0 1 |0|0|0|0|0|0|0|0|0| 1|
0 0 0 0 0 0 0 0 0 1 |0|0|0|0|0|0|0|0|0| 1|
0 0 0 0 0 1 0 0 0 0 |0|0|0|0|0| 1|0|0|0|0|
0 0 0 0 0 1 0 0 0 0 |0|0|0|0|0| 1|0|0|0|0|
0 0 0 0 0 0 0 0 1 0 |0|0|0|0|0|0|0|0| 1|0|
0 1 0 0 0 0 0 0 0 0 |0|1|0|0|0|0|0|0|0|0|
0 0 0 0 1 0 0 0 0 0 |0|0|0|0| 1|0|0|0|0|0| 17 of 20 recognized correctly
0 0 0 0 0 1 0 0 0 0 |0|0|0|0|0| 1|0|0|0|0|
0 0 0 0 1 0 0 0 0 0 |0|0|0|0| 1|0|0|0|0|0| 85% accuracy
0 0 0 0 0 0 0 0 1 0 |0|0|0|0|0|0|0|0| 1|0|
0 0 0 1 0 0 0 0 0 0 |0|0|0| 1|0|0|0|0|0|0|
0 1 0 0 0 0 0 0 0 0 |0|0|0|0|0|0|0|0|0|0|
0 0 0 0 0 0 1 0 0 0 |0|0|0|0|0|0| 1|0|0|0|
0 0 0 0 0 1 0 0 0 0 |0|0|0|0|0| 1|0|0|0|0|
Results summary

1. Our simple, 1-layer network achieved 85% accuracy.

2. Given the simplicity of architecture, 85% is a pretty good result.
3. Close to the result obtained in 1998 by Yann Lecun - 88%, using a similar
1-layer network.
4. The latest solutions, based on complex architectures (including convolutional
networks) achieve accuracy of 99.97%.
Execution times for matrix operations
Number of Execution time Execution time %
calls (total)

add 111 120 0,31 s 0,09 %

sub 138 900 70,55 s 19,80 %

multiply 111 120 47,25 s 13,26 %

multiplyByMatrix 55 560 0,14 s 0,04 %

dot 194 460 230,06 s 64,56 %

transpose 83 340 7,20 s 2,02 %

apply 111 124 0,76 s 0,22%

Total: 356,27 s
Total execution time

Total script execution time: 368,83 s

Total time execution time of the matrix operations: 356,27 s

96,6 % of execution time is spent on matrix operations !

Number epochs 30

Number of images 926

How do we speed
up matrix
operations ?
Nvidia / GPU / CUDA / CUBLAS

● CUDA - is a parallel computing platform and application programming

interface (API) model (Compute Unified Device Architecture)

● CUBLAS - implementation of BLAS (Basic Linear Algebra Subprograms) using

CUDA
TensorFlow / Keras / PyCUDA

Keras - open source, high level neural networks Python library. Designed for easy
and fast experimentation .

TensorFlow - open source, low level library for numerical computations based on
data flow graphs.

PyCUDA - allows to access API Nvidia CUDA API from Python’a

Example GPU/CUBLAS benchmark
Single-precision (SP)/double precision (DP) multiplication of (24576 x 32) i (32 x
24576) matrices, giving in result a matrix (24576 x 24576). Hardware: Tesla P100
(1 GPU) https://www.sharcnet.ca/help/index.php/Linear_algebra_on_the_GPU

SP time (s) DP time (s)

CPU - 1 core 1.53 3.07

CPU - 16 core 0.13 0.25

122x
CUBLAS - without memory 0.021 0.025
transfer

CUBLAS - with memory transfer 0.43 0.62

Call To Action !

There is job to do: binding to CUDA in PHP !

https://qiita.com/d_nishiyama85/items/0d1a6939adfe0e29fb0e

Japan/Tokyo: d_nishiyama85 / updated on 2018-03-02

Nishiyama did a comparison of matrix multiplication using:

1. ‘pure’ PHP
2. C extension
3. OpenBLAS (http://www.openblas.net/)
4. cuBLAS - (BLAS by NVIDIA CUDA)
Performance tests - dot product of squared mat.
PHPCUDA? / PHPOpenBLAS ?

https://github.com/dnishiyama85/PHPMatrix

This is a PHP extension which computes float matrix products. It supports

OpenBLAS sgemm and cuBLAS sgemm function.

sgemm - Single precision floating General Matrix Multiply

Running sample code:

$ php -d extension=./modules/matrix.so sample.php

There is already
NN lib in PHP
Binding to
FANN Fast Artificial Neural
Network

No BLAS/CUDA support
Image classification
Object detection
Object detection and attribution
Style transfer

=
+
Image completion
Reminder of the presentation
goals
Who understand now how NN work?

Who wants to do some NN experimenting on his/her own ?

Meet me at the conference

● staying at the Clarion Hotel

● leaving Sunday 3pm
● happy to share knowledge about:
○ Machine Learning / Deep Learning
○ Software Development Processes
○ Internet of Things
kprzybyszewski@gmail.com