CHE 509: Process Dynamics

MODULE 6:
Empirical Modeling Using Artificial Neural Networks (ANN)

Ayorinde Bamimore, Ph.D.

Department of Chemical Engineering
Obafemi Awolowo University
 Module Objectives
 To be able to define ANN model,
 To Compare and contrast ANN with
biological neurons,
 To state the various applications of ANN,
 To develop ANN model for different types
of systems.

Obafemi Awolowo University 2

Chemical Engineering
 What is an Artificial Neural Network?
Definition: According to Simon Haykin [1999]:
A neural network is a massively parallel distributed processor
made up of simple processing units, which has a natural
propensity for storing experiential knowledge and making it
available for use. It resembles the brain in two respects:
1. Knowledge is acquired by the network from its
environment through a learning process
2. Interneuron connection strengths, known as synaptic
weights, are used to store the acquired knowledge.

Simon Haykin (1999) Neural networks - A comprehensive foundation, second edition,

Prentice Hall Inc., NJ, USA.

Obafemi Awolowo University 3

Chemical Engineering
 Other Terms used in connection with ANN

Other terms often used to describe neural network includes:

 Neurocomputers
 Connectionist networks
 Parallel distributed processors
 Artificial intelligence
 Machine learning
 Deep and shallow learning
Other things that should catch your fancy:
 Big data
 Data analytics etc

Obafemi Awolowo University 4

Chemical Engineering
 Biological Neural Network

Fig. 6.20: The biological neuron

Obafemi Awolowo University 5

Chemical Engineering
 Artificial Neural Network (ANN)

Fig. 6.21: The Artificial neuron

Obafemi Awolowo University 6

Chemical Engineering
 A Biological and Artificial Neuron

Fig. 6.22: Comparison between biological and artificial neuron

Obafemi Awolowo University 7

Chemical Engineering
 Uses of Neural Networks

Neural networks have been used extensively in recent

years to model a wide range of physical and chemical
phenomena and to model other non-engineering
situations such as:
[1.] Function approximation
[2.] Stock market analysis
[3.] Chess strategies
[4.] Speech and pattern recognition
[5.] Medical diagnoses
[6.] Dynamic system modeling and control
[7.] Time series analysis, etc.
Obafemi Awolowo University 8
Chemical Engineering
 Types of Neural Networks

Common types include:

 Feedforward neural network (FFNN)

 Recurrent neural network (RNN)

 Radial-basis function network

 Convolution neural network (CNN)

Obafemi Awolowo University 9

Chemical Engineering
 Mathematical Representation of FFNN model

An artificial neural network consists of many simple processing elements called

neurons, operating in parallel. An individual processing element (neuron) in a

neural network can have any number of inputs, but only one output that is

generally related to the inputs by a transfer function. A schematic representation

a typical neuron is shown in Figure 6.23

The output of such a neuron is given as:

(6.40)

Obafemi Awolowo University 10

Chemical Engineering
 Mathematical Representation of FFNN model

where: input signal; weight; bias; output signal and is a

transfer (activation) function, the most frequently used activation function

includes:

 sigmoid function
 hyperbolic tangent function
 sine function
 linear and saturated linear function
 ReLU
 SPOCU
Obafemi Awolowo University 11
Chemical Engineering
 The Structure of an ANN model

Fig. 6.23: An individual processing element of a neural network

Obafemi Awolowo University 12

Chemical Engineering
 Topology of a FFNN

Fig. 6.24a: Multilayer FFNN with three layers (5 – 3 – 2)

Obafemi Awolowo University 13

Chemical Engineering
 Topology of a FFNN

Fig. 6.24b: Multilayer FFNN with three layers (5 – 3 – 2)

Obafemi Awolowo University 14

Chemical Engineering
 Mathematical Representation of FFNN model

The output of FFNN according to Fig.6.24(b) can be computed thus:

(6.41)

(6.42)

Substituting (6.41) into (6.42) gives

(6.43)

where , and represent vectors of input, hidden layer and

output respectively.

Obafemi Awolowo University 15

Chemical Engineering
 Mathematical Representation of FFNN model

and represent input weight matrix and output weight matrix.

and are input bias matrix and output bias matrix. is the activation

function for the hidden layer. Tansig activation function is given by:

(6.44)

is the activation function for the output layer. It is selected as a linear

function in this work.

Obafemi Awolowo University 16

Chemical Engineering
 Topology of a RNN

Fig. 6.25a: Recurrent neural network with three layers (5 – 3 – 2)

Obafemi Awolowo University 17

Chemical Engineering
 Topology of a RNN

Fig. 6.25b: Recurrent neural network with three layers (5 – 3 – 2)

Obafemi Awolowo University 18

Chemical Engineering
 Mathematical Representation of RNN model

The output of RNN, according to Fig. 5.25b, can be represented in the

general nonlinear state space form:

(6.45)

(6.46)

where is a vector of hidden layer output, is a vector of

network output. is a unit delay. is the hidden layer weight matrix.

All other variables remain the same.

Obafemi Awolowo University 19

Chemical Engineering
 Training of an ANN Model

The unknown parameters in an ANN model are the weights (output and
input) and the biases (output and input). These parameters are usually
estimated using input/output data set.

The iterative process of calculating the weights and the biases in a neural
network is referred to as training. It is a highly intensive optimization
process but algorithms abound in literatures for doing this. MATLAB
(2015a) neural network toolbox is employed in this work for the training.

Usually the input/output data set is divided into the training set and the
validation set.

MATLAB toolbox will further divide the training set into:

Training data, testing data and validation data

Obafemi Awolowo University 20

Chemical Engineering
 Worked Example 1

Example 1:The following data were collected during experiment:

The following model is proposed to fit the data:

(6.47)

(a) Determine the model parameters and using nonlinear regression.

(b) Fit a neural network model to the data using MATLAB function
“fitnet”.
(c) Make a comparison between the predictions by the models derived in
(a) and (b) and the original data.

Obafemi Awolowo University 21

Chemical Engineering
 Solution to Worked Example 1

(a) Follow these steps to generate model parameters in MATLAB

STEP 1: Create MATLAB function file named “fSSR”
function f = fSSR(a,xm,ym)
yp=a(1)*xm.*exp(a(2)*xm) ;
f = sum((ym-yp).^2);

STEP 2: Enter the data below and invoke the function as follows:

x = [0.1, 0.2, 0.4, 0.6, 0.9, 1.3, 1.5, 1.7, 1.8];

y = [0.75, 1.25, 1.45, 1.25, 0.85, 0.55, 0.35, 0.28, 0.18];
[xopt, fval]=fminsearch(@fSSR, [1, 1], [], x, y)

9.8974 2.5319,

Obafemi Awolowo University 22

Chemical Engineering
 Solution to Worked Example 1

(b) The subroutine in MATLAB for fitting FFNN to this data is called
“fitnet”. Enter the following into the command prompt/m-file:
x = [0.1, 0.2, 0.4, 0.6, 0.9, 1.3, 1.5, 1.7, 1.8];
y = [0.75, 1.25, 1.45, 1.25, 0.85, 0.55, 0.35, 0.28, 0.18];
p=[x] ;
t=[y] ;
net=fitnet(5) ;
net.divideFcn = '' ;
net.trainParam.epochs = 6000;
net.trainParam.max_fail = 10000;
net.trainParam.min_grad=1e-15;
net.trainParam.show = 50;
net.trainParam.lr=0.01;
net.trainParam.goal = 0;
%%%%%%%%%%%%%%%%%%%%
net = init(net);
[net, tr] = train(net,p,t);
out1 = sim(net,p);

Obafemi Awolowo University 23

Chemical Engineering
 Solution to Worked Example 1

out1=out1';
genFunction(net,'save_net', 'MatrixOnly','yes');
figure(1)
plot(p, t, '*b',p,out1,'r','LineWidth',2)
legend('Actual Data','ANN model prediction')
xlabel('x')
ylabel('y')

The following results were obtained:

Fig. 6.26: (1 – 5 – 1) FFNN architecture

Obafemi Awolowo University 24

Chemical Engineering
 Solution to Worked Example 1

Fig. 6.27: A comparison between the actual data and the ANN model prediction

Obafemi Awolowo University 25

Chemical Engineering
 Solution to Worked Example 1

Fig. 6.28: Parity plot of Output prediction vs the target

Obafemi Awolowo University 26

Chemical Engineering
 Solution to Worked Example 1

Fig. 6.29: MATLAB Neural Network training tool

Obafemi Awolowo University 27

Chemical Engineering
 Solution to Worked Example 1

Table 6.10: Neural Network Simulation function

Obafemi Awolowo University 28

Chemical Engineering
 Solution to Worked Example 1

Table 6.10: Neural Network Simulation function (Cont’d)

Obafemi Awolowo University 29

Chemical Engineering
 Solution to Worked Example 1

Table 6.10: Neural Network Simulation function (Cont’d)

Obafemi Awolowo University 30

Chemical Engineering
 Solution to Worked Example 1

Table 6.11: Comparison between actual, regressed and FFNN model prediction

x y FFNN prediction FFNN prediction

Nhidden layer=5 Nhidden layer=3
0.1 0.75 0.77 0.75 0.75
0.2 1.25 1.19 1.25 1.25
0.4 1.45 1.44 1.45 1.45
0.6 1.25 1.30 1.25 1.25
0.9 0.85 0.91 0.85 0.85
1.3 0.55 0.48 0.55 0.54
1.5 0.35 0.33 0.35 0.37
1.7 0.28 0.23 0.28 0.25
1.8 0.18 0.19 0.18 0.20

Obafemi Awolowo University 31

Chemical Engineering