Summary Report

Multi-Layer Perceptron Training Optimization Using Nature Inspired Computing

Abstract: A wide variety of classification and regression problems can be solved with Multi-
Layer Perceptron (MLP) artificial neural networks. However, there are some limitations
pertaining to the training and optimization of MLP networks. One of them is associated with the
optimization techniques used to train MLP such as stochastic gradient descent (SGD), because it
is mathematically tractable. This limitation can cause undesirable outputs.

The colonial selection algorithm (CSA) can be used to solve the problem of optimizing MLP
networks. CSA is one of the nature-inspired algorithms that can be used to accurately train
MLP networks. This paper presents a novel implementation of CSA to train MLP networks and
solve real-world problems such as breast cancer diagnosis, active sonar target classification,
wheat classification, and flower classification.

The CSA is used to find optimal weights and biases that can significantly improve the accuracy
of MLP networks. The performance of the proposed system was compared with other popular
nature-inspired algorithms such as genetic algorithm, ant colony optimization, particle swarm
optimization, Harris hawks optimization, moth-flame optimization, flower pollination algorithm,
and backpropagation. This comparison is benchmarked using five real-world data sets: Iris
Flower, Sonar, Wheat Seeds, Breast Cancer Wisconsin, and Haberman's Survival.

Comparative study results illustrate significant improvement gained by training MLP with CSA
over other training methods, hence it can be considered as a competitive approach to train
MLP networks to solve real-world problems in a variety of different disciplines.

Introduction: Optimization refers to make a system robust, as much functional and efficient
as possible. It also refers to the selection of best and optimal values from a range of given sets
of values. Training MLP can be articulated as an optimization problem, hence it can be solved by
using multiple optimization methods. Stochastic Gradient Descent (SGD) is one of the
optimization algorithms used to train MLPs, as it is flexible and mathematically tractable. SGD
uses backpropagation to calculate the gradient of each parameter and update weights and
biases. But this technique often converges to a local minimum or sub-optimal weight, resulting
in failure to find the accurate solution. Normally it is very difficult to find optimal or near-
optimal solutions but this problem can be solved by taking inspiration from nature to produce
near-optimal values of parameters.
Several nature-inspired algorithms have been used to train MLPs with promising results such as
GA, PSO Algorithm, Differential Evolution, Artificial Bee Colony, Grey Wolf optimizer, Lightning
Search Algorithm, Multi-Verse Optimizer, and Whale Optimization Algorithm are also other
popular nature-inspired algorithms that have been used to train MLPs and have shown good
performance. Hybrid systems which often combine two algorithms to get desirable results were
also used to train MLP models effectively. The aforementioned algorithms have numerous
advantages which them popular; these include the:

 Handling complex and large spaces with a lot of parameters

 Optimization of both discrete and continuous functions as well as multi-objective
problems.
 Do not require derived knowledge that might not be available for a lot of real-world
problems
 Faster and more effective than classical solutions and provide not just one but a number
of different solutions

This paper focuses on training MLP networks with CSA to avoid local minima and get optimal
values of the results to get desirable solution. CSA is inspired by biological immune systems and
fetches ideas and metaphors from them. It searches the solution space by creating a population
of candidate solutions called antibodies. The affinity of the candidate solutions is calculated and
compared with a candidate solution called an antigen. Those antibodies with higher affinity
were then selected and a number of clones were created proportional to the affinity of the
antibody. These cloned antibodies undergo a hyper mutation process to further refine the
antibodies, hyper mutation is inversely proportional to the affinity. Finally, reselection is
performed to get the optimal mutate clone and it becomes the antigen for the next generation,
and the same process is repeated until a desirable solution is obtained.