Self-Organizing Map (or Kohonen Map or SOM) is a type of Artificial

Neural Network which is also inspired by biological models of neural systems

from the 1970s. It follows an unsupervised learning approach and trained its
network through a competitive learning algorithm. SOM is used for clustering
and mapping (or dimensionality reduction) techniques to map multidimensional
data onto lower-dimensional which allows people to reduce complex problems
for easy interpretation. SOM has two layers, one is the Input layer and the other
one is the Output layer.
The architecture of the Self Organizing Map with two clusters and n input
features of any sample is given below:

An input data of size (m, n) where m is the number of training examples and n is
the number of features in each example. First, it initializes the weights of size (n,
C) where C is the number of clusters. Then iterating over the input data, for each
training example, it updates the winning vector (weight vector with the shortest
distance (e.g Euclidean distance) from training example). Weight updating rule is
given by :

wij = wij(old) + alpha(t) * (xik - wij(old))

where alpha is a learning rate at time t, j denotes the winning vector, i denotes the
ith feature of training example and k denotes the kth training example from the
input data. After training the SOM network, trained weights are used for
clustering new examples. A new example falls in the cluster of winning vectors.

Algorithm

Training:
Step 1: Initialize the weights wij random value may be assumed. Initialize the
learning rate α.
Step 2: Calculate squared Euclidean distance.
D(j) = Σ (wij – xi)^2 where i=1 to n and j=1 to m
Step 3: Find index J, when D(j) is minimum that will be considered as winning
index.
Step 4: For each j within a specific neighborhood of j and for all i, calculate the
new weight.
wij(new)=wij(old) + α[xi – wij(old)]
Step 5: Update the learning rule by using :
α(t+1) = 0.5 * t
Step 6: Test the Stopping Condition.

Learning Vector Quantization

Learning Vector Quantization ( or LVQ ) is a type of Artificial Neural

Network which also inspired by biological models of neural systems. It is based
on prototype supervised learning classification algorithm and trained its network
through a competitive learning algorithm similar to Self-Organizing Map. It can
also deal with the multiclass classification problem. LVQ has two layers, one is
the Input layer and the other one is the Output layer. The architecture of the
Learning Vector Quantization with the number of classes in an input data and n
number of input features for any sample is given below:

How Learning Vector Quantization works?

Let’s say that an input data of size ( m, n ) where m is the number of training
examples and n is the number of features in each example and a label vector of
size ( m, 1 ). First, it initializes the weights of size ( n, c ) from the first c number
of training samples with different labels and should be discarded from all training
samples. Here, c is the number of classes. Then iterate over the remaining input
data, for each training example, it updates the winning vector ( weight vector
with the shortest distance ( e.g Euclidean distance ) from the training example ).
The weight updation rule is given by:
if correctly classified:
wij(new) = wij(old) + alpha(t) * (xik - wij(old))
else:
wij(new) = wij(old) - alpha(t) * (xik - wij(old))

where alpha is a learning rate at time t, j denotes the winning vector, i denotes the
ith feature of training example and k denotes the kth training example from the
input data. After training the LVQ network, trained weights are used for
classifying new examples. A new example is labeled with the class of the
winning vector.

Competitive learning in the Kohonen network

Competitive learning is a fundamental concept in Self-Organizing Maps (SOM),

which is a type of artificial neural network used for data visualization and
dimensionality reduction. Competitive learning is the key mechanism through
which SOMs organize and represent input data in a lower-dimensional space.
Here's an overview of how competitive learning works in SOM:

Neuron Activation: A SOM consists of a grid of neurons arranged in a two-

dimensional lattice. Each neuron is associated with a weight vector of the same
dimension as the input data. When presented with an input data point, each neuron
calculates its activation level based on the similarity between the input and its
weight vector. Common similarity measures include the Euclidean distance or
cosine similarity.

Winner-Takes-All: In competitive learning, only one neuron is allowed to "win"

and be the most activated neuron for a given input. This neuron is referred to as the
"winner" or the "Best Matching Unit" (BMU). The BMU is the neuron whose
weight vector is most similar to the input data.

Weight Update: After determining the BMU, the weights of the BMU and its
neighboring neurons are updated. Typically, a neighborhood function is used to
determine which neurons are considered neighbors and to what extent they will be
updated. The purpose of updating the weights is to bring the BMU and its
neighbors closer to the input data point. Over time, this process results in the
SOM's weight vectors becoming more aligned with the input data distribution.

Learning Rate: The learning rate is a parameter that controls the magnitude of
weight adjustments during training. Initially, the learning rate is set to a relatively
high value, and it gradually decreases over time. This schedule ensures that the
SOM converges to a stable representation of the data.

Neighborhood Function: The neighborhood function defines the spatial extent over
which neurons' weights are updated. It starts with a larger radius and gradually
shrinks during training. Neurons close to the BMU have their weights updated
more significantly, while those farther away are updated to a lesser extent. This
mechanism helps the SOM organize the input data in a spatially meaningful way.

Iterative Training: The competitive learning process is repeated iteratively,

presenting the input data points one by one or in batches, and adjusting the neuron
weights according to the competitive and cooperative principles outlined above.
This process continues until convergence or until a set number of iterations is
reached.

The result of competitive learning in a SOM is a topological map where similar

input data points are mapped to nearby neurons in the lattice, preserving the
inherent structure and relationships within the data. SOMs are often used for tasks
like data visualization, clustering, and reducing high-dimensional data to a lower-
dimensional representation that captures the essential features of the data.

Mexican Hat Networks:

Mexican Hat networks, also known as "Mexican Hat filters" or "Mexican Hat
wavelets," are a type of artificial neural network architecture that is inspired by the
mathematical concept of the Mexican Hat wavelet or the Laplacian of Gaussian
(LoG) function. These networks are primarily used in image processing and
computer vision for tasks such as edge detection, texture analysis, and feature
extraction.

The Mexican Hat wavelet, or the LoG function, is a two-dimensional spatial filter
that is characterized by a positive central peak and a surrounding ring of negative
values. It looks like a sombrero or "Mexican hat," hence the name. This wavelet is
used to highlight edges and transitions in images and is particularly effective at
detecting fine details and patterns.

In the context of neural networks, Mexican Hat networks are designed to apply the
principles of this wavelet to process image data. Here are some key features of
Mexican Hat networks:
Convolutional Layers: Mexican Hat networks typically use convolutional layers to
apply the Mexican Hat wavelet to the input image. Convolution involves sliding a
small filter (the Mexican Hat wavelet) across the image and computing the dot
product between the filter and the image at each position.

Edge Detection: Mexican Hat wavelets are good at detecting edges in images
because they respond strongly to areas of rapid intensity change. The positive
central peak highlights edges, while the negative ring suppresses noise and
enhances the edge structure.

Feature Extraction: Mexican Hat networks can be used to extract relevant features
from images, making them useful for tasks like texture analysis and pattern
recognition. These networks can be customized by adjusting the size and shape of
the Mexican Hat wavelet.

Tunable Parameters: Mexican Hat networks allow for the adjustment of parameters
like the size of the central peak and the width of the surrounding ring. These
parameters control the sensitivity to edge details and the extent of noise
suppression.

Preprocessing: In some cases, Mexican Hat networks are used as a preprocessing

step before feeding the extracted features into other neural network architectures
for further processing and classification.

It's worth noting that Mexican Hat networks are not as commonly used as
traditional convolutional neural networks (CNNs) in modern computer vision
tasks. CNNs have shown significant success in a wide range of applications and
have become the go-to choice for many image-related tasks. However, Mexican
Hat networks can still be useful in specific scenarios where fine-grained edge
detection and feature extraction are essential.

In summary, Mexican Hat networks are a specialized type of neural network

architecture that applies the mathematical properties of the Mexican Hat wavelet
for image processing tasks, particularly for edge detection and feature extraction in
computer vision applications.
Cooperative and Adaptive Processes in SOM
In Self-Organizing Maps (SOM), cooperative and adaptive processes are essential
mechanisms that help the network organize and represent input data effectively.
These processes work together to create a topological map of the input data,
preserving its underlying structure. Here's an explanation of these processes in
SOM:

Cooperative Process:

The cooperative process in SOM involves the competitive interaction among

neurons. When an input data point is presented to the SOM, all neurons compute
their similarity to the input and compete to become the Best Matching Unit
(BMU). The BMU is the neuron whose weight vector is most similar to the input.

Once the BMU is determined, the cooperative process comes into play as
neighboring neurons also adapt their weights based on the input. These
neighboring neurons are determined by a neighborhood function, typically a
Gaussian function, which defines the extent of cooperation.

Neurons closer to the BMU cooperate more with the BMU and undergo more
significant weight adjustments. This cooperation among neighboring neurons helps
the SOM capture the global and local structure of the input data.

Adaptive Process:

The adaptive process in SOM refers to the adjustment of neuron weights based on
the input data. After identifying the BMU and its neighbors, the weights of these
neurons are updated to become more similar to the input data. The degree of
weight adjustment depends on the learning rate, which decreases over time.

The learning rate controls the size of weight updates. It starts with a relatively high
value and gradually decreases as training progresses. This adaptive learning rate
allows the SOM to converge and stabilize its representation of the data.

Additionally, the adaptive process may involve other factors, such as the
adaptation of the neighborhood size (the spatial extent over which neurons
cooperate and adapt). The neighborhood size typically shrinks as training proceeds,
which leads to a finer level of organization as the network converges.
Cooperative and adaptive processes work hand in hand to ensure that a SOM
creates a meaningful topological representation of the input data. The competitive
process identifies the most relevant neurons for a given input, while the
cooperative process helps the network to organize the data in a spatially
meaningful way by considering neighboring neurons. The adaptive process fine-
tunes the weights of these neurons to align them with the input distribution,
gradually refining the map's representation.

This combination of competitive, cooperative, and adaptive processes makes

SOMs useful for tasks such as data visualization, clustering, and dimensionality
reduction, as they capture and preserve the inherent structure and relationships in
the input data.