Kohonen self-

organizing map
(Kohonen SOM)
also deep SOM
 • modification of the stimulus by
lateral feedback
• network topology
• arrangement of neurons
(rectangular, hexagonal)
• matching criterion
• the winner
Contents • topological environment
(neighborhood)
• learning
– spatially correlated learning
– general rules of learning
• illustration of mapping using
SOM
• deep SOM

AIDP, M.Oravec, ICSM FEI STU

 Kohonen SOM

• belongs to:
✓SELF-ORGANZINIG SYSTEMS BASED ON
COMPETITIVE LEARNING

• self-organizing learning
– i.e. unsupervised learning
• SOM, Self-Organizing Map
AIDP, M.Oravec, ICSM FEI STU
 Modification of stimulus by lateral
feedback
• lateral feedback - depends on the lateral distance from
the point of its action
• one-dimensional lattice of neurons with forward
connections and lateral feedback connections (lateral
are shown only for the middle neuron):

AIDP, M.Oravec, ICSM FEI STU

 Modification of stimulus by lateral
feedback
• lateral feedback - usually expressed by the
function of a “Mexican hat function”
output

Three types of lateral interactions

between neurons:
• narrow area of ​lateral excitation
• area of ​lateral inhibition
• area of ​weaker excitation that
surrounds the inhibition zone
(this area is usually neglected) input

The effects of the lateral feedback described by the

„Mexican hat function“ are replaced by the
introduction of the topological neighborhood of
AIDP, M.Oravec, ICSM FEI STU
active neurons.
 SOM
• Principle of creating a topographic map:
– the position of the output neuron corresponds to the input data
feature

• 1D, 2D, 3D, ..., grid allows to have adjacent neurons

• rectangular or hexagonal arrangement of neurons

x = [ x1 , x2 ,..., x p ]T R p - input vector connected in parallel to all

neurons i in the network

w i = [ wi1 , wi 2 ,..., wip ]T  R p - weight vector of neuron i

AIDP, M.Oravec, ICSM FEI STU

Rectangular arrangement of neurons in a self-
organizing map

AIDP, M.Oravec, ICSM FEI STU

Hexagonal arrangement of neurons in a self-
organizing map

AIDP, M.Oravec, ICSM FEI STU

 SOM
• matching criterion
o dot product
o Euclidian distance

x − w c = min x − w i
i

• winner
o minimal distance defines „winner" wc

• spatially correlated learning

o learning neurons do not work independently of each other, but
as subsets of neurons in a mutual topological relationship

• topological neighborhood Nc(n)

AIDP, M.Oravec, ICSM FEI STU N – means neighborhood

 Topological neighborhood
Examples of the topological neighborhood N c (n) ,
where n1  n2  n3
The neighborhood is very
wide at the beginning and
shrink monotonically over
time.

Initial wide neighborhood

allows the map to have an
approximate, "coarse"
global alignment of the
network, then shrinking the
neighborhood will improve
the spatial resolution of the
map, but the obtained
global alignment is not
disturbed.
AIDP, M.Oravec, ICSM FEI STU
 Topological neighborhood
Examples of the topological neighborhood N c (n) ,
where n1  n2  n3

AIDP, M.Oravec, ICSM FEI STU

 (n) is the learning rate parameter
SOM learning 0  ( n)  1 , ( n) decreases with time

w i ( n + 1) = 

w i ( n) + ( n) x( n) − w i ( n)  if i  N c ( n)
ak
 w i ( n) if i  N c ( n)
ak
hci(n) – scalar kernel function


w i (n + 1) = w i (n) + hci (n) x(n) − w i (n) 
hci (n) = ( n) inside N c (n) and hci (n) = 0 out of N c
• hci(n): vectors rc and ri are coordinates of neurons c and i
(
hci (n ) = h0 (n )exp - ri − rc /  2 (n )
2
)
where h0 = h0 (n) and  =  (n ) are suitable decreasing function of time
AIDP, M.Oravec, ICSM FEI STU
 General rules for learning the map:

• weight initialization

• number of learning steps

SOM • learning rate parameter

learning – phase of global map alignment

– phase of fine-tuning the map

• topological neighborhood
– phase of global map alignment
– phase of fine-tuning the map

AIDP, M.Oravec, ICSM FEI STU

 SOM learning
• weight initialization:
– random selection of initialization weights, the only
limitation is that they should be different

• number of learning steps:

– the number of steps (by step we mean the introduction of
one input sample) at least 500 times greater than the
number of network elements

– if only a small number of samples are available, they must

be repeated to complete the required number of steps

AIDP, M.Oravec, ICSM FEI STU

 SOM learning
• Parameter (n) :

o for approximately the first 1000 steps the value (n ) can be close to 1,
then it must decrease monotonically

o the exact rule is not important,  = (n) can be a linear or exponential

function, or it can be inversely proportional to n
o e.g. (n) = 0.9 (1 − n / 1000)

o During this initial phase, the sorting of w i will take place (phase of global
map alignment), other steps are needed to fine-tune the map

o after the sorting process  = (n) can acquire small values (e.g. 0.01)
during a long learning time (phase of fine-tuning the map)

AIDP, M.Oravec, ICSM FEI STU

 SOM learning
• Topological neighborhood N c (n) :

o If the neighborhood is too small at the beginning of learning, the map will
not be sorted globally - instead, it will be variously "subdivided".

o We can prevent this phenomenon if at first the neighborhood

N c = N c (0) is wide enough and then shrinks over time.
▪ The initial radius of N c can even be larger than half the diameter of
the net.

o During approximately the first 1000 steps, when the correct global
alignment of the map occurs and  = ( n) is large enough, the radius may
decrease linearly until it contains only one map element.

o During the fine-tuning process, N c may still contain the nearest adjacent
neurons of the winner c.

AIDP, M.Oravec, ICSM FEI STU

 Note

• SOM does not suffer by overtraining

AIDP, M.Oravec, ICSM FEI STU

 ILUSTRATIONS OF
MAPPING BY SELF-
ORGANIZING MAP

AIDP, M.Oravec, ICSM FEI STU

2D input space -> 2D SOM

SOM 10x10
2D evenly distributed inputs
x = [x1, x2]
-1<x1<1
-1<x2<1

a) randomly initialized weights

b) SOM after 50 iterations
c) SOM after 1000 iterations
d) SOM after 10000 iterations

Kohonen,T.: The Self-Organizing

Map, Proc. of the IEEE, Vol.78,
No.9, 1990, pp. 1464-1480
AIDP, M.Oravec, ICSM FEI STU
2D input space -> 1D SOM

SOM 1x100
2D evenly distributed
inputs x = [x1, x2]
-1<x1<1
-1<x2<1

a) randomly initialized weights

b) SOM after 10000 iterations
c) SOM after 25000 iterations

Kohonen,T.: The Self-Organizing

Map, Proc. of the IEEE, Vol.78,
No.9, 1990, pp. 1464-1480
AIDP, M.Oravec, ICSM FEI STU
 • https://www.youtube.com/watc
h?v=eLK1Taj0EJo

Self-Organizing
• https://www.youtube.com/watc
Map 1D and 2D h?v=VVQm8ZfuuSA
Visualization
• https://www.youtube.com/watc
h?v=0BM1wwc3Nco

AIDP, M.Oravec, ICSM FEI STU

 Kohonen,T.: The Self-Organizing
Map, Proc. of the IEEE, Vol.78,
No.9, 1990, pp. 1464-1480

2D input space -> 2D SOM

2D evenly distributed inputs
x = [x1, x2], -1<x1<1, -1<x2<1

AIDP, M.Oravec, ICSM FEI STU

 Kohonen,T.: The Self-Organizing
Map, Proc. of the IEEE, Vol.78,
No.9, 1990, pp. 1464-1480

2D input space -> 1D SOM

2D evenly distributed inputs
x = [x1, x2]

AIDP, M.Oravec, ICSM FEI STU

 Kohonen,T.: The Self-Organizing
Map, Proc. of the IEEE, Vol.78,
No.9, 1990, pp. 1464-1480

3D input space -> 2D SOM

3D inputs x = [x1, x2, x3]

AIDP, M.Oravec, ICSM FEI STU

 Kohonen,T.: The Self-Organizing
Map, Proc. of the IEEE, Vol.78,
No.9, 1990, pp. 1464-1480

3D input space -> 2D SOM

3D inputs x = [x1, x2, x3]

AIDP, M.Oravec, ICSM FEI STU

 Enter to web browser
• web traffic self-organizing map
• social behaviour self-organizing map
• customer behaviour self-organizing map
• ...

AIDP, M.Oravec, ICSM FEI STU

 DEEP SOM

Deep self-organizing maps

AIDP, M.Oravec, ICSM FEI STU

 DSOM, Deep Self-Organizing Map for
Visual Classification, 2015
• proposal of a Deep Self-Organizing Map (DSOM) algorithm

• It consists of
– layers of alternating self-organizing map
• The self-organizing layer is made up of certain numbers of SOMs, with
each map only looking at a local region block on its input.
– sampling layers.
• The winning neuron's index value from every SOM in self-organizing
layer is then organized in the sampling layer to generate another 2D
map, which could then be fed to a second self-organizing layer.

» N. Liu, J. Wang and Y. Gong, "Deep Self-Organizing Map for visual

classification," 2015 International Joint Conference on Neural Networks
(IJCNN), Killarney, 2015, pp. 1-6, doi: 10.1109/IJCNN.2015.7280357.

AIDP, M.Oravec, ICSM FEI STU

 DSOM architecture

A DSOM architecture for handwritten digits recognition. This architecture consists of 3 self-
organizing layers (C1, C2, C3) and 2 sampling layers (S1, S2).
C1, C2, and C3 contains 100, 25 and 1 SOMs respectively. Each map in the self-organizing layer
models a sub-region of its input. Every sampling layer is formed by the wining neurons in its
previous self-organizing layer.

N. Liu, J. Wang and Y. Gong, "Deep Self-Organizing Map for visual

AIDP, M.Oravec, ICSM FEI STU classification," 2015 International Joint Conference on Neural Networks
(IJCNN), Killarney, 2015, pp. 1-6, doi: 10.1109/IJCNN.2015.7280357
 DSOM self-organizing layer

Illustration of the self-organizing layer. There are Nmap x Nmap SOMs in the layer, each
focusing on a different sub-region on the original input.

N. Liu, J. Wang and Y. Gong, "Deep Self-Organizing Map for visual

AIDP, M.Oravec, ICSM FEI STU classification," 2015 International Joint Conference on Neural Networks
(IJCNN), Killarney, 2015, pp. 1-6, doi: 10.1109/IJCNN.2015.7280357
 DSOM sampling layer

Illustration of the sampling layer. The color nodes highlight the winning neurons on each
map. The Nmap x Nmap winners are organized on a two-dimensional map.

N. Liu, J. Wang and Y. Gong, "Deep Self-Organizing Map for visual

AIDP, M.Oravec, ICSM FEI STU classification," 2015 International Joint Conference on Neural Networks
(IJCNN), Killarney, 2015, pp. 1-6, doi: 10.1109/IJCNN.2015.7280357
 UDSOM, 2020

• Unsupervised Deep Self-Organizing Map

(UDSOM) algorithm for feature extraction

» Sakkari, Mohamed & Zaied, Mourad. (2020). A

Convolutional Deep Self-Organizing Map Feature
extraction for machine learning. Multimedia Tools and
Applications. 79. 10.1007/s11042-020-08822-9.

AIDP, M.Oravec, ICSM FEI STU

 UDSOM
• Feature extraction algorithm:

• UDSOM consists of layers of an alternating self-

organizing layer, a sampling layer and a ReLU function
AIDP, M.Oravec, ICSM FEI STU Sakkari, Mohamed & Zaied, Mourad. (2020). A Convolutional Deep Self-Organizing
Map Feature extraction for machine learning. Multimedia Tools and Applications. 79.
10.1007/s11042-020-08822-9.
 UDSOM
• The self-organizing layer is composed of some numbers of 2D maps,
with each map focusing on modelling a local sub-region of the input
space.

• Algorithm:
➢ 1. Split the input image into N sub-regions according to a first
parameter setting.
➢ 2. Build N SOMs with each map focusing on modelling a local sub-
region.
➢ 3. Do Sub-sampling (Convolution and pooling).
➢ 4. ReLU.
➢ 5. Get the image abstraction provided in 3.
➢ 6. Spread the weights to the top layer.
➢ 7. Repeat the procedure (1,2,3,4,5 and 6) applying the new settings.

AIDP, M.Oravec, ICSM FEI STU Sakkari, Mohamed & Zaied, Mourad. (2020). A Convolutional Deep Self-Organizing
Map Feature extraction for machine learning. Multimedia Tools and Applications. 79.
10.1007/s11042-020-08822-9.
 UDSOM
• Sampling module:

AIDP, M.Oravec, ICSM FEI STU Sakkari, Mohamed & Zaied, Mourad. (2020). A Convolutional Deep Self-Organizing
Map Feature extraction for machine learning. Multimedia Tools and Applications. 79.
10.1007/s11042-020-08822-9.
 UDSOM
• ReLu module:
• ReLU: neurons in black have never been activated so
they will have crushed

AIDP, M.Oravec, ICSM FEI STU Sakkari, Mohamed & Zaied, Mourad. (2020). A Convolutional Deep Self-Organizing
Map Feature extraction for machine learning. Multimedia Tools and Applications. 79.
10.1007/s11042-020-08822-9.
 UDSOM
• Why not using a CNN and what is the difference between UDSOM
and CNN?

➢ the deep architecture of UDSOM uses local receptive fields as CNN,

➢ the function of hidden layers, based on a non-linear projection of
high-dimensional data over a small space using Kohonen maps, is
completely different from CNN

• The CNN hidden layer consist of a convolution function followed by a pooling

operation allows to generate a feature map for the top-level layer.
• In the UDSOM, the hidden layer process local sub-region with SOM map and
aggregate the BMUs to generate a feature abstraction level based a bottom-up
hierarchical classification procedure. The neurons on the SOM map are
grouped in such a way to give a more global view of the map, by grouping the
closest neurons, then the closest groups. The bottom-up method proposed is
based on the principle of minimization of inter-class inertia.

AIDP, M.Oravec, ICSM FEI STU Sakkari, Mohamed & Zaied, Mourad. (2020). A Convolutional Deep Self-Organizing
Map Feature extraction for machine learning. Multimedia Tools and Applications. 79.
10.1007/s11042-020-08822-9.
 UDSOM
• Feature extraction based BMUs clustering
o BMU=Best Matching Unit) = winning neuron

The only similarity between the deep SOM proposed and the CNN
architecture is the notion of the local receptive field.

AIDP, M.Oravec, ICSM FEI STU Sakkari, Mohamed & Zaied, Mourad. (2020). A Convolutional Deep Self-Organizing
Map Feature extraction for machine learning. Multimedia Tools and Applications. 79.
10.1007/s11042-020-08822-9.
 UDSOM
• Computational complexity:

• The computational complexity of the UDSOM is much

lower than the CNN.

» Since, in any CNN the maximum time of training goes in

the Back-Propagation of errors in the Fully Connected
Layer (depends on the image size). Also, the maximum
memory is also occupied by them.
» Contrairwise, the UDSOM do not require any Back-
Propagation process.

AIDP, M.Oravec, ICSM FEI STU Sakkari, Mohamed & Zaied, Mourad. (2020). A Convolutional Deep Self-Organizing
Map Feature extraction for machine learning. Multimedia Tools and Applications. 79.
10.1007/s11042-020-08822-9.