Self-Organization

• Economic representation of data with all their interrelationships is one of

the most central problems in information sciences.

• ➔Such an operation is obviously characteristic of the operation of the

brain.

• In thinking, and in the sub-conscious information processing, there is a

general tendency to compress information by forming reduced
representations of the most relevant facts, without loss of knowledge
about their interrelationships.

• The purpose of intelligent information processsing seems in general to be

creation of simplified images of the observable world at various levels of
abstraction.
• Cerebral cortex is is organized according according to different sensory
modalities

• There also areas performing specialized tasks, e.g. speech control and
analysis of sensory signals (visual, auditory, somatosensory, etc.)

• Associative areas onto which signals of different modality converge

• Fine structure of the areas: the visual, somatosensory, etc., response
signals are abtained in the same topographical order on the cortex
in which they were received at the sensory organs.
What is Self-Organization

•
 What is Self-Organization?

• Self-organization is a process of attraction and repulsion in which

the internal organization of a system, normally an open system,
increases in complexity without being guided or managed by an
outside source

• Self-organizing systems typically (but not always) display emergent

properties
 What is Self-Organization?

Self-organization is the spontaneous often seemingly purposeful formation of spatial,

temporal, spatio-temporal structures or functions in systems composed of few or many
components. In physics, chemistry and biology self-organization occurs in open systems
driven away from thermal equilibrium.

The process of self-organization can be found in many other fields also, such as
economy, sociology, medicine, technology.
Motivation: The Problem Statement

The problem is how to find out

semantics relationship among lots
of information without manual labor
How do I know, where to put
my new data in, if I know
nothing about information‘s
topology?
When I have a topic, how can I
get all the information about it,
if I don‘t know the place to
search them?

JASS 05 Information Visualization with SOMs © sebis 10

Motivation: The Idea

Computer know automatically information classification and put them together

Input Pattern 1

Input Pattern 2

Input Pattern 3

JASS 05 Information Visualization with SOMs © sebis 11

Motivation: The Idea

Text objects must be automatically produced with semantics relationships

Semantics Map

Topic1

Topic2

Topic3

JASS 05 Information Visualization with SOMs © sebis 12

• The Self-Organizing Map (SOM) is one of the most popular neural network models.
• It belongs to the category of competitive learning networks.
• The Self-Organizing Map is based on unsupervised learning, which means that no
human intervention is needed during the learning and that little needs to be known
about the characteristics of the input data.
• We could, for example, use the SOM for clustering data without knowing the class
memberships of the input data. The SOM can be used to detect features inherent to
the problem and thus has also been called SOFM, the Self-Organizing Feature Map.

• The Self-Organizing Map was developed by professor Kohonen.

 Self-Organization

Phonetic
typewriter
• The SOM algorithm is based on unsupervised, competitive learning.
• It provides a topology preserving mapping from the high dimensional space to map
units.
• Map units, or neurons, usually form a two-dimensional lattice and thus the mapping is
a mapping from high dimensional space onto a plane.
• The property of topology preserving means that the mapping preserves the relative
distance between the points. Points that are near each other in the input space are
mapped to nearby map units in the SOM.
• The SOM can thus serve as a cluster analyzing tool of high-dimensional data.
• Also, the SOM has the capability to generalize. Generalization capability means that
the network can recognize or characterize inputs it has never encountered before. A
new input is assimilated with the map unit it is mapped to.
(http://users.ics.aalto.fi/jhollmen/dippa/node9.html)
SOM videos:

SELF ORGANISING MAPS: INTRODUCTION - YouTube

https://www.youtube.com/watch?v=g8O6e9C_CfY
 SOM algorithm – weight adaptation

16 September 2021
Self-Organizing Maps

SOM – Algorithm Overview

1. Randomly initialise all weights

2. Select input vector x = [x1, x2, x3, … , xn]
3. Compare x with weights wj for each neuron j to determine winner
4. Update winner so that it becomes more like x, together with the
winner’s neighbours
5. Adjust parameters: learning rate & ‘neighbourhood function’
6. Repeat from (2) until the map has converged (i.e. no noticeable
changes in the weights) or pre-defined no. of training cycles have
passed

JASS 05 Information Visualization with SOMs © sebis 23

Initialisation

(i)Randomly initialise the weight

vectors wj for all nodes j

JASS 05 Information Visualization with SOMs © sebis 24

 Input vector
(ii) Choose an input vector x from the training set

In computer texts are shown as a Region

frequency distribution of one word.

A Text Example:
Self-organizing maps (SOMs) are a
data visualization technique invented by
Professor Teuvo Kohonen which reduce
the dimensions of data through the use
of self-organizing neural networks. The
problem that data visualization attempts Self-organizing 2
to solve is that humans simply cannot maps 1
visualize high dimensional data as is so data 4
visualization 2
technique are created to help us technique 2
understand this high dimensional data. Professor 1
invented 1
Teuvo Kohonen 1
dimensions 1
...
Zebra 0

JASS 05 Information Visualization with SOMs © sebis 25

Finding a Winner

(iii) Find the best-matching neuron w(x), usually the neuron whose weight vector
has smallest Euclidean distance from the input vector x

The winning node is that which is in some sense ‘closest’ to the input vector
‘Euclidean distance’ is the straight line distance between the data points, if they
were plotted on a (multi-dimensional) graph
Euclidean distance between two vectors a and b, a = (a1,a2,…,an), b =
(b1,b2,…bn), is calculated as:
=  (a −b )
2
a, b i d i
i
Euclidean distance

JASS 05 Information Visualization with SOMs © sebis 26

Weight Update

SOM Weight Update Equation

wj(t +1) = wj(t) + (t) w(x)(j,t) [x - wj(t)]

“The weights of every node are updated at each cycle by adding

Current learning rate × Degree of neighbourhood with respect to winner ×
Difference between current weights and input vector
to the current weights”
Example of (t) Example of w(x)(j,t)
L.
rate

–x-axis shows distance from winning node

No. of cycles –y-axis shows ‘degree of neighbourhood’ (max. 1)

JASS 05 Information Visualization with SOMs © sebis 27

Example: Self-Organizing Maps
The animals should be ordered by a neural networks.
And the animals will be described with their attributes(size, living space).
e.g. Mouse = (0/0)

Size: Living space:

small=0 medium=1 big=2 Land=0 Water=1 Air=2
Mouse Lion Horse Shark Dove
Size small medium big big small
Living space Land Land Land Water Air
(0/0) (1/0) (2/0) (2/1) (0/2)

JASS 05 Information Visualization with SOMs © sebis 34

Example: Self-Organizing Maps
After the fields of map will be initialized with random values, animals will be
ordered in the most similar fields. If the mapping is ambiguous, anyone of
fields will be seleced.

(0/0) (0/2)
(2/2)
Mouse (0/0), Lion (1/0) Dove (0/2)

(2/1) (2/0)
(0/0)
Shark (2/1) Horse (2/0)

(1/1) (1/1) (0/0)

JASS 05 Information Visualization with SOMs © sebis 35

Example: Self-Organizing Maps

Auxiliary calculation for the field of left above:

Old value in the field: (0/0) (0/0)
Lion (1/0)
Direct ascendancies:
Difference Mouse (0/0): (0/0)
Difference Lion (1/0): (1/0)
Sum of the difference: (1/0)
Thereof 50%: (0.5/0) Training
Influence of the allocations of the neighbour fields:
Difference Dove (0/2): (0/2)
Difference Shark (2/1): (2/1)
Sum of the difference: (2/3)
(1/0.75)
Thereof 25%: (0.5/0.75) Lion (1/0)
New value in the field: (0/0) + (0.5/0) + (0.5/0.75)= (1/0.75)

JASS 05 Information Visualization with SOMs © sebis 36

Example: Self-Organizing Maps
This training will be done in every field. After the network had been trained,
animals will be ordered in the similarest field again.

(1/0.75) (0.25/1)
(1.5/1.5)
Lion Dove

(2/0)
(1.25/0.5) (1/0.75)
Horse

(1.25/1) (0.5/0)
(1/1)
Shark Mouse

JASS 05 Information Visualization with SOMs © sebis 37

Example: Self-Organizing Maps
This training will be very often repeated. In the best case the animals should be at
close quarters ordered by similarest attribute.

(0.1875/1.25)
(0.75/0.6875) (1.125/1.625)
Dove

(1.5/0)
(1.375/0.5) (1/0.875)
Hourse

(1.625/1) (1/0.75) (0.75/0)

Shark Lion Mouse

Land animals

JASS 05 Information Visualization with SOMs © sebis 38

 Example: Self-Organizing Maps
Animal names and their attributes
Dove Hen Duck Goose Owl Hawk Eagle Fox Dog Wolf Cat Tiger Lion Horse Zebra Cow
Small 1 1 1 1 1 1 0 0 0 0 1 0 0 0 0 0
is Medium 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0
Big 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1
2 legs
4 legs
1
0
1
0
1
0
1
0
1
0
1
0
1
0
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
A grouping according to similarity has
has Hair
Hooves
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
0
1
0
1
0
1
0
1
0
1
0
1
1
1
1
1
1 emerged
Mane 0 0 0 0 0 0 0 0 0 1 0 0 1 1 1 0

likes
Feathers
Hunt
1
0
1
0
1
0
1
0
1
1
1
1
1
1
0
1
0
0
0
1
0
1
0
1
0
1
0
0
0
0
0
0
peaceful
Run 0 0 0 0 0 0 0 0 1 1 0 1 1 1 1 0
to Fly 1 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0
Swim 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0

birds

hunters
[Teuvo Kohonen 2001] Self-Organizing Maps; Springer;

JASS 05 Information Visualization with SOMs © sebis 39

Self-Organizing Maps

SOM – Result Example

Classifying World Poverty

‘Poverty map’ based on 39 indicators from World Bank statistics (1992)

JASS 05 Information Visualization with SOMs © sebis 43
 SOM applications

• Soms have been used in many applications; most

often in vision systems for vector quantisation.
• Vision based autonomous
robot navigation systems
historically use pre-
determined models of the
environment.
• Kohonen SOMs have
recently been investigated to
allow such systems to develop
their own internal
representation of the external
environment.
16 September 2021
 Some Theory
Property 1. Approximation of the Input
Space
• The feature map , represented by the set
of synaptic weight vectors {wj} in the input
space A, provides a good approximation to
the input space H.
• The theoretical basis of the idea is rooted
in vector quantization theory.
 Some Theory
Property 2. Topological Ordering
• The feature map  computed by the SOM
algorithm is topologically ordered in the sense
that the spatial location of a neuron in the
lattice corresponds to a particular domain or
feature of input patterns.
 Some Theory
Property 3. Density Matching

• The feature map  reflects variations in the statistics of the input

distributions: regions in the input space H from which the sample vectors
x are drawn with a high probability of occurrence are mapped onto larger
domains of the output space A, and therefore with better resolution than
regions in H from which sample vectors x are drawn with a low probability
of occurrence.
• Minimum-distortion encoding, according to which the curvature terms
and all higher-order terms in the distortion measure due to noise model
() are retained. m(x)  fx ⅓(x)
• Nearest-neighbor encoding, which emerges if the curvature terms are
ignored, as in the standard form of the SOM algorithm. m(x)  fx ⅔(x)
 Some Theory
Property 4. Feature Selection
• Given data from an input space with
an nonlinear distribution, the self-
organizing map is able to select a set
of best features for approximating
. the
underlying distribution.