Introduction to Self-Organizing Maps (SOMs)

In this article, we’re going to introduce self-organizing maps. We assume the reader has prior experience with neural networks.

Plan of Attack:

1. Introduction to SOMs
2. How SOMs work
3. Components of Self Organization
4. Steps of training a SOM
5. SOM implementations
6. Introduction to MiniSom
7. Build a Simple SOM using minisom

Introduction to SOMs

Self-organizing maps are a class of unsupervised learning neural networks used for feature detection. They’re used to produce a low-dimension space of training samples. Therefore, they’re used for dimensionality reduction.

SOMs differ from other artificial neural networks because they apply competitive learning as opposed to error correlated learning, which involves backpropagation and gradient descent. In competitive learning, nodes compete for the right to respond to the input data subset. The training data usually has no labels and the map learns to differentiate and distinguish features based on similarities.

How SOMs work

The figure below illustrates how we train a self-organizing map. The purple blob is the distribution of the training data. The small white disc is the current training datum drawn from that distribution. At first, the SOM nodes are arbitrarily positioned in the data space. The node (highlighted in yellow) nearest the training datum is selected. It’s moved towards the training datum, as are its neighbors on the grid. After many iterations, the grid tends to approximate the data distribution (right).

The Euclidean distance to all weight vectors is computed when we feed the training data into the network. The neuron whose weight vector is most similar to the input is called the best matching unit (BMU). The weights of the BMU and neurons close to it in the SOM grid are adjusted towards the input vector. Once the BMU has been determined, the next step is to calculate which of the other nodes are within the BMU’s neighborhood.

Components of Self Organization

• Initialization: all connection weights are initialized to random values.
• Competition: Output nodes compete against themselves to be activated. Only one of them is activated at a time. The activated neuron is called a winner-takes-all neuron. Because of this competition, the neurons are forced to organize themselves, forming a self-organizing map (SOM).
• Cooperation: The spatial location of a topological neighborhood of excited neurons is determined by the winning neuron. This provides the basis for cooperation among neighboring neurons.
• Adaptation: Excited neurons decrease individual values of the discriminant function. This is done in relation to the input pattern via suitable adjustment of the associated connection weights. This way, the response of the winning neuron to the subsequent application of a similar input pattern is enhanced. The discriminant function is defined as the squared Euclidean distance between the input vector x and the weight vector wj for each neuron j:

Steps for training a Self-Organizing Map

Training a self-organizing map occurs in several steps:

1. Initialize the weights for each node. The weights are set to small standardized random values.

2. Choose a vector at random from the training set and present to the lattice.

3. Examine every node to calculate which one’s weight is most like the input vector. This will allow you to obtain the Best Matching Unit (BMU). We compute the BMU by iterating over all the nodes and calculating the Euclidean distance between each node’s weight and the current input vector. The node with a weight vector closest to the input vector is marked as the BMU.

4. Calculate the radius of the neighborhood of the BMU. Nodes found within the radius are deemed to be inside the neighborhood of the BMU.

5. Weights of the nodes found in step 4 are adjusted to make them more like the input vector. The weights of the nodes closer to the BMU are adjusted more.

6. Repeat step 2 for N iterations.

Self-Organizing Map Implementations

SOMs are commonly used in visualization. Below is a visualization of the world’s poverty data by country. The countries with higher quality of life are clustered towards the upper left while the most poverty-stricken nations are clustered towards the lower right.

Other implementations of self-organizing maps include:

• Data compression
• Speech recognition
• Separating sound sources
• Fraud detection

Introduction to MiniSom

MiniSom is a minimalistic Numpy-based implementation of self-organizing maps. MiniSom can be installed using the Python package management utility pip.

pip install minisom

How to Use MiniSom

In order to use MiniSom, your data has to be organized as a Numpy array, with the rows representing a single observation.

data = [[ 0.80, 0.55, 0.22, 0.03],
[ 0.82, 0.50, 0.23, 0.03],
[ 0.80, 0.54, 0.22, 0.03],
[ 0.80, 0.53, 0.26, 0.03],
[ 0.79, 0.56, 0.22, 0.03],
[ 0.75, 0.60, 0.25, 0.03],
[ 0.77, 0.59, 0.22, 0.03]]

Once the dataset is set up you can implement MiniSom as follows. The first step is to import the MiniSom class after installing minisom.

from minisom import MiniSom

Next, we can initialize a 6-by-6 SOM with a learning rate of 0.5.

som = MiniSom(6, 6, 4, sigma=0.3, learning_rate=0.5)
Then we train the SOM on 100 iterations.
som.train_random(data, 100)

MiniSom implements two types of training: train_random and train_batch. In train_random, we train the model by picking random samples from our data. In train_batch, the samples are picked in the order that they are stored. A method called random_weights_init initializes the weights by picking random samples from the data.

Build a simple SOM using MiniSom

Now let’s use what we’ve learned so far to build a simple color quantization model. Color quantization is a process that reduces the distinct colors used in an image with the intention of making the new image as visually similar as possible to the original image.

This technique is useful in displaying images with many colors on devices that only display a limited number of colors. This challenge can arise due to memory allocation limitations. We’re going to work on a model that will learn the colors in an image of houses, and then use that to reconstruct the original image.

The first step is to import the MiniSom class from MiniSom, as well as numpy and Matplotlib. If you’re using Jupyter Notebooks, you’ll have to include %matplotlib inline in order to see your graphs in the Notebook.

from minisom import MiniSom
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

Loading the image

As noted earlier, MiniSom is a numpy implementation for self-organizing maps. We therefore have to load in our image as an array. We do this using Matplotlib’s imread utility.

img = plt.imread('house.jpg')

You can view the loaded image using the imshow function from Matplotlib.

plt.imshow(img)

Let’s check the shape of the image we’ve loaded

We notice it’s a 3D matrix. We need to convert the array into a 2D array by multiplying 730 and 1368 as seen below. Afterwards, we check the shape using the Numpy shape attribute.

pixels = np.reshape(img, (img.shape[0]*img.shape[1], 3))

SOM initialization and training

We initialize the MiniSom object and assign it to a variable som. The first parameter it takes is the dimensions of the self-organizing map. In our case, we’ll build a 3-by-3 SOM. This means that the final colors we get will be 3 * 3 which is 9. Feel free to experiment with this figure and see the different results you get. Obviously the larger the self-organizing map, the longer it will take to train.

The second parameter is input_len, which is the number of features in our dataset. In our case, we use 3, which corresponds to the shape of the pixels array. If you use a number other than this you’ll get a value error indicating that MiniSom couldn’t broadcast the input array from shape (3) to the number you’ve chosen.

The next parameter is sigma. Sigma is the radius of the different neighbors in the SOM. The default value for this is 1.0.

The last parameter is the learning_rate, which determines how much weights are adjusted during each iteration. After several experiments, I found a sigma of 0.1 and a learning rate of 0.2, which give some decent results.

som = MiniSom(x= 3, y = 3, input_len = 3, sigma=0.1, learning_rate=0.2)

The next step is to initialize the SOM’s weights to small, standardized random values. We achieve this using the random_weights_init function and by passing in our data (the pixels).

som.random_weights_init(pixels)

We now need to save the starting weights, which represent the image’s initial colors. We’ll visualize this later. The way we save these weights is by using the get_weights function and Python’s copy to ensure we get the weights before they get updated.

starting_weights = som.get_weights().copy()

We then train the pixels by running the train_random function. It takes two parameters. The first parameter is the dataset to train on and the second parameter is the number of iterations.

som.train_random(pixels, 100)

Vector quantization

The next step is to quantize each pixel of the image. In this process, we’ll reduce the number of colors in the image. MiniSom enables us to do this using the quantization utility.

qnt = som.quantization(pixels)

Building new image

If you’ll recall, at the beginning of this process we had to transform the image into a 2D array. However, now we need to build the image as a 3D image. We can kick of this process by creating a matrix with just zeros using the image dimensions we loaded in the beginning. Numpy allows us to do this easily using its zeros utility.

clustered = np.zeros(img.shape)

The next thing we need to do is place the quantized values into a new image. In order to do this, we use Numpy’s unravel_index functionality. This function takes three arguments:

• indices : array_like
• An integer array whose elements are indices into the flattened version of an array of dimensions dims.
• dims : tuple of ints

The shape of the array to use for unraveling indices.dex value:

• order : {‘C’, ‘F’}, optional
• Determines whether the indices should be viewed as indexing in row-major (C-style) or column-major (Fortran-style) order.

for i, q in enumerate(qnt):
  clustered[np.unravel_index(i, dims=(img.shape[0], img.shape[1]))] = q

Show the results

Matplotlib enables us to plot images on the axes using imshow. We do this using one Matplotlib figure and 4 subplots. The first and second subplots will show the original image and the new image. The third and fourth subplot will show the initial image colors and the learned colors.

We create a 12-by-6 figure. We then create the subplots. The subplots take 3 parameters: the number of rows, number of column, and the plots index. When these number are less than 10, they can be passed as three digits, as seen below. We then use imshow from Matplotlib to plot the images.

plt.figure(figsize=(12, 6))
plt.subplot(221)
plt.title('Original')
plt.imshow(img)
plt.subplot(222)
plt.title('Result')
plt.imshow(clustered)

We create the third and the fourth subplots in a similar manner.

plt.subplot(223)
plt.title('Initial Colors')
plt.imshow(starting_weights)
plt.subplot(224)
plt.title('Learnt Colors')
plt.imshow(som.get_weights())

We use the tight_layout functionality to ensure that our plots don’t become squeezed together. Finally we plot them using plt.show().

plt.tight_layout()
plt.show()

Conclusion

MiniSom has been used in unsupervised fraud detection, Neural Network Based Association Rule Mining, Learning hand-eye coordination for a humanoid robot using SOMs among others. To learn more about MiniSom and further implementations visit MiniSom’s Github page.

Discuss this post on Hacker News and Reddit.