A self-organizing map of sigma–pi units

doi:10.1016/j.neucom.2006.05.014

Neurocomputing

Volume 70, Issues 13–15, August 2007, Pages 2552-2560

https://doi.org/10.1016/j.neucom.2006.05.014 Get rights and content

Abstract

By frame of reference transformations, an input variable in one coordinate system is transformed into an output variable in a different coordinate system depending on another input variable. If the variables are represented as neural population codes, then a sigma–pi network is a natural way of coding this transformation. By multiplying two inputs it detects coactivations of input units, and by summing over the multiplied inputs, one output unit can respond invariantly to different combinations of coactivated input units. Here, we present a sigma–pi network and a learning algorithm by which the output representation self-organizes to form a topographic map. This network solves the frame of reference transformation problem by unsupervised learning.

Section snippets

Model architecture

The network architecture is schematically displayed in Fig. 2. With the number of units in the corresponding layers being $N_{x}$ , $N_{y}$ and $N_{z}$ , the total number of possible sigma–pi connections ${w_{ijk}}$ is $N_{x} \cdot N_{y} \cdot N_{z}$ . This is in the order of, but still less than, the case of the basis function networks [6], [12], [16]. A unit i on the top layer is activated by the input vectors $x$ and $y$ via the relation $a_{i} = \sum_{j, k} w_{ijk} x_{j} y_{k} .$ Hence, a sigma–pi weight $w_{ijk}$ is effective, if unit j of the input vector $x$ is

General idea

For simplicity of notation we will term the output quantity the “sum location”, envisaging the relation $μ_{x} + μ_{y} = μ_{z}$ as a paramount example. For a given sum location $μ_{z}$ , there are many possible pairs of inputs $(μ_{x}, μ_{y})$ which lead to the same sum. Therefore, learning is about generating responses that are invariant to variations of input pairs which belong to the same sum location.

In order to generate these invariances, we will supply the learning algorithm with sets of input pairs that shall lead to

One-dimensional maps

Simple case: Fig. 6 shows the resulting connections of trained networks. In Fig. 6(a) the weights of each unit fall onto a diagonal line in input space along which the sum $μ_{x} + μ_{y}$ is a constant. This constant decreases linearly from left to right, indicating an “inverted” polarity of the map. Different initial random values of the weights can lead to another polarity. Test transformations of this network are displayed in Fig. 7(b) in response to the input shown in Fig. 7(a). The map units

Discussion

Based on our recent approach of a neural frame of reference transformation which was trained by supervised learning [17], we intend to use the model presented in this paper in the context of a neurally controlled robot docking maneuver. The supervised system has been tested on a robot simulator, and Fig. 10 explains the geometry on our PeopleBot robot.

The overall neural system which controls a robot to pick up an object will consist of three parts: (i) a visual system provides the horizontal

Acknowledgments

This research is part of the MirrorBot project supported by a EU, FET-IST programme, Grant IST-2001-35282, coordinated by Prof. Wermter.

References (18)

E. Sauser et al.
Three dimensional frames of references transformations using recurrent populations of neurons
Neurocomputing
(2005)
L.H. Snyder
Coordinate transformations for eye and arm movements in the brain
Curr. Opinion Neurobiol.
(2000)
C. Weber et al.
Robot docking with neural vision and reinforcement
Knowl.-Based Syst.
(2004)
C.H. Andersen, D.C. van Essen, B. Olshausen, Directed visual attention and the dynamic control of information flow, in:...
C.M. Bishop, Neural Networks for Pattern Recognition, Oxford University Press, Oxford,...
C.A. Buneo et al.
Direct visuomotor transformations for reaching
Nature
(2002)
Y.E. Cohen et al.
A common reference frame for movement plans in the posterior parietal cortex
Nat. Rev. Neurosci.
(2002)
J.D. Crawford et al.
Spatial transformations for eye–hand coordination
J. Neurophysiol.
(2004)
S. Deneve et al.
Efficient computation and cue integration with noisy population codes
Nat. Neurosci.
(2001)

There are more references available in the full text version of this article.

Cited by (39)

A multi-in and multi-out dendritic neuron model and its optimization
2024, Knowledge-Based Systems
Artificial neural networks (ANNs), inspired by the interconnection of real neurons, have achieved unprecedented success in various fields such as computer vision and natural language processing. Recently, a novel mathematical ANN model, known as the dendritic neuron model (DNM), has been proposed to address nonlinear problems by more accurately reflecting the structure of real neurons. However, the single-output design limits its capability to handle multi-output tasks, significantly lowering its applications. In this paper, we propose a novel multi-in and multi-out dendritic neuron model (MODN) to tackle multi-output tasks. Our core idea is to introduce a filtering matrix to the soma layer to adaptively select the desired dendrites to regress each output. Because such a matrix is designed to be learnable, MODN can explore the relationship between each dendrite and output to provide a better solution to downstream tasks. We also model a telodendron layer into MODN to simulate better the real neuron behavior. Importantly, MODN is a more general and unified framework that can be naturally specialized as the DNM by customizing the filtering matrix. To explore the optimization of MODN, we investigate both heuristic and gradient-based optimizers and introduce a two-step training method for MODN. Extensive experimental results performed on 11 datasets on both binary and multi-class classification tasks demonstrate the effectiveness of MODN, with respect to accuracy, convergence, and generality.
A novel nature inspired firefly algorithm with higher order neural network: Performance analysis
2016, Engineering Science and Technology, an International Journal
Citation Excerpt :
It was able to find an appropriate input–output mapping of various chaotic financial time series data with a good performance in learning speed and generalization capability. A sigma-pi network trained with an online learning algorithm for solving the frame of reference transformation problem has been presented by Cornelius Weber and Stefan Wermter [45]. An online gradient algorithm for Pi-Sigma neural networks with stochastic inputs with improved computational efficiency have been proposed by X. Kang et al. [46].
The applications of both Feed Forward Neural network and Multilayer perceptron are very diverse and saturated. But the linear threshold unit of feed forward networks causes fast learning with limited capabilities, while due to multilayering, the back propagation of errors exhibits slow training speed in MLP. So, a higher order network can be constructed by correlating between the input variables to perform nonlinear mapping using the single layer of input units for overcoming the above drawbacks. In this paper, a Firefly based higher order neural network has been proposed for data classification for maintaining fast learning and avoids the exponential increase of processing units. A vast literature survey has been conducted to review the state of the art of the previous developed models. The performance of the proposed method has been tested with various benchmark datasets from UCI machine learning repository and compared with the performance of other established models. Experimental results imply that the proposed method is fast, steady, reliable and provides better classification accuracy than others.
A novel Chemical Reaction Optimization based Higher order Neural Network (CRO-HONN) for nonlinear classification
2015, Ain Shams Engineering Journal
Citation Excerpt :
Li [48] has suggested a memory based Sigma–Pi–Sigma neural network for excellent learning convergence along with reducing the memory size and overcoming the possible extensive memory requirement problem. Weber and Wermter [49] have presented a sigma-pi network trained with an online learning algorithm for solving the frame of reference transformation problem. For financial time series prediction a novel application of Ridge polynomial network formed by adding different degrees of Pi–Sigma neural networks has been suggested by Ghazali et al. [50] which is able to find an appropriate input output mapping of various chaotic financial time series data with a good performance in learning speed and generalization capability.
In this paper, a Chemical Reaction Optimization (CRO) based higher order neural network with a single hidden layer called Pi–Sigma Neural Network (PSNN) has been proposed for data classification which maintains fast learning capability and avoids the exponential increase of number of weights and processing units. CRO is a recent metaheuristic optimization algorithm inspired by chemical reactions, free from intricate operator and parameter settings such as other algorithms and loosely couples chemical reactions with optimization. The performance of the proposed CRO-PSNN has been tested with various benchmark datasets from UCI machine learning repository and compared with the resulting performance of PSNN, GA-PSNN, PSO-PSNN. The methods have been implemented in MATLAB and the accuracy measures have been tested by using the ANOVA statistical tool. Experimental results show that the proposed method is fast, steady and reliable and provides better classification accuracy than others.
Convergence of batch gradient learning algorithm with smoothing L<inf>1/2</inf> regularization for Sigma-Pi-Sigma neural networks
2015, Neurocomputing
Citation Excerpt :
Sigma–Pi–Sigma neural networks (SPSNNs) are considered as efficient high-order neural networks which can learn to implement static mapping that multilayer neural networks and radial basis function networks usually do [1], since the output of the SPSNNs has the sum of product-of-sum form. A self-organizing map of Sigma–Pi units was provided in [2]. The applicability of networks built on Sigma–Pi units with Elman topology was explored in [3].
Sigma–Pi–Sigma neural networks are known to provide more powerful mapping capability than traditional feed-forward neural networks. The L_1/2 regularizer is very useful and efficient, and can be taken as a representative of all the $L_{q} (0 < q < 1)$ regularizers. However, the nonsmoothness of L_1/2 regularization may lead to oscillation phenomenon. The aim of this paper is to develop a novel batch gradient method with smoothing L_1/2 regularization for Sigma–Pi–Sigma neural networks. Compared with conventional gradient learning algorithm, this method produces sparser weights and simpler structure, and it improves the learning efficiency. A comprehensive study on the weak and strong convergence results for this algorithm are also presented, indicating that the gradient of the error function goes to zero and the weight sequence goes to a fixed value, respectively.
A new Sigma-Pi-Sigma neural network based on L<inf>1</inf> and L<inf>2</inf> regularization and applications
2024, AIMS Mathematics
A Multi-In and Multi-Out Dendritic Neuron Model and its Optimization
2023, arXiv

View all citing articles on Scopus

Cornelius Weber is a Junior Fellow at the Frankfurt Institute for Advanced Studies in Germany since March 2006. He graduated in physics in Bielefeld, Germany in 1995 and received his PhD in computer science in Berlin in 2000. Then he worked in the group of Alexandre Pouget in Brain and Cognitive Sciences, University of Rochester, USA. From 2002 to 2005 he worked in Hybrid Intelligent Systems at the University of Sunderland, UK throughout the EU-funded MirrorBot project. His research interests are in computational neuroscience, focusing on visual and motor systems, and robotic applications. In December 2003 he won the Machine Intelligence Prize of the British Computer Society in Cambridge, demonstrating the “visually guided grasping robot MIRA”. This publication is motivated by extending the robot's grasping range for such a scenario.

Stefan Wermter is professor in Intelligent Systems at the University of Sunderland, UK and is the Director of the Centre for Hybrid Intelligent Systems. His research interests are in intelligent systems, neural networks, cognitive neuroscience, hybrid systems, language processing and learning robots. He has a Diploma from the University of Dortmund, an MSc from the University of Massachusetts and a PhD and Higher Doctorate (Habilitation) from the University of Hamburg, all in computer science. He was a Research Scientist at ICSI, Berkeley in 1997 before accepting the Chair in Intelligent Systems at the University of Sunderland in 1998.

Professor Wermter has written or edited five books and published about 150 articles on this research area, including books like “Hybrid Connectionist Natural Language Processing” or “Connectionist, Statistical, and Symbolic Approaches to Learning for Natural Language Processing”, “Hybrid Neural Systems”, “Emergent Neural Computational Architectures based on Neuroscience” and “Biomimetic Neural Learning for Intelligent Robots”.

He is an Associate Editor of the journals “Connection Science”, the “International Journal for Hybrid Intelligent Systems” and the “Knowledge and Information Systems”. He is on the editorial board of the journals “Neural Networks”, “Cognitive Systems Research”, “Neural Computing Surveys”, “Neural Information Processing” and “Journal of Computational Intelligence”. Furthermore, he is leading the EU project MirrorBot on biomimetic multimodal learning in a mirror neuron-based robot and coordinates the EmerNet network on “emerging computational neural architectures based on neuroscience”.

View full text

A self-organizing map of sigma–pi units

Abstract

Section snippets

Model architecture

General idea

One-dimensional maps

Discussion

Acknowledgments

Neurocomputing

Curr. Opinion Neurobiol.

Knowl.-Based Syst.

Direct visuomotor transformations for reaching

Nature

A common reference frame for movement plans in the posterior parietal cortex

Nat. Rev. Neurosci.

Spatial transformations for eye–hand coordination

J. Neurophysiol.

Efficient computation and cue integration with noisy population codes

Nat. Neurosci.