Skip to main content

About this book

Autonomous robots may become our closest companions in the near future. While the technology for physically building such machines is already available today, a problem lies in the generation of the behavior for such complex machines. Nature proposes a solution: young children and higher animals learn to master their complex brain-body systems by playing. Can this be an option for robots? How can a machine be playful? The book provides answers by developing a general principle---homeokinesis, the dynamical symbiosis between brain, body, and environment---that is shown to drive robots to self- determined, individual development in a playful and obviously embodiment- related way: a dog-like robot starts playing with a barrier, eventually jumping or climbing over it; a snakebot develops coiling and jumping modes; humanoids develop climbing behaviors when fallen into a pit, or engage in wrestling-like scenarios when encountering an opponent. The book also develops guided self-organization, a new method that helps to make the playful machines fit for fulfilling tasks in the real world.

The book provides two levels of presentation. Students and scientific researchers interested in the field of robotics, self-organization and dynamical systems theory may be satisfied by the in-depth mathematical analysis of the principle, the bootstrapping scenarios, and the emerging behaviors. But the book additionally comes with a robotics simulator inviting also the non- scientific reader to simply enjoy the fabulous world of playful machines by performing the numerous experiments.

Table of Contents



Robots and their relation to mankind have taken a long and diversified development. Starting from the romantic desire to have a workmate and/or playmate centuries ago, the modern history of robot control starts with the birth of artificial intelligence (AI) about 50 years ago. In the hype of that time, robots were considered as machines under total control of an artificial intelligence thought to understand the world and the physics of the body well enough in order to control the robot by a set of rules defining its behavior.

Ralf Der, Georg Martius

Self-Organization in Nature and Machines

Self-organization in the sense used in natural sciences means the spontaneous creation of patterns in space and/or time in dissipative systems consisting of many individual components. Central in this context is the notion of emergence meaning the spontaneous creation of structures or functions that are not directly explainable from the interactions between the constituents of the system. This chapter presents at first several examples of prominent self-organizing systems in nature with the aim to identify the underlying mechanisms. While self-organization in natural systems shares a common scheme, self-organization in machines is more diversified. An exception is swarm robotics because of the similarity to a system of many constituents interacting via local laws as encountered in physics (particles), biology (insects), and technology (robots). This chapter aims at providing a common basis for a translation of self-organization effects to


robots considered as complex physical systems consisting of many constituents that are constraining each other in an intensive manner.

Ralf Der, Georg Martius

The Sensorimotor Loop

This chapter aims at providing a basic understanding of the sensorimotor loop as a feedback system. First we will give some insights into the richness of behavior resulting from simple closed loop control structures in a robotic system called the


. This richness is a lesson we can learn from dynamical systems theory: even very simple systems can produce highly complicated behavior. Nearly everything is possible in such a feedback system that is provided with enough energy from outside. Surprisingly, this is accomplished even with extremely simple, fixed controllers, to which we will restrict ourselves here. In later chapters we will see how the homeokinetic principle makes theses systems adaptive and drives them towards specific working regimes of moderate complexity, loosely speaking somewhere between order and chaos.

Ralf Der, Georg Martius

Principles of Self-Regulation — Homeostasis

We have seen in Chap. 3 that there are specific working regimes in closed sensorimotor loops where the agents exhibit interesting behaviors. The challenge is now to develop general principles so that the agent finds these regions by itself. One essential point at this level of autonomy is the ability to survive in hostile situations, which, as a first prerequisite, requires a certain stability against external perturbations. An example of this is homeostasis, one of the prominent self-regulation scenarios in living beings. This chapter introduces Ashby’s homeostat as a concrete example from cybernetics and develops a general principle of self-regulation as a first step towards a general basis for the self-organization of behavior.

Ralf Der, Georg Martius

A General Approach to Self-Organization — Homeokinesis

In this chapter we will introduce the concept of homeokinesis, formulate it in mathematical terms, and develop a first understanding of its functionality. The preceding chapter on homeostasis made clear that the objective of “keeping things under control” cannot lead to a system which has a drive of its own to explore its behavioral options in a self-determined manner. This is not surprising since the homeostatic objective drives the controller to minimize the future effects of unpredictable perturbations. This chapter uses a different objective, the so called time-loop error, derives learning rules by gradient descending that error and discusses first consequences of the new approach. Minimizing the time-loop error is shown to generate a dynamical entanglement between state and parameter dynamics that has been termed homeokinesis since it realizes a dynamical regime jointly involving the physical, the neural, and the synaptic dynamics of the brain-body system.

Ralf Der, Georg Martius

From Fixed-Point Flows to Hysteresis Oscillators

Homeokinesis realizes the self-organization of artificial brain-body systems by gradient descending the time-loop error, a quantity that is truly internal to the robot since it is defined exclusively in terms of its sensorimotor dynamics. Homeokinesis can therefore be considered as a self-supervised learning procedure with the special effect of making the brain-body system self-referential. We will study this phenomenon here in an idealized one-dimensional world in order to identify key features of our self-referential dynamical systems independently of any specific embodiment effects. In particular, we will gain some insight into the entanglement of state and parameter dynamics and investigate the way how the latter induces behavioral variability.

Ralf Der, Georg Martius

Symmetries, Resonances, and Second Order Hysteresis

This chapter is a continuation of the preceding chapter to two-dimensional systems. We will identify new features of our self-referential dynamical system again independently of any specific embodiment effects. The additional dimension opens the possibility for state oscillations, where the entanglement of state and parameter dynamics will lead to interesting phenomena and induces behavioral variability. The most prominent effects are driving oscillatory systems into a second order hysteresis (by a self-organized frequency sweeping effect) and getting into resonance with latent oscillatory modes of the controlled system.

Ralf Der, Georg Martius

Low Dimensional Robotic Systems

In this chapter we will demonstrate the performance of the homeokinetic control system when applied to physical robots. We will recognize many of the effects observed in idealized world conditions, shown in the previous chapters, but most dominantly witness new features originating from the interaction of the learning dynamics with the respective embodiment. Among them are non-trivial sensorimotor coordination, excitation of resonance modes, the adaptation to different environments - all emerging from the unspecific homeokinetic learning rules. The entanglement effect is seen to make emerging motion patterns transient so that the behavioral options are explored and a playful behavior is observed. In order to keep things simple enough for analysis we consider here only low-dimensional systems and leave the high-dimensional ones for Chap. 10.

Ralf Der, Georg Martius

Model Learning

This chapter discusses several aspects concerning the simultaneous learning of controller and internal model. We start with discussing the bootstrapping dilemma arising in this context and the consequences of insufficient sampling. It appears that homeokinetic learning solves these problems naturally, which we illustrate in several examples. Further, we extend the implementation of the internal model by a sensor-branch. This is seen to increase the applicability of the homeokinetic controller because it allows for situations where the sensor values are subject to an action-independent dynamics. The extended model is prone to an ambiguity in the learning process, which can lead to instabilities. The problem can be resolved if the time-loop error is used as an additional objective for the model learning.

Ralf Der, Georg Martius

High-Dimensional Robotic Systems

This chapter contains many applications of homeokinetic learning to high-dimensional robotic systems. The examples chosen for investigation and proposed as experiments to the reader comprise various robots ranging from dog-like, to snake-like up to humanoid robots in different environmental situations. The aim of the experiments is to understand how the controller can learn to “feel” the specific physical properties of the body in its environment and manages to get in a kind of functional resonance with the physical system. In order to better bring out the characteristics of homeokinetic learning in these systems, we use a kind of physical scaffolding, for instance suspending the


like a bungee jumper, putting it in the


, or hanging it at the high bar. Interestingly, in all situations the robots develop whole-body motion patterns that seemingly are related to the specific environmental situation: the


starts playing with a barrier eventually jumping or climbing over it; the


develops coiling and jumping modes; we observe emerging climbing behaviors of a


like trying to get out of a pit; and wrestling like scenarios if a


is encountering a companion. Eventually, in our robotic zoo all kinds of robots are brought together so that homeokinesis can prove its robustness against heavy interactions with other robots or dynamical objects. Essentially this chapter provides a phenomenological overview and invites to play around with numerous simulations to see the “playful machine” in action.

Ralf Der, Georg Martius

Facing the Unknown — Homeokinesis in a New Representation

Homeokinesis has been introduced and analyzed in the preceding chapters on the basis of the time-loop error (TLE). This chapter presents an alternative approach to the general homeokinetic objective by introducing a new representation of the sensorimotor dynamics. This new representation corrects the state dynamics for the predictable changes in the sensor values so that the transformed state is constant except for the interactions with the unknown part of the dynamics. The single-step interaction term will be seen to be identical to the TLE so that the learning dynamics is not altered. However, besides giving an additional motivation for the TLE, this chapter will extend the considerations to the case of several time steps and will eventually consider infinite time horizons making contact with the global Lyapunov exponents and chaos theory.

Ralf Der, Georg Martius

Guided Self-Organization — A First Realization

We introduce here an in the following chapters

guided self-organization

as the combination of specific goals with self-organizing control. As a first realization we propose in this chapter the guidance with supervised learning signals. First, we investigate how these signals can be incorporated into the learning dynamics and present then a simple scenario with direct motor teaching signals.We find that the homeokinetic controller explores around the given motor patterns and thus may find a more suitable behavior for the particular body. Second, we transfer this into a teaching at the level of sensor signals, which is very natural in our setup. Thismechanism of guidance builds the basis for higher level guiding mechanisms as discussed in Chap. 13.

Ralf Der, Georg Martius

Channeling Self-Organization

Many desired behaviors are distinguished by a certain structure in the motor or sensor activity. In particular the phase relation between different motors or sensors capture a lot of this structure. We will now propose a way to embed these relations as soft constrains to the learning system, such that we break certain symmetries and let desired behaviors emerge. Starting from the guidance by teaching we introduce the concept of crossmotor teaching that allows to specify abstract relations between motor channels. First we study simple pairwise relations and shape the behavior of the


robot to drive mostly straight by a relation between both motor neurons. Then we will consider a high-dimensional robot-the


and demonstrate fast locomotion behaviors from scratch by guided self-organization.

Ralf Der, Georg Martius

Reward-Driven Self-Organization

In this chapter we investigate how to guide the self-organization process by providing an online reward or punishment. The starting point for the following considerations is that the homeokinetic controller explores the behavioral space of the controlled system and that those behaviors which are well predictable will persist longer than others. The idea we pursue in this chapter is to regulate the lifetimes of the transient according to the reward or punishment. The mechanism is applied to the


with two goals, fast motion and curved rolling.

Ralf Der, Georg Martius

Algorithmic Implementation

This chapter presents a unified algorithm implementing the homeokinetic learning rules including a number of extensions partly discussed already in earlier chapters of this book. We continue with some guidelines and tips on how to use the homeokinetic “brain.” We discuss techniques and special methods to make the self-supervised learning of embodied systems more reliable from the practical point of view. This includes the regularization procedures for the singularities in the time-loop error and different norms of the error for the gradient descent. The internal complexity of the controller and the model is extended by the generalization to multilayer networks. Apart from that the computational complexity of the learning algorithm will be reduced essentially by easing non-trivial matrix inversions. This is important for truly autonomous hardware realizations.

Ralf Der, Georg Martius

The LpzRobots Simulator

In this chapter we describe our robot simulator. We start with the overall structure of the software package containing the controller framework, the physics simulator and external tools. The controller framework makes it very easy to develop and test our algorithms, be it in simulations or with real robots. The physics simulator can handle rigid bodies with fixed geometric representation that are connected by actuated joints. Particular efforts have been undertaken to develop an elaborated treatment of physical object interactions including friction, elasticity, and slip. The chapter also briefly discusses the generation of virtual creatures, the user interface and the most important features of the


simulation environment.

Ralf Der, Georg Martius

Discussion and Perspectives

This book started with a question. Now, more than 300 pages and, if you were eager, nearly 30 experiments later, let us try to draw a balance. Have we solved the problem of getting robots into controlled activities without telling them what to do, without giving a task, a goal, or any other external pressure for development? The answer to this question is a clear yes. We have formulated a general principle—homeokinesis—that was seen to provide, in a natural and unbiased way, the desired drive to activity while still “keeping things under control.”

Ralf Der, Georg Martius


Additional information

Premium Partner

Neuer Inhalt
image credits