Self-localization in non-stationary environments using omni-directional vision

Abstract

This paper presents an image-based approach for localization in non-static environments using local feature descriptors, and its experimental evaluation in a large, dynamic, populated environment where the time interval between the collected data sets is up to two months. By using local features together with panoramic images, robustness and invariance to large changes in the environment can be handled. Results from global place recognition with no evidence accumulation and a Monte Carlo localization method are shown. To test the approach even further, experiments were conducted with up to 90% virtual occlusion in addition to the dynamic changes in the environment.

Introduction

One of the most essential abilities needed by robots is self-localization (“where am I?”), which can be divided into geometric and topological localization. Geometric localization tries to estimate the position of the robot as accurately as possible, e.g., by calculating a pose estimate $(x,y,\theta )$, while topological localization gives a more abstract position estimate, e.g., “I’m in the coffee room”. There has been much research on using accumulated sensory evidence to improve localization performance, including a highly successful class of algorithms that estimate posterior probability distributions over the space of possible locations [1], [2], [3], [4], [5], [6]. This approach enables both position tracking and relocalization from scratch, for example, when the robot is started (no prior knowledge of its position) or becomes lost or “kidnapped” (incorrect prior knowledge).

In recent years, panoramic or omni-directional cameras have become popular for self-localization because of their relatively low cost and large field of view. This makes it possible to create features that are invariant to the robot’s orientation, for example, using various colour histograms [7], [8], [9] or Eigenspace models [10], [11], [12].

Other innovations include increased robustness to lighting variations [11] and multi-view registration to deal with occlusions [13]. Recent work has combined panoramic vision with particle filters for global localization, including feature matching using Fourier transform [14], PCA [15] and colour histograms [16].

Most previous work on robot mapping and localization assumes a static world, i.e., that there are no changes to the environment between the time of mapping and the time of using the map. However, this assumption does not hold for typical populated indoor environments. Humans (and other robots) are not merely “dynamic obstacles” that may occlude the robot’s sensors — they also make changes to the world. For example, they may leave temporary objects such as packages, or move the furniture. In addition to these sudden changes, there may be gradual changes such as plants growing, coloured paint fading, etc.

Our approach to self-localization in non-static environments uses an image matching scheme that is robust to many of the changes that occur under natural conditions. Our hypothesis is that a map that is out of date can still contain much useful information. Thus the important question is how to extract features that can be used for matching new sensor data to a map that is only partially correct. We present an appearance-based approach to matching panoramic images that does not require calibration or geometric modelling of the scene or imaging system, thus it should be applicable to any mobile robot using omni-directional vision for self-localization. The hypothesis is validated through experiments using sensor data collected by a mobile robot in a real dynamic environment over a period of two months.

The image matching algorithm uses local features extracted from many small subregions of the image rather than global features extracted from the whole image, which makes the method very robust to variations and occlusions. For example, Fig. 2 shows some of the local features that were matched between two different panoramic images of a laboratory environment, recorded 56 days apart, despite changes such as a television appearing, chairs moving and people working. The approach is similar to other approaches using local features for self-localization [17], [18], [19], with the differences that our method is adapted for panoramic images and was specifically designed and tested to work in long-term experiments conducted in a real dynamic environment.

We use a version of Lowe’s SIFT algorithm [20], which is modified so that stored panoramic images are only recognised from a local area around the corresponding location in the world (Section 2.3). A novel scheme is introduced for combining local feature matching with a particle filter for global localization (Section 2.4), which minimizes computational costs as the filter converges. See also Fig. 1 for a brief overview of the method. To evaluate the method, we also compare its performance with that of several other types of features, including both global and local features (Section 3). We also show how image matching performance can be further improved by incorporating information about the relative orientation of corresponding features between images (Section 3.3). Our experiments were designed to test the system under a wide variety of conditions, including results in a large populated indoor environment (up to 5 persons visible) on different days under different lighting conditions (Section 4).

The results demonstrate that the robot is able to localize itself from scratch, including experiments in “kidnapping”, and that the performance shows a graceful degradation to occlusions (up to 90% of the robot’s field of view).