Superpixel-based appearance change prediction for long-term navigation across seasons

Highlights

• This work is about place recognition across seasons.

• We present the novel idea of an additional prediction step.

• We propose a superpixel based implementation: SP-APC.

• SP-APC can learn to predict e.g. winter images from summer images.

• Extensive evaluation is done on the Nordland dataset.

Abstract

Changing environments pose a serious problem to current robotic systems aiming at long term operation under varying seasons or local weather conditions. This paper is built on our previous work where we propose to learn to predict the changes in an environment. Our key insight is that the occurring scene changes are in part systematic, repeatable and therefore predictable. The goal of our work is to support existing approaches to place recognition by learning how the visual appearance of an environment changes over time and by using this learned knowledge to predict its appearance under different environmental conditions. We describe the general idea of appearance change prediction (ACP) and investigate properties of our novel implementation based on vocabularies of superpixels (SP-ACP). Our previous work showed that the proposed approach significantly improves the performance of SeqSLAM and BRIEF-Gist for place recognition on a subset of the Nordland dataset under extremely different environmental conditions in summer and winter. This paper deepens the understanding of the proposed SP-ACP system and evaluates the influence of its parameters. We present the results of a large-scale experiment on the complete 10 h Nordland dataset and appearance change predictions between different combinations of seasons.

Introduction

Long term navigation in changing environments is one of the major challenges in robotics today. Robots operating autonomously over the course of days, weeks, and months have to cope with significant changes in the appearance of an environment. A single place can look extremely different depending on the current season, weather conditions or the time of day. Since state of the art algorithms for autonomous navigation are often based on vision and rely on the system’s capability to recognize known places, such changes in the appearance pose a severe challenge for any robotic system aiming at autonomous long term operation.

The problem has recently been addressed by few authors, but so far no congruent solution has been proposed. Milford and Wyeth  [1] proposed to increase the place recognition robustness by matching sequences of images instead of single images and achieved impressive results on two across-seasons datasets. Exploring into a different direction, Churchill and Newman  [2] proposed to accept that a single place can have a variety of appearances. Their conclusion was that instead of attempting to match different appearances across seasons or severe weather changes, different experiences should be remembered for each place, where each experience covers exactly one appearance. Both suggested approaches can be understood as the extreme ends of a spectrum of approaches that spans between interpreting changes as individual experiences of a single place on one hand and increasing the robustness of the matching against appearance changes on the other hand. Our work presented in the following is orthogonal to this spectrum.

What current approaches to place recognition (and environmental perception in general) lack, is the ability to reason about the occurring changes in the environment. Most approaches try to merely cope with them by developing change-invariant descriptors or matching methods. Potentially more promising is to develop a system that can learn to predict certain systematic changes (e.g. day–night cycles, weather and seasonal effects, re-occurring patterns in environments where robots interact with humans) and to infer further information from these changes. Doing so without being forced to explicitly know about the semantics of objects in the environment is in the focus of our research and the topic of this paper.

Fig. 1 illustrates the core idea of our work and how it compares to the current state of the art place recognition algorithms. Suppose a robot re-visits a place under extremely different environmental conditions. For example, an environment was first experienced in summer and is later re-visited in winter time. Most certainly, the visual appearance has undergone extreme changes. Despite that, state of the art approaches would attempt to match the currently seen winter image against the stored summer images.

Instead, we propose to predict how the current scene would appear under the same environmental conditions as the stored past representations, before attempting to match against the database. That is, when we attempt to match against a database of summer images but are in winter time now, we predict how the currently observed winter scene would appear in summer time or vice versa.

The result of this prediction process is a synthesized summer image that preserves the structure of the original scene and is close in appearance to the corresponding original summer scene. This prediction can be understood as translating the image from a winter vocabulary into a summer vocabulary or from winter language into summer language. As is the case with translations of speech or written text, some details will be lost in the process, but the overall idea, i.e. the gist of the scene will be preserved. Sticking to the analogy, the error rate of a translator will drop with experience. The same can be expected of our proposed system: It is dependent on training data, and the more and the better training data is gets, the better can it learn to predict how a scene changes over time or even across seasons.

This paper build upon our previous work  [3] where we introduced the novel idea of predicting extreme scene changes across seasons to aid place recognition for the first time. We prove the feasibility of our idea and describe an implementation based on superpixel vocabularies. We demonstrate how we can predict the appearance of natural scenes across winter and summer time, as illustrated in Fig. 1. By applying this approach, we are able to significantly improve the place recognition performance of SeqSLAM  [1] and BRIEF-Gist [4] on the new, publicly available large-scale Nordland dataset  [5] that traverses an environment in winter and summer under extremely different environmental conditions. While the first results we reported in  [3] were based on a small subset of the Nordland dataset, this paper presents new results on the complete Nordland track. We furthermore evaluate predictions between different combinations of seasons. An extensive evaluation of important parameters deepens our understanding of the proposed prediction system and its parameters.

In the following section, we put the proposed prediction system in the context of related work, before we describe its algorithmic steps in Section  3. Section  4 introduces the Nordland dataset we used for training, validation and testing. The results Section  5 presents comprehensive place recognition experiments on this dataset using FAB-MAP, BRIEF-Gist and SeqSLAM in combination with the proposed SP-ACP system. The paper is concluded by a discussion on limitations of the current system and directions for future work in Section  6. Additional information and videos can be obtained from our project website.¹