Lp-slam: language-perceptive RGB-D SLAM framework exploiting large language model

Visual SLAM exclusively uses vision for external sensory perception. MonoSLAM [30] was the first real-time visual SLAM system that employed a monocular camera and Extended Kalman Filter (EKF) to estimate camera location and construct a sparse 3D map. PTAM [31] was proposed as the first SLAM system that has utilized graph optimization instead of filtering. PTAM’s keyframe approach achieved superior performance with lower cost, and it introduced the concept of front-end tracking and back-end optimization to improve efficiency. ORB-SLAM [32], which has used FAST corner points and Oriented Brief (ORB) descriptors, running speed and accuracy have been improved as compared to PTAM. Additionally, ORB-SLAM incorporated Loop Closure Detection to optimize the map’s pose and reduce accumulated drift errors.ORB-SLAM2 [2] extended the framework to support binocular and RGBD cameras, while also adding a pre-processing module in the tracking thread to handle more information from these devices, thus improving precision. Furthermore, it introduced full BA to rectify the map and added a localization mode. ORB-SLAM3 [33] is the latest release of the series and supports an even wider range of equipment and functions, such as pinhole, fisheye, and visual-inertial odometry. It is a multi-map system that creates a new map when VO is lost, and automatically merges with the previous map when the scene is recovered. ORB-SLAM3 has demonstrated robustness comparable to state-of-the-art systems while achieving higher precision. In recent years, SLAM systems using direct methods were also developed [34, 35] and showed advantages in efficiency. The algorithms leveraging machine learning also achieve higher accuracy. There are two ways to introduce machine learning into SLAM system. The modules such as feature extraction, feature matching, and loop closing can be replaced by machine-learning-based modules [36, 37], which improves the overall accuracy of the system. The machine learning algorithms can also be side-loaded along with SLAM to provide more information to the original system, such as object detection and semantic segmentation. Some works have used STR to extract the text information to help in localization as mentioned above. However, the previous systems could not get an understanding of the texts or only judged them as special geometry-level information. In this work, encouraged by the emerging LLMs, the potential of text information in SLAM system is explored.

The fundamental technology of our system is STR, which detects and recognizes scene texts. There are many text recognition approaches have been proposed but, Convolutional Recurrent Neural Network (CRNN) [38], which combines convolutional neural network (CNN), recurrent neural network (RNN), and connectionist temporal classification (CTC), has shown promising results in recognizing text in natural scenes. CNNs are used to extract features from the input images. The output is then fed into a bidirectional long-short-term memory (LSTM) layer to generate feature sequences and the network uses a CTC module to decode the sequence into corresponding text output. ASTER [39] is a recognition network based on the attention mechanism that can capture the spatial relationships between characters and recognize text accurately. The attention mechanism is beneficial for text recognition in natural scenes, where the text can appear at different scales, orientations, and positions in the image. Semantic reasoning network (SRN) [40] introduces a global semantic reasoning module (GSRM) to capture context, which is more robust and efficient than traditional methods. Meanwhile, SRN can be trained in an end-to-end approach and achieve state-of-the-art on multiple benchmarks.

Though STR has been largely developed in the past years, the accuracy in large-scale scenes is still not high enough for our proposed LP-SLAM. Two major errors appear frequently and severely influence further processing and understanding. Mis-detection stands for the case where some text-like objects in the scene are judged as texts, and some non-sense texts are extracted from them. Mis-recognition stands for the case where the correctly detected texts are judged into texts with spelling errors. Traditional algorithms are unable to recognize the errors due to the lack of prior knowledge of the world. However, trained by sufficient language data, LLM can correctly judge the errors and even correct them. In this work, the LLM is used as a key components to deal with the STR errors.

A large language model (LLM) is a type of artificial intelligence (AI) system designed to process natural language and generate coherent, contextually appropriate responses to user inputs. LLMs are typically based on deep learning algorithms that are trained on massive datasets of text, allowing them to recognize patterns and relationships in language and generate highly accurate and relevant responses. One of the earliest examples of a large language model was the Statistical Language Model (SLM) [41], which was developed in the 1990s. This model was based on statistical techniques such as n-grams [42] and maximum likelihood estimation and was used for tasks such as speech recognition and machine translation. The most popular Large Language Models in use today are based on the Transformer architecture [43]. The Transformer architecture incorporates the self-attention mechanism, which allows the model to attend to various parts of the input sequence and capture complex relationships between them. The Transformer architecture offers several advantages over traditional RNNs and CNNs. It can process sequences in parallel, which makes it more efficient than RNNs and CNNs. It also enables the model to capture long-range dependencies and relationships between words and phrases, which is essential for natural language processing applications, such as language translation and language modeling.

GPT-3 (Generative Pre-trained Transformer 3) [28] is one of the largest and most powerful language models to date, with 175 billion parameters. GPT-3 was trained on a massive corpus of text data, including web pages, books, and articles, and can generate highly coherent and fluent text. Its advanced capabilities have significant implications for the field of natural language processing and offer exciting opportunities for applications in areas such as chatbots, content generation, and language translation.

LLMs have exhibited their power in natural language processing, facilitating robots in comprehending and responding to human commands with increased human-like proficiency [44]. This multifaceted nature of LLMs enables a more profound integration of linguistic comprehension and contextual understanding within the robotic domain [45]. Introduces an LLM-based approach to translate natural language commands into linear temporal logic (LTL) specifications for robot tasks [46]. Finds that when paired with LLMs to deconstruct complex natural language instructions into subgoals, their robots accomplish intricate, multi-tier tasks in real-world scenarios.

The essence of efficient task planning and scheduling lies in the ability to comprehend and interpret intricate instructions and commands, alongside the capability to strategize and allocate resources effectively. LLMs enable robotic systems to ingest and comprehend diverse sets of data, ranging from textual instructions to real-time environmental cues, and subsequently formulate optimized task plans and schedules [47]. Introduces a scene representation framework integrating contextual information into LLMs planners through VLMs, enabling robots to generate context-aware plans in real-world tasks without predefined object lists or fixed executable options [48]. Proposes a programmatic LLMs prompt structure for generating context-aware plans, demonstrating improved success rates in household and tabletop tasks across virtual and physical environments.

LLMs also have some other applications in the robotics field, such as the development of dialog agents and visual grounding [49]. Introduces a multisensory perception approach for robotic manipulation, emphasizing the coordination of visual, tactile, and auditory perception to handle complex situations, accompanied by a user-friendly mobile app based on LLMs [50]. Presents an interactive visual grounding system that effectively handles open-world scenes with ambiguous natural language instructions, through its integration of large-scale vision-language models and traditional decision-making processes. While SLAM is becoming one critical module in autonomous robots, the potential of LLMs in SLAM is not explored yet.

The overall framework of LP-SLAM is shown in Fig. 2. LP-SLAM has 4 parts: geometry feature based SLAM, runtime text mapping, distilling, and natural user interface. Three modules leverage LLM and are marked as red.

The geometry feature based SLAM has four typical modules including tracking, local mapping, loop closing, and full BA. It estimates the pose for each frame and generates point cloud for the map. The estimated pose \(T_{wc}\) will be used to calculate the position of texts in the 3D world coordinates. In this work, ORB-SLAM3 is used as the base SLAM system. The tracking module runs in a frontend thread while the other modules run in backend threads. Thus tracking module runs in real-time and other modules do not require real-time execution.

The whole runtime text mapping part is executed along with the tracking module of the SLAM system in real-time. The input RGB image is firstly processed by the deep neural network (DNN) based scene text recognition (STR) module to extract the set of visible texts \(T_{i}={t_{i1},t_{i2},...,t_{in}}\), where \(t_{in}\) denotes the \(n-th\) text in the \(i-th\) image. The corresponding 2D pixel position of each text is set as the center of the detection box. One text may appear in many frames and the extraction results (both content and position) differ in frames due to the limited accuracy of STR. The texts extracted from the same object in the real-world scene in different frames are called homologous texts. One real-world text object will produce many homologous texts, some of which have spelling errors (mis-recognition). The 2D pixel positions of homologous texts also differ due to the change of viewpoint and measurement error. There are mis-detection cases when some textures in the scene are similar to texts. In these cases, meaningless texts will be produced. To process the homologous texts, and deal with mis-recognition and mis-detection, TECC is proposed and used after the STR. TECC includes three modules: similarity classification, two-stage memory, and text clustering. The TECC modules will be discussed in detail in the following subsections. The TECC modules will finally generate the accurate text item for each group of homologous texts, correcting the spelling errors. The mis-detection cases will also be deleted.

The distilling part, containing two functions, is executed along with or after runtime text mapping. The text items generated by TECC are judged whether are task-relevant leveraging LLM. The texts that are not task-relevant will be discarded. Position clustering is performed on the reserved items to generate the corresponding positions in the 3D world coordinates by clustering algorithms from the stored positions. The items will be inserted in the 3D world map at the corresponding position. The point cloud & text map is thus constructed With the text items, and the geometry feature points calculated by the tracking module of the base SLAM system.

The navigation part is executed after the above two parts when the point cloud & text map is established. The query from users in natural language is input and processed by LLM to understand the users’ demand. Then LLM provides position instructions according to the map and the demand, which can be used for the follow-up navigation algorithms.

Some texts, while meaningful, may not be relevant to the specific task at hand. For instance, when the task is navigation in a mall, the slogans are not relevant, while the shop names are relevant. The irrelevant texts consume storage space and increase the token length when LLM is used to process the texts according to human queries in the NUI. Therefore we propose a landmark judgment realized by LLM to decide whether a text extracted is task-relevant. The prompt of task-relevant selection is shown in Fig. 5. The pre-training process consists of two key components. Initially, we claim the task that LLM is expected to accomplish and subsequently, we provide relevant examples to ensure optimal performance.

The position of text in world coordinates is determined by its pixel position from STR and the current camera pose in SLAM tracking part. When one point is projected onto the camera image, the point \(P(x_{w},y_{w},z_{w},1)\) is firstly transformed into the camera coordinate:where \(T_{CW}\) is the 4\(\times \)4 transformation matrix from world coordinate to camera coordinate. Then \(P(x_{C},y_{C},z_{C},1)\) is projected onto the image coordinates as:Finally, the pixel position is calculated by affine transformation based on the intrinsic of the camera:Inversely, the position P in real-world coordinates can be calculated from the pixel position (u,v) as:The pixel position of the detected text is calculated as the average of the four counters of the detection box. Each class of homologous texts has many estimated positions from different frames in different points of view. The clustering method is implemented to generate the final position of each class of homologous texts. For each class, N iterations of clustering are executed to remove the outliers. In each iteration, the average position is calculated, which is also the optimum of least squares. Then the 20% farthest points from the average position are judged as outliers and removed.

Different navigation algorithms are implemented on the constructed map of SLAM systems. However, the current algorithms focus on route planning given the start and end position. Compared with the explicit queries asking position of specific text, the implicit queries that only give the demand are more difficult. In this case, the framework should understand the texts and the demands, and logically bridge them. In this work, we explore the possibility of LLM to understand both the constructed map and the human queries. The natural use interface is designed to use the framework in real-time and natural language. The LLM will take in the text lists of the map, and give position instructions according to human queries in natural language. The prompt is shown in Fig. 6.

The real-world experiment is conducted in a mock mall environment containing shops, slogans, and public facilities shown in Fig. 7. The platform robot is a Roban child-sized humanoid robot from Leju Robotics with an Intel RealSense D435i RGBD camera. The ChatGPT\(-\)3.5 is used as LLM in this work. The functionality of the modules will be validated in real-world experiments, and the thresholds will be adjusted accordingly for the larger simulation experiments.

Though the real-world experiments have validated the proposed method, the quantified results are lacking due to the absence of ground truth. Thus the virtual experiments are conducted for more complex environments and the quantified results. The Unreal Engine and Airsim are used to construct an environment of a mall with different shops and slogans. The unmanned aerial vehicle (UAV) is used to explore the scene recording the RGB-D frames. The overall scene is shown in Fig. 14, which contains 28 shops and random non-shop slogans. The UAV is used to explore the whole environment and the ground-truth trajectory is recorded. Three goals are to be reached by the LP-SLAM: 1. Constructing the map of the whole environment; 2. Extracting the task-relevant texts in the scene and mapping them onto the corresponding position; 3. Understanding the texts and providing instructions for human users.

The proposed framework leverages multiple neural-network-based modules. PP-OCRv2 [52] is implemented as the STR module in our experiment. The size of the implemented PaddleOCR model is 155.1M. The text clustering, task-relevant selection, and natural user interface are executed by ChatGPT 3.5. The size of ChatGPT 3.5 is approximately 175B. The other non-neural-network-based modules including similarity classification, two-stage memory, position calculation, and position clustering have a very low complexity and account for a very small proportion of the total time.

The STR is implemented offline and the ChatGPT is executed online via APIs from OpenAI. Thus the execution speed of the STR module is restricted by the local hardware and the LLM-based modules are restricted by the response time of ChatGPT. The STR module and the three LLM-based modules are executed in two individual threads along with the tracking thread of SLAM. When a new frame arrives, the tracking thread and runtime text mapping thread process the image in parallel. The runtime mapping thread that contains the STR module costs more time than the tracking thread and thus becomes the bottleneck of the two threads. The distilling thread further processes the texts in a background thread that is decoupled with the tracking thread. Therefore the distilling thread will not influence the frame rate. The natural user interface is executed after the whole mapping process and also does not influence the frame rate. In our experiment, the platform is one desktop computer with Intel i7 CPU and Nvidia 3080ti GPU and the measured frame rate is 25fps, which meets the requirement of real-time.

The implemented STR network and LLM networks in the experiment are replaceable. There are two major considerations when selecting the models. Firstly, the model should be selected by the functionality. Different STR models are trained for various languages and various environments such as cities or villages. The further fine-tuned LLMs are more suitable for specific tasks in addition to the mall navigation tasks. The different LLMs are also more capable in different languages. For example, WenXinYiYan has better accuracy in a Chinese environment. Secondly, the model size should be considered according to the target platform. The trade-off between execution time and accuracy should be designed based on specific tasks. The base SLAM algorithm is also replaceable in consideration of accuracy and computation complexity.