Traffic Pretext Tasks for Trajectory Prediction with Large Langugage Models

Trajectory prediction is critical to autonomous driving. It enables autonomous vehicles to anticipate the future motions of surrounding agents and plan accordingly. Recently, self-supervised approaches have significantly improved prediction performance by incorporating trajectory and map reconstruction during pretraining. However, limited by the narrow scope of pretraining objectives, these methods fail to capture complex traffic semantics and social interactions. To address this issue, we extend the self-supervised objectives into a series of semantically rich traffic pretext tasks and leverage a pre-trained Large Language Model (LLM) to obtain expressive representations for downstream prediction tasks. First, a set of design principles is introduced to ensure that the pretext tasks can effectively model the intricate spatio-temporal dependencies inherent in traffic scenes. Then, we enhance conventional masked reconstruction with map-level and agent-level tasks to enable structured relational modeling and multi-scale abstraction. The pre-trained LLM is post-pretrained on these tasks to align its prior knowledge with the traffic domain. Finally, the model is fine-tuned with a Laplace decoder to generate multimodal trajectory predictions. Experiments on the Argoverse2 (AV2) dataset show that our approach outperforms the latest supervised and self-supervised methods across all metrics, demonstrating the effectiveness of combining LLM with carefully designed traffic pretext tasks for accurate and robust trajectory prediction.