Using LLM to Perform Actions in a Virtual 3D Environment with Consideration to Limitations

This chapter delves into the innovative use of large language models (LLMs) to perform actions within virtual 3D environments, with a particular focus on the gaming industry. The integration of LLMs into games has been largely text-based until now, but this work introduces a pipeline that leverages LLMs to create intelligent Non-Playable Characters (NPCs) capable of performing contextually appropriate actions within a 3D game world. The chapter begins with an overview of the current state of LLMs, highlighting their versatility and the ease of fine-tuning them for specific tasks. It then presents a detailed methodology for selecting and implementing the most suitable LLM for the project, emphasizing the importance of reasoning benchmarks in predicting actions based on context and constraints. The proposed pipeline involves three key stages: prediction, correction, and fine-tuning. The prediction stage involves feeding a fixed prompt to the LLM to generate a sequence of actions, which are then sent via an API to the game engine. The correction stage addresses any actions that do not align with the game's logic or limitations, using a fallback function to ensure coherence. The fine-tuning stage employs QLoRa to create an adapter specific to the game, updating the model's parameters for optimal performance. The chapter concludes with a discussion on the limitations of the approach and future work, including the development of a lightweight model for single-player experiences and the improvement of response times. The results are demonstrated through a demo environment, showcasing the NPC's ability to interact with objects and navigate the game world based on player instructions. This chapter offers a comprehensive exploration of the potential of LLMs in enhancing gameplay and immersive experiences, providing a foundation for future innovations in the gaming industry.