A Flexible Framework For Using NLP In XR

The increasing use of Extended Reality (XR) brings a need for more advanced Human-Computer Interaction (HCI) technologies to allow for intuitive and robust collaboration. However, challenges arise when also ensuring interactions are both natural and effective, particularly when incorporating flexible verbal communication. Prior research has explored many multi-modal interaction technologies, yet there exists a need for a framework to allow for human-computer communication in specifically XR applications. This work proposes a novel framework that can incorporate advanced means of Natural Language Processing (NLP) to handle flexible verbal communication while ensuring computer interpretation is robust. Drawing from prior research that identified limitations of information given in XR visuals and challenges in using widely available NLP tools, the proposed framework uses a rule-enforced command extraction pipeline to ensure consistent human-like processing, while preserving the flexibility of more open-domain verbal dialogue. The design employs a sequence of NLP techniques for spoken utterance analysis, language specific rules to extract usable game commands, and verification on the structured command with a user-reinforced hyper-graph method of data storage. The framework and its processing pipeline was evaluated using a wide-range of human-like phrasings of commands deemed representative of verbalized instructions for human-computer collaboration in an XR context. Framework performance was then compared against a Large Language Model (LLM) that was customized to extract commands using the same rules. These results serve to showcase the potential of the proposed framework through demonstrated examples and early results while also noting areas of needed performance enhancements.