Using GPT-3 to Achieve Semantically Relevant Data Sonificiation for an Art Installation

Large Language Models such as GPT-3 exhibit generative language capabilities with multiple potential applications in creative practice. In this paper, we present a method for data sonification that employs the GPT-3 model to create semantically relevant mappings between artificial intelligence-generated natural language descriptions of data, and human-generated descriptions of sounds. We implemented this method in a public art installation to generate a soundscape based on data from different systems. While common sonification approaches rely on arbitrary mappings between data values and sonic values, our approach explores the use of language models to achieve a mapping not via values but via meaning. We find our approach is a useful tool for musification practice and demonstrates a new application of generative language models in creative new media arts practice. We show how different prompts influence data to sound mappings, and highlight that matching the embeddings of texts of different lengths produces undesired behavior.