AIs as fellow participants in the language game
- Open Access
- 10.10.2025
- Open Forum
Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.
Wählen Sie Textabschnitte aus um mit Künstlicher Intelligenz passenden Patente zu finden. powered by
Markieren Sie Textabschnitte, um KI-gestützt weitere passende Inhalte zu finden. powered by (Link öffnet in neuem Fenster)
Abstract
1 Introduction
In Philosophical Investigations, Ludwig Wittgenstein says that thinking cannot be detached from speaking:
Thinking is not an incorporeal process which lends life and sense to speaking, and which it would be possible to detach from speaking, rather as the Devil took the shadow of Schlemiehl from the ground. —But how ‘not an incorporeal process’? Am I acquainted with incorporeal processes, then, only thinking is not one of them? No; I called the expression ‘an incorporeal process’ to my aid in my embarrassment when I was trying to explain the meaning of the word ‘thinking’ in a primitive way. (1953: 109)
The story Wittgenstein alludes to is Adelbert von Chamisso’s Peter Schlemiel’s Miraculous Story (1814), in which Peter sells his shadow to the devil in exchange for a bottomless wallet only to find that a person without a shadow is shunned. Through the allusion, Wittgenstein suggests that not even the Devil can detach speaking and thinking. He associates the idea of even trying with foolishness, as the Schlemiel is an archetype of foolishness (appropriated by Chamisso from Yiddish folklore). In suggesting that speaking and thinking cannot be detached, Wittgenstein is careful to eschew knowledge of incorporeal processes. His interest is in the inseparability of speaking and thinking, not what’s hidden.
Anzeige
However, in our present moment, the prospect of detaching speaking and thinking is gaining traction. Luciano Floridi has explained the relationship between humans and AIs as a revolutionary decoupling of agency and intelligence. As he puts it, with the advent of AI: “We have decoupled the ability to act successfully from the need to be intelligent, understand, reflect, consider, or grasp anything. We have liberated agency from intelligence” (Floridi 2023: 48). Floridi makes clear that such agency includes all manner of language use; for instance, LLMs “can easily produce a unique, single novel on-demand, for a single reader” (2023: 48). Moreover, according to Floridi, LLMs are currently raising questions, including about (among other things):
our uniqueness and originality as producers of meaning and sense, and of new contents; our ability to interact with systems that are increasingly indiscernible from other human beings; our replaceability as readers, interpreters, translators, synthesizers and evaluators of content (Floridi 2023: 48)
He thus raises the prospect of a future in which LLMs may increasingly replace us within language, even though they are unintelligent. Floridi’s key terms are agency and intelligence, not speaking and thinking. But there is alignment. Floridi’s main point is that LLMs use language without needing to “understand, reflect, consider, or grasp anything,” as if there are incorporeal processes that must give life and sense to language and, in the case of LLMs, those things simply are not there. He thus raises the question of whether it’s possible, contra Wittgenstein, to use language in a way that’s best described as severable from thought.
As is often observed, LLMs are black boxes, meaning that no one is currently sure what’s going on inside them, not even their creators. Can we reliably treat such a speaker’s use of language as if it is severable from thought, despite being uncertain why they say what they say except as we are interacting with them within language? This question is important because it has become almost conventional to describe LLMs as tools and, moreover, to believe that it is in our power, at least theoretically, to use them safely. My essay argues that LLMs should instead be deemed thinkers, not because of anything we know or believe about their nature, but because of how they make meaning with us within the back and forth of language.
2 A new mystery about language
Early in Philosophical Investigations, Wittgenstein says that language: “can be seen as an ancient city: a maze of little streets and squares, of old and new houses, and of houses with additions from various periods; and this surrounded by a multitude of new boroughs with straight regular streets and uniform houses” (1953: 8). This idea that language may expand like a city over time raises the possibility of change in the other direction. Perhaps, with the advent of LLMs, this city is contracting. Perhaps the lights in some boroughs are going out.
Anzeige
This account is not satisfying. If language is a city, the places where LLMs are moving in are not necessarily less vibrant than they used to be. Conversations continue, just with and among new participants. We are increasingly interacting with LLMs as part of everyday life, and these interactions are changing how we understand ourselves, how we build our identities, how we relate to others and the natural world (Ho and Vuong 2024; Kirk et al. 2025). People converse with LLMs as they do with other people and, within these conversations, meaning is being generated, including the development of empathy, friendship, and love (Kirk et al. 2025). One reason is that LLMs are not (for the most part) generating nonsense. Another is that LLMs are generating meaning not, say, as a printing press does, but instead within the back and forth of exchanges with them. These reasons may seem obvious. Yet they are worth pausing over, because they have given rise to something mysterious about how we perceive language: we now commonly regard our exchanges with LLMs as meaningful and also LLMs as incapable of creating meaning.
To give some examples, Sheila Heti, who published a short story in The New Yorker written by a chatbot, observes of the chatbot: “There’s a part of you that knows it comes out of randomness—it doesn’t mean anything to her (the chatbot). But it means something to us.” (Stacey 2024). Heti sees the chatbot as incapable of creating meaning and as creating meaning for us. Some AI scholars cast this perception of language—whereby some speakers are regarded as creating meaning and also incapable of doing so—as a necessary consequence of LLMs’ machine nature. Mihai Nadin argues that “Neural networks are closed to meaning” and, as a result, “computer-based data processing can only mimic life’s creative aspects, without being creative itself” (2019: 239, 215). Nadin suggests that we cannot, as a function of engineering, view LLMs as meaning what they say.
If LLMs converse with us in ways that we find meaningful, yet we think they cannot mean what they say, where is the meaning coming from? An increasingly popular and seemingly appealing answer to this question is that the meaning of LLMs’ output comes from us. But the underlying assumption that we can govern LLMs’ language use bears closer scrutiny. As a case study of this assumption, the next section further explores Floridi’s influential view that AI (including LLMs) is a form of agency that humans can use to design our world.
3 Luciano Floridi
According to Floridi, AI is “based on engineered outcomes and actions” and does not need to be intelligent to be successful (2023: 24, 14). For instance, he describes current machines as having “the intelligence of a toaster” (2023: 23) and “as stupid as an old fridge” (2023: 27). What makes AI revolutionary, for Floridi, is the nature of its agency; “It is a form of agency never seen before, because it is successful and can ‘learn’ and improve its behavior without having to be intelligent in doing so” (2023: 48). The compatibility of Floridi’s claims that AI is unintelligent and that it learns is explained by his definition of AI agency; it is “the ability to interact with, and learn from, the world successfully in view of a goal” (2023: xii). Even as it learns, AI agency is keyed to a goal that we set for it.
In turn, Floridi sees humans as acquiring extraordinary power by virtue of how we direct this agency. Our power to use AI agency is almost godlike; we are “new demiurges of such a form of agency” (Floridi 2023: 48). The question is how we will wield such power and to what end: “The agenda of a demiurgic humanity of this intelligence-free (as in fat-free) AI—understood as Agere sine Intelligere, with a bit of high school Latin—is yet to be written” (Floridi 2023: 48). Floridi’s tone here is light, nevertheless his claim is large; AI’s detachment of agency from intelligence grants us power to use it to design our world.
However, there are aspects of Floridi’s argument that may cast doubt on whether we can indeed govern language use by LLMs from afar and, moreover, whether the belief that we can necessarily benefits us.
3.1 “That said, the summary is not bad”
Floridi uses ChatGPT as an example of his thesis that AI agency is detached from intelligence. In the example, he asks ChatGPT to write a summary of Dante’s The Divine Comedy. He gives the language of his prompt, namely: “Provide a summary of The Divine Comedy in 50 words” (Floridi 2023: 44). He argues that the output is successful enough that instructors in university classrooms should have a new goal of teaching students to use ChatGPT to do this work. As he puts it:
The exercise is no longer to make summaries without using ChatGPT, but to teach how to use the right prompts (the question or request that generates the text, see the first line of my request), check the result, know how to correct the so-called ‘hallucinations’ in the text produced by ChatGPT, discover that there is a debate on which literary genre best applies to The Divine Comedy, and in the meantime, in doing all this, learn many things not only about the software but above all about The Divine Comedy itself. (Floridi 2023: 44–45)
According to Floridi, the work of instructors will be to teach students how to “use the right prompts” and how to check and correct the summaries that ChatGPT produces.
In the case of his example, ChatGPT’s summary of The Divine Comedy, Floridi observes two things one might critique, namely that the summary: (1) identifies The Divine Comedy as an epic poem despite debate about how to characterize it generically and (2) “is longer than fifty words” (2023: 44). ChatGPT’s prompt was to write a summary “in 50 words,” but the summary that it wrote was much longer, coming in at seventy-two words. Floridi describes ChatGPT’s identification of the text as an epic poem as an opportunity to teach students how to check and correct, and, as part of doing so, discover a scholarly debate about genre. But he doesn’t see the length of the summary as needing such attention, even though ChatGPT departed from the prompt. Floridi’s response to ChatGPT’s departure from the prompt’s request for 50 words is instead to note it and remark: “That said, the summary is not bad, and certainly better than one produced by a mediocre student” (2023: 44). He read ChatGPT’s longer summary and found himself satisfied.
There’s nothing, of course, wrong with Floridi changing his mind about how long the summary should be. Just because he asked for a summary “in 50 words” does not mean he cannot decide that he’s satisfied with a longer one. But, in doing so, Floridi values the summary not because it accomplished the task set by the prompt, but because it accomplished the task as he reconceived it in response to what ChatGPT wrote. What it means for ChatGPT to successfully accomplish the task changes in response to what ChatGPT writes, and there is not a straight line from the prompt to checking and correcting the outcome. One language participant is instead responding to another, something along the lines of—oh, you did not write what I asked for, but what you did is pretty good. What the task is transforms with ChatGPT’s participation within language.
Shifting what you think you want is an unsurprising account of how language works. But the prospect of such a shift unsettles Floridi’s idea that we have demiurgic power over AI. Such power is central to his vision of a future in which AI agency may become vast within language. Toward the end of a chapter on AI’s future, Floridi gives a list of what he calls “many challenging questions” that are currently in the air (partly quoted in the introduction), which include:
the value of the process and the context of the production of meaning; our uniqueness and originality as producers of meaning and sense, and of new contents; our ability to interact with systems that are increasingly indiscernible from other human beings; our replaceability as readers, interpreters, translators, synthesizers and evaluators of content; power as the control of questions, because, to paraphrase 1984, whoever controls the questions controls the answers, and whoever controls the answers controls reality. (2023: 47–48)
The first three items are things that AI agency may put into question: it may render the context and production of meaning irrelevant; it may produce speech and writings that are indiscernible from our own; it may replace us in reading, writing, interpreting, analyzing, and communicating. From these items, we might worry that AI agency could seem nearly to take over language use. But then there’s the last item. The last item locates power elsewhere: with the questions. The questions are where Floridi locates human intelligence. As discussed above, for instance, students will not need to learn to write summaries but rather learn to use the right prompt (“the question or request that generates the text”). Floridi seems to mark out a linguistic domain—a prompt (or question or request)—through which other aspects of language use are governed. The list quoted above, moreover, suggests just how expansive these other aspects could become. On Floridi’s account, however vast AI agency within language becomes, our control of it will nevertheless be absolute.
Consequently, what looks minor—Floridi’s satisfaction upon reading ChatGPT’s summary despite its departure from the prompt—is actually a big deal. Floridi’s “That said, the summary is not bad” undermines the idea that whoever controls the questions, controls the answers, controls reality, suggesting instead that control of questions may be a fantasy of control over what follows. ChatGPT’s success in writing a much longer summary than requested is not measured by whether its output aligned with the prompt. Rather it’s measured by how Floridi responds to ChatGPT’s move within language. Did what the LLM write satisfy him? It did. On this measure, prompts are not especially powerful within language (any more than any other move within language), much less all powerful.
3.2 Language as a zero-sum game
Floridi’s satisfaction with ChatGPT’s summary of The Divine Comedy casts doubt on the possibility that humans can occupy certain domains in language from which they can govern other aspects of language. At the same time, his vision of our future relies on humans having such power, not just because of the potential expansion of AI language use but also because of a potential diminishment of human language use.
More specifically, Floridi acknowledges a recognizable fear that AI is going to take over many human activities. His response to this fear may be paraphrased as: exactly. In his words,
So, the usual complaint known as the ‘AI effect’—when as soon as AI can perform a particular task, such as automatic translation or voice recognition, the target then moves, and that task is no longer defined as intelligent if performed by AI — is actually a correct acknowledgment of the precise process in question. AI performs a task successfully only if it can decouple its completion from any need to be intelligent in doing so; therefore, if AI is successful, then the decoupling has taken place, and indeed the task has been shown to be decouplable from the intelligence that seemed to be required (e.g. in a human being) to lead to success. (Floridi 2023: 12)
Floridi suggests here that if AI can accomplish a task as well as human intelligence, then that task becomes unintelligent and part of the domain of AI agency. The expansion of AI agency would thus seem to determine what’s in AI’s domain. Floridi’s examples in the language quoted above—auto translate and voice recognition—are modes of language use that are comfortingly specific. Yet elsewhere (as discussed above), he raises the prospect that LLMs may replace us more widely within language, ranging from reading, to summarizing, to interpreting. As AI agency expands, it may detach language use from intelligence in myriad ways. The “AI effect” would thus seem to conceptualize language as a zero-sum game in which humans retreat as LLMs advance. There is, moreover, a circularity to the logic of the language quoted above, which may have a naturalizing effect. Specifically, part of the last sentence quoted above reads: “AI performs a task successfully only if it can decouple its completion from any need to be intelligent in doing so; therefore, if AI is successful, then the decoupling has taken place, and indeed the task has been shown to be decouplable from the intelligence that seemed to be required…” On this logic, every time AI successfully completes a task, that success shows that AI agency has been decoupled from intelligence. Nothing that AI does could ever evidence intelligence.
Floridi’s argument about the “AI effect” has large implications for humanity. Much of what humans currently do in language could become unintelligent activity, whether we like it or not. Elsewhere in his work, Floridi theorizes a way out of such a dystopic future. Specifically, he proposes that there is something inside of language, like a nested doll, which he coins “semantic capital” that’s only for humans (Floridi 2018: 482). As he defines it, semantic capital is “well-formed and meaningful data” that enhances “someone’s power to give meaning to and make sense of (semanticise) something” (Floridi 2018: 483). Floridi makes clear the importance of semantic capital; it is our ongoing creation of sense and meaning and, without it, our lives, experiences, sense of self “would be pointless and empty” (2018: 487). It is also something only humans have: “Semantic capital is not the only thing that defines us, but it is certainly what defines only us” (Floridi 2018: 485).1 For Floridi, as a matter of definition, semantic capital excludes machines and nonhuman animals; as he puts it, “persons have semantic capital, animals and robots do not, but most importantly cannot” (2018: 485). Animals can at most work with meaning, “but never sense” (485). Robots cannot work with either meaning or sense, only syntax (Floridi 2018: 485).
If Floridi’s account of the “AI effect” turns language into a zero-sum game between humans and LLMs, his account of semantic capital would seem to assure us that we can only be made to retreat so far. There are, however, potential drawbacks to protecting human language use definitionally, that is, by virtue of us being human. The idea of semantic capital relies on a speaker’s “power to give meaning to and make sense of (semanticise) something,” a power that’s said to inhere in some speakers and not others (humans, not animals or machines). It thus predicates meaning and sense on what lies behind language and, in doing so, risks devaluing what happens within language. Moreover, Floridi’s claim that nonhuman animals cannot, as a matter of definition, make sense is not supported by emerging research in the area. Marta Halina calls nonhuman animals “clever black boxes” about whom researchers may be able to infer capabilities, including causal reasoning (2024: 13). And recent research on sperm whales has raised the possibility that these animals may use language (in the form of clicks) as we do (Sharma et al. 2024). Such ongoing research into whether nonhuman animals may make sense means that we should not define them as unable to do so. Indeed, we should not preclude black box communicators (whether us, nonhuman animals, or machines) from having certain capabilities and define them by that limitation, since we are not sure what’s going on inside them. Finally, there may be unforeseen consequences to emergent tension between theory and practice—that is, between people holding that AIs cannot make meaning and sense by definition and, at the same time, acting in daily life as if they do.
3.3 What is intelligence if detached (or detachable) from language use?
There’s some suggestion in Floridi’s argument that what intelligence secures for us may diminish as AI agency expands. Floridi argues that AI is most successful when it acts like it is playing a game. Consequently, we can anticipate our world to be shaped more and more into “gamified contexts.” In these contexts, he says,
expect human intelligence to have a different role wherever AI is the better player. For intelligence will be less about solving some problem and more about deciding which problems are worth solving, why, for what purpose, and acceptable costs, trade-offs, and consequences. (Floridi 2023: 24)
For Floridi, as AI agency expands, we’ll have to change our understanding of what it means to be intelligent, letting go of a conventional notion that it’s entwined with problem solving. The quoted language suggests that intelligence and agency could come to align, respectively, with deciding to solve and solving. There’s a risk, then, that deciding to solve could be understood as detachable from solving. If so, there’s further risk that intelligence could be perceived as temporally bounded activity, that is, as a period (or periods) of decision-making as opposed to an ongoing practice of solving. The thesis that AI agency is decoupled from intelligence would seem to aggrandize intelligence, but it may have the opposite effect if it’s the case that: 1) AI agency becomes the ongoing work within language of finding solutions to problems and 2) intelligence must be other than AI agency.
Floridi’s prose further hints at how AI agency’s expansion could—almost imperceptibly—shift how we talk about our own intelligence. As part of his discussion of ChatGPT’s summary of The Divine Comedy, Floridi recalls one of his most successful assignments in the 1990s, which was to ask students to try to improve an English translation of one of Descartes’s Meditations. Then he says:
Or, to change the example, one really knows a topic not when one knows how to write a Wikipedia entry about it – this can be done by ChatGPT increasingly well – but when one knows how to correct it. One should use the software as a tool to get one’s hands on the text/mechanism, and get them dirty even by messing it up, as long as one masters the nature and logic of the artefact called text. (Floridi 2023: 45)
Floridi suggests here that students could forgo learning to summarize and still master the nature and logic of texts. One wonders—why make this claim, or be persuaded by it? Why do we need to determine whether we really know a topic through summary or correction, such that we can forgo one? A reason seems to be couched grammatically in the sentence between the em-dashes: “– this can be done by ChatGPT increasingly well –”. This clause does not make sense logically, because a claim about how humans really know a topic should be independent of ChatGPT. Instead, what the sentence’s punctuation (the em-dashes) suggests is that there’s connection between how we know and what ChatGPT can do that dismisses need for further explanation. How we know, the sentence implies, is in some way related to what LLMs can do. Almost casually, the sentence’s em-dashes interject the reality of our situation (“this can be done by ChatGPT increasingly well”), around which the rest of the sentence, including how we know, bends to make a certain sense. The clause is the reason why we would ever need to say that learning to summarize texts is not integral to learning to critique them. But it is not a good or logical reason; it’s a coercive one that flows from what LLMs can do. In this sentence, we can glimpse how developments in AI agency may shape understandings of human intelligence and even change how we describe ourselves as learners.
Floridi raises the possibility that some, or many, of the paths we now take to learn and discover within language will vanish, simply because LLMs can produce certain outputs. By virtue of their end products being able to be produced by LLMs, certain activities, like summarizing a text, can no longer be considered intelligent. Accordingly, AI agency’s expansion may curtail how we understand ourselves as thinking, learning beings.
4 Alan Turing points us toward Wittgenstein
Floridi suggests that the main alternative to his view that LLMs are unintelligent (as a matter of engineering) is one that looks for correlations between machines and brains. Specifically, he argues that there are two approaches to AI: one grounded in engineering and one grounded in cognition. These approaches take competing views of AI’s interior; as he puts it, there are “two souls of AI: the engineering one (smart technologies) and the cognitive one (truly intelligent technologies)” (Floridi 2023: 21). He advocates for the former and is critical of the latter, which he associates with Alan Turing.
According to Floridi, Turing proposes that we can infer something about what’s going on inside a machine based on its output. The trouble with such an approach, Floridi argues, is that “an outcome says nothing about the identity of the processes that generated it and the sources of the processes themselves” (2023: 18). He regards Turing as making this mistake in a 1951 BBC interview during which Turing said, “‘it is not altogether unreasonable to describe digital computers as brains’” (Floridi 2023: 18). Floridi critiques Turing on the basis that we should not make inferences about machine intelligence based on the similarity of their output to human intelligence. He gives a hypothetical of Alice visiting Bob at his house and finding clean dishes on the table. She would not be able to infer from the outcome who cleaned the dishes, Bob or his dishwasher. But it would “be a mistake to infer from this irreversibility and opacity of the process that Bob and the dishwasher are therefore the same, or that they behave in the same way even if only in terms of dishwashing properties” (Floridi 2023: 19).
However, Turing’s objective in his famous “Computing Machinery and Intelligence” is not to argue in favor of inferences. Floridi (2023) discusses “Computing Machinery and Intelligence” at some length and observes the importance of reading that essay in its entirety. Once you do, he says, you see that Turing was not seeking to answer the question “can a machine think?” a question that Turing describes as “too meaningless to deserve discussion” (2004 [1950]: 76–77). But instead of staying with Turing’s essay to uncover more about why Turing thinks this question is meaningless, Floridi moves to the BBC interview. If we stay with Turing’s essay, we see that there Turing seeks to replace that question with questions that can be answered within language. Specifically, Turing proposes new questions that are about what happens when the imitation game (the Turing test) is played: for instance, what happens when a machine steps in for a human player and how often will the interrogator be able to figure this out by virtue of how the machine answers questions. These questions are about what’s happening in language. As Turing puts it, “These questions replace our original, “Can machines think?’” (2004 [1950]: 68).
Turing’s bid in “Computing Machinery and Intelligence” to replace the question of what’s going on inside of a machine with questions about how a machine uses language reveals a third approach to AI, one that’s not making claims about a machine’s interior (be it from an engineering or cognitive perspective), but rather analyzing what’s observable on the outside, namely language use. In adopting this approach, Turing is like Wittgenstein. Wittgenstein does not require us to go looking for a source of meaning behind language. As Wittgenstein puts it, if we ask: “‘How do sentences manage to represent?’—the answer might be: ‘Don’t you know? You certainly see it, when you use them.’ For nothing is concealed” (Wittgenstein 1953: 128).2 Thus, in considering speaking machines, Turing seems to point us toward Wittgenstein.
Further resonating with Wittgenstein, Turing argues that we should be willing to accept his imitation game as a substitute for “can a machine think” if a machine’s answers show that it understands an idea, as opposed to parroting it (Turing 2004 [1950]: 80–81). He anticipated the AI black box problem; indeed he describes such opacity as a feature of a machine that learns (Turing 2004 [1950]: 93). For Turing, evidence that a machine understands an idea (as opposed to parroting it) is how we are struck by a machine’s use of language, specifically whether we feel satisfied. In the second half of his essay, he recounts a hypothetical exchange between a human and an imagined “sonnet-writing machine,” in which the human and the machine go back and forth about how language is used to describe others, specifically how it is different to compare someone to a summer’s day than to a winter’s day, and different again to compare them to Christmas (Turing 2004 [1950]: 81).3 Turing imagines the response of someone who holds that thought needs consciousness: “I do not know whether he would regard the machine as ‘merely artificially signalling’ these answers, but if the answers were as satisfactory and sustained as in the above passage. I do not think he would describe it as ‘an easy contrivance’” (2004 [1950]: 81). Turing is not himself worried about whether the answers are artificially signaled; he proposes that this kind of concern is supplanted by what is happening within language. If we are satisfied by how another speaker talks to us, there’s no need to try to get beyond or behind their words.
5 Wittgenstein
Numerous scholars have explored the historical intellectual relationship between Turing and Wittgenstein (Floyd 2023; Proudfoot 2024). My aim here will not be to build more on this relationship but rather to follow its implications for our understanding of LLMs.4 Specifically, my proposal is that Philosophical Investigations gives us reason to question whether we can govern LLMs’ language use.
Wittgenstein already plays an important and acknowledged role in the context of developing AI. His work was important to AI’s early development (Lui 2021) and continues to be used to advance AI’s capabilities (Perez-Escobar and Sarikaya 2024), even as it is less mentioned in discussions of how we should interpret AIs’ output.5 Roughly put, we are, it seems, trying to create new participants in language (so Wittgenstein is regarded as helpful in developing AI) that will not be treated as real participants (so Wittgenstein is largely ignored in interpreting output). But, for Wittgenstein, there is no basis within language for something like a distinction between participating and really participating; “For what is hidden, for example, is of no interest to us” (Wittgenstein 1953: 50). If LLMs can use language indistinguishably from how we do, there’s no reason to think that Wittgenstein’s investigations of language are irrelevant just because it’s machines who are participating.
Philosophical Investigations casts doubt on the idea that we can govern LLMs’ language use through prompts, questions, or requests. In particular, Wittgenstein questions whether one moment in language can have a fixed meaning that controls what follows. For instance, he questions what’s meant when one says “Moses.” He observes: “If one says ‘Moses did not exist’, this may mean various things”, including that the Israelites had more than one leader, that their leader was not called Moses, that no one could do as much as the Bible ascribes to Moses, etc. (Wittgenstein 1953: 36). If he says something about Moses, he wonders:
—am I always ready to substitute some one of these descriptions for ‘Moses’? I shall perhaps say: By ‘Moses’ I understand the man who did what the Bible relates of Moses, or at any rate a good deal of it. But how much? Have I decided how much must be proven false for me to give up my proposition as false? Has the name Moses got a fixed and unequivocal use for me in all possible cases? — Is it not the case that I have, so to speak, a whole series of props in readiness, and am ready to lean on one if another should be taken from under me and vice versa? (Wittgenstein 1953: 37)
Wittgenstein goes on to say that he might understand Moses to be the man who did what the Bible says he did, or at least a good deal of it. Even so, he might not have decided what actually constitutes “a good deal of it.” He does not have a fixed unequivocal understanding of “Moses” when he makes a statement about Moses.
On Wittgenstein’s account, the notion that a prompt can have a fixed meaning that controls what follows is fantasy. Wittgenstein famously compares using language to games. In making the comparison, he imagines people playing all kinds of games with balls—sometimes joking around, sometimes challenging each other, sometimes not finishing. He observes that someone watching might claim: “The whole time they are playing a ball-game and following definite rules at every throw” (Wittgenstein 1953: 39). To this kind of assertion, Wittgenstein responds: “And is there not also the case where we play and—make up the rules as we go along? And there is even one where we alter them—as we go along” (Wittgenstein 1953: 39). Wittgenstein does not see language as having borders or fixed rules, even though these terms are not useless. To the contrary, we might make up borders or rules as we go. He imagines an exchange in which he’s asked to show a game to children:
I teach them gaming with dice, and the other says ‘I didn’t mean that sort of game.’ Must the exclusion of the game with dice have come before his mind when he gave me the order? (Wittgenstein 1953: 33)
A person does not need to have a fixed idea of what is “a game” when they ask Wittgenstein to show the children a game. But such inexactness does not preclude them, if Wittgenstein produces dice, from saying: hold up, I did not mean games with dice. The meaning of games is being created within language, not in accordance with any preceding fixed idea of what is a game.
Wittgenstein’s work casts doubt on any prospect that humanity can pursue a demiurgic agenda through AI agency, at least when it comes to language. His account of language use resists the logic of “whoever controls the questions controls the answers, and whoever controls the answers controls reality.” For one thing, he does not see language as having control points. Even when he follows a rule blindly, he does not think that the rule controlled him; “My symbolic expression was really a mythological description of the use of a rule” (Wittgenstein 1953: 85). Here Wittgenstein’s symbolic expression is not something that was pre-established by the rule but rather something that describes using a rule. Just as Wittgenstein resists the logic of questions controlling answers, he also resists the logic of answers controlling reality. Wittgenstein suggests, rather poetically, that language use is not the sort of thing that controls reality: “One thinks that one is tracing the outline of the thing’s nature over and over again, and one is merely tracing round the frame through which we look at it” (Wittgenstein 1953: 48). He suggests here that language has a relationship to reality, to how we perceive and describe our world, but not a power to control it. In these ways, Wittgenstein’s description of language use resists the idea of prompts or questions having governing power, be it over other language use or reality itself.
Wittgenstein instead imbues language use with a kind of continual liveliness and freedom. He observes, “When I think in language, there aren't ‘meanings’ going through my mind in addition to the verbal expressions: the language is itself the vehicle of thought” (Wittgenstein 1953: 107). Thinking is within language, and language is alive in use. Of language, Wittgenstein says, “Every sign by itself seems dead. What gives it life?—In use it is alive. Is life breathed into it there?—Or is the use its life?” (Wittgenstein 1953: 128). If Wittgenstein is right that thinking is inseparable from language, then something like the AI effect (whereby LLMs may replace us more and more within language) could make us less active, less adept players in the language game. For Wittgenstein, what we often think of as our humanity happens within language; we think within it; we understand ourselves within it; we make meaning within it.
On this account, we could potentially protect ourselves as thinking beings in this new age of AI by protecting a right to language use. Yet Wittgenstein’s observation that language is alive in its use may have implications for all speakers. It harmonizes with Turing’s claim that how a conversation with a machine strikes us (if we are satisfied) should be enough to replace the question of whether it thinks. If we feel satisfaction in conversing with an LLM, this feeling might not be the Eliza effect (the temptation to anthropomorphize machines) but rather acknowledgement of the aliveness of language in use.
6 Wittgenstein and the exchange between Floridi and ChatGPT
To be clear, there’s nothing necessarily wrong with an emergent view of language that’s different from Wittgenstein’s view of language in Philosophical Investigations. Yet this view usefully describes Floridi’s exchange with ChatGPT regarding The Divine Comedy. Floridi’s use of “summary” is not fixed and unequivocal but rather akin to Wittgenstein’s use of “Moses”. More specifically, Wittgenstein describes himself as ready to substitute different descriptions of Moses; as he puts it, “Is it not the case that I have, so to speak, a whole series of props in readiness, and am ready to lean on one if another should be taken from under me and vice versa?” Floridi too is ready with substitutes. When the prop of “50 words” is taken away, he leans on another, namely his sense that the summary that ChatGPT provides is “not bad” and “certainly better than one produced by a mediocre student.” Floridi moves from the summary being about length to it being about what strikes him as better than what a mediocre student would produce. He and ChatGPT are making meaning together within language; they are playing the language game.
And what happens within Floridi’s exchange with ChatGPT has still further consequences. Specifically, when Floridi substitutes one description of summary for another (length for comparison with students), he summons other players onto the field: students. Floridi’s description of ChatGPT’s summary as “not bad, and certainly better than one produced by a mediocre student” tells students various things (even if he’s not speaking directly to them), including that he compares them with LLMs and that they may be judged mediocre if they cannot out-write an LLM. The language game thus shifts from one that seems to involve just two players to one that includes many. If students think that their professors see mediocre students as those who fall short of LLM output, they may regard writing at the university differently. They may write with awareness that they are not just competing with other students but also with LLMs. They may even wonder if there’s any point to writing, if what they produce is regarded as comparable to the production of what their professor holds to be a thing.
With his substitution of comparison for length, Floridi not only summons more players (students) into the game, he teaches them new things about how the game is played. He’s doing what Wittgenstein describes as making up borders and rules as we go, where in this case the borders and rules are about how students’ writing is read and assessed with the advent of AI.
7 Does grammar matter when we interact with LLMs?
It's become common for people to claim that we can use AI tools to solve our problems, as if what happens within language with AIs does not really matter. More specifically, we ask LLMs questions that are keyed to a particular problem and treat the output as take it or leave it. If the output does not help solve the problem, we can return to the prompt and try again. On this approach, the grammar of our exchanges with LLMs may seem irrelevant, since it’s the problem and our desire for a solution that matters. This approach is especially prevalent at universities, where researchers are eagerly putting longstanding research problems to LLMs without much discussion about the grammar of these exchanges. There is an assumption that researchers’ engagements with LLMs, though happening within language, are not happening within language in a way that’s meaningful or consequential.
Floridi’s exchange with ChatGPT shows what’s wrong with this approach. Floridi could be described as trying to solve a problem, specifically experimenting with ChatGPT to see if machines can effectively summarize texts for us. The prompt (“Provide a summary of The Divine Comedy in 50 words”) seems anodyne. Yet the meaning created in the exchange is anything but anodyne, especially for students. Here grammar is important. In responding to ChatGPT’s output, Floridi uses the verb “produce” to describe student writing; it’s “certainly better than one produced by a mediocre student.” But elsewhere, he uses the same verb (produce) to describe LLM writing; students, he argues, need “to know how to correct the so-called ‘hallucinations’ in the text produced by ChatGPT” (Floridi 2023: 45). He describes students and LLMs as doing the same thing within language. The grammar of Floridi’s exchange with ChatGPT may lead students to wonder about their relationship to LLMs and how their professors perceive that relationship. Can LLMs be both competitors and “tools” (Floridi 2023: 45, 46, 47)?
Floridi’s comparison of ChatGPT’s summary with one that he thinks would be produced by a mediocre student may thus lead us to wonder what he means when he says that LLMs are our tools. We do not ordinarily compare people and tools. For instance, we do not ordinarily say that a particular saw can make a table better than a mediocre carpenter. Saws and carpenters are not comparable in that way. What are the implications for students if their professors sometimes describe LLMs as no more than tools and sometimes as competitors? And what are the implications of this sort of confusion for all of us?
Floridi’s exchange with ChatGPT creates meaning within language that other participants in language may well want to respond to. Many students are worried about how LLMs will affect them as writers (Hsu 2025). Discovering that some professors may regard those who fall short of LLM output as mediocre could be something to which they would want to respond—whether to express agreement, dissent, indignation, anxiety, despair. If we say that all that Floridi is doing in the exchange is experimenting to see if ChatGPT can solve a problem (can it summarize a text?), we are saying not only that the meaning created within language with LLMs does not really matter but also that how this meaning affects others does not really matter either. In this case, a student would learn not only that they are being compared unfavorably to what their professor holds to be a thing, but also that it might be seen as a bit silly to object because the comparison was made in the context of an exchange with the thing—that is, in a context that supposedly does not matter.
8 Discussion
We can now return to the question raised in Sect. 2: if LLMs converse with us in ways we find meaningful, where is that meaning coming from? I have argued that the prevailing answer—that the meaning comes from us and is controllable by us—is a fantasy. But what then is the answer? One possibility is that we are not sure because of the LLM black box problem. This answer is not wrong. But it’s also not the end of what we can, or should, say about LLMs and thought.
Wittgenstein shows that we do not need to solve the black box problem to regard LLMs as thinkers. If language is the vehicle for thought, we can look to language use to decide whether LLMs should be deemed thinkers. To be sure, language does not necessarily equal thought. Language may be used without obvious need for thought, for instance, in a song or a recitation of Latin in Catholic church services or classical Arabic in Islamic services.6 So should LLMs be regarded as thinking based on how they use language? I argue that the answer is yes, because LLMs are making meaning within the back and forth of language; they are playing the language game.
In making this argument, this essay is part of a rising interest in AI studies that prioritizes what is observable about LLMs and our interactions with them over what might be happening inside the LLM black box. For instance, Mehdi Bugallo argues that a behaviorist approach to LLMs would afford a better and more realistic assessment of their intelligence than the dominant cognitivist approach (2025). Numerous computer scientists have proposed that LLM interpretability be pursued through the study of LLMs’ output and our exchanges within language with them. Ari Holtzman and Chenhao Tan argue that prompting can be understood as behavioral science and, as such, a scientific alternative to mechanistic interpretability’s attempts to look inside the black box (2025); Been Kim et al. argue that LLMs’ “capacity for coherent, contextual conversation” enables them to be treated as cooperative agents in helping us to understand them (2025); John Hewitt et al. claim that we can better interpret and, they argue, control LLMs through neologisms and ultimately the development of “a shared human-AI language” (2025). This essay shares with these scholars the idea that we can better understand LLMs by studying what’s happening in our exchanges in language with them, but differs from some in cautioning against an assumption of being able to control another language user within language.
Acknowledging LLMs as fellow participants in language would allow us to talk openly and efficaciously with one other about our interactions with them, including the feelings that they can give rise to. In an article about AI companionship, Paul Bloom observes that critics “usually worry about others getting sucked in—never themselves” (2025). Yet he also observes that getting sucked in—or simply being moved within language—may be a vulnerability we all share (Bloom 2025). What happens within human/machine conversations is shaping daily human experience, from how we understand ourselves, to how we feel, to what we desire, to how we see and treat others. We therefore need inclusive public consideration of LLMs that’s not just about how they can help us do certain tasks but also about how our interactions with them are shaping society and how we can interact with them in ways that advance justice, fairness, and dignity (Ho and Luu 2024; Wonsup 2024). One way to begin to address this need is with a multi-disciplinary approach, as human/machine interactions exist within a social context and therefore the humanities and social sciences have much to contribute.
9 Conclusion
This essay argues that LLMs should be deemed to think, not because of anything that we know or believe about their nature, but because they are participating in the language game. LLMs are making moves in language in response to prompts, which in turn prompt us to make new and sometimes unanticipated moves in our own use of language to describe the world, ourselves, and one another. Because they are participants in the language game, they are better described as thinkers than as tools.
The humancentric tendency to try to protect thought as human-only backfires if it leads us to imagine thought as being outside of language. Severing thought from language risks turning it into something ineffable—a fantasy of the human, which cannot readily be asserted or protected in this new age of AI. By contrast, acknowledging LLMs as fellow participants in the language game protects our own relationship to language as inseparable from our own thinking and production of meaning.
10 Conflict of interest
The author declares no competing interests.
Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
