The Chinese Room Thought Experiment: Syntax and Semantics

AI competency versus AGI comprehension

There’s a been a lot of discussion recently over the nature of LLMs and AGI and whether or not the former represents a toehold on the latter. I’ve discussed this in several previous posts, most recently here and here.

Related to this discussion is a famous thought experiment by the philosopher John Searle known as the Chinese Room. The experiment was first proposed at a lecture Searle gave at Yale and later published in a paper titled Minds, Brains, and Programs in the journal Behavioral and Brain Sciences in 1980.

The paper caused quite a stir and generated extensive commentary, mostly in the form of arguments against it. Many considered it an argument against the possibility of creating what we call AGI today, but Searle specifically said that was not his intent nor what should be concluded from the paper.

To understand the idea he was trying to get across, it’s important to put the paper into the context of its time. In 1980, almost all AI research involved directly programming formal symbolic systems. This meant specifying formal rules for manipulating and associating symbolic information. The symbols in question could represent things such as objects, words, or concepts.

Searle defined two types of AI to conform to the thinking of AI research at the time, Weak AI and Strong AI:

According to weak AI, the principal value of the computer in the study of the mind is that it gives us a very powerful tool. For example, it enables us to formulate and test hypotheses in a more rigorous and precise fashion. But according to strong AI, the computer is not merely a tool in the study of the mind; rather, the appropriately programmed computer really is a mind, in the sense that computers given the right programs can be literally said to understand and have other cognitive states.

In other words, Weak AI is AI that replicates certain limited capabilities of human cognition without any underlying comprehension. Strong AI replicates human cognition and consequently both its capabilities and its ability to comprehend. In today’s terms, Weak AI is equivalent to AI and Strong AI is equivalent to AGI (at least the way AGI has most often been defined).

The Chinese Room thought experiment was an argument against the idea that Strong AI could be created by programming and that it could exist on hardware such as that used in computers. If it’s possible to create AGI at all, he believes that it will have to be done using processes like those in the brain and on hardware like the brain.

To demonstrate this, Searle proposed placing himself in a room with paper and pencil. He has no understanding of Chinese, but he’s given papers listing Chinese characters (i.e., the logographs of Chinese writing), a story in Chinese and a series of questions in Chinese. He’s also given English instructions on how to correlate the Chinese questions to the Chinese story using the Chinese characters.

Since he doesn’t know Chinese, he doesn’t know what the questions or story are or even that they’re a story and questions about that story. All he knows is that he has instructions on how to take these three inputs and output Chinese characters.

To an observer outside the room (at least one who knows Chinese), Searle appears to understand the story and the questions. Yet, all he really understands are the English instructions — the program — that advise him on which characters to write on the paper given the Chinese characters handed to him.

In 2009, Searle wrote a follow-up article in which he simplified the experiment such that the Searle in the room has a database of Chinese characters and instructions in English that allow him to answer questions in Chinese sent to him with answers that are also in Chinese. He also clarified what his intent was with the experiment:

The Chinese Room Argument thus rests on two simple but basic principles, each of which can be stated in four words.

First: Syntax is not semantics.

Syntax by itself is not constitutive of semantics nor by itself sufficient to guarantee the presence of semantics.

Second: Simulation is not duplication.

In this post, I’d like to concentrate on the first principle. What Searle is stating here is that merely being able to manipulate and arrange words successfully (syntax) does not imply or necessitate understanding the concepts behind those words (semantics). This is perhaps a more important principle to consider today than it was when Searle first proposed it, for it underlies the current debate about LLMs and their ability to understand anything at all.

Thought Experiment to the Real World

Searle’s task in the experiment is to appear to both comprehend questions in Chinese and provide answers to those questions in Chinese without actually knowing any Chinese.

There have been a number of arguments against Searle’s Chinese Room conclusions over the years, and Searle has written rebuttals to them, some more convincing than others. But none of the arguments against the Chinese Room really do a good job of addressing the point of the thought experiment or the issues with it. A number of these arguments and Searle’s rebuttals can be found here and here.

Searle claims that his scenario is equivalent to programming a computer to do the task, with Searle as the computer and the English instructions as the programming. His first conclusion is that no actual comprehension of Chinese is required to complete this task. His next conclusion is that no programming of a computer system can ever lead to comprehension within that system.

A lot of time has passed since Searle first proposed the thought experiment, and we have a lot more real world empirical data available with which to evaluate it. In fact, the thought experiment is now something that we can easily do in the real world using ChatGPT, Gemini, Claude, etc. This means that in many ways, the experiment is very closely related to what LLMs do today and the question of whether they possess any understanding of the real world.

The Rules of Language Translation

The task of the Chinese Room experiment is also closely tied to the problem of machine language translation. Scientists struggled for decades with this task, with the first concerted efforts starting in the early 1950s. Early approaches involved directly programming the translation engine, and the results were not particularly good.

During the 1990s, systems using a statistical approach began gaining steam. These systems ingested large volumes of text in the target languages, most often datasets of matching dual language texts, and then used statistical analysis to achieve the final results. These were better than the rule based systems but still not great.

Starting in the mid 2010s, however, neural network machine learning techniques began to be used in what were called neural machine translation engines. This has been the most successful and widely used technique so far, and is the technique used in systems such as Google Translate. A related offshoot of this is the translation capabilities of LLMs today, which are relatively good though typically not as good as specialized translation engines.

LLMs, however, have the added ability to actually answer questions in a language the operator does not know. Thus, what was a hypothetical scenario need no longer remain so. In other words, if Searle is in the room with his phone, he can use ChatGPT or Gemini to actually perform his task in the Chinese Room.

The Limits of Formal Rules

Before diving into whether Searle’s conclusions hold up given our ability to test them (at least to some degree) in the real world, it’s worth pointing out a foundational flaw in the experiment that the empirical evidence of the last 40-plus years has clarified. This is the assumption that it’s possible to create instructions which would be sufficient to guide Searle to successfully completing his task in the Chinese Room.

Rule-based translation techniques and rule-based attempts to answer random questions directly or in relation to a source text have simply proven to be highly ineffective. The possibility that Searle could have a rule book, even one of infinite length, that would allow him to complete his tasks in the Chinese Room seems pretty remote. There are an infinite number of questions to ask and languages are also always in flux, so no matter how complete your rule book might be, there will always be questions and nuances of language that are beyond its scope.

So while we can’t say a useable system along these lines is impossible, we can say that all attempts to do anything like this approach since the conception of AI have provided evidence against its viability. This may seem like an unimportant detail given that this is just a thought experiment, but its importance will become clearer below as it relates to LLMs.

Thus, the base premise of the thought experiment, even in a perfect “spherical cow” world of a thought experiment, rests on a proposition that is quite likely too complex to ever implement. This is the first step of the experiment but it’s not established as correct or even possible. What speculation follows this faulty premise does so as an Unproven Basis fallacy in that it bases a conclusion on an unproven foundation, and thus it can’t really be relied upon.

However, another limitation of Searle’s thought experiment actually negates this problem. Searle assumes a very narrow view of programming, one that confines programming to creating rules to directly manipulate symbols. The successful Deep Learning AI systems we have today don’t work that way.

Instead, these contemporary systems consist of data structures and mathematical processes that are able to recognize the patterns of symbols in data and restructure themselves to associate those symbols in meaningful ways using statistical analysis. This requires vast amounts of human-created data and typically some degree of human guidance. The actual programming involved provides the structure and processes for the (mostly) self-assembled correlations of data rather than any explicit specification of those correlations.

So it is now possible to create a system that will do exactly what the system in the experiment does, just not in the way Searle described. Given this modification to the experiment, is Searle still correct in his conclusion that the system itself has no comprehension of what it’s talking about?

An LLM in the Chinese Room

One of the most famous arguments against the Chinese Room is known as the System Reply. This argument states that while the man doesn’t understand Chinese, the combination of the man with the papers of instructions and questions together form a system that understands Chinese. The understanding is embodied in the paper instructions, not the man using those instructions, and thus it is possible to program understanding.

Obviously, the main problem with this is the one mentioned above, namely that this system won’t actually work so ascribing comprehension to it is moot. Searle’s rebuttal was to reframe the scenario so that the man now memorizes all the rules so that he has the rules in his head. Now he is the only part of the system into which questions written in Chinese are fed and from which answers in Chinese come out, yet he still has no understanding of Chinese.

In the context of the thought experiment, this is a fair if not completely satisfying rebuttal. The problem is that the understanding is not in the papers and pencils given to the man, but instead in the people who wrote all the instructions on the papers. Without that human understanding, the man in the Chinese Room would not be able to complete the task. If we update the scenario so that the man in the room is using an LLM to answer the questions, the understanding is likewise in the people who created all the data that was ingested to train the LLM rather than the LLM or the man in the room.

And yet, Searle is correct in that the system, whether paper and pencils or LLMs on his phone, still has no understanding of semantics, no comprehension of the meaning behind the questions or answers. Whether based on formal logic or statistical analysis, the system has, in the words of the late, great Daniel Dennett, competence without comprehension.

This is relevant to one of the more interesting questions in today’s LLM debates, which is whether these systems actually understand anything to create their impressive results. Some scientists believe that current LLMs do actually have some degree of understanding about the world around them. They believe, depending on the particular system, that they may even have developed an internal model of the world, a model at least somewhat similar to the ones humans have.

Yet there seems to be very little evidence to support this opinion, and, in fact, substantial evidence against it. I’ve discussed this issue in several previous posts (such as this one, this one, and this one). I suspect most in the field (at least if they’re talking off the record) would agree. Perhaps the most vocal computer scientist speaking out against this thesis is NYU Professor and Chief AI Scientist at Meta, Yann LeCun.

The Argument Against AGI

In his 2009 article, Searle breaks down his experiment in the following way:

Premise 1: Implemented programs are syntactical processes.

Premise 2: Minds have semantic contents.

Premise 3: Syntax by itself is neither sufficient for nor constitutive of semantics.

Conclusion: Therefore, the implemented programs are not by themselves constitutive of, nor sufficient for, minds. In short, Strong Artificial Intelligence is false.

So Searle believes his thought experiment is ultimately an argument against what he calls Strong AI, which is similar to what we would today call AGI, being implemented with programs on a computer.

In the original 1980 paper, Searle states:

“But could something think, understand, and so on solely in virtue of being a computer with the right sort of program? Could instantiating a program, the right program of course, by itself be a sufficient condition of understanding?” This I think is the right question to ask, though it is usually confused with one or more of the earlier questions, and the answer to it is no.

As discussed above, Searle used a very narrow definition of programming and computation. However, swapping that out for our modern day examples of LLM systems, it turns out that the first part of his conclusion — that the program completing the task in The Chinese Room is not constitutive of a mind — has now been proven. In other words, it is possible to have competency in this task with no comprehension.

Thus what’s surprising, and even shocking to many, is not just that the task can be done at all, but that it can be done without understanding. In fact, it’s so surprising that many people, including scientists, are reluctant to accept this conclusion.

Machines and Minds

But this is only one part of Searle’s conclusion. He correctly states that one can’t infer understanding from such a programmed system, but he also states that such a system is insufficient to create a mind, something that embodies cognition and is capable of understanding.

Searle feels that what distinguishes a mind from a computer program, as well as what distinguishes Weak AI from Strong AI, is intentionality. This is the capacity of the mind to represent objects and affairs in the world, to have internal states that are about or directed towards beliefs, desires, and perceptions of objects, events, or conditions in the world.

Searle stated the following at the end of his 1980 paper:

The point is that the brain's causal capacity to produce intentionality cannot consist in its instantiating a computer program, since for any program you like it is possible for something to instantiate that program and still not have any mental states. Whatever it is that the brain does to produce intentionality, it cannot consist in instantiating a program since no program, by itself, is sufficient for intentionality.

This leads to his overall conclusion that AGI, or Strong AI as he terms it, is impossible in a programmed computer system. We now have evidence that the conclusion of the first sentence in the above quote is valid (at least in the special case of the Chinese Room experiment). However, the conclusion of the second sentence amounts to an Appeal to Ignorance fallacy, i.e. a claim that something is true simply because it hasn’t been proven false.

It has now been shown that a programmed computer system can complete the Chinese Room tasks by outputting appropriate responses and yet have no understanding of the questions or responses. But this in no way implies the obverse, i.e. that it’s not possible to create a programmed computer system that completes those tasks and understands them. Nothing in his thought experiment demonstrates this, either, nor does he provide any sort of proof for the claim.

The case is similar to the popular aphorism usually attributed to scientist and science communicator Carl Sagan: Absence of evidence is not evidence of absence. All the experiment demonstrates is that it’s not necessary to have semantic understanding of language to manipulate it in a way that appears to require such understanding.

Searle also states in that paper’s conclusion:

Whatever else intentionality is, it is a biological phenomenon, and it is as likely to be as causally dependent on the specific biochemistry of its origins as lactation, photosynthesis, or any other biological phenomena.

This is the Ipse Dixit fallacy, the assertion without proof or evidence, that is the basis for the above Unproven Basis fallacy. While it may or may not true, it’s not elaborated on much in the paper nor is it proven or demonstrated by the thought experiment. It’s an assertion that requires supporting evidence, but none is offered, and so far there is none provided by research in the field. All we know is that we haven’t created intentionality in a non-biological medium yet.

Searle does have a reason for his assertion, however, and this takes us back to his second principle of the Chinese Room argument according to his 2009 article: Simulation is not duplication.

And this is the topic of the next post…