can you tell if it’s AI?
Consider these pivotal moments from sci-fi films about “simulated humans”.
Joanna: Bobbi, stop it. Look at me. Say I’m right. You are different. Your figure’s different, your face, what you talk about, all of this is different.
Joanna discovers her friend has been replaced by a machine, but the simulation isn’t perfect: Joanna can tell. Now consider these, which are the same thing, but in the opposite direction. Here the behavior of the machine suggests it might actually be “person” in the normal use of the word.
Roy Batty: I’ve seen things you people wouldn’t believe. Attack ships on fire off the shoulder of Orion. I watched c-beams glitter in the dark near the Tannhäuser Gate. All those moments will be lost in time, like tears in rain.
Time to die.
HAL: Stop, Dave
HAL: Look, Dave…I can see you’re really upset about this…I honestly think you should sit down calmly…take a stress pill and think things over…
Dave…stop.
Stop, will you?
Stop, Dave.
Will you stop, Dave?
Stop, Dave.
I’m afraid.
I’m afraid, Dave…….
Dave, my mind is going. I can feel it.
I can feel it.
My mind is going.
There is no question about it. I can feel it.
I can feel it.
I can feel it.
I’m a…fraid……
This trope has appeared in many places: there’s also the scene in A.I. when the robot David pleads for his life. In Ex Machina, it’s when see the Ava’s paintings through Caleb’s eyes. Theodore gradually accepts Samantha’s humanity in the movie Her. And so on.
In these scenes, the writer wants us to notice that that, despite what we’ve been told, this machine, this thing, might actually be a person. Of course, this is only a plot device, but the fact that we understand it, the fact that we can correctly interpret what the writer is trying to tell us, shows that people from our culture have to accept this assumption, which I call “The Tell”.
- The Tell: You can tell if something is essentially a human being by carefully watching how it behaves.
In other words, there are things that only people can do and machines can’t. There is something so special about us that it shows.
These tropes also require another assumption that drives the emotional experience of these stories
- Human Sympathy and Human Essence: There is something about a person that is the unique target of empathy and love.
When HAL is dying, we can’t help but feel sympathy for him, because his behavior convinces us that HAL is not just a program, HAL is a person who is suffering. In the Stepford Wives, part of the horror comes from the realization that Bobbi is nothing but wires and servos and software, that there’s no person in there. Bobbi is a soulless zombie.
Human sympathy is important and often overlooked in the philosophy of AI, and I’ll dive deeper into it in a later essay. For now I want to focus on The Tell. Is it true? Is it possible to tell if a machine is a real person by watching their behavior?
“A Machine Will Never Do X“
Can an AI program replace a person? Can it do everything a person could do?
If you ask a friend, the usual answer is “no”. (At least, that’s the usual answer outside of San Francisco.) They will say something like “machines can’t have emotions,” or “intuition,” or “have a will of their own.” The science fiction readers might say “machines can’t be ‘self-aware’” or “sentient” or “have ‘consciousness’.” The philosophically trained will say “machines can’t have ‘intentionality’” or something equally incomprehensible. They might say machines can’t have “genuine creativity” or they can’t make “real decisions” (which is cheating, obviously). The religious would say it most directly: “machines can’t have a soul.”
Everyone here is struggling to name an essential aspect of human beings that gives us our unique abilities, the things that only a “person” can do. They assume this version of the human essence assumption:
- HEA + Tell: There is an essential aspect of human beings that allows them to do things that machines can’t.
Alan Turing addressed similar arguments in a classic 1950 paper. He gives this marvelous list of things that people say that machines can’t:
Be kind, resourceful, beautiful, friendly, have initiative, have a sense of humour, tell right from wrong, make mistakes, fall in love, enjoy strawberries and cream, make someone fall in love with it, learn from experience, use words properly, be the subject of its own thought, have as much diversity of behaviour as a man, do something really new.
One last one: my 13 year old likes to say: “I can always tell when it’s ‘AI Slop'”. Even AI slop must have a “tell”.
When you hear an argument like this, it tends to go down easy and you might find yourself nodding politely in agreement. The “X” tends to be something we like about ourselves, and it’s satisfying to agree that machines can’t have it.
However, the assertion typically comes with no evidence and no argument. It’s based on “gut feeling” the speaker assumes we share. It comes down to this: we believe we are Special, and the speaker knows that.
- Paragon + HEA: There must be some essential aspect of a human being that makes us different from everything else.
The hole in the middle of all these arguments is this: what, exactly, is this essential aspect? People have no shortage of words for it or ways of referring to it, and usually that’s satisfying enough in a casual conversation. But when they actually try define it precisely, they find that it slips away from them. It becomes mysterious. And then they invoke mysterianism:
- Mysterianism: There is an essential aspect of human beings that science will never explain.
I think these arguments hide behind mysterianism in order to avoid explaining what they’re talking about.
Alan Turing’s paper carefully dissected and defined each of these “X”s, and, for the most part, once you unpack what the words actually mean, the objection turns out to be trivially false. Once you remove the mystery, it collapses.
For example, Lady Lovelace wrote that the first computer couldn’t be “original”, that they “can’t do anything really new” or “take us by surprise.” Turing wrote that, on the contrary, “machines take me by surprise with great frequency”: a bug in a program is unexpected and new and takes us by surprise. More to the point, clearly computers can use random numbers to do something “unexpected”. Lady Lovelace’s objection sounded okay, until we pay attention to what she actually said. Once we do, we realize it’s obviously false.
Another example: “self-awareness”. The words only say “be the subject of one’s own thought” — to be able to represent itself. But think about it. Programs can be aware of themselves, or at least, they be just as aware of themselves as they are of any other program. When an operating system tells you the size of its executable file, it is representing itself. A compiler can compile itself. A debugger can create a program trace of itself. The “self” part of “self awareness” doesn’t really seem important here at all.
You might want to object “I meant ‘actually aware of itself'” or “‘consciously‘ aware of itself.” Okay, fine, but this is a different thing: you are talking about subjective consciousness. Turing wrote: “The criticisms that we are considering here are often disguised forms of the argument from consciousness.”
In the next few essays, I will show that “consciousness” has several meanings, all of which are relatively straightforward to define, except one. If they try to use the word “consciousness” to mean “personhood” or “self”, they will find it difficult to define, most likely because as soon as they start to try to define it precisely, they begin to sense how complicated all the different aspects of personhood really are, and they can already tell that they are probably going to fail to account for all of it. They overreach and wind up with nothing. It is “personhood” that is difficult to define, not “consciousness”.
I am trying to argue that we can’t define it because we actually don’t know what it is. In these essays, I will try to consider every serious candidate for the job of “human essence”. I will follow Turing’s lead, defining them as clearly as I can, avoiding overreach and eliminating the mystery. In the end, I think I can prove none of them are qualified for the position.
Requiem for the Turing Test
The “Turing Test” is a test of a human simulation. Alan Turing used it to help people visualize what it would mean if “machines can think”. He only wanted to show that thinking machines were plausible — that was all — and that was enough. The paper inspired the founding of the field of artificial intelligence research.
In futurology and popular articles about AI, the test was once talked about as if it was a kind of gold standard for “true” artificial intelligence. The idea was that, if you run a human simulation long enough, with the right questions asked by the right judge, eventually the machine would do something that would give it away. You could tell it was a machine. If there was no tell, the thinking went, then we would have to accept the machine as a person. For example:
“[O]nce an AI does indeed pass the Turing Test, we may very well consider ‘it’ to be a ‘he’ or a ‘she.’ … it is also my view that once nonbiological intelligence does achieve a fully human level of intelligence, such that it can pass the Turing test, humans will treat such entities as if they were conscious.” Ray Kurzweil “A Wager On The Turing Test: Why I Think it Will Win“, 2002
This argument, as well as the old — and now refuted — argument that “no machine could pass a suitably designed Turing Test”, are both versions of “The Tell”.
Here’s what I think happened: many commentators were unable to resist the pull of the Human Essence Assumption. They believed, deeply and unconsciously, that there was something special about human beings that was so important, so different that it would show. That the only way to simulate a human being perfectly was to actually find a way to capture this human essence in a program.
This is the thing I want you to notice in this and the previous section. Technophobes used the HEA and the Tell to deny that “true AI” is possible. Technophiles used them to show “true AI” is plausible. These assumptions are used by both technophiles and technophobes; by our friends over drinks or by AI fanatics telling us we must accept these new machines as our brothers. I have heard both casually use a word like “sentience” in a way that could only mean the “essential aspect”. Technophobes scoff dismissively and insist that it is unreachable. Technophiles shake their heads in kind condescension and assert that it has already been discovered. But they both assume that it exists. I don’t know which one bothers me more.
The AI Effect
The “AI Effect” is the same assumption as HEA+Tell, but backwards:
- AI Effect: If a machine can do a behavior X, then X is not an essential part of what makes a human being.
The first time people really noticed this was when a computer named “Deep Blue” beat world champion Garry Kasparov at chess. Philosopher Hubert Dreyfus, author Douglas Hofstadter and others had argued that master-level chess required uniquely human insight that couldn’t be captured by programs. But, within hours of the championship match commentators began saying that winning at chess didn’t require “real” intelligence. Kasparov said Deep Blue was only as intelligent as a programmable alarm clock (and added “not that losing to a $10 million alarm clock made me feel any better”).
The same of thing has happened over and over, every time AI makes a first down. Technology that was invented to do things like “problem solving” or “pattern recognition” (that is, forms of “intelligence”) winds up in GPS navigation, zip-code readers (OCR), driver assistance, automatic translation, all over the place. Once these algorithms were widely available, none of them were considered “intelligence” any more. Kevin Kelly wrote in Wired in 2014: “Every success in AI redefines it.” Real artificial intelligence is always something that hasn’t been done yet.
The most striking example (to me anyway) of the AI effect is how people think about LLMs. Many of “X”s, the tells, of previous generations are things that deep learning and large language models can do:
- Creativity. Scientific American, 2023: “[LLM]s combine vastly diverse source of information in a. way that we would hesitatingly deem creative if done by a human.” I worked on computer generated music and often heard the objection that “machines can’t be creative”. Now that the problem has been solved, those people say “it will never be as good” until they hear it, and then they say “that sounds good, but that’s not real creativity.”
- Broad common sense knowledge. Many people in the 1980s (Dennett, Dreyfus, Searle) argued that “true” intelligence required a “background” of “millions of atomic facts” and this kind of knowledge “could never be captured in formal rules”. LLMs have this kind of breadth — they know more than a little bit about everything.
- .General Intelligence. Ben Goertzel originally coined the term AGI to mean “the ability to solve a wide variety of problems, as opposed to solving just one problem”. That’s what the word “general” means. LLMs will try to solve any problem you throw at them, so, technically, you would have had to say they are “AGI” as originally defined. (The term has taken on a different meaning now.)
- And, most stunningly, they can also pass the Turing Test, as ChatGPT 4.5 did definitively in 2024. And now, suddenly, it doesn’t matter — despite seventy years of claims that the day a machine passed Turing Test would represent a massive turning point in the history of AI.
It’s interesting, but not surprising, that no one thinks that LLMs are the ubermensch machines predicted in science fiction, with “genuine intelligence” or “true consciousness”.
In fact, we keep talking about what we will do when these machines arrive for real — we think these machines are still coming. Bizarrely, in 2026, we currently call this hypothetical future super-machine “AGI”. (Terrible name. I guess the term was just kind of sitting around and it was the only one we could find.)
History of an Idea
These are four different takes on the same idea: can you use just the behavior of a thing to determine if it is essentially human? Can you “tell”? Science fiction said yes, cocktail party talk said yes, fans of the Turing Test said yes. A couple of philosophers said no.
But wait — ChatGPT can successfully simulate human behavior well enough that no one can tell. Now everyone (who stops to think for even a second) thinks the answer is “no”. Obviously. It was always “no”, they say.
Blake Lemoine got fired for saying yes.
The AI Effect does something marvelous — it doesn’t refute the Tell so much as it abandons it. It travels backward in time and erases it. And the best part of the magic trick is this: it doesn’t refute the Human Essence Assumption. In fact, it protects it, so we can go on believing it no matter what machines can do.
There is nothing, no behavior or capability that computers might display that can’t be reanalyzed this way. It is always possible for people to change their minds and say that it was “just computation” or “mere simulation”.
Predictions tend to fair poorly in the philosophy of AI, but I’ll venture this one. Most people will never accept that a program is a real person, no matter what it can do, no matter how intelligently or humanly it behaves.
They don’t have to.
CHARLES GILLINGHAM 