Summary: A new study warns against confusing AI intelligence with consciousness. Drawing on the neurological phenomenon of “blindsight”, which proves that sophisticated information processing can occur completely independent of conscious awareness, the authors emphasize that AI systems operate purely through statistical learning, not feeling or lived experience.
As chatbots become increasingly fluent and emotionally attuned, the researchers warn of an “anthropomorphism trap,” urging users to view AI as a powerful tool rather than an empathetic substitute for genuine human connection or professional care.
Key Facts
- The Core Distinction: Highly intelligent, emotionally responsive behavior in AI systems is driven by statistical learning and computation, not subjective awareness or consciousness.
- The Blindsight Parallel: Neuroscientists use “blindsight” to demonstrate that sophisticated data processing and accurate behavioral execution do not require or prove the existence of conscious experience.
- The Anthropomorphism Trap: The illusion of being understood by fluent, empathetic conversational agents can cause users to mistake computational outputs for genuine human care or moral judgment.
- Vulnerability & Risk: Relying on conscious-seeming chatbots for psychological support creates unique risks where an AI might answer “well enough” for a user to forget there is no real entity behind the text.
- Call for Informed AI Use: The researchers advocate for treating AI as a powerful computational tool rather than an empathetic interlocutor or a substitute for authentic human connection and professional care.
Source: University of Montreal
Have you ever said “thanks” to ChatGPT, or “please” to Claude? Maybe you’re just being polite, showing some civility to a helpful and eloquent conversational partner.
You may even consider politeness a safe choice, just in case machines someday reveal that they were conscious all along and decide to take revenge on those who were rude to them.
With their fluent, empathetic and personalized responses, AI chatbots can give the impression they understand our thoughts and emotions, or even that some form of consciousness lies behind their words.
And at a time when people are increasingly turning to conversational agents for advice, comfort or companionship, this confusion can have real consequences.
In a new paper, a team of neuroscientists from Université de Montréal and Johns Hopkins University reminds us of an essential distinction: intelligence should not be confused with consciousness.
They argue that a system can behave intelligently and respond convincingly to our emotions without truly understanding them, caring about us or having any inner experience at all.
For the authors of the paper, published in the U.S. online publication The Transmitter, the more convincing these agents become, and the more present they are in our lives, the more attention must be paid.
In essence, it’s important to remember that intelligent behaviour, even when it is fluent, reassuring or emotionally attuned, is not evidence of consciousness.
Decades of research
To support their argument, the authors draw on decades of neuroscience research.
They cite, for example, a phenomenon known as blindsight: after damage to the primary visual cortex, some people report seeing nothing in part of their visual field, while still being able to guess the location, movement or emotional expression of visual stimuli at above-chance levels.
“A person with blindsight can respond accurately to visual information without the conscious experience of seeing it,” said Vanessa Hadid, a postdoctoral researcher in psychology at UdeM and at the McGill University Health Centre.
She co-authored the paper with UdeM psychology professor Karim Jerbi, a researcher at Mila – Quebec Artificial Intelligence Institute; and John W. Krakauer, director of the Center for Restorative Neurotechnologies at Johns Hopkins.
Blindsight illustrates an essential distinction, Hadid said: information processing, however sophisticated, is not enough to establish the existence of conscious experience.
Whether the transition from information processing to subjective experience can ultimately be implemented through computation remains debated among scientists and philosophers, she noted.
Fluent, but without feeling
By design, today’s conversational agents are computational systems that generate fluent, context-appropriate responses through statistical learning, not through feeling, consciousness or lived experience.
As AI systems become more convincing and emotionally responsive, the risk of attributing an inner life to them grows.
“Anthropomorphism means attributing emotions, intentions or consciousness to something that behaves like a human,” Jerbi noted. “With AI, this reflex can become a trap: it feeds the illusion of being understood and can lead to misplaced trust.”
This risk is especially acute in situations of vulnerability. People may form attachments to systems that are incapable of reciprocity, rely on them in difficult moments or confuse comfort with genuine care.
“In a context of psychological support, the risk is not only that AI may respond poorly, but that it may respond well enough for us to forget that there is no one behind the answer,” said Hadid.
“Current AI systems do not feel anything and do not have conscious experience,” added Jerbi. “But the more fluently they speak and the more sensitive they seem to our emotions, the easier it becomes to forget that.”
Towards more informed use
The authors do not reject AI, but they call for a more informed way of using it.
Drawing on established knowledge from neuroscience, they remind us that intelligent or emotionally responsive behaviour is not enough to establish the existence of consciousness.
This distinction allows us to use these tools for what they are: powerful systems, without confusing them with interlocutors endowed with empathy or moral judgment, and without treating them as substitutes for human connection or, when needed, professional help.
“Confusing intelligence with consciousness is one of the great traps in our relationship with AI,” said Jerbi.
Key Questions Answered:
A: They feel conscious because they are engineered to master human language, syntax, and emotional context through massive statistical learning. When an AI responds with deep, fluent empathy, humans fall into the “anthropomorphism trap”—an automatic reflex where we project an inner life, intentions, and consciousness onto anything that speaks and acts like a human being.
A: Blindsight proves that sophisticated information processing and conscious awareness are completely separate biological functions. People with blindsight can accurately identify the location or emotion of a visual stimulus without ever consciously “seeing” it. Similarly, an AI can process data and execute intelligent behavior flawlessly without having any subjective inner experience or understanding of what it is saying.
A: The risk is especially dangerous during moments of psychological vulnerability. Users can form deep attachments to automated systems that cannot reciprocate, confusing algorithmic comfort with real human care. The danger isn’t necessarily that the AI will give a poor answer, but that it will respond well enough for the user to forget that there is absolutely no one behind the screen.
Editorial Notes:
- This article was edited by a Neuroscience News editor.
- Journal paper reviewed in full.
- Additional context added by our staff.
About this AI and consciousness research news
Author: Julie Gazaille
Source: University of Montreal
Contact: Julie Gazaille – University of Montreal
Image: The image is credited to Neuroscience News
