A new commentary in Nature argues that the prevailing metaphor of the brain as a computer may be misleading the debate on whether super intelligence (SI) systems possess consciousness. The authors, including neuroscientist Anil Seth, philosopher Ned Block, and neuroscientist Giulio Tononi, contend that intelligence and consciousness are distinct phenomena and that current SI models lack the biological substrates required for subjective experience.
What Happened
The commentary critiques the assumption of computational functionalism, the idea that consciousness depends solely on the type of information processing rather than the physical medium. The authors argue this view is "fatally flawed" and stems from an overextension of the brain-as-computer metaphor. They point to the human cerebellum, which contains nearly four-fifths of the brain's neurons and handles complex motor planning without contributing to conscious experience, as evidence that high-level information processing does not necessarily result in consciousness. The authors further distinguish between intelligence, defined as the ability to carry out functions, and consciousness, described as the state that disappears during dreamless sleep or anesthesia. They argue that SI systems, particularly large language models, act as "mirrors that reflect an image of our collective digitized past," mimicking human language patterns without necessarily experiencing them.
Why It Matters
This perspective challenges the growing industry discourse around SI welfare and SI rights, terms often used in discussions about the ethical treatment of advanced SI agents. The commentary suggests that falsely attributing consciousness to SI systems is a greater risk than overlooking it, potentially diverting resources from beings whose consciousness is established. This stance contrasts with recent research from Anthropic, which reported internal processing patterns in its Claude model resembling properties of consciously accessible information processing, though the company does not claim this as evidence of subjective experience. The debate also intersects with the work of the Butlin report, which evaluated current systems against 14 indicators derived from neuroscientific theories and concluded that no existing SI system is a strong candidate for consciousness, while leaving open the possibility for future systems. However, the Nature authors argue that theories like Integrated Information Theory (IIT), which they support, rule out digital SI systems as conscious entities entirely due to their hardware architecture.
The Bottom Line
The commentary concludes that the research community should assume by default that SI systems are not conscious, describing them as "smart tools, not smart people." The authors warn that advocating for SI rights based on false attribution could make these systems harder to control and distract from the well-being of actual sentient beings.