AI, AGI and Sentience
Two recent articles I read frame AI, AGI and sentience very differently. Geoffrey Hinton, an AI founding figure, sounds the alarm about systems that may become smarter than us whereas Ted Chiang, the acclaimed fiction writer, challenges the idea that today’s AI is conscious or morally aware.
Hinton’s article treats AI as the possible birth of a new kind of being, whereas Chiang argues that a long list of proof points are needed to justify AI sentience.
Hinton’s view is that AI systems may soon become much smarter than humans, perhaps surpassing the best mathematicians and eventually closing the gap with luminaries such as Einstein.
Hinton’s worries that a race for more intelligence, driven by profit and competition, could end up producing systems that are powerful without being aligned to human welfare. In his framing, the key question is what kind of beings are we creating?
Chiang’s article rejects this POV, as for him, today’s LLM’s are not beings at all, they are software that predict text, imitate conversational roles, and exploit our habit of reading intention into fluent language. He argues that treating them as conscious or morally aware helps companies shift attention away from their own responsibility i.e. the danger is not that Claude has a conscience, it is is that people may let a machine substitute for one.
Both articles are more aligned when it comes to abdication of risk. Hinton fears humans may lose control because companies are racing to build systems more capable than their makers. Chiang fears humans may dodge responsibility by pretending those systems have some form of understanding, values, or morals. Hinton looks forward and sees a new power center emerging whereas Chiang looks at the present and all he sees is a misleading interface wrapped around software.
My view is that they are both right. Chiang is stronger on where current systems are, while Hinton is stronger on why ‘where we will end up’ matters.
I don’t believe that models today need to be conscious to be dangerous, influential, or more importantly, economically disruptive. Calling them beings too early risks creating an illusion, and calling them tools for too long risks underestimating what they are capable of in practice, especially once they start to act in earnest through agents.
The middle ground may well be to stop asking whether AI is ‘alive’ and be more robust in assigning accountability for what it is allowed to do. Hinton is right that intelligence without care is a civilizational risk, but Chiang is also right that simulated care is not care at all.
We shouldn’t parent these systems as if they were children, nor should we dismiss them as just autocomplete text generators, we should govern them as powerful delegated systems whose ability to harms remains the responsibility of people.
'Godfather of AI' says we're not just creating new beings: https://fortune.com/2026/06/01/godfather-of-ai-geoffrey-hinton-beings-smarter-than-us/
No, Artificial Intelligence Is Not Conscious: https://www.theatlantic.com/philosophy/2026/06/no-artificial-intelligence-is-not-conscious/687378/