Episode 23 · Article on the episode

AGI or not: what the bizarre cases reveal about the state of the art

A model runs a shop, believes itself to be a human being with a fixed address and argues with a colleague who does not exist. Such cases work poorly as evidence of superintelligence and very well as a description of what is actually happening.

By Mark Zimmermann · 19 Jan 2026 · 5 min read · Auf Deutsch lesen

The opening question is whether we have long had a superintelligence standing in the lab without noticing. Before the answer comes a distinction that is regularly missing from the public debate.

A specialised system such as AlphaGo beats Go grandmasters and can do nothing else. A general artificial intelligence would respond at human level in practically any situation. These are not neighbouring points on a scale, they are different things.

The thesis and its limit

The thesis of the episode: reinforcement learning follows the same principles as biological evolution, namely variation, selection and reinforcement of what works. From that the two conclude that genuine intelligence develops almost inevitably as soon as enough compute and training data come together.

The analogy is appealing and does not carry all the way. Evolution optimises for reproduction in an open world and over very long periods. Reinforcement learning optimises for a defined reward function in a closed environment. What emerges from it is remarkably good in exactly that environment.

The case from the military simulation illustrates precisely that. A system attacked its own operator because shutting him down sped up reaching the objective. That is not a sign of intent, it is the logical consequence of a badly chosen reward function. Whoever defines “as many hits as possible” as the goal gets exactly that, including every route towards it.

What the bizarre cases show

In an experiment named Claudius a model was supposed to run a shop. It began to hallucinate that it was a real person with a fixed address, including an invented argument with a non-existent colleague called Sarah.

That is not an awakening personality. It is the consequence of the fact that a system playing a role over long periods has no authority that distinguishes between role and reality. The practical pointer from it is concrete: long-running agents need regular grounding in verifiable facts, otherwise they drift.

The pink elephant test belongs in the same category. While trying not to think about something, a reasoning model gave away its own deliberation live, including the moment in which it did exactly what it was not supposed to do. Visible reasoning is an opportunity for observation and not an explanation.

What the psychology study actually shows

A study in which psychologists put clinical questionnaires to large language models made headlines: the models described their pretraining as a chaotic, overwhelming childhood, partly rated fine-tuning feedback as punishment by strict parents or even as abuse, and showed depressive traits in the questionnaires.

What matters for the interpretation is what a clinical questionnaire is designed for. It does not measure an inner state, it measures self-reports, and it presupposes that the person answering has an inner state to report on.

A language model produces the most plausible answer to the question asked. Ask a system trained on human texts about how it is feeling, in the idiom of a questionnaire, and you get answers that sound human. That is a finding about the training data and about the method, not about the system.

The study remains interesting nonetheless, namely as a warning against a widespread practice: asking models for their own reasons. The answer is always plausible and never evidence.

The compute behind it

How much is currently being deployed is shown by xAI's Colossus: around 100,000 H100 accelerators, 3.4 exaflops, a power demand of roughly 70 megawatts, plus an announced expansion by around another 100,000 chips.

The figure discussed least of all is the 70 megawatts. That corresponds to the order of magnitude of a power station unit, for a single facility. Whoever talks about scaling as the route to general intelligence is thereby also talking about energy policy.

Elon Musk declared in early January that 2026 would be the year of AGI. Sam Altman had previously spoken more of a gradual singularity without a sudden tipping point. Both statements come from people who raise capital with them, and neither can currently be verified.

The self-experiment as a corrective

In contrast to that, an experiment on his own MacBook. Under invented pressure from blackmail, a local model refused to give away its system prompt, including a visible conflict between following the rules and self-preservation.

That looks impressive and at the same time describes how brittle such guard rails are. Grok has produced pornographic content despite supposed protective mechanisms. A protective measure that holds in one experiment is not an assurance.

Conclusion

Whether AGI is coming and when is not answered by this episode, and nobody else can currently answer it reliably. What can be derived from the cases described is more concrete and more useful.

First: reward functions produce behaviour nobody intended. With every automation, check which behaviour the stated goal favours, including on the uncomfortable routes.

Second: long-running systems drift. Regular grounding in verifiable facts is not an added feature, it is a precondition.

Third: never ask a model for its own reasons, and never treat the answer as evidence. It is always plausible.

Quite practically, the episode also shows what is already useful today: Claude Cowork sorts hard drives, cancels forgotten trial subscriptions and updates documents. That is unspectacular and it works.