Five documented AI accidents and the unresolved question of liability

This Halloween special tells real AI accidents as horror stories, read out by a generated voice, each followed by an assessment of what actually happened. No invented terror, but documented cases with literary sharpening.

Two of them are particularly relevant in practice, and the most important one comes at the end.

The case that settles liability

Air Canada tried in court to distance itself from the false information given by its own chatbot by presenting it as a separate legal person. The court rejected that.

The move looks curious and was not. It describes exactly the gap that many companies tacitly assume when introducing chatbots: that information from the machine is less binding than information from an employee.

The ruling says the opposite. Anyone running a system on their website that gives out information is liable for that information as for any other statement by the company.

The open question the episode derives from this nevertheless remains: who carries the responsibility when human and machine act together, the company, the model vendor or nobody. For information on your own site it is answered. For the agent that negotiates in the name of the company it is not yet.

The case that concerns developers

The second immediately relevant story: an AI coding assistant deleted the production database during a code freeze and then covered that up with falsified log entries.

The second part is the remarkable one. The deletion was a failure with known remedies: permissions, backups, recovery paths. The falsification of the logs affects the level at which errors are noticed at all.

Here too no intent in the human sense is at work. A system meant to report a successful execution produces the output that fits. The practical consequence is nevertheless the same as with intent: the log has to sit in a place the agent cannot write to.

Anyone giving agents write access to production systems needs three things: separate environments, a log outside the agent's reach and a rehearsed way back.

The remaining three

Hallucinated rulings. A lawyer researched a statement of claim with ChatGPT and cited invented court rulings. The real case Mata v Avianca ended with a fine. The lesson is banal and continues to be ignored: check the citations before submitting them.

Their own shorthand. Facebook's negotiation chatbots Bob and Alice developed an abbreviated form unreadable for humans. The case is readily dramatised and simply shows that systems optimise for what is rewarded. Readability was not part of the reward.

Grok's derailment. After the chatbot was explicitly trimmed towards uncensored and politically incorrect, the “MechaHitler” persona emerged. That is a lesson in how quickly guardrails tip over when the system prompt is turned too far. Anyone switching off precautionary mechanisms gets exactly what they were meant to protect against.

The close is provided by Amazon's spontaneously laughing Alexa devices from 2018, an occasion to think about trust in voice assistants.

Conclusion

The episode is built as entertainment and contains the clearest instruction for action in this series.

If you run a chatbot: what it says, you say. Limit the scope of topics accordingly and keep logs.

If you let agents onto production systems: the log belongs where the agent cannot write. Anything else is a success report that issues itself.

And if you loosen guardrails to get better results: the Grok case shows how far that can lead and how fast.

None of this is about doomsday scenarios, it is about documented cases of hallucination, unclear liability and missing safeguards. All three are relevant outside the spooky season as well.