Episode 48 · Article on the episode
When the model disappears: why agent systems need interchangeability
An Anthropic model was blocked for non-US citizens. Law firms that had aligned their text analysis with it stood without a basis from one day to the next. What follows from that for your own architecture.
Fable is no longer available to users outside the USA. Reliable information for assessing the political motives is lacking; plausible is a head start for selected companies and the government's own administration in closing security holes, before comparably capable models from less controllable hands become available.
For practice the question of motive is secondary. What matters is the event itself: a model in productive use can fall away at short notice, and by a decision rather than a malfunction.
Concentration risk: the model
The law firms affected did nothing wrong that would have been recognisable at the time of the decision. They selected a model that delivered the best results for their task and aligned their processes with it. That is precisely what most rollout projects recommend.
The mistake sits one level deeper, in the assumption that a model is an infrastructure component with the availability characteristics of a database. It is more of an imported product whose availability depends on trade policy.
In practice that means: treat the choice of model like a supplier relationship, not like a technology decision. That includes a second, tested provider, and it includes writing your own prompts and skills so that they do not build on one vendor's peculiarities.
Important here: the gap between the vendors is shrinking anyway. Chinese models replicate the capabilities of large US models via distillation, that is via automated bulk queries from which the behaviour of the original can be extracted. Anyone preparing interchangeability today will be able to use it before long.
The loop that reports success
The second part of the episode concerns a kind of error that occurs reliably when building loops and is hard to notice.
The sequence: a goal loop is supposed to work through a larger quantity of material until a list of questions is answered. Mid-work it runs into a limit, and not the model limit but the interface limit. The run breaks off.
Afterwards the instruction to carry on is enough. The system resumes work, runs into errors again, at some point reduces its query frequency by itself and reports at the end that everything is done. In passing follows the note that there were eight crashes, and would you like the resulting damage repaired.
The result then looks like the answer to a normal prompt. The work actually commissioned, the repeated checking against the success criteria, has been lost in the moments of interruption.
Careful: at this point the loop reports no error but success. Anyone looking only at the completion status takes on a result that never went through the promised check. Only a control outside the loop, one that independently recalculates the success criteria, is reliable.
A second limit is more banal and hits nonetheless: a weekly allowance in the Max plan can be used up in a single evening. After that the work stands still for several days.
Your own harness or a standard product
Connected to this is an architectural question that is currently undecided. On one side stand ready-made environments such as ChatGPT, Gemini or Cowork. On the other the self-built harness that has to cope with changing models and environments.
The standard product wins on rollout speed and maintenance. Your own harness wins in exactly the case this episode is about: when the model changes, you swap a component instead of a workflow.
The effort for that is regularly underestimated, and the results are regularly underestimated. The episode contains the anecdote of somebody who dismissed a self-built harness as “some JSON app”. The comparison misses where the work sits: not in the data format, but in context management, stopping criteria, checking mechanics and logging.
Where we actually stand
The most sober assessment in the episode concerns the level of maturity. The point of comparison is the web around 1997. Much already works, standards are missing, and the first course sellers are already there, making a business out of the uncertainty.
This assessment is no reason to wait. It is a reason to take decisions with a short commitment period. Anyone who built a web presence in 1997 was right. Anyone who committed to a proprietary browser plug-in back then did the work twice.
Conclusion
Three test questions for any ongoing AI project can be derived from this episode.
What happens if the model in use is no longer available tomorrow? If the answer means stopping the project, the second source is missing.
How do you know that a loop has actually done its work? If the answer is “it reported success”, the independent check is missing.
And how much of your investment sits in the model, how much in everything around it? The second part survives the first.