An AI process that produced fine quotes or customer emails last month suddenly gets sloppier, while your prompt did not change. That is often not in your head. After launch, vendors adjust the settings around a model, and they are not required to say so.
What happened
In a video from 18 November, Nate Herk looked into why models seem to get worse after launch. He read the vendors' post-mortems, daily tracking sites and the benchmarks from launches.
What he found
- Anthropic and OpenAI both admitted in writing that their models performed worse for a while after launch.
- He found no proof that the model itself gets swapped. The weights, the actual model, appear to stay the same.
- What does change are four dials around it: how much the model may think, the vendor's hidden instructions, which model your request is routed to, and bugs in the app itself.
- According to the video, nothing obliges a vendor to tell you when such a dial is turned. Compute is scarce, so the pressure to cut back is real.
The four dials in plain language
- Thinking time. According to Herk, the default in the app is lower than the setting used to run benchmarks.
- Instructions. The vendor wraps its own rules around the model. A rule that makes answers shorter can hurt quality.
- Routing. A request can end up at a different model without notice, for example through a safety filter.
- App bugs. A bug in the app can make the answer worse while the model itself is fine.
Most of the measurements in the video come from the creator's own tests. We could not verify them and leave them out here.
Our view: a launch benchmark says nothing about your work next month. What you don't measure yourself, you can't correct.
What this means for your organisation
If you run a fixed AI process every week, such as quotes, customer emails or meeting summaries, then quality is not a given. It can shift quietly because of a vendor's choice. That is a process risk, just like a supplier who replaces a component without telling you.
What to do now
- Record per process which model and which setting you use. Write it down with the work instruction.
- Keep a small fixed test set, for example five questions with the answer you expect, and run it monthly and after every noticeable change.
- Compare against your own work, not against the demo from the launch video.
- Also choose a vendor on openness about changes, not just on this month's best demo.
How it works under the hood
A model is essentially a file of weights. The behaviour you see comes from the whole chain around it: the effort setting, the vendor's system prompt and harness instructions, routing to another model, and the application. If you build on an API yourself, log the model ID, the effort setting and a version of your prompt and system instructions with every call, and check that the model in the response is the same as the model you requested.
This news touches chapters 11, 22 and 24 of the course AI(l) or nothing by Newway. If you follow the course, this subject returns there at a calmer pace.
What you can do with this
Want to know what this means for your business? Plan an introduction: half an hour, no sales pitch, but an honest answer to the question where to begin. Rather read on first? The course AI(l) or nothing is free and starts with the basics.
Written in response to a publication by Nate Herk (YouTube).
S.O.S