A silent model update can break automation without any code change
A support-ticket step had been running under the same model name for about three months. Its internal evaluation accuracy stayed near 94%. One morning, accuracy fell to about 91%, even though there had been no deployment, code change, prompt change, or data change.
The model name in the was also unchanged. The cause was a provider update behind the same model id, so the version used in production was no longer the version the team had tuned against. The newer model may have been fine on broad , but it behaved differently on short, angry tickets that mixed languages.
That input group was about 8% of total volume, and the model started putting too many of those tickets into one category. Downstream routing handled that category poorly, and finding the cause took half a day.
Key points
- The same model name can point to changed model behavior if the provider updates it silently.
- Accuracy dropped from about 94% to about 91% with no local system change.
- The failure showed up mainly in short, angry, mixed-language tickets.
- That input type was about 8% of volume but affected downstream routing.
- and input-group monitoring can catch this earlier.