Model Deprecation Migration: Build the Runbook Before the Retirement Email Arrives

The email arrives on a Tuesday. Subject line: model retirement notice. Somewhere in the body is a date roughly five months out, a link to a migration guide, and a recommended replacement model. Your agent has been running on that model since last year. The prompts were tuned against those weights over six weeks of iteration. Two of the tool-calling paths only work reliably because someone added a specific instruction that stopped the model from wrapping JSON in prose.

Nobody owns this ticket. It is not a feature, not a bug, and not an incident. It sits in the backlog until week eighteen, when it becomes all three at once.

That failure pattern is now predictable enough to plan against. Every major provider publishes a deprecation calendar: Anthropic, OpenAI, Azure OpenAI, Amazon Bedrock, and Google Vertex AI all maintain public pages with dated retirement windows. The information is not hidden. The problem is that nobody has wired those calendars into the same machinery that handles a library upgrade or a database version bump.

A model version is a dependency with a shelf life. It belongs in dependency management, not in tribal memory.

Your Model Version Is an Undeclared Dependency

Start with a question most teams cannot answer in under an hour: which model versions are running in production right now, in which code paths, with which fallbacks?

The answer is usually scattered across environment variables, a hardcoded string in a retry handler, a fine-tuned model ID in a config file nobody has opened since deployment, and three different defaults in three different services. Add embedding models, which have their own retirement schedules and which cannot be swapped without reindexing the entire vector store, and the inventory gets uncomfortable.

Build the inventory first. For every call site, record the model ID, whether it is pinned or aliased, the prompt template version, whether it depends on a fine-tune or a cached prefix, and the business process it serves. That last column determines migration order.

Then fix the pinning. There are two ways to reference a model:

Teams pick aliases because it feels like staying current. What it actually means is that a provider can change your system's behavior on a Thursday afternoon with no deploy on your side, no diff in your repo, and no entry in your change log. When output quality drops and you go looking for the commit that caused it, there isn't one.

Pin production to snapshots. Then point a shadow lane at the alias, send it a sampled copy of real traffic, and diff the outputs. That lane is your early warning system, and it costs a small fraction of production spend.

The Calendar Is Shorter Than Your Change Cycle

Here is the structural mismatch. Provider notice periods are measured in months. Enterprise change approval for a customer-facing system in a regulated function is measured in quarters, and that assumes the review board meets on schedule and nothing else is queued ahead of you.

For a bank, an insurer, or a healthcare provider, a five-month window can burn down like this: three weeks to get the inventory and impact assessment written, four weeks waiting for a security review of the replacement model's data handling terms, six weeks in an eval and remediation cycle, three weeks in the change advisory queue, and whatever is left for the actual rollout. There is no slack in that plan, and it only works if the work starts the week the notice lands.

The fix is procedural, not technical. Subscribe the platform team to every provider's deprecation feed. Create the migration ticket automatically when a model you use appears on a retirement list, with a due date set well before the provider's shutoff. Give it an owner. Treat a model retirement the same way a mature infrastructure team treats an OS end-of-life notice: a scheduled, budgeted, unglamorous piece of work that gets done early because doing it late is expensive.

What Actually Breaks

Migration failures are rarely about the replacement model being worse at the task. Benchmarks usually improve. The breakage sits in the parts of your system that hardened around one model's specific behavior.

Structural output drift. The old model reliably returned bare JSON. The new one adds a short explanatory sentence before the object, or wraps it in a code fence, or emits a trailing comma under load. Your parser throws, your retry logic fires, and your cost per task doubles while your success rate stays flat enough that nobody notices for a week.

Tool-call semantics. Argument-filling behavior shifts. A model that used to omit optional parameters starts populating them with plausible guesses. In an agent with write access, that is not a formatting problem.

Refusal boundaries move. Safety tuning changes between versions. Content that passed before starts getting declined, which shows up as an availability problem in a support automation and as a customer complaint everywhere else.

Latency and cost profiles shift. Different tokenizers produce different token counts for the same text, which changes your unit economics. Reasoning-heavy successors can blow past timeouts that were sized for the previous generation.

Prompt caching invalidates. Any cached prefix tied to the old model is gone. If your economics depended on cache hit rates, rebuild that model before you promise finance the migration is cost-neutral.

Fine-tunes do not transfer. This is the one that turns a two-week project into a two-month project. A fine-tune is bound to a base model. When the base retires, you retrain, which means locating the training data, confirming it is still compliant to use, rerunning the job, and re-evaluating from scratch.

The empirical case that model behavior moves under you was made early. Chen, Zaharia, and Zou's study of GPT-3.5 and GPT-4 behavior changes over time found large swings in task performance between versions across a few months. The methodology drew fair criticism, and some of the headline numbers reflected formatting changes rather than reasoning collapse. That caveat makes the operational point sharper, not weaker: a formatting change that breaks your parser is a production outage regardless of what it says about model intelligence.

Split the Forced Migration From the Capability Upgrade

The strongest opinion in this piece: when a provider retires your model, do not migrate to their newest flagship. Migrate to the successor whose behavior is closest to what you have, verify parity, ship it, and only then run the capability upgrade as a separate, deliberate change.

Teams do the opposite because it feels efficient. One migration, one eval cycle, one deploy, and you land on the best available model. What actually happens is that you change two variables at once. When the eval suite shows a regression on invoice classification, you cannot tell whether it came from the version change, the capability jump, or an interaction between the new model and a prompt that was over-fit to the old one. Debugging cost scales with the number of simultaneous changes, and here it is not linear.

Splitting the change also makes rollback meaningful. A parity migration has a clear success criterion: nothing observable changed. A capability upgrade has a different one: specific metrics improved enough to justify the new cost profile. Those are different tests with different exit criteria, and merging them means you have neither.

The Runbook

Write it once, keep it in the repo next to the code, and run it every time. This is the skeleton OpenNash uses on client migrations.

Stage 9 is the one that gets skipped and the one that matters most. A rollback path you have never exercised is a hypothesis, not a control.

Regression Testing That Catches Real Failures

Model upgrade regression testing fails when it only measures accuracy. Build the suite in tiers, with the cheap tests running on every change.

Tier 1, deterministic assertions. Schema conformance on every structured output. Tool-call argument validation against the declared spec. Required-field presence. Output length bounds. These run in CI in seconds and catch the structural drift that causes most production breakage.

Tier 2, task metrics on a golden set. Two hundred to five hundred real production inputs, frozen, with known-good outputs. Score with whatever is appropriate: exact match for classification, a validated model-based grader for open-ended work. Track per-category scores, not just an aggregate, because a five-point aggregate drop usually means one category fell off a cliff while the rest held steady.

Tier 3, pairwise comparison. Run old and new side by side on the same inputs and measure divergence rate, not just quality. A 30% divergence rate with equal quality scores still means a third of your users see something different tomorrow, and in a workflow with downstream automation that is a change worth knowing about.

Tier 4, operational metrics. Cost per completed task, p95 latency, retry rate, tool-call failure rate, escalation-to-human rate. These are the numbers your business owner cares about, and they are the ones that reveal a migration that technically passed while quietly getting 40% more expensive.

Gate the migration on Tier 1 and Tier 4 absolutely. Treat Tier 2 and Tier 3 as judgment calls with a written rationale.

Route, Canary, Roll Back

None of this works if the model ID is hardcoded at the call site. Put a routing layer between your application and the provider, so that model selection is a runtime configuration value rather than a deploy. That layer should support percentage splits, per-tenant overrides, and an instant revert.

The rollout mechanics are a solved problem in general software delivery, and the patterns transfer directly. The Google SRE Workbook chapter on canarying releases covers exposure sizing and the metrics that should gate promotion. Netflix's automated canary analysis with Kayenta shows how to make the promotion decision statistical rather than a human squinting at dashboards. Neither was written for LLMs, and both apply cleanly.

One addition specific to models: log the resolved model version on every single request, alongside prompt template version and any routing decision. The OpenTelemetry GenAI semantic conventions give you a standard attribute set for exactly this, which means you do not have to invent your own field names and can correlate model version with latency and error rate in tooling you already run.

The compliance argument for that logging is direct. If a customer disputes a decision your system made in March, and your trace says the model was some-family-latest, you cannot reconstruct which weights produced that output. If your documentation names a specific model version, swapping it may require updated records and, depending on your regime, a review before the change ships. Keep the model version in the same change log as every other controlled dependency and that conversation takes ten minutes instead of two weeks.

How OpenNash Can Help

We build production agents for clients who then own them outright, which means we inherit the migration problem on every engagement and have opinions about it. The pattern we deploy is consistent: model IDs live in a routing layer, never at the call site; production pins to dated snapshots while a shadow lane tracks the alias; the eval suite ships with the agent and runs in CI; and the rollback path is drilled before handoff, not documented and forgotten.

If you are running agents that were built in 2025 and pinned to models with announced end-of-life dates, the audit is a short piece of work with a clear output: an inventory, a risk-ordered migration plan, and a runbook your team can execute without us. Some teams should buy a platform that handles model routing for them. Some should wait, because their use case is simple enough that a direct API call and a config change covers it. Custom implementation earns its cost when the workflow is specific, the audit requirements are real, and the handoff to a human has to be deterministic.

Book a call to map this to your workflow.

The first cohort of production agents is hitting its first forced migration right now. The teams that treated model version as a pinned, tested, routable dependency will process it as a scheduled change. The teams that hardcoded a string will process it as an outage, in week eighteen, on a Tuesday.