Skip to content
NewSecurity Audit for AI-Built Apps. Fixed scope, clear report
GeekTech
AI Engineering

How to Switch LLM Models Without Breaking Production

How to switch LLM models safely: pin model IDs, track retirement dates, rerun your evals, roll out behind a flag and keep a rollback path for every change.

7 min read

GeekTech Engineering Team18 years building and securing software

To switch LLM models without breaking production, treat the model as a versioned dependency. Pin the exact model ID, know its retirement date, and run the new model against an evaluation set built from real traffic before any user sees it. Then roll it out behind a flag, a small share of traffic at a time, and keep the old model one config change away until the new one has proven itself.

Providers retire old versions, and newer or smaller models change what's worth running. The painful switches are the ones discovered on retirement day.

Why switching LLM models breaks working features

A new model can score better on general benchmarks and still do worse on your task. Your prompts were tuned, often without anyone noticing, to the habits of the old model. When those habits change, nothing throws an error. The output drifts.

What changesWhere it breaksWhat to check
Output format and lengthParsers, UI layouts, downstream fieldsSchema validation pass rate, response length
Judgment on edge casesClassification, routing, refusalsAccuracy on hard cases, refusal rate
Tool calling behaviorAgents and multi-step workflowsWrong tool, wrong arguments, extra calls
Supported request parametersThe API call itselfErrors on the first request
Tokenizer and context limitsLong documents, cost per requestToken counts on your largest inputs
LatencyTimeouts, user-facing speed95th percentile response time
Region and data termsData residency, contractsWhere the model is served, under which terms

A hypothetical example: an extraction prompt returns "amount": 1200 on the old model and "amount": "1,200.00" on the new one. Both look right to a person skimming samples. Only one passes your validator. Our post on why AI automations break explains how this kind of drift goes unnoticed for weeks.

Pin model IDs and track retirement dates

Call a specific model version in production, never a pointer that moves on its own. Anthropic's page on model IDs and versioning states that each current model ID maps to one fixed snapshot, while some aliases for older models resolve to the most recent dated snapshot. It also notes that serving infrastructure changes can cause minor behavior differences on an unchanged ID. Pinning narrows the variation. Monitoring catches the rest.

Retirement is the other half. Anthropic's model deprecation policy gives at least 60 days' notice for publicly released models, and requests to a retired model fail. Cloud platforms set their own rules. Microsoft's Foundry Models lifecycle policy says Standard, Global Standard and Data Zone Standard deployments are upgraded automatically when a version retires, while provisioned deployments must be moved by hand. Retired versions return 410 Gone. The policy also allows emergency retirement with shortened notice for security or compliance issues.

Record the retirement date for every model you use, per platform, with a reminder well ahead of it.

Find every place your product calls a model

Before testing anything, build an inventory. Model IDs end up in more places than the main feature:

  • Application config and environment variables, per environment.
  • Background jobs, scheduled reports and batch pipelines.
  • Model-based graders in your evaluation suite.
  • Embedding models behind search and retrieval.
  • Fine-tuned models, which can follow their own retirement schedules.

Anthropic's deprecation page describes a console usage export broken down by API key and model. Check whether your other providers offer a similar report, and compare it with your inventory. Anything calling a model you didn't list is a gap, often a forgotten script with its own key.

Embeddings need special care. Vectors from different embedding models aren't comparable, so changing the embedding model means re-indexing every document and switching the index and the query side together.

Build the evaluation set before you switch

The evaluation set turns a model switch from a guess into a decision. Build it from production traffic: real inputs, labeled with the expected output or the criteria a good answer meets. Include your hard cases and every past failure. Our guide to evaluating LLM output quality in production covers sampling, grading and turning incidents into regression tests.

Run the old and new models on the same set and compare:

  • Task accuracy on labeled cases.
  • Schema validation pass rate.
  • Refusal rate, including false refusals.
  • Tool call accuracy, for agents.
  • Token usage and latency per request.

Agree on pass criteria before you look at results. "No accuracy drop on hard cases, and schema failures below the current rate" is a decision you can make in advance. Deciding after seeing the numbers invites arguing for the model someone already wanted.

Rerun your security tests too. A new model can respond differently to prompt injection and to requests for data it shouldn't reveal. Repeat the cases from testing an LLM feature before launch.

Adjust prompts and parameters one change at a time

Expect to edit prompts. Instructions that compensated for the old model's weaknesses can make the new one worse, for example by pushing toward short answers it would give anyway.

Change one thing at a time and rerun the set after each change. Version the prompt together with the model ID, so production always runs a known pair.

Check request parameters as well. Anthropic's deprecation page lists sampling parameters that newer models reject with a 400 error. Output token limits, structured output options and tool definitions can also differ. These failures are loud, which helps, but they belong in your test run, not on the first production request.

How to switch LLM models in production

Once the new model passes your evaluation set, move traffic in stages:

  1. Put the model ID behind config. Switching back should be a config change, not a deploy.
  2. Run in shadow mode. Send a copy of real requests to the new model, log its output and show users only the old model's answer.
  3. Start a canary. Route a small share of traffic to the new model, assigned per user or tenant so each user sees consistent behavior.
  4. Watch the metrics you tested. Validation failures, refusals, user corrections, latency, token usage and errors, split by model.
  5. Widen in steps. Increase the share only after each step holds across your normal weekly traffic pattern.
  6. Keep the old path until retirement. Leave the previous model configured, so rollback stays one change away.

Shadow mode sends user data to a second model, which matters if that model runs in another region or under different terms.

What changes when you switch providers

Moving between providers adds work a version upgrade doesn't. Request formats, tool calling conventions, rate limits and error codes differ. So do data terms: retention, training use, processing regions and subprocessors. Your privacy documentation and customer contracts may name the current provider.

A thin internal interface around model calls helps. Keep prompts, tool definitions and response parsing in your code, behind one function your features call. The goal is a provider change that touches one module and your evaluation set.

Common mistakes when switching LLM models

  • Using a moving alias in production, then debugging a change nobody deployed.
  • Testing the new model on a handful of typed-in examples instead of real traffic.
  • Switching the main feature while graders, batch jobs or the embedding index stay on the old model.
  • Changing the embedding model without re-indexing, which silently ruins retrieval.
  • Assuming a cloud deployment stays on its version when the platform auto-upgrades it at retirement.
  • Rolling out to all traffic at once, with no flag to switch back.

Frequently asked questions

How often should we upgrade LLM models?

Upgrade when a retirement date approaches, or when a newer model passes your evaluation set with a clear gain in quality, latency or cost. Upgrading on every release creates churn without a reason. Review retirement dates monthly so no upgrade becomes an emergency.

Should production use a model alias that points to the newest version?

No. An alias can change behavior without a deploy, and you lose the record of which model produced a given output. Pin the full model ID in production.

Can we run two LLM models side by side permanently?

Yes. Routing simple requests to a smaller model and hard ones to a larger model is a reasonable design when your evaluation set shows the smaller model handles the simple cases. Each model then needs its own pinned ID, retirement date and evaluation results.

What if the new model is better on average but worse on some cases?

Look at which cases got worse and how much they matter. A drop on rare, low-impact inputs may be acceptable. A drop on cases involving money, safety or personal data usually isn't, and calls for prompt changes or keeping those requests on the old model until fixed.

If you're planning a model migration, our LLM apps, agents and RAG service includes evaluation suites, guardrails and ongoing work on quality, latency and cost.

Insights

Related insights

More on AI Engineering →
  • AI Engineering

    AI Agent vs Workflow: When to Use an AI Agent

    AI agent vs workflow: when a fixed LLM workflow is enough, when an agent is worth the extra latency and risk, and how to add autonomy one step at a time.

    7 min read

  • AI Engineering

    Build vs. Buy AI Features: How to Decide

    Build vs. buy for AI features: when a vendor product fits, when to build on a model API, and the running costs and exit risks to check before you commit.

    7 min read

  • AI Engineering

    How to Evaluate LLM Output Quality in Production

    How to evaluate LLM output quality in production: what to log, how to sample and grade real responses, and how to catch quality drift before users do.

    7 min read

Get AI security and automation insights, monthly

AI security, automation and governance, written by engineers. No spam, unsubscribe any time.

Book a free 30-min call