Switching models safely

Lesson 09 was about choosing a model. This one is about the fact that you will choose again. Prices fall, better models ship, and providers retire the one you built on. The teams that handle this well are not the ones with the best model — they are the ones for whom changing model is boring.

1Your prompt is partly tuned to one model's quirks

You did not intend this. It happened the same way it always happens: something did not work, you added a line to the prompt, it worked, you moved on. Half of those lines are general instruction. The other half are corrections for one specific model's habits, and they mean nothing to a different one — or mean something worse.

Here is what actually differs between models, all of which your prompt has absorbed:

None of these is a bug in either model. They are personalities, and you fitted your prompt to one of them.

The conclusion is short and worth taping to the wall: a model swap is a re-test, not a config change. The config change is how you deploy it. It is not how you decide it.

2The migration recipe

This takes an afternoon if you already have the twenty cases from lesson 09, and a week if you do not — which is itself the argument for having them.

1. Freeze the eval set. Do not edit cases while comparing; you need one fixed ruler.
2. Run both models on it. Same inputs, same temperature, same everything except the model id.
3. Diff the failures, and sort them into three piles. Cases both models fail — pre-existing, not your problem today. Cases only the old one fails — free wins. Cases only the new one fails — this is the entire migration.
4. Fix the prompt against the new model. Read those failures, find the instruction the new model reads differently, rewrite it. Then re-run everything, because a fix for one case often breaks another.
5. Never patch the output instead. If the new model adds a preamble, do not strip it with a regex. That regex is permanent, it hides the real difference, and you now maintain two models' habits at once.
6. Check cost and p95 latency too. A model with a lower price per token can still cost more if it is chattier, and can still miss your latency budget. Measure the bill, not the rate card.
7. Then ship it, gradually. Section 3.

Fix the prompt for the new model. Post-processing its output back into the old model's shape means you are now maintaining the quirks of a model you no longer use.

One more thing that survives every migration: keep a small set of cases that check the boring plumbing, not the intelligence. Does it return valid JSON. Does it call the tool with the right argument names. Does it stay under the length limit. Those are the failures that reach users fastest, and they are the cheapest to catch.

3Keep the switch one config value away

The engineering half of this is small, and doing it before you need it is the whole trick.

One adapter module. Every call goes through one function. Model id, base URL and default parameters come from config. No provider SDK type escapes into the rest of your app. When most providers speak the OpenAI request shape, the adapter is genuinely a few lines — and as lesson 08 showed, a locally served open-weight model answers on the same shape too.

# pip install openai
import os
from openai import OpenAI

# One place. The rest of the app calls ask() and never names a provider.
client = OpenAI(
    base_url=os.environ["LLM_BASE_URL"],  # https://api.openai.com/v1, or http://localhost:11434/v1
    api_key=os.environ["LLM_API_KEY"],    # a local server accepts any placeholder here
)
MODEL = os.environ["LLM_MODEL"]           # pin an exact dated id, never a moving alias

def ask(system: str, user: str) -> tuple[str, str]:
    r = client.chat.completions.create(
        model=MODEL,
        temperature=0,
        max_tokens=500,
        messages=[{"role": "system", "content": system},
                  {"role": "user", "content": user}],
    )
    # Return the id the provider actually served, and store it with the output.
    return r.choices[0].message.content, r.model

text, served_by = ask("Answer in one sentence.", "What is a KV cache?")
print(served_by, "->", text)

That second return value earns its keep the first time someone reports a bad answer from three weeks ago. Without it you cannot tell which model produced the row.

Pin versions. Use exact dated model ids, not aliases that point at the latest. An alias moves without a deploy on your side, which means your behaviour changed and your git history shows nothing. Pinning turns a surprise into a scheduled task.

Watch the deprecation pages. Every major provider publishes one, and they are the only reliable warning you get. Subscribe, put the retirement dates in the same calendar you use for certificate expiry, and treat them the same way — because they are the same kind of problem.

Keep a second provider warm. Credentials that work, an adapter path that works, and a smoke test in CI that actually calls it. A fallback you have never exercised is not a fallback; it is a plan to debug authentication during an outage.

Cut over with shadow traffic. Send a copy of real requests to the candidate model without returning its answers to anyone. Compare offline against what the live model produced, on genuinely representative inputs — the ones your eval set never thought of. Sample a small slice, 1 to 5 percent, since you pay for the copies. Then ramp: a small percentage of live traffic, watch your quality signal and error rate, increase, and keep a kill switch that puts the old model back in one config change.

That last sentence is the point of the whole lesson. If reverting takes a deploy and an argument, you will hesitate at exactly the wrong moment.