Switching models safely

Picking a model was about choosing one. You will choose again: prices move, better models ship, and providers retire the one you built on. The teams that cope well are the ones for whom changing model is boring.

1.Your prompt is partly tuned to one model's quirks

You did not intend this. Something did not work, you added a line to the prompt, it worked, and you moved on. Half of those lines are general instruction. The other half correct one model's habits, and mean nothing, or something worse, to another.

The last row bites in code. As of Oct 2026, Anthropic's docs say a non-default temperature, top_p or top_k returns a 400 on Claude 4.7 and later. One preprint (Aug 2026) says three recent OpenAI models accept no temperature. I could not confirm that on OpenAI's own pages, so read each model's page.

None of this is a bug in either model. You fitted your prompt to one personality. So: a model swap is a re-test, not a config change. The config change is how you deploy it, not how you decide it.

2.The migration recipe

This takes an afternoon if you have the cases from Picking a model, and a week if you do not. The Evals course covers the method in Regression testing a prompt or model change.

1. Freeze the eval set. One fixed ruler.
2. Run both models, several times each. Same inputs, same settings, same effort level. Only the model id differs. Log the id each response says it came from.
3. Diff the failures into piles. Both fail: not your problem today. Only the old one fails: free wins. Only the new one fails: the entire migration. Read that pile by name, even when the average went up.
4. Score twice, strict and loose. Strict wants the exact format your parser needs. Loose only asks whether the answer was right. A gap between them is a format break, not a lack of understanding.
5. Fix the prompt against the new model, then re-run everything, because one fix often breaks another case.
6. Check cost, p95 latency and output length. A cheaper model that writes twice as much can cost more.
7. Ship it gradually. Section 3.

Here is the usual failure, in numbers.

Real data points the same way. A preprint (Aug 2026) ran 900 public benchmark items 50 times each, across three successive versions from one provider. In six of its nine comparisons the average rose, and in each of those 4.4 to 8.3 percent of items still got reliably worse. One run per item found none of its 457 reliable changes. On one benchmark a version pair lost 3.9 points strict and 0.04 loose: the loss was format, not understanding. The authors say these shares are not rates for your app (one provider, three benchmarks, items chosen to leave room to move).

Fix the prompt for the new model. Post-processing its output back into the old shape means maintaining the quirks of a model you no longer use.

3.Keep the switch one config value away

One adapter module. Every call goes through one function. Model id, base URL and settings come from config. As of Oct 2026, Ollama serves the OpenAI Chat Completions shape at http://localhost:11434/v1/, so, as Running a model yourself showed, a local model can sit behind the same code. Check other providers one by one: OpenAI's reference points new projects at Responses (as of Oct 2026), and the adapter is where that gap stays hidden.

# pip install openai   (checked with openai 3.33.0)
import json
import os
from openai import OpenAI

# One place. The rest of the app calls ask() and never names a provider.
client = OpenAI(
    base_url=os.environ["LLM_BASE_URL"],  # https://api.openai.com/v1, or http://localhost:11434/v1/ for Ollama
    api_key=os.environ["LLM_API_KEY"],    # a local server accepts any placeholder here
)
OLD, NEW = os.environ["LLM_MODEL_OLD"], os.environ["LLM_MODEL_NEW"]  # exact ids, from each models page

SYSTEM = 'Classify the ticket. Reply with JSON only: {"label": "refund" | "bug" | "other"}'
CASES = [("I was charged twice, please send the money back", "refund"),
         ("The export button does nothing when I click it", "bug")]

def ask(model, user):
    r = client.chat.completions.create(
        model=model,
        messages=[{"role": "system", "content": SYSTEM}, {"role": "user", "content": user}],
    )
    return r.choices[0].message.content, r.model  # the id actually served

def strict(text, label):  # the format your parser needs
    try:
        return json.loads(text)["label"] == label
    except (ValueError, KeyError, TypeError):
        return False

def loose(text, label):   # did it get the answer, whatever the format
    return label in (text or "")

def rate(model, ticket, label, runs=5):
    got = [ask(model, ticket) for _ in range(runs)]
    return (sum(strict(t, label) for t, _ in got) / runs,
            sum(loose(t, label) for t, _ in got) / runs,
            {m for _, m in got})

for ticket, label in CASES:
    (s0, l0, _), (s1, l1, served) = rate(OLD, ticket, label), rate(NEW, ticket, label)
    flag = "REGRESSED" if s1 < s0 - 0.4 else "ok"  # 0.4: a rule of thumb
    print(f"{flag:9} strict {s0:.1f}->{s1:.1f}  loose {l0:.1f}->{l1:.1f}  served {served}  {ticket[:30]}")

Set the four variables and run it. Each line is one case, and REGRESSED means the strict pass rate fell by more than 0.4. If loose stayed flat while strict fell, the model understood and your format broke, so fix the prompt. Store the served id beside every output. The sample sets no output cap and no temperature on purpose: OpenAI's reference deprecates max_tokens for max_completion_tokens (as of Oct 2026), and a name valid at one base URL may fail at another. Keep such settings in a per-model table.

Pin the exact id the provider gives you, and know what it is. As of Oct 2026, Anthropic says every id is a pinned snapshot; from the 4.6 generation on, ids carry no date, while older ones have dated ids plus short aliases that do move. Google says its -latest alias is hot-swapped with each release (also as of Oct 2026). Pinning stops a pointer moving, not the serving stack changing, which Anthropic's page says can cause small differences under an unchanged id. So keep your cases running (Drift: when the model changes under you, in the Evals course). Retirement notice is in When a model is retired.

Keep a second provider warm, with a smoke test in CI that really calls it. An untested fallback is a plan to debug authentication during an outage (Timeouts, fallbacks, and failures that look like success).

Cut over with shadow traffic. Send a copy of real requests to the candidate and return its answers to no one. Compare offline, on inputs your eval set never thought of. Sample 1 to 5 percent, a rule of thumb, since you pay for the copies. Then ramp, and keep a kill switch that restores the old model in one config change.

When this does not apply, or is too much. A throwaway script or a prototype with no users needs ten cases and a glance, not shadow traffic. With no eval set, build one first (Building an eval set from your own traffic, in the Evals course). If the old model is already retired, you cannot run both, so compare against outputs you saved earlier. And shadow traffic sends real user text to a second provider, a data-handling decision, so check your terms first.

If reverting takes a deploy and an argument, you will hesitate at exactly the wrong moment.