Picking a model

This lesson makes the decision: build a small test from your own inputs, then use a routing pattern so you do not have to pick only one model.

1.First check that the model is the problem

Public rankings do one job well: they cut a long list to about three (Reading a leaderboard covers how). Price, context window, rate limits and region (What you actually pay for, Rate limits and reliability) cut it further. Then stop reading rankings.

Before you compare, ask whether a different model would even help. The authors of an evals FAQ (Sep 2026) advise against treating a model swap as the main lever: read real failing outputs first. They sell a paid course, but the logic holds. If the failures come from an unclear prompt, poor retrieval or missing context, a new model gives you the same failures at a different price.

When this does not apply. If the product is new and has no failing outputs, a comparison is premature. Pick a mid-tier model, ship behind a flag, and collect real inputs. Invented cases are shaped by what you already expect.

2.Your own cases decide it

The prompt-engineering course builds a test like this in Versioning and testing prompts; the Evals course goes deeper in Building an eval set from your own traffic and How to trust a judge. The short version, pointed at one question, which model to buy:

1. Read first. Look at 20 to 50 real outputs from your current setup, note each failure, and group the notes into a few named failure types.
2. Pull cases from real traffic. Take them from logs or tickets: as a rule of thumb, about half ordinary and half awkward (long inputs, missing fields, known edge cases).
3. Write the rubric after the reading. The FAQ authors say a rubric written first gets in the way of seeing what actually fails.
4. Grade pass or fail. The FAQ authors prefer yes or no to a 1 to 5 score, noting that adjacent points on a scale get judged inconsistently. Use an equality check where one exists. Where a model must judge, check it against one domain expert's labels first.
5. Run every candidate on the same cases with the settings you would ship, next to cost and p95 latency.

Twenty is a floor, not a verdict. As of Sep 2026 the same FAQ says to start with 20 to 50 reviewed outputs, review until failures stop appearing (at least 100 traces), and label 100 to 200 examples to validate a model judge. It names no number for trusting a result. My rule of thumb: 100 or more cases before you believe a small gap; a three-case gap on twenty is a coin toss.

Run each case more than once. A model samples its answer, so one run is one draw. A preprint from August 2026 ran each of 900 benchmark items 50 times against three OpenAI model versions, and found that single runs recovered none of the 457 item-level changes the repeats showed reliably. It covers one vendor and three public benchmarks, so its percentages are not rates for your app. The narrow lesson: ask which cases flipped, not only what the average did.

A failure case (invented, to show the shape). A team runs a cheap and a strong model on twenty tickets, once each: 17 and 17. They ship the cheap one. Run five times each, the cheap model passes the invoice tickets only some of the time, the strong one nearly always.

Hold the settings fixed, since thinking effort changes quality and cost (Thinking and effort, model by model). Record output tokens too: the same preprint saw one model pair's mean output on a benchmark roughly halve, so cost can move with no price change.

Keep the set. It makes Switching models safely and When a model is retired routine.

3.Cost, latency, quality, and the routing pattern

For a given task you get two of these three. Deciding which to give up is a product decision.

A starting point by task type, to test rather than to believe:

The routing pattern is how you stop choosing. Send every request to the small model first. Escalate to the strong one only when a signal you can compute says the cheap answer is not good enough: a failed schema or business-rule check, an input-length or complexity score over a threshold, or a customer tier that pays for the better answer. A model's own "I am confident" flag is the weakest of these, so check it against your cases before you rely on it.

When not to route. Skip it if you have no computable signal, because then you are guessing. Skip it for the high-stakes rows of the first table, which go straight to the strong model. An escalated request pays for and waits on both calls, so routing helps only if most requests stay cheap.

# Python 3.10+; pip install openai   (run on SDK 3.33.0, 2026-10-17)
import os
import json
from openai import OpenAI

client = OpenAI()
CHEAP = os.environ["CHEAP_MODEL"]
STRONG = os.environ["STRONG_MODEL"]

SCHEMA = {
    "type": "object",
    "properties": {
        "label": {"type": "string", "enum": ["billing", "bug", "other"]},
        "confident": {"type": "boolean"},
    },
    "required": ["label", "confident"],
    "additionalProperties": False,
}


def classify(text: str, model: str) -> dict | None:
    r = client.responses.create(
        model=model,
        instructions="Label the support ticket. Set confident to false if unsure.",
        input=text,
        max_output_tokens=2000,  # room for hidden thinking as well as the answer
        text={"format": {"type": "json_schema", "name": "ticket", "strict": True, "schema": SCHEMA}},
    )
    if r.status != "completed":  # cut off at max_output_tokens: a failure
        return None
    try:
        out = json.loads(r.output_text)
    except json.JSONDecodeError:  # a refusal is "completed" but has no JSON
        return None
    out["model"] = r.model  # log which model really answered
    return out


def route(text: str) -> dict:
    cheap = classify(text, CHEAP)
    if cheap and cheap["confident"]:  # a computable signal, not a guess
        return cheap
    return classify(text, STRONG) or {"label": "other", "model": None}


print(route("I was charged twice for the same invoice last Tuesday."))
print(route("It. Does the thing. Again. Maybe."))  # vague, so expect escalation

Set CHEAP_MODEL and STRONG_MODEL to two ids from GET /v1/models. I ran it on 2026-10-17 with two small models from one OpenAI family. The first ticket stayed with the cheap model; the vague one was answered by the stronger one. The returned model was a dated snapshot name, not the short id I sent, so log it per request, or you cannot tell whether your escalation rate is 5 percent or 60.

A refusal is completed with no JSON, so classify() returns None; if both models refuse, the caller gets {"label": "other", "model": None}.

Three choices are dated. OpenAI's text guide (Oct 2026) recommends the Responses API "over the older Chat Completions API"; Chat Completions was not on the deprecations page on 2026-10-17. As of Oct 2026 the structured-outputs guide says to always use Structured Outputs instead of JSON mode when possible. The sample sends no temperature, because some models reject it; check each model's page (as of Oct 2026) and see Sampling, and the knobs providers are taking away.