Picking a model

Everything so far has been about what the models are and what they cost. This lesson is the decision itself. It is deliberately unglamorous, because the glamorous version — reading comparisons and picking the winner — is the version that goes wrong.

1The leaderboard is a filter, not a decision

Public benchmarks are not fake. They are careful work and they measure something real. The problem is narrower than that and harder to get around: they measure a general capability on public data, and your job is a specific capability on your data.

Three things degrade these scores further:

• Contamination. Benchmarks are published on the internet. Training data is scraped from the internet. A model that has effectively seen the answer key scores well without being better, and nobody can fully rule this out for any public test.
• Saturation. Once every strong model scores in the nineties on a test, the remaining differences are noise, and vendors move to a harder test. A one-point gap is not a decision.
• Self-reported harnesses. Vendors report their own numbers, on their own prompts and settings. Reproducing them is genuinely hard.

Use them anyway, for the one job they are good at: cutting a list of twelve candidate models down to three that are plausibly in the right class for your task, along with the constraints that lessons 05 and 06 covered — price, context window, rate limits, latency, region. Then close the tab.

2Twenty cases of your own decide it

The build-with-llms course covers evals properly in its lesson 08, Evals: proving it works. Read it — this is the same tool, pointed at a different question. There you use an eval to tell whether your prompt edit helped. Here you use it to tell which model to buy.

The build is small on purpose:

1. Pull twenty real inputs out of your logs, your ticket queue, your existing data. Real ones. Inputs you invent are inputs shaped by what you already expect.
2. Weight it towards trouble. Roughly half ordinary cases, half the awkward ones — long inputs, missing fields, hostile users, the two edge cases you already know break things.
3. Write down the expected outcome for each. Exact value where there is one, a short rubric where there is not.
4. Grade with the cheapest method that fits. An equality check beats a regex, which beats a model judging a model.
5. Run all three candidates, at temperature 0, on the same twenty, and put the results in one table alongside cost and p95 latency.

Twenty real cases from your own traffic will tell you more in an afternoon than every leaderboard on the internet, and they keep being true after the leaderboards change.

Keep the set. It is the artefact that makes lesson 10 possible: when a model is deprecated or a cheaper one appears, you already own the thing that answers whether to switch.

3Cost, latency, quality — and the routing pattern

For any given task, you get two of these three. Deciding which one you are giving up is most of the work, and it is a product decision, not a technical one.

Then a table by task type, as a starting point to test rather than a conclusion:

The routing pattern is how you stop choosing. Send everything to a small cheap model first. Escalate to the big one only when a signal you can actually compute says the cheap answer is not good enough. Typical signals: the output failed a schema or business-rule validator, the model reported low confidence when asked for one, an input-length or complexity score crossed a threshold, or the request came from a tier that pays for the better answer.

The economics work because in most real workloads the easy majority is a large majority. The proportion that stays cheap varies a lot by task, so measure yours rather than trusting anyone's percentage, including a number in a lesson.

# pip install openai
import os, json
from openai import OpenAI

client = OpenAI()
CHEAP  = os.environ.get("CHEAP_MODEL",  "gpt-4o-mini")   # your own ids here
STRONG = os.environ.get("STRONG_MODEL", "gpt-4o")

def classify(text: str, model: str) -> dict | None:
    r = client.chat.completions.create(
        model=model,
        temperature=0,
        max_tokens=100,
        response_format={"type": "json_object"},
        messages=[
            {"role": "system", "content": 'Reply as JSON: {"label": one of billing|bug|other, "confident": true or false}'},
            {"role": "user", "content": text},
        ],
    )
    try:
        out = json.loads(r.choices[0].message.content)
    except json.JSONDecodeError:
        return None                                   # unparseable counts as a failure
    return out if out.get("label") in {"billing", "bug", "other"} else None

def route(text: str) -> dict:
    cheap = classify(text, CHEAP)
    if cheap and cheap.get("confident"):              # a measurable signal, not a guess
        return cheap
    return classify(text, STRONG) or {"label": "other"}

print(route("I was charged twice for the same invoice last Tuesday."))

Log which model served each request. Without that line you cannot tell whether the escalation rate is 5 percent or 60, and the whole point of routing is that number.