Latency and cost as first-class metrics

Every lesson before this one has been about correctness — does the output match what you want. This one is about the two axes a correct answer can still fail on, which is why Why "looks good to me" fails opened this course with a demo that read well and told you almost nothing: the same trap catches teams here, on a different pair of metrics. An answer can be exactly right and still be a bad answer to ship, if it arrives too slowly or costs too much to be worth what it delivers.

1Latency is a distribution, not a number

The average latency of your endpoint is close to the least useful number you can report about it, because it hides exactly the shape you need to see.

Say ninety-five requests out of a hundred finish in half a second and five take eight seconds — a retry, a cold cache, a longer input. The average lands around 0.8 seconds, which sounds fine, and reports nothing about the one user in twenty who waited eight seconds and is now telling a colleague the feature is broken. Percentiles fix this because they describe the tail directly instead of blending it into a number that hides it.

Time to first token deserves its own line because it measures something total latency cannot. Two responses that both finish in four seconds do not feel the same if one starts streaming after 200 milliseconds and the other sits silent for three and a half seconds before dumping the whole answer at once. If your interface streams, track this separately — it is often the bigger lever on perceived speed than total generation time, and it is invisible if all you watch is the number at the end.

An average with no percentile next to it is a claim about a typical request, dressed up as a claim about all of them.

2Cost is the same shape as latency

Cost per request behaves exactly like latency: it has a distribution, it has a tail, and a change that looks fine on average can still be a real regression for a meaningful slice of traffic. A prompt that occasionally triggers a much longer response — a retry loop, a verbose edge case, an unusually long tool result stuffed into context — costs a lot more on those requests, and if you only ever look at an average cost per request, that tail is invisible in exactly the way a p95 latency spike is invisible in an average latency number.

Track it the same way: total tokens in and out per request, split by the step that used them if a run has more than one model call, and a distribution rather than a single average. The two levers worth watching by name, both measurable without needing a price card:

• Tokens per request, input and output separately, since a change that inflates the prompt and a change that inflates the response are different bugs with different fixes.
• Cache hit rate, if you use prompt caching for a system prompt or a repeated context block — a dropping hit rate is often the real cause behind a cost or latency regression that looks unrelated at first glance, because a cache miss silently reprocesses everything a hit would have skipped.

Rates and dollar figures belong in your own provider's current pricing page, not in a lesson — they move, and a number printed here would be wrong before you finish reading this sentence. What does not move is the habit: measure tokens and cache behaviour as metrics you track over time, the same way you already track quality, so a regression shows up as a graph moving instead of an invoice at the end of the month.

from dataclasses import dataclass, field

@dataclass
class RequestMetrics:
    input_tokens: int
    output_tokens: int
    time_to_first_token_ms: float
    total_latency_ms: float
    cache_read_tokens: int = 0

def record(m: RequestMetrics, history: list[RequestMetrics] = []):
    history.append(m)

def p95(values: list[float]) -> float:
    if not values:
        return 0.0
    ordered = sorted(values)
    index = int(len(ordered) * 0.95)
    return ordered[min(index, len(ordered) - 1)]

def report(history: list[RequestMetrics]) -> dict:
    return {
        "p95_latency_ms": p95([m.total_latency_ms for m in history]),
        "p95_ttft_ms": p95([m.time_to_first_token_ms for m in history]),
        "avg_output_tokens": sum(m.output_tokens for m in history) / len(history),
        "cache_hit_rate": sum(1 for m in history if m.cache_read_tokens > 0) / len(history),
    }

This is the shape of the thing, not a library to adopt — the point is that latency, time to first token, tokens, and cache behaviour all live on one record per request, the same request your trace from Tracing a multi-step run already produced, so nothing here requires a second collection system next to the one you already built.

3Treat both as SLOs, not a monthly surprise

An SLO — a service level objective — is a target you set on purpose and alert against, the same posture this course has taken toward quality since Why “looks good to me” fails: a number you check on a schedule, not one you learn about from a complaint. Latency and cost deserve the identical treatment, and for the identical reason — both degrade quietly, and both are cheap to catch early and expensive to catch late.

Set a p95 latency target per feature, not one target for the whole system — a background summarization job and a live chat response have completely different tolerances, and a single global number will be too loose for one and too strict for the other. Set a cost-per-request ceiling the same way, scoped to what that feature can justify. Alert when either one crosses its line, exactly the way Drift alerts on a golden set score crossing its baseline — the mechanism is identical, only the metric changed.

The habit this closes is the one the whole course opened with. A demo of five good answers told you nothing about the other five thousand. A model that scored well on a leaderboard told you nothing about your traffic. And a change that passes every quality check you have can still ship a regression, if speed and cost were never on the list of things anyone was checking. Watching them is not a lesser concern bolted onto evaluation — it is the same discipline, aimed at the two dimensions that never show up in a correctness score, and it is worth the same seriousness as everything that came before it.

You now have the whole loop: a way to build a test set from real traffic, a way to judge output at scale, a way to ship a change without breaking what worked, and a way to watch a system that is quietly changing on its own. None of it is exotic. It is a habit of measuring instead of assuming, applied consistently, to the parts of the system that go wrong silently if nobody is looking.