Evals and Observability

What the detail shows

1. Live requests run across the top of the sheet. Three dotted taps hang off it — one at retrieval, one at the model, one at the output. A dotted line watches and carries nothing.
2. The three taps run into one dotted line that feeds everything below.
3. At the left it turns down into a hopper. The wide mouth is all your traffic; what drops out of the spout is the handful of cases you keep, ruled up as an eval set.
4. The set is replayed offline, and splits to two instruments. The exact check has two stops and a needle against one of them: it passed or it did not. The judge is a model, drawn as one, and gives a reading on a scale.
5. Both readings converge on one dial, the score.
6. A dotted tap takes the judge off to a second, smaller dial, where its answers on A-or-B pairs are checked against a human’s.
7. Along the bottom the same set is run before a change and after it. Two scales, a marker on each, meeting at a comparator.
8. The difference has to pass a gate before anything ships. Past the gate the line is solid again.
9. After it ships a dotted line keeps reading, onto a strip chart where the score drifts down across its own threshold on its own.
10. At the right the output tap lands on a trace: one run end to end, its steps drawn as bars, with the field that is never logged struck out.
11. The same tap carries on down a rail, and three gauges hang off it: quality, latency and cost.