Tracing a multi-step run

Reading a trace after it breaks starts from a record that already exists and asks what went wrong. This lesson is about the step before that one, which nobody gets to skip: making the record exist in the first place. Reading a trace is a skill you use once, on one bad run. Building the thing that recorded it is infrastructure you build once and every future run benefits from — including the one that breaks next month, in a way you have not imagined yet.

1A run is a tree, not a line

A single request into an LLM system rarely does one thing. It retrieves, then calls a model, then calls a tool, then calls the model again to read the tool's result. Each of those is a step with its own inputs, its own outputs, and its own way of going wrong — and the pipeline as a whole succeeds or fails based on all of them, not on the one that happened to produce the final text.

Treat each step as a span: one record covering one unit of work, with what went in, what came out, when it started, and how long it took. A run is the tree of spans it produced, in order. That tree is the thing you actually want when something goes wrong later — not the final answer, and not a log line saying the request completed, but the sequence of steps that produced it.

The distinction matters because the temptation, especially early on, is to log one line per request: "request finished, 4.2 seconds, 200 OK." That line is true and it is nearly useless. It tells you the request happened. It cannot tell you which of the four steps inside it took three of those seconds, or which one returned the empty result that the model then guessed around. A trace with no internal structure is not a smaller trace — it is a different, much less useful kind of record, and no amount of squinting at it later recovers the steps you never wrote down.

2What a span has to carry to be answerable later

You do not know today which question you will be asking about a given run in three weeks. That is the actual argument for instrumenting broadly now rather than narrowly, adding fields only once a specific bug demands them. The fields below are cheap to write and expensive to have missed.

input and output verbatim are the two people cut first, because they are the biggest fields and the storage adds up. They are also the two that answer the question you will actually ask, which is "what did this step actually see and actually return." A field you summarized at write time encodes a guess about what mattered, made before you knew what would go wrong. The next lesson, What to log, and what never to, is entirely about the one exception to logging everything verbatim — some of what a step sees should never be written down at all, and that lesson draws the line. This one assumes you have decided what is safe to keep and covers making sure you actually keep it.

import contextlib
import json
import logging
import time
import uuid

log = logging.getLogger("trace")

@contextlib.contextmanager
def span(run_id: str, kind: str, name: str, **input_fields):
    started = time.monotonic()
    record = {
        "run_id": run_id,
        "span_id": uuid.uuid4().hex[:12],
        "kind": kind,           # "retrieval", "model", or "tool"
        "name": name,           # retriever name, model id, or tool name
        "input": input_fields,
    }
    try:
        yield record
    finally:
        record["duration_ms"] = round((time.monotonic() - started) * 1000)
        log.info(json.dumps(record))

# Usage around one step of a run:
run_id = uuid.uuid4().hex
with span(run_id, "tool", "lookup_order", order_id="A-1029") as s:
    result = lookup_order("A-1029")
    s["output"] = result       # set once the step actually finishes

That is a context manager, not a library — it works with whatever logging sink you already have, and it costs nothing to add around a new step. The specific shape matters less than the discipline of wrapping every step, not just the ones that have bitten you before.

A trace you wish you had is not a trace. It is a run that happened once, told you nothing, and cannot be replayed to tell you anything more.

3Instrument the pipeline, not the incident

The natural order most teams instrument in is backwards: add tracing to the step that just broke, ship it, wait for the next thing to break somewhere else, repeat. Each addition is reasonable on its own and the result is a pipeline with three well-traced steps and two silent ones — exactly the two that will produce the next unanswerable ticket, because the pattern that got you here has no reason to stop.

The fix is to trace at the level of "a step in the pipeline," as a rule applied once, rather than "a step that has caused a problem," applied after each incident. Every retrieval call gets a span. Every model call gets a span. Every tool call gets a span. New steps get one too, as part of adding the step — not as a follow-up ticket that competes with everything else for priority and usually loses.

Two things make this durable rather than a one-time cleanup:

• Wrap the call site, not the business logic. A span() context manager around the actual retrieval, model, or tool call means every future call through that same function inherits tracing automatically. Scatter the instrumentation through the reasoning code instead, and every new code path is a new place to forget it.
• Reach for OpenTelemetry's GenAI conventions once you have more than a couple of pipelines. They define agreed span kinds and attribute names for model calls, tool calls, and agent steps, so a trace from your code looks like a trace from anyone else's, and existing tools can render the tree without custom glue. The conventions are still moving, so check their current shape before you commit to field names — but the underlying idea, a run as a tree of typed spans with a shared id, is the one this lesson has been building toward the whole time, however you end up storing it.

None of this predicts which run will break next, or why. What it buys you is the one thing you cannot get any other way: when it does break, the record already exists, and reading it is the next lesson's problem instead of a re-run that may never reproduce.