What to log, and what never to

Tracing a multi-step run told you to log inputs and outputs verbatim, and that advice was correct for the fields it meant — arguments, results, timing. It did not mean every field, and it left the actual line-drawing to this lesson, because the two questions are genuinely different. That lesson asks what makes a run answerable. This one asks what makes a trace store safe to have built in the first place.

1A trace store is a data store

Once tracing works, it is tempting to treat it as a debugging scratchpad — a place that only engineers look at, only when something breaks, so ordinary data handling rules feel like they do not apply. They apply exactly as much as they do anywhere else. A trace store holding full prompts and responses from real users is a store of whatever those users typed and whatever the model returned, which includes anything they mentioned: a name, an address, a medical detail, a account number, pasted into a chat because that is what chats are for.

The store does not know the difference between a field that is safe to keep and one that is not — it writes what it is given. The judgment has to happen before the write, because a trace store is generally optimized for search and retention, not for the access controls and deletion guarantees a system holding personal data is supposed to have. Treating it as exempt from those rules because it is "just logs" is how a debugging tool becomes the least protected copy of your users' data in the whole system.

That last row is the whole risk. A schema constrains what can end up in a database field. A trace of a free-text conversation has no such constraint — the user can type anything, the model can echo it back, and both ends land in the trace unless something explicitly stops them.

2Three categories, decided before the write

Every field a span could capture falls into one of three buckets. Deciding which bucket up front, at instrumentation time, is what makes this workable — sorting it out after millions of trace rows already exist is a much worse version of the same job.

Always log. Structure, timing, token counts, span kind, model id, tool name, status. None of this identifies a person or exposes a secret, all of it is exactly what makes a trace answerable, and none of it should ever be the thing you strip to be careful — a trace with the content redacted but the shape intact is still useful; a trace with the shape gone is not a trace at all.

Redact or hash. Free-text fields a user controls — the message content, a tool argument built from user input, a model's response that might echo something the user said back. These carry the debugging value the previous lesson cared about, so the fix is not to drop them; it is to run them through a scrubber that masks structured patterns like phone numbers, emails, and national id formats before the write, and hash the rest if you need to correlate "the same user hit this twice" without keeping what they said both times.

Never log, no exception. Secrets: API keys, auth tokens, session cookies, credentials of any kind. There is no safe redacted form of a credential sitting in a log line, so the answer is not a scrubber for this category — it is making sure these values never reach the logging call to begin with, by keeping them out of the objects that get passed to span() in the first place.

import re

# A short list, not exhaustive - extend to match what your own inputs look like.
PATTERNS = {
    "email": re.compile(r"[\w.+-]+@[\w-]+\.[\w.-]+"),
    "ssn": re.compile(r"\b\d{3}-\d{2}-\d{4}\b"),
    "phone": re.compile(r"\b\d{3}[-.\s]?\d{3}[-.\s]?\d{4}\b"),
}

def scrub(text: str) -> str:
    for label, pattern in PATTERNS.items():
        text = pattern.sub(f"[{label}]", text)
    return text

def safe_span_input(user_message: str, tool_args: dict) -> dict:
    return {
        "user_message": scrub(user_message),
        "tool_args": {k: scrub(str(v)) for k, v in tool_args.items()},
        # never include: api_key, auth_token, or anything from a credentials object
    }

A pattern-based scrubber like this one will miss things — an id format you did not anticipate, a name with no fixed pattern at all. Treat it as a floor, not a guarantee, and pair it with the retention policy below so a miss has a shelf life instead of living forever.

The safest field in a trace store is the one that was never written. Everything after that is damage control.

3Sampling and retention are part of the same decision

Two more knobs sit next to redaction, and both reduce exposure the same way redaction does — by shrinking what is kept and for how long.

Sample, don't capture everything at full detail forever. Trace every run at the structural level — timing, status, token counts — because that is cheap and the always-log bucket is not the risk. Capture full redacted content for a percentage of runs, weighted toward the ones you actually need: errors, low eval scores, anything flagged for review. A representative slice with content is worth more than a complete set of shapes with nothing in them, and it is far less exposure than capturing every single run at full detail by default.

Set a retention window, and let things actually expire. Storage being cheap is not a reason to keep data indefinitely — every day a redacted-but-imperfect field sits in the store is another day a scrubber miss is sitting there too. A 30- or 90-day window on raw content, with the structural, non-identifying fields kept longer for trend analysis like the drift checks in the previous part of this course, gives you enough history to debug last week's incident without accumulating a permanent archive of everything anyone ever typed.

Both of these are decisions to make once, at the same time you build the tracing pipeline, not patches to bolt on after a store already has months of unredacted history in it. The cheapest time to decide what a trace store should never hold is before the first row is written.