Versioning and testing prompts

This is not a lesson about evaluating AI products. The Evals course does that, properly. This is the minimum that comes first: the prompt in git, twenty cases, one script, and a committed result. Get that running before you need anything bigger.

1.A prompt is code, and so is the model beside it

A prompt decides what your system does, which is the definition of code. So it gets the treatment code gets. It lives in the repository, in a real file such as prompts/classify_ticket.py, not inline three functions down and not in a database row. It is in git, so you have history and a way back. A change to it is a pull request where someone reads the before and after. It deploys the way your code deploys, so the version running is the version you can see.

The habit to break is editing the live prompt in a vendor dashboard. There is no diff, no review, no rollback, and no link between the edit and the incident three days later. If your team needs to change behaviour without a deploy, that is a feature-flag problem, and the flag should still point at a prompt that lives in the repo. Prompt-management tools have a second cost: Hamel Husain and Shreya Shankar (September 2026) say they often cannot run your own tools or retrieval, so you end up testing something other than what ships. That is their opinion from practice, not a measurement.

Keep the stable part stable while you are in there. The durable rules go in the system prompt (The system prompt and the user turn), at the front of the request where the cache can use them (Prompt layout and the cache). An edit changes that prefix, so expect one cache write after each prompt change, as of October 2026.

Then pin the rest of the artefact. A prompt is tuned against one model, at one effort, with thinking on or off, and the same words can behave differently when any of the three moves. The vendor's Sonnet 5.5 guide (October 2026) says effort levels were recalibrated against the previous model, so run a fresh sweep instead of carrying a level forward. Store all three beside the prompt. Changing any of them is a change to test, exactly like editing a word.

A prompt edit is a deploy, and so is a model swap. Review both as a diff.

2.Twenty cases, run more than once

Change a word and you have not changed one branch. You have nudged the influence on every input the system will see. "Be concise" may also drop the caveat that mattered on long documents. So trying one ticket and shipping measures one point on a distribution you just moved everywhere. What works is the same fixed set, before and after, compared.

Prefer a code check to a model judge: exact match, a substring, a rule. A judge needs its own labelled examples and upkeep, and the Evals course covers when it is worth it in Deterministic checks and LLM-as-judge. For where cases come from, see Building an eval set from your own traffic. Twenty is where you start. Hamel and Shankar put a regression set at 100 or more as failures accumulate, a rule of thumb with no study behind it. Retire or harden any check that has never failed.

Run each case more than once. A June 2025 Wharton study, on older models and one science benchmark, found that against a baseline told to answer with no working, "think step by step" lifted the average but the share of questions answered right on every trial fell for three of five models (Thinking out loud has the numbers). The method travels: report the share of cases right every time, next to the average. A case that flips between runs of an unchanged prompt is borderline, and the harness is telling you so. Lowering randomness, where your model lets you, narrows that flipping but does not remove it.

The bar is regression, not perfection. You are not chasing twenty of twenty. The score must not go down, and every case whose answer changed gets looked at.

3.The harness, a failure it missed, and when to skip it

One file holds the pair, so the prompt and the settings it was tuned on change in the same diff. The parameter spellings below are copied from vendor docs as of October 2026, not run here, so try them on your SDK version first. A one-word label needs no thinking, so it is turned off.

# prompts/classify_ticket.py — prompt, model and settings travel together
MODEL = "claude-sonnet-5-5"   # as of October 2026
EFFORT = "high"
THINKING = {"type": "between_tools"}   # thinking off: a label needs none
SYSTEM = "Classify the ticket: billing, bug, feature_request or other. Reply with the label only."

# evals/run.py
import json

from anthropic import Anthropic

from prompts.classify_ticket import EFFORT, MODEL, SYSTEM, THINKING

client = Anthropic()
REPEATS = 3

def text_of(resp) -> str:
    return "".join(b.text for b in resp.content if b.type == "text")

def run_case(case: dict) -> str:
    resp = client.messages.create(
        model=MODEL, max_tokens=20, thinking=THINKING, output_config={"effort": EFFORT},
        system=SYSTEM, messages=[{"role": "user", "content": case["input"]}],
    )
    if resp.stop_reason != "end_turn":
        return f"[{resp.stop_reason}]"   # record it as the answer, so the case fails and the run goes on
    return text_of(resp).strip().lower()

cases = json.load(open("evals/tickets.json"))   # ~20 real cases: id, input, expected
results = {c["id"]: [run_case(c) for _ in range(REPEATS)] for c in cases}
ok = {c["id"]: [r == c["expected"] for r in results[c["id"]]] for c in cases}
print(f"right every time: {sum(map(all, ok.values()))}/{len(cases)}",
      f"runs right: {sum(map(sum, ok.values()))}/{len(cases) * REPEATS}")

with open("evals/last_run.json", "w") as f:   # commit this file
    json.dump({"model": MODEL, "effort": EFFORT, "results": results}, f, indent=2, sort_keys=True)

Sorted keys and indented JSON are what make the next run a small, readable diff and not a reformatted blob. Twenty cases at three runs each is sixty cheap calls. The worked example is the one in the figure: edit, run, and read the two cases that moved.

A failure the harness missed. Picture a set that passes twenty of twenty for months, and passes again after a model upgrade. Then tickets in a language that appeared in none of the twenty started landing in other. The harness was fine; the set had stopped growing. The fix is a habit: each production failure becomes a case the day it is found, with its expected label, before you fix the prompt. Then the fix has something to prove.

The same blind spot applies to security. If the prompt reads untrusted text, keep a few injection cases in the set (Prompt injection). They are a canary that can only go wrong. Passing proves little: the October 2025 paper by Nasr and colleagues, "The attacker moves second," found that defences tested against fixed attack strings fell to attacks tuned against them, though it tested no Claude model.

When not to use it. Skip the full harness for a script nobody will edit again. Do not build it on invented cases before you have read any real outputs; read a few dozen by hand first, and you will find the cases. And when the task has no checkable answer yet, such as open prose, the code-check harness has nothing to hold, so write down what a good answer must contain before you automate anything. It guards against regressions. It does not prove the prompt is good.

Next: a prompt that has passed this test for months has also grown a line for every old failure, and Trimming a prompt that grew is about taking those lines back out.