Let a model write the prompt

Versioning and testing prompts gave you a scoreboard: a fixed set of cases, a pass count, a diff. It grades whatever you hand it and stops there. This lesson is not about handing the whole job to a model. It is the narrow version: a model reads your failures and drafts rewrites, and your harness decides whether any of them is better.

1.Show it the failures, and why they failed

Ask a model to "improve this prompt" with the prompt alone and you tend to get polish — smoother wording, the same score. It cannot tell which sentence is broken. So give it the misses. For each one, hand over the input, the expected output, what the prompt actually produced, and one line saying why that is wrong. The last item matters most. Agrawal and colleagues (a February 2026 revision, the authors' own results on small and mid-size models, none of them Claude) argue that a written account of each failure carries more signal per run than a score. Searching over prompts returns to that paper.

Where does the reason come from? From you. Hamel Husain and Shreya Shankar (September 2026) argue, as opinion rather than a measured result, that an optimiser only improves failures you already know about, and that your idea of "good" shifts as you read outputs. Read the failing outputs by hand, write a short note on each, and keep the note in the case file. In their view the safer pattern is this one: the model revises the prompt from your notes and you review the result. It works the way Retries and fallbacks did, with the error attached — one level up.

The model drafts the candidates. The harness picks. Neither does the other's job.

2.Several candidates, scored on cases it never saw

Ask for three rewrites in one call, each a different kind of change: tighter rules, one example lifted from the misses, a clearer line between the two labels it confuses. One draft is one unmeasured guess; three let the harness tell you which kind of change helped. The same move works on your examples. Anthropic's prompting guide (October 2026) says you can ask Claude to critique or extend your example set.

Fix in code what a rewrite may not touch. Here the four labels must survive, so a candidate that loses one is discarded before it costs a single call. A request in the prompt is a request; a check is a rule — though the one below is only a cheap guard, since a label can sit in the text without being used.

Then split the cases in two. The working set supplies the misses the drafter sees and scores every candidate. The holdout is never shown to the drafter and never used to pick. You read it once, for the winner only. Twenty cases is too few to spare any, so grow the file to forty: thirty working, ten holdout, with the split stored in the file so it never shifts. Ten is still coarse, since one case is ten points. The full three-set version belongs to Searching over prompts.

The parameter spellings below are copied from vendor docs as of October 2026, not run here, so try them on your SDK version first.

import json
import re

from anthropic import Anthropic
from pydantic import BaseModel

from prompts.classify_ticket import EFFORT, MODEL as CLASSIFIER, SYSTEM, THINKING   # pinned with the prompt, as before

client = Anthropic()
LABELS = ["billing", "bug", "feature_request", "other"]
# MODEL here is the drafter; CLASSIFIER is the model being tuned
MODEL = "claude-sonnet-5-5"   # as of October 2026

def text_of(resp) -> str:
    return "".join(b.text for b in resp.content if b.type == "text")

def run_with(prompt: str, case: dict) -> str:
    """Same call as run_case in evals/run.py, with the prompt as an argument."""
    resp = client.messages.create(
        model=CLASSIFIER, max_tokens=20, thinking=THINKING, output_config={"effort": EFFORT},
        system=prompt, messages=[{"role": "user", "content": case["input"]}],
    )
    if resp.stop_reason != "end_turn":
        return f"[{resp.stop_reason}]"   # the case fails and the run goes on
    return text_of(resp).strip().lower()

def score(prompt: str, cases: list[dict]) -> int:
    return sum(run_with(prompt, c) == c["expected"] for c in cases)

class Candidates(BaseModel):
    prompts: list[str]

def draft(current: str, misses: list[dict]) -> list[str]:
    resp = client.messages.parse(
        model=MODEL, max_tokens=4000,   # thinking is on and counts against this
        system="Rewrite a ticket-classifier prompt. Return 3 full replacements, each fixing the "
               "misses a different way. Keep the four labels and the one-word reply.",
        messages=[{"role": "user", "content": f"Prompt:\n{current}\n\nMisses:\n{json.dumps(misses, indent=2)}"}],
        output_format=Candidates,
    )
    if resp.stop_reason != "end_turn":
        raise RuntimeError(f"no usable draft: {resp.stop_reason}")
    # cheap guard only: each label appears as a whole word; it cannot tell the label is still used
    return [p for p in resp.parsed_output.prompts if all(re.search(rf"\b{l}\b", p) for l in LABELS)]

cases = json.load(open("evals/tickets.json"))   # 40 cases: id, input, expected, split, and your note on known misses
working = [c for c in cases if c["split"] == "working"]   # 30
holdout = [c for c in cases if c["split"] == "holdout"]   # 10

got = {c["id"]: run_with(SYSTEM, c) for c in working}
misses = [{"input": c["input"], "expected": c["expected"], "got": got[c["id"]], "why": c.get("note", "")}
          for c in working if got[c["id"]] != c["expected"]]
if not misses:
    raise SystemExit("no misses")
candidates = draft(SYSTEM, misses[:5])   # a sample of the misses, not all of them
if not candidates:
    raise SystemExit("no candidate kept the labels")

scores = [score(p, working) for p in candidates]
best = max(range(len(candidates)), key=scores.__getitem__)   # picked on the working set alone
print(f"working: current {len(working) - len(misses)}, winner {scores[best]} of {len(working)}")
print(f"holdout: current {score(SYSTEM, holdout)}, winner {score(candidates[best], holdout)} of {len(holdout)}")

3.A worked example, the way it fails, and when to skip it

A worked example. Say the current prompt gets 22 of 30 on the working set and 7 of 10 on the holdout. Eight misses, mostly tickets about a double charge that the prompt labels billing and your notes say are bug. The drafter returns three candidates. The numbers are an illustration, not a measurement.

A failure. In another round, a candidate pastes four of the misses it was shown into the prompt as examples. It scores 30 of 30 on the working set and 5 of 10 on the holdout, against the current 7. It learned the shown tickets: near-twins of them pass and anything new does not. Because you picked on the working set alone, you see 5 against 7 and refuse the swap. Add the holdout into the pick score and 35 beats the current 29, and the overfit prompt ships.

Even a clean win needs care. B is not safe just because it won. Ten holdout cases cannot tell a one-case gain from noise, so treat 8 as support, not proof. A drafted prompt also tends to pick up narrow rules, so read the winner as if a colleague wrote it, and prune it later with Trimming a prompt that grew. And if you draft with a bigger model than you ship, the harness is your only evidence: Anthropic's Claude Code team said in July 2026 that it has no hard data on whether a bigger model's detailed prompt lifts a smaller one (an informal conversation, read through Simon Willison's write-up, not checked here).

When not to use it. That is real money and real time next to typing a sentence — so spend it where it pays. Use it when you are stuck after three or four hand edits, when the misses are too many or too varied to hold in your head, or when you would otherwise lose an afternoon to guessing. Skip it when the prompt already passes, when the script is one nobody will touch again, and before the harness exists. Skip it too when your labels are the problem — a model cannot repair a prompt for a case you marked wrong. If you have not read the failing outputs yourself, do that first, because a search only climbs the metric you gave it.

Next: one round is the baseline version, and Searching over prompts runs it many times over, with a third set kept out of the way.