Let a model write the prompt

Versioning and testing prompts turned prompt edits into something measurable: twenty real cases, a pass count, a diff you can review. That was the whole point, and it works. What it does not do is tell you which word to change next. You still stare at three failing cases, guess at a rewrite, run the harness, and find out you were wrong - now with a number confirming it instead of a feeling. This lesson automates that last, most tedious step: drafting the next candidate.

1Show it the failures, not just the prompt

Asking a model to "improve this prompt" cold produces exactly what you would expect from a vague request - Why prompts fail's whole argument, one level up. The model has no idea which part is broken, so it polishes wording, adds a sentence about being thorough, and hands back something that reads better and scores the same.

The fix is the one you already know from Retries and fallbacks: show it the error, not just the task. Instead of the prompt alone, hand the drafting call three things - the current prompt, a handful of cases it is currently failing, and for each one the input, the expected output, and what it actually produced. That is a narrower, easier question than "write a better prompt." It is closer to "here is specifically what is wrong, fix that," which is the same move that made repair retries work in Retries and fallbacks, aimed at the prompt itself instead of at one response.

Neither column removes the harness. Both still need it, for the same reason - the row about intuition running out is really Versioning and testing prompts's argument again: a person eyeballing one example is measuring almost nothing, and a model reading three failing cases is only slightly better informed unless something else checks its work.

2Generate several, and let the harness pick

The single biggest mistake in this technique is asking for one rewrite and shipping whichever one comes back. That is the same trap as judging a prompt edit from one example - a single candidate is one point on a distribution, and you have no idea whether it is better everywhere or just better on the three cases it was shown.

Ask for three to five candidates in one call, each making a different kind of change - one tightening the instructions, one adding an example from the failing cases, one restructuring the output contract. Then run every candidate through the exact same twenty-case harness from Versioning and testing prompts, unchanged, and compare pass counts directly. This is where the technique earns its keep: you are not trusting the drafting model's opinion of its own work, you are trusting the same measurement you already trusted for hand edits.

The model drafting the prompt is not the judge of the prompt. Your twenty cases are, every time.

Two guardrails keep this from drifting into a wholesale rewrite that happens to score well by accident:

• Constrain what is allowed to change. Tell the drafting model to keep the output schema, the system/user split, and the field names fixed, and only rewrite the instructions. Otherwise a "winning" candidate might have changed three unrelated things at once, and you are back to not knowing which change did the work - precisely the problem Why prompts fail opened the whole course with.
• Hold a few cases back. Keep two or three cases out of what the drafting model ever sees, and score every candidate against them too. A candidate that aces the twenty cases it was shown and drops on the ones it was not is not improving at the task - it has learned the specifics of your set. That gap is the tell, and it is cheap to check for.

import json
from anthropic import Anthropic

client = Anthropic()

DRAFTER_SYSTEM = """You improve a prompt used for a specific task.
You will be given the current prompt and cases it is currently failing.
Propose 3 candidate rewrites that would fix these specific failures.
Keep the output schema and the field names exactly as they are.
Return one JSON array of 3 strings - the full replacement prompt text only."""

def draft_candidates(current_prompt: str, failures: list[dict]) -> list[str]:
    resp = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=2000,
        system=DRAFTER_SYSTEM,
        messages=[{
            "role": "user",
            "content": (
                f"Current prompt:\n{current_prompt}\n\n"
                f"Failing cases:\n{json.dumps(failures, indent=2)}"
            ),
        }],
    )
    return json.loads(resp.content[0].text)

def score(prompt: str, cases: list[dict]) -> int:
    # Reuses the exact runner and grader from the eval harness in Versioning and testing prompts.
    return sum(run_case(prompt, c) == c["expected"] for c in cases)

with open("evals/tickets.json") as f:
    cases = json.load(f)
holdout, working_set = cases[:3], cases[3:]          # never shown to the drafter

with open("prompts/classify_ticket_current.txt") as f:
    current = f.read()

failing = [c for c in working_set if run_case(current, c) != c["expected"]]
candidates = draft_candidates(current, failing[:5])   # a sample, not the whole list

results = [(score(c, working_set) + score(c, holdout), c) for c in candidates]
best_score, best_prompt = max(results, key=lambda r: r[0])
print(f"current: {score(current, working_set) + score(current, holdout)}  best candidate: {best_score}")

run_case is the same function Versioning and testing prompts already built - nothing about the harness changes, only where the next candidate comes from. The holdout score sitting right next to the working-set score is the whole safeguard: a candidate that wins on one and loses on the other has not earned the swap.

3What it costs, and when hand-editing is still faster

Be honest about the arithmetic before reaching for this. One drafting call, then every candidate run once against your full case set, is 1 + (candidates × cases) model calls per round - five candidates against twenty-three cases is over a hundred calls to test one round of ideas. That is real money and real time, and it dwarfs the cost of typing a sentence yourself.

It earns that cost back in specific situations, not by default:

• You are stuck, not starting. You have made three or four hand edits, the score has not moved, and you are out of ideas for what to try next. That is exactly when a model reading the actual failing cases outperforms another guess from you.
• The failure set is large or unfamiliar. Ten failing cases across a handful of distinct causes is more than a person reliably holds in their head at once; a model reading all ten in one pass does not have that limit.
• You are about to spend an afternoon iterating anyway. If the cost of the search is smaller than the cost of the hours you would otherwise spend hand-tuning, automate the tuning.

It is close to pure waste on a prompt that already passes cleanly, on a script nobody will touch again, or - this one matters - before Versioning and testing prompts's harness exists at all. Meta-prompting has nothing to search against without a fixed set of cases and a grader; build that first, or you are just asking a model to guess at improvements and taking its word for it, which is the exact failure mode this whole lesson exists to avoid.

The technique is not a replacement for understanding your own prompt. It is a way to generate more candidates, faster, once you already know how to tell a good one from a bad one - which is the harness, still doing the job Versioning and testing prompts gave it.