Sampling and voting

Before this lesson, every reliability tool in the course assumed you had something to react to: a schema violation, a timeout, a 429. This one is for the call that gives you nothing to react to at all — it just quietly gave you the wrong answer, formatted correctly.

1Why retrying the same call doesn't help here

Retries and fallbacks is built around a specific shape of failure: the response fails validation, you attach the error, you ask again. That works because the error tells the model something concrete to fix. A classifier that returns urgency: 4 when the true urgency is 2 hasn't failed anything. It's a valid integer, in range, a plausible reading of an ambiguous ticket. There's no exception to catch and nothing to attach to a repair turn, because as far as your code can tell, nothing went wrong.

This is a different failure family: well-formed and wrong, rather than malformed. Retrying the identical prompt against it looks pointless at first glance — if the call already succeeded once, what does calling it again get you?

More than you'd think, and here's why. Sampling, and the knobs providers are taking away, in the LLMs course, covers the mechanism: providers rescale or truncate the distribution over the next token before sampling from it, and — on Anthropic's current models — you can no longer even touch those knobs; temperature, top_p, and top_k outside their defaults are a flat 400. It would be reasonable to assume that means two identical calls now return identical output. They don't. Default sampling still draws from a distribution rather than always taking the single highest-probability token, so two calls with the same input remain two independent draws, not two copies. That's not a workaround for the locked knobs — it's the same fact that always made temperature 0 a weaker guarantee than people treated it as. Retries and fallbacks touched this from the other side, in its note that a stuck repair loop needs to change the request rather than lean on sampling variation.

Five calls to the same prompt are therefore five real opinions, not five copies of one opinion. That's the entire premise of this lesson.

A retry with an error attached asks a narrower question. A vote asks the same question several times and listens for whether the answers agree.

2Counting the votes

Once you have several independent answers, the question is how to turn them into one. That depends entirely on the shape of what came back.

Closed answers vote cleanly. A label from a fixed set, an enum, a small integer range, yes/no — these are comparable. Five draws, take the mode, done. This is the case self-consistency was built for: Wang et al.'s original paper ran a model several times on the same reasoning problem and kept the final answer that recurred most often, on the theory that a wrong derivation is far less likely to land on the same wrong number twice than a right one is to land on the right number five times.

Open prose doesn't vote at all. Five calls asking for a one-paragraph summary give you five different, individually reasonable paragraphs. There's no meaningful "majority" paragraph to extract — you'd need a second model call just to judge which of the five is best, which is a different technique (LLM-as-judge) belonging to a different course, not a vote.

Ties need a rule you set in advance. A clean 3-2 split has a winner. A 2-2-1 split across three labels does not, and treating the first-arrived answer as the tiebreaker throws away exactly the signal a split vote is giving you — that the input is genuinely ambiguous. Escalate ties to a stronger model, or queue them for a human, the same fallback-ladder instinct from Retries and fallbacks. Never resolve a tie by coin flip; you'd be manufacturing false confidence on the cases you most need to flag.

import anthropic
from collections import Counter
from concurrent.futures import ThreadPoolExecutor

client = anthropic.Anthropic()

def classify(ticket_text):
    resp = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=10,
        system="Classify the ticket urgency as low, medium, or high. Reply with one word.",
        messages=[{"role": "user", "content": ticket_text}],
    )
    return resp.content[0].text.strip().lower()

def vote(ticket_text, n=5):
    with ThreadPoolExecutor(max_workers=n) as pool:
        votes = list(pool.map(lambda _: classify(ticket_text), range(n)))

    counts = Counter(votes)
    winner, top_count = counts.most_common(1)[0]

    # A genuine tie: more than one label hit the same top count.
    if list(counts.values()).count(top_count) > 1:
        return None, counts  # signal "escalate", don't guess

    return winner, counts

label, tally = vote("Customer says checkout is broken for everyone, right now.")
print(label, tally)

The ThreadPoolExecutor is doing the real work here — five calls fired at once, not five calls in a row. That's the whole latency argument from the table above made concrete.

3What it costs, and when it's worth it

The cost is linear and unavoidable: N calls means N times the tokens of one call, billed in full every time, whether the votes agree or not. Where voting earns that back is latency, not money — because the calls don't depend on each other, they run concurrently, and five parallel calls at roughly the same latency as one is a very different trade than the two sequential repair attempts a retry loop pays for.

Spend it where being wrong is expensive and there's no ground truth to check against — a support ticket getting misrouted to the wrong queue, a moderation call, a judgment classification with real consequences downstream. Skip it on anything cheap and low-stakes, or anything a schema can already fully validate; if there's a decisive check, use it, and save voting for the cases where "decisive" isn't available because the question is genuinely a judgment call.

One more thing worth being honest about: voting doesn't make the model right more often on any single call — it makes you more likely to notice when the model isn't converging. A model that's confidently, consistently wrong will vote for its wrong answer just as cleanly as it would vote for a right one. What you're really buying is the disagreement signal on the cases that are actually ambiguous, and a slightly better hit rate on the cases where the model was directionally right but noisy. It's a reliability tool, not a correctness guarantee — keep that distinction in mind and you won't oversell it to whoever is reading your postmortem.