Sampling and voting

Every reliability tool so far reacts to something: a schema violation, a timeout, a 429. This one is for the call that gives you nothing to react to. It returned a valid answer, in the right shape, and the answer is wrong.

1.Why a second call can disagree with the first

Retries and fallbacks works because a failure carries a message you can attach to the next attempt. A ticket rated urgency: 2 when it deserved a 4 carries nothing. It is an integer in range and a plausible reading of an ambiguous ticket. Your code cannot tell that anything went wrong.

So the question is what calling again could buy you. More than it seems. Generation draws from a distribution rather than always taking the top token, and Sampling, and the knobs providers are taking away in the LLMs course explains the mechanism. Whether you can still set the temperature depends on the provider and the model, so check yours. Where you can, a low value narrows the spread without removing it. The Wharton team (Meincke and colleagues, June 2025, older models) ran every question 25 times at temperature 0, and questions still flipped between runs. They report "100 percent correct" and mostly correct as separate measures because the two differ.

Five calls to one prompt are five independent draws. If the model is merely noisy on a ticket, the noise shows up as disagreement, and disagreement is something you can count.

A retry reacts to a failure you can see. A vote listens for a doubt you cannot.

2.Counting the votes

How you combine answers depends on their shape. Closed answers vote cleanly. A label, an enum, a digit from one to five, yes or no: take the most common one. This is the counting idea behind self-consistency, from Wang et al.'s paper on chain-of-thought, here without the reasoning chain. The idea is that wrong derivations scatter, while right ones tend to land on the same answer. Open prose does not vote. Five summaries are five different good paragraphs, with no majority among them. Picking the best needs a judge, which is a different technique and belongs to the Evals course.

Use an odd N so that two labels cannot split evenly. With three or more labels a tie is still possible, which is why the rule below asks for more than half of N, not merely the most.

from collections import Counter
from concurrent.futures import ThreadPoolExecutor

from anthropic import Anthropic

client = Anthropic()
MODEL = "claude-sonnet-5-5"   # as of October 2026
SYSTEM = "Rate this support ticket's urgency from 1 (can wait) to 5 (outage). Reply with the digit only."

def text_of(resp) -> str:
    return "".join(b.text for b in resp.content if b.type == "text")

def ballot(ticket: str) -> str | None:
    resp = client.messages.create(
        model=MODEL,
        max_tokens=20,
        thinking={"type": "between_tools"},   # the off setting for a one-digit answer
        system=SYSTEM,
        messages=[{"role": "user", "content": ticket}],
    )
    if resp.stop_reason != "end_turn":
        return None   # cut off or refused: a spoiled ballot, not a vote
    digit = text_of(resp).strip()
    return digit if digit in {"1", "2", "3", "4", "5"} else None

def vote(ticket: str, n: int = 5):
    with ThreadPoolExecutor(max_workers=n) as pool:
        ballots = [b for b in pool.map(lambda _: ballot(ticket), range(n)) if b]
    counts = Counter(ballots)
    if not counts:
        return None, counts
    winner, top = counts.most_common(1)[0]
    # Spoiled ballots still count toward n, so a damaged vote needs more agreement.
    return (winner if top > n / 2 else None), counts   # None means escalate

The parameter spellings come from the vendor docs as of October 2026 and were not run here, so try them on your SDK version first. The pool is doing the work: five calls fired together, not five in a row. The majority line is the rule you set in advance. It folds ties, three-way splits and mostly spoiled ballots into one outcome, and that outcome is a person or a stronger model, never a coin flip. Errors are not caught here: an API error inside pool.map propagates and kills the vote, and Retries and fallbacks covers them.

3.What it costs, where it fails, and when to skip it

Five votes cost five times the tokens, billed whether they agree or not. What you save is time, since the calls run side by side. One cost tip comes from the vendor's caching page: a cache entry exists only after the first response begins. If the votes share a long prompt, above the 512-token minimum for this model as of October 2026, fire one call first and the other four after it. Otherwise all five may pay the write price and none reads. That trade is a little extra latency for a cheaper bill, so only make it for a long shared prefix.

The failure case is the third run in the figure. Voting measures how much the draws disagree, not whether they are right. When the model lacks a fact, the miss is correlated: every draw makes it, and the vote confirms it with a straight face. The fix is in the prompt, or in an example of the policy, as in Examples beat explanations. More votes do not help. Treat a unanimous result as "no noise detected", and keep a sample of unanimous cases in your tests.

Score a vote by how stable each case is across repeated runs of vote(), not by an average, since the user gets one answer; Versioning and testing prompts is where that harness lives.

When not to use it.

• A decisive check exists: a schema, a lookup, a test. Use the check, which is cheaper and certain.
• The stakes are low. A mislabelled newsletter is not worth five calls.
• The answer is open prose, or has no short comparable form.
• You are close to your rate limits. Five votes spend five times the request and token budget.
• The model is confidently wrong. That calls for a better prompt, not more draws.

There is also a rival to test. For a hard closed question, one call at higher effort may beat five votes at default, or the reverse. The vendor's steering page offers effort as the first lever but publishes no comparison, and none of the sources used here measures one, so run both on your own cases. The effort arm needs thinking back on and a larger max_tokens, because between_tools only accepts high effort or lower. Changing effort between calls also breaks the cache, so set it once per conversation.

Voting buys a disagreement signal. It is a reliability tool, not a guarantee of correctness. Say that in the postmortem before someone else does.

Next: you now have several tools and need to know whether a change helped, so Versioning and testing prompts opens Part 4 with the harness.