LLM Eval Noise: Run an A/A Test Before You Trust a Prompt Win

Here's a PR description you've probably approved:

Tightened the citation instruction. Eval: 82 → 87 on the 100-case set. LGTM?

The diff is one line. The evidence is two numbers. And there's a decent chance the old prompt, run a second time with nothing changed, would also score 87.

This post is the practical version of a simple idea: an eval score has a noise floor, and you should measure it before you trust any change against it. You'll get three tools (an A/A test, a flip count, and a holdout rule) and about 30 lines of Python that implement them.

Three sources of noise in an LLM eval

A pass/fail eval score moves for three reasons that have nothing to do with your prompt getting better:

Question noise has a closed form. For pass rate p on n cases, the standard error is sqrt(p * (1 - p) / n):

On 100 cases, "85%" means "somewhere from 78 to 92". That's not pessimism; it's the arithmetic Evan Miller (Anthropic) argues for in Adding Error Bars to Evals (2024): report evals with standard errors like any other experiment.

Answer noise is the one most people underestimate. Even temperature 0 often isn't fully repeatable on hosted models (more on why in Temperature 0 Won't Fix Your AI. It Just Hides the Doubt.). Grader noise and grader bias are covered in Your AI Judge Has Favorites.

Step 1: Run an A/A test

Borrowed straight from web experimentation: before trusting an A/B setup, run both arms on the identical variant. Any "winner" is pure noise, and its size tells you how loud the noise is.

For prompts: score the current prompt twice on the same cases. Record two things:

1. The gap between the runs. That's your floor.
2. The flipped cases, the ones that passed in one run and failed in the other. That list is your set of unstable cases, and it's the most informative slice of your test set.

To get a feel for the size, here's a toy model: 100 cases, of which 75 always pass, 10 always fail, and 15 are coin flips. True score 82.5.

import numpy as np

rng = np.random.default_rng(0)
RUNS = 200_000

def score(size):
    # 100 cases: 75 always pass, 10 always fail, 15 coin flips
    return 75 + rng.binomial(15, 0.5, size)

a, b = score(RUNS), score(RUNS)                     # A/A: same prompt twice
print("A/A gap >= 4:", np.mean(abs(b - a) >= 4))    # ~0.20
print("A/A gap >= 5:", np.mean(abs(b - a) >= 5))    # ~0.10

Two runs of the same prompt land 4+ points apart about one time in five, and 5+ points apart about one time in ten. The PR above claims +5. In this toy world, the identical copy manages that about one time in twenty, in the upward direction alone.

Your eval isn't the toy model. That's why you run the A/A on yours.

Step 2: Compare flips, not averages

Store results per case, not per run. Then compare two runs case by case:

import math

def compare(base: dict[str, bool], cand: dict[str, bool]):
    """base/cand map case_id -> passed? for the SAME case ids."""
    fixed  = [c for c in base if not base[c] and cand[c]]
    broken = [c for c in base if base[c] and not cand[c]]
    flips  = len(fixed) + len(broken)
    net    = len(fixed) - len(broken)
    bar    = 2 * math.sqrt(flips)
    return {"fixed": fixed, "broken": broken, "flips": flips,
            "net": net, "bar": round(bar, 1), "clears_bar": abs(net) > bar}

Why 2 * sqrt(flips)? If a change did nothing and every flip were a coin toss, the net gain would scatter around zero with a standard deviation of sqrt(flips). Twice that is a rough 95% line. It's a mental-math version of McNemar's test, the standard test for paired yes/no outcomes.

Both prompts above report +5. Only one of them is a fix:

• Prompt B: 5 fixed, 0 broken. Bar = 4.5. Clears it, narrowly.
• Prompt C: 15 fixed, 10 broken. Bar = 10. Nowhere near. It churned a quarter of the test set.

With fewer than about ten flips, use the exact test instead of the shortcut:

from scipy.stats import binomtest

def exact_p(n_fixed: int, n_broken: int) -> float:
    # exact McNemar = two-sided sign test on the flipped cases only
    return binomtest(n_fixed, n_fixed + n_broken, 0.5).pvalue

exact_p(5, 0)    # 0.0625  -> promising, confirm on holdout
exact_p(15, 10)  # 0.42    -> coin toss

Pairing also buys precision. Miller's paper notes that when two variants agree on which questions are hard, a paired comparison cancels that shared difficulty, so you can detect the same effect with fewer cases than comparing two independent averages.

And read the broken list. Cases a change broke usually share a cause, often an instruction that over-applies to inputs it was never meant for.

Step 3: Beat the winner's curse with a holdout

Trying twelve rewrites and keeping the best one is the most reliable way to manufacture a win from nothing. The max of noisy scores selects for favorable noise.

best_of_12 = score((RUNS, 12)).max(axis=1)          # 12 IDENTICAL "variants"
print("best of 12 beats baseline by >= 4:",
      np.mean(best_of_12 - a >= 4))                 # ~0.43

Twelve copies of one prompt. The "winner" averages 85.6, about three points above a baseline run, and beats it by 4+ points 43% of the time. Ship it and production comes in lower. That's regression to the mean, not drift.

The fix:

• Dev split: iterate as much as you like.
• Holdout split: run the chosen variant against the baseline once per ship decision. If you tweak after seeing it, it's a dev set now. Refresh it from production logs.

Step 4: Buy precision when the win is genuinely small

• Repeat each case K times. Miller shows averaging K samples per question divides the answer-noise variance by K. If your eval calls share a long, byte-identical prefix, prompt caching cuts the cost of repeats (how prompt caching breaks).
• Don't drop temperature to hide variance. The same paper warns it can bias scores. Users still hit those knife-edge cases at your shipping temperature.
• Add cases that look like your flips. Random cases help with √n (4× the cases for 2× the precision). Cases near the unstable ones help faster.
• Count clusters, not rows. Thirty cases from five documents aren't thirty independent cases. On one real benchmark in Miller's paper, clustered standard errors were more than 3× the naive ones.
• Log per case, per run: case_id, prompt_version, run, output, verdict. Nothing above works without it.

The Noise Floor Protocol

1. Run A/A before A/B. Score the current prompt twice; keep the gap and the flip list.
2. Compare case by case. Fixed vs broken, for every change.
3. Clear the bar. Net gain beats both the A/A gap and 2 * sqrt(flips). Under ten flips, use exact_p.
4. Explore on dev, confirm once. One holdout look per ship decision.
5. Report the floor with the score. 87 vs 82, 25 flips, A/A gap 4, never just 87.

Skip all this when the change is huge (60 → 90), when you're fixing one specific failure and can read that it stopped, or when it's cheap to revert and you're watching production. It matters most in the long tail of polishing, where every change is worth two or three points.

Key Takeaways

• An eval score is a measurement with three noise sources: answers, questions, and grader.
• On 100 pass/fail cases at 85%, the 95% range is about ±7 points.
• An A/A test (same prompt, two runs) measures your noise floor and surfaces unstable cases.
• Compare flips, not averages. A net gain under 2 * sqrt(flips) is a coin toss; use exact McNemar under ~10 flips.
• Picking the best of many variants on one set is the winner's curse. Confirm on a holdout you touch once.
• Shrink noise with repeat runs, targeted cases, and cluster-aware counting, not by lowering temperature.

FAQ

How many test cases do I need for an LLM eval?

It depends on the size of the change you want to detect. For pass/fail scoring, the standard error is sqrt(p(1-p)/n): at 85%, 100 cases give about ±7 points (95%), 300 give ±4, and 1,000 give ±2.2. Paired comparisons on the same cases need fewer than that.

What is an A/A test for prompts?

Running your current prompt twice on the same test set with nothing changed. The score gap between the two runs is your noise floor, and the cases that flipped are your unstable cases. Any improvement smaller than the A/A gap isn't visible to that eval.

Is a 5-point improvement on 100 test cases statistically significant?

Not on its own. It depends on how many cases flipped. Five fixed and none broken gives an exact p of about 0.06 (promising); fifteen fixed and ten broken gives about 0.42 (noise). Count the flips, then confirm on a holdout.

What is McNemar's test and why use it for LLM evals?

It's the standard test for comparing two systems on the same yes/no items. It ignores cases both got right or both got wrong and looks only at the disagreements. For prompt evals, that means comparing fixed vs broken cases, which is far more sensitive than comparing two averages.

Should I set temperature to 0 to make evals reproducible?

Not as a way to reduce noise. Temperature 0 often isn't fully deterministic on hosted models, and the Adding Error Bars to Evals paper warns that lowering temperature to cut variance can bias the score. Repeat each case a few times and average instead.

What is the winner's curse in prompt engineering?

When you try many prompt variants on the same test set and keep the highest scorer, you're partly selecting for lucky noise, so the winner's score overstates its real quality. In a simple simulation, the best of 12 identical prompts beats a baseline run by 4+ points 43% of the time. A holdout set fixes it.

Related reading

• Temperature 0 Won't Fix Your AI. It Just Hides the Doubt.: where answer noise comes from
• Your AI Judge Has Favorites: grader bias, the third noise source
• Prompt Caching for AI Agents: Stop Breaking Your Own Cache: making repeat eval runs cheaper
• Structured Outputs Still Hallucinate: another place a passing check hides a guess
• New to the series? Start Here

Get one of these in your inbox each week: Simply Explained on Substack.

The bottom line

Before comparing your new prompt to the old one, compare the old one to itself. It costs one run, and it tells you how big a win has to be before you can see it.

If the identical copy can win, your win isn't evidence yet.

Adam Jaber is a software engineer who writes Simply Explained: complex topics, made simple. No jargon, no hype.