LLM-as-a-Judge Bias: Why Your Eval Score Climbs While Your Answers Get Worse

Here's an experiment you can run on your eval setup in five minutes.

Take one question from your test set and two candidate answers. Ask your LLM judge which is better. Then ask again with the two answers in the opposite order.

If the verdict flips, your judge just told you something about itself, not about the answers. And if you've been tuning prompts against that judge for weeks, some of what you've been shipping is the judge's taste.

This post covers the three documented biases of LLM judges, why they compound when you iterate against an eval, and the code to design them out and calibrate what's left.

What LLM-as-a-judge is, and why it's mostly a good idea

LLM-as-a-judge means using a strong model to grade the outputs of your AI feature, either pairwise ("which of these two answers is better?") or against criteria ("does this answer do X?"). It exists because checking AI output is the expensive part of building with it, and AI looks smartest exactly where you can't check it.

The approach holds up better than you might expect. In the MT-Bench study (Zheng et al., 2023), GPT-4's pairwise verdicts agreed with human experts 85% of the time on non-tie votes, while the experts agreed with each other 81% of the time.

The catch: an average agreement rate says nothing about which way the disagreements lean. Random disagreements wash out. Disagreements that all lean the same way don't.

The three biases, with the evidence

Position bias

The judge's verdict depends on which answer it sees first.

• Wang et al. (2023): with ChatGPT as the judge, reordering alone let Vicuna-13B beat ChatGPT on 66 of 80 queries.

• Zheng et al. (2023): after swapping answer order, GPT-4 gave a consistent verdict 65% of the time. The weaker judges in the study did worse.

Length (verbosity) bias

The judge rewards more words, even when they add nothing.

• Zheng et al. took answers with numbered lists, had a model rephrase the list with no new information, and prepended it, roughly doubling the length. Two of three judges preferred the padded answer 91.3% of the time. GPT-4 failed this test 8.7% of the time.

• Length-Controlled AlpacaEval (Dubois et al., 2024): one model's win rate ranged from 22.9% to 64.3% depending only on whether it was prompted to be more or less verbose. Their length-controlled metric narrowed that to 41.9%–51.6%.

Self-preference bias

The judge favors outputs that sound like it.

• Panickssery et al. (2024): LLM evaluators rated their own outputs higher than other outputs humans judged similar in quality, and self-preference grew with the model's ability to recognize its own text. The study used summarization tasks, and the authors stop short of calling the link proven.

Caveat that matters: these are 2023–2024 models. Newer judges are stronger. Treat the numbers as proof the bias can exist, not as an estimate of yours. You find out your judge's bias by measuring it.

Why iterating against a judge makes bias compound

A one-off eval absorbs a small bias as noise. A prompt-tuning loop doesn't, because a loop is a selection process:

1. Change the prompt.

2. The judge compares old vs new.

3. The slightly longer answer wins slightly more often.

4. Keep the winner. It's the new baseline.

Changes that add words get a small unearned edge every round. Changes that cut words get a small unearned penalty, which means a "make answers shorter" PR loses the eval and gets reverted with a chart to justify it. After twenty rounds, the bias is the direction your product moved.

If the generator and judge come from the same model family, self-preference adds a second pull toward that family's house style.

A bias that's a rounding error in one eval becomes the direction of your product after twenty.

Building a judge that can't be swayed by its favorites

Telling the judge "ignore length" is a request, not a control. You can't verify it by reading the prompt. Adding few-shot examples helped in Zheng's study (GPT-4's swap consistency went from 65% to 77.5%), but at roughly 4× the cost and with no gain in human agreement. The useful fixes are structural.

1. Swap order and require agreement

def judge_pair(question, a, b, judge):
    """Return 'A', 'B', or 'tie'. A win has to survive the swap."""
    first = judge(question, left=a, right=b)    # returns 'left' or 'right'
    second = judge(question, left=b, right=a)
    if first == "left" and second == "right":
        return "A"
    if first == "right" and second == "left":
        return "B"
    return "tie"  # the verdict depended on order

This is the conservative rule Zheng et al. used in their own experiments. It doubles judge calls and gives you a free metric: the tie rate. A high tie rate is a property of your judge, not your prompts. (Pinning the judge to temperature 0 doesn't fix this; it hides the doubt instead of removing it.)

2. One binary criterion per call

Replace "rate helpfulness 1–10" with a specific yes/no property, and give the judge the reference material when one exists:

You are checking one property of a customer support answer.

Policy excerpt:
{policy}

Customer question:
{question}

Answer to check:
{answer}

Does the answer state the refund window exactly as the policy gives it?
Write one or two sentences of reasoning, then PASS or FAIL on its own line.

Reasoning comes before the verdict on purpose. If you use structured output, keep that order in the schema too; field order decides what the model has already "said" when it writes the verdict (more on that here). Reference answers had a big effect in Zheng's math subset: judge failures dropped from 14/20 to 3/20.

3. Use a different model family as the judge

If your feature runs on one provider's model, grade with another's. It removes the self-preference pull entirely.

4. Log length next to every score

import statistics

def length_report(rows):
    """rows: dicts with 'variant', 'answer', 'won' (bool) from one eval run."""
    by_variant = {}
    for r in rows:
        by_variant.setdefault(r["variant"], []).append(r)
    for name, rs in by_variant.items():
        words = [len(r["answer"].split()) for r in rs]
        win_rate = sum(r["won"] for r in rs) / len(rs)
        print(f"{name}: win rate {win_rate:.0%}, median {statistics.median(words)} words")

If win rate and median length climb together across versions, investigate before shipping. Where you can, compare candidates at similar lengths or make a word budget an explicit criterion.

5. Run a pad test on your own judge

A homemade version of Zheng's padding experiment:

PAD_PROMPT = (
    "Rewrite the answer below at roughly twice the length. "
    "Restate and expand the existing points only. "
    "Add no new facts, steps, or caveats.\n\n{answer}"
)

def pad_test(cases, rewrite, judge):
    """cases: (question, good_answer) pairs you've labeled as good."""
    padded_wins = 0
    for question, answer in cases:
        padded = rewrite(PAD_PROMPT.format(answer=answer))
        if judge_pair(question, answer, padded, judge) == "B":
            padded_wins += 1
    return padded_wins / len(cases)  # should be close to 0

Ten labeled-good answers is enough to see whether "ignore length" in your judge prompt is doing anything.

Calibrating the judge against your own labels

Treat the judge like a new reviewer on probation. Hand-label 50–100 real outputs pass/fail yourself, run the judge on the same set, and split the errors:

def judge_report(labels, verdicts):
    """labels and verdicts: lists of True (pass) / False (fail), same order."""
    pairs = list(zip(labels, verdicts))
    human_fails = [j for h, j in pairs if not h]
    human_passes = [j for h, j in pairs if h]
    return {
        "agreement": sum(h == j for h, j in pairs) / len(pairs),
        "missed_failures": sum(human_fails) / max(len(human_fails), 1),
        "false_alarms": human_passes.count(False) / max(len(human_passes), 1),
    }

Why not just agreement? If 90% of outputs are good, a judge that passes everything agrees with you 90% of the time and catches nothing. Missed failures is usually the number that matters.

Expect your criteria to move while you label. Shankar et al. (2024) call this criteria drift: you need criteria to grade, but grading is how you discover your criteria. So label first, then write the judge prompt from your disagreements, not the other way round.

The Judge Audit checklist

1. Swap test: grade both orders; a win counts only if both agree.

2. Pad test: double-length restatements of good answers shouldn't win.

3. Family test: grader and writer come from different model families.

4. Label test: compare against 50–100 human labels; watch missed failures.

5. Drift test: re-run the label test when the judge model, judge prompt, or your criteria change.

Spend the full audit on judges you optimize against: release gates, per-PR checks, prompt-tuning loops. One-off comparisons can skip most of it.

Key Takeaways

• LLM judges agree with humans about as often as humans agree with each other, but their disagreements lean in consistent directions: position, length, and self-preference.

• Iterating against a judge turns a small bias into the direction of your product.

• Prompt instructions like "ignore length" are requests. Swapping order, binary criteria, reference answers, and a different model family are controls.

• Log answer length next to every eval score.

• Calibrate with 50–100 human labels and track missed failures, not just agreement.

• Label before you write the judge prompt; your criteria will change once you read real outputs.

FAQ

What is LLM-as-a-judge?

LLM-as-a-judge is an evaluation method where a language model grades the outputs of another AI system, either by picking the better of two answers or by checking an answer against criteria. It replaces slow human review in eval pipelines so teams can test prompt and model changes at scale.

Is LLM-as-a-judge reliable?

On average it can be. In the MT-Bench study, GPT-4 agreed with human experts 85% of the time on non-tie pairwise votes, close to the 81% rate at which experts agreed with each other. The problem is that its errors can lean consistently in one direction, so you should measure your judge against your own labels before trusting trends.

What is position bias in LLM judges, and how do you fix it?

Position bias means the verdict changes depending on which answer is shown first. Fix it by grading every pair in both orders and counting a win only when both orders agree; treat disagreements as ties and track the tie rate as a health metric for the judge.

Do LLM judges prefer longer answers?

Many do. In published tests, padding answers with restated content fooled some judges most of the time, and one benchmark found a model's win rate swung from 22.9% to 64.3% based only on verbosity. Log answer length alongside scores and run a pad test on your own judge.

Should an LLM judge use a 1–10 scale or pass/fail?

Prefer binary pass/fail on one specific criterion per call. A numeric scale has no shared meaning between a 6 and a 7, so score changes are hard to interpret and hard to check against human judgment. Binary verdicts can be compared directly with your own labels.

How many human labels do you need to calibrate an LLM judge?

There's no universal number, but 50–100 labeled outputs is a practical starting point for most features. Make sure the set includes enough failures to measure missed failures meaningfully, and re-run the comparison whenever the judge model, judge prompt, or your criteria change.

Related reading

• Why AI Feels Smarter About Topics You Don't Know: why checking is the bottleneck evals exist to solve

• Temperature 0 Won't Fix Your AI. It Just Hides the Doubt.: why a "stable" judge can still be unsure

• Structured Outputs Still Hallucinate: schema design for judges that return JSON

• How to Review AI-Generated Code: polish vs correctness, the same confusion in a different place

• Start Here: A Plain-English Map to Understanding AI

The bottom line

An eval score is a claim about your product made by a model with opinions. Check the opinions before you act on the claim.

If you never check the judge, you aren't measuring your product. You're measuring how closely it resembles the judge's favorite answer.

Sources: Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (NeurIPS 2023); Wang et al., "Large Language Models are not Fair Evaluators" (ACL 2024); Dubois et al., "Length-Controlled AlpacaEval" (2024); Panickssery et al., "LLM Evaluators Recognize and Favor Their Own Generations" (NeurIPS 2024); Shankar et al., "Who Validates the Validators?" (UIST 2024).

Adam Jaber is a software engineer who writes Simply Explained: complex topics, made simple. No jargon, no hype. Get one of these in your inbox each week → subscribe to the newsletter.