LLM-as-Judge Position Bias: Measure It Before You Ship

Take your pairwise eval set. Run it once as (A, B), then run the identical set again as (B, A) with nothing else changed. Count how often the judge picks the same answer — not the same slot. If you have never done this, brace yourself: on hard, closely-matched pairs it is normal for a strong judge to contradict itself on a double-digit percentage of comparisons. Those flips are not noise you can average away by adding more eval examples. They are a systematic offset, and they are why the model you shipped "won" your eval.

LLM-as-judge position bias is the tendency of a judge model to prefer a candidate because of where it sits in the prompt rather than what it says. It is the single most common reason internal eval leaderboards fail to replicate.

Key takeaways

• Position bias is systematic, not random. More eval examples shrink variance, not bias. Only swapping order cancels it.
• Measure it with a two-pass swap. Report consistency rate (agreement between the two orderings) and positional preference (how far the first-slot win rate sits from 50%) as first-class eval metrics.
• Fix it by scoring each pair in both orders and treating disagreement as a tie, then aggregate with a tie-aware Bradley–Terry model plus bootstrap confidence intervals.
• Pointwise 1–10 scoring dodges position bias but buys you scale compression and drift — most probability mass lands on 7 and 8, and the meaning of "8" wanders across runs.
• Verbosity and self-preference are separate confounds. Control for length explicitly, and never let a model family judge its own outputs in a head-to-head that decides a launch.

Why does an LLM judge prefer the first answer at all?

Because a decoder-only transformer does not see two candidates side by side in any symmetric way. Answer A is conditioned on the rubric alone; answer B is conditioned on the rubric plus all of answer A. Those are different computations over different KV states, so there is no architectural reason the judge's verdict would be invariant to the swap.

Three mechanisms stack on top of that asymmetry:

1. Anchoring in the residual stream. The first candidate establishes a reference for "what a good answer looks like here." The second is evaluated as a delta against it. Deltas read as deviations, and deviations read as flaws.
2. Recency and the answer template. If you ask the judge to end with Verdict: A or Verdict: B, the token nearest the decision point is influenced by whatever was just read. Some judges skew late rather than early — direction varies by model and by prompt format, which is exactly why you must measure it per-setup instead of assuming a fixed direction.
3. RLHF-shaped output priors. Judges trained to be helpful have opinions about formatting: headers, bullets, hedging, length. Those priors interact with slot position in ways that are not stable across prompt templates.

The practical consequence: position bias is a property of (judge model, prompt template, task domain) — not of the judge model alone. Change the rubric wording and the offset changes. So it has to be re-measured whenever the harness changes, not measured once and assumed.

How do you measure LLM-as-judge position bias?

Run every pair twice with the order swapped and compute two numbers:

• Consistency rate = fraction of pairs where both orderings pick the same underlying answer. This is your ceiling on judge reliability.
• Positional preference = first-slot win rate across all comparisons. A perfectly unbiased judge sits at 0.5. Anything meaningfully off 0.5 is a slot preference, not a quality signal.

import asyncio
from anthropic import AsyncAnthropic

client = AsyncAnthropic()
JUDGE = "claude-sonnet-4-5"

RUBRIC = """You are grading two candidate answers to the same question.
Judge only on: factual correctness, then instruction adherence, then concision.
Do NOT reward length, formatting, or confident tone.
Reply with exactly one line: A, B, or TIE."""

async def judge_once(question, first, second):
    msg = await client.messages.create(
        model=JUDGE,
        max_tokens=8,
        temperature=0,
        system=RUBRIC,
        messages=[
            {"role": "user", "content":
                f"<question>{question}</question>\n"
                f"<answer_A>{first}</answer_A>\n"
                f"<answer_B>{second}</answer_B>"},
            # Prefill forces the label token immediately, no preamble to parse.
            {"role": "assistant", "content": "Verdict:"},
        ],
    )
    return msg.content[0].text.strip().upper()[:1]  # 'A' | 'B' | 'T'

async def judge_pair(question, x, y):
    """Return winner in terms of x/y, order-cancelled."""
    fwd, rev = await asyncio.gather(
        judge_once(question, x, y),   # x in slot A
        judge_once(question, y, x),   # x in slot B
    )
    # Map both verdicts back onto x/y identity.
    fwd_x = fwd == "A"
    rev_x = rev == "B"
    if fwd == "T" or rev == "T" or fwd_x != rev_x:
        return "tie", (fwd, rev)      # disagreement == no reliable signal
    return ("x" if fwd_x else "y"), (fwd, rev)

Note temperature=0 and max_tokens=8. You want the judge's modal verdict, and you want the flips you observe to come from ordering rather than from sampling. (Temperature 0 is not bit-deterministic under batched serving, but it removes the dominant source of run-to-run variance.)

Two other details in that snippet matter more than they look:

• The assistant prefill ("Verdict:") eliminates preamble tokens and makes parsing exact. It also stops the judge from writing a paragraph of rationalization that it then has to stay consistent with — free-form reasoning before a verdict tends to amplify whichever slot it started narrating about.
• Neutral slot names. Use A/B or answer_1/answer_2, never baseline vs candidate. Semantically loaded labels leak your hypothesis into the judge.

If you do want chain-of-thought in the judge — and for hard reasoning tasks it genuinely helps accuracy — keep it, but re-measure consistency with and without it. CoT usually raises agreement with human labels and simultaneously raises position sensitivity. Those trade off; know which side you are on.

Should you just use pointwise 1–10 scoring instead?

Pointwise scoring removes position bias by construction, and it introduces two worse problems.

Scale compression. Ask a modern instruction-tuned model to score on 1–10 and the distribution collapses onto 7–8. You lose almost all discriminative power in exactly the region where your candidates live. A 1–10 scale with a compressed mode is effectively a 3-point scale with extra steps.

Drift. "8" means something slightly different across prompt revisions, model versions, and even batches. Pairwise comparison is relative and self-normalizing; pointwise is absolute and has no anchor.

If you need pointwise — and you do for regression monitoring, where you have no natural opponent — anchor it hard: give 2–3 fixed exemplar answers with their scores inside the rubric, and use a short discrete scale with explicit criteria per level (a 4-point scale where each level is a described behavior beats a 10-point scale of vibes). Then run a pinned reference set every time and track the distribution shift on that set, not just the mean. If your anchors' scores move, the judge moved, and every other number in that run is suspect.

How do you turn swapped comparisons into a ranking you can trust?

Use a tie-aware Bradley–Terry (Rao–Kupper) fit over your order-cancelled outcomes, and bootstrap it. Do not just count wins.

import numpy as np
from scipy.optimize import minimize

def fit_bt(records, n_models, tie_scale=1.0):
    """records: list of (i, j, outcome) with outcome in {1: i wins, 0: j wins, 0.5: tie}"""
    def nll(theta):
        s = 0.0
        nu = tie_scale
        for i, j, o in records:
            d = theta[i] - theta[j]
            p_i = 1.0 / (1.0 + np.exp(-d) * (1.0 + nu))
            p_j = 1.0 / (1.0 + np.exp(d) * (1.0 + nu))
            p_t = max(1.0 - p_i - p_j, 1e-9)
            s -= np.log(p_i if o == 1 else p_j if o == 0 else p_t)
        return s + 1e-3 * np.dot(theta, theta)   # ridge, kills separation blowups
    theta = minimize(nll, np.zeros(n_models), method="L-BFGS-B").x
    return theta - theta.mean()

def bootstrap_ci(records, n_models, B=1000, seed=0):
    rng = np.random.default_rng(seed)
    idx = np.arange(len(records))
    draws = np.array([
        fit_bt([records[k] for k in rng.choice(idx, len(idx), replace=True)], n_models)
        for _ in range(B)
    ])
    return draws.mean(0), np.percentile(draws, [2.5, 97.5], axis=0)

Ties are load-bearing here. If you collapse "the judge flipped when swapped" into a coin-flip win, you inject the judge's slot preference straight back into the ratings. Treating it as an explicit tie tells the model this pair carries no information about which is better, which is the truth.

And report the bootstrap interval. With a few hundred comparisons, rating gaps that look decisive routinely have overlapping 95% intervals. A leaderboard without intervals is a ranking of sampling noise.

What about verbosity and self-preference?

They are separate biases that swapping does not fix, because they attach to the answer, not the slot.

Verbosity. Judges reward length well past the point where length adds information. Diagnose it by regressing outcome on token-count delta across your comparison set. If longer answers win at a rate that tracks length independent of correctness, your rubric is not controlling it. Two mitigations that actually work: state the criteria in strict priority order (correctness → adherence → concision, as in the snippet above), and add a length-matched slice to your eval set where both candidates are constrained to similar token budgets. If the effect survives length matching, it was real quality; if it vanishes, it was verbosity.

Self-preference. Models tend to prefer text distributed like their own. If Claude Opus 4.x is judging a head-to-head that includes a Claude candidate, you have a conflict of interest. Use a judge from a different family for launch-gating comparisons, or run a panel of judges from different families and require agreement. Panel disagreement rate is itself a useful signal — pairs where judges from different families disagree are usually pairs where "better" is genuinely underdetermined, and those belong in front of a human.

Failure modes worth a checklist

• Measured bias once, never again. It is per-template. Re-measure on every rubric edit.
• Reused the swapped run as two data points. Forward and reverse are one comparison, not two. Doubling them inflates your effective sample size and shrinks your CIs by a bogus √2.
• Cached the forward pass and not the reverse. With prompt caching, the shared rubric prefix is cacheable but the candidate order is not — cost per pair is closer to 2× than to 1×. Budget for it, or reserve swapping for the pairs whose forward verdict is low-confidence.
• No human-agreement anchor. Hold out a few hundred human-labeled pairs and report judge–human agreement alongside consistency. High self-consistency with low human agreement means you built a precise instrument pointed at the wrong thing.

The short answer

LLM-as-judge position bias is a systematic preference for a candidate's slot in the prompt, caused by the asymmetric conditioning of a decoder-only model over two candidates and amplified by anchoring and RLHF output priors. You measure it by running every pair in both orders and reporting consistency rate and first-slot win rate; you correct it by scoring both orders, treating disagreement as an explicit tie, and aggregating with a tie-aware Bradley–Terry fit with bootstrap confidence intervals. Pointwise scoring avoids the bias but substitutes scale compression and drift, so use it only with anchored exemplars and a pinned reference set. And keep verbosity and self-preference in a separate column — swapping does nothing for either.