One average eval score was hiding two different failure modes

A mean faithfulness of 0.75 sounds like a model that is usually right and occasionally slips. Mine was near-perfect on half the data and near-zero on the other half, and 0.75 described neither slice.

I spent a week chasing the wrong problem because I trusted an average. Our RAG system reported faithfulness 0.75 on the eval set, and 0.75 reads like "mostly grounded, some drift." So I tuned the prompt to nudge the drift down. The mean did not move. It did not move because there was no drift to fix. The distribution was not centered anywhere near 0.75. It was two spikes, one clump of scores at 0.95-plus and another clump at 0.05-ish, and the mean was sitting in the empty valley between them where almost no example actually lived.

This is the oldest trap in descriptive statistics, and Anscombe made it unforgettable in "Graphs in Statistical Analysis" (The American Statistician, 1973, 27(1), 17-21). His four datasets share the same mean, variance, correlation, and regression line, and look completely different the moment you plot them. Eval scores are Anscombe's point with a modern coat of paint: the summary statistic is identical across wildly different realities, and only the shape tells you which reality you are in.

Why the mean is the wrong summary for eval scores specifically

There is a reason this bites eval work harder than most data. A lot of eval scores are not smooth quantities. Faithfulness, correctness, pass/fail graded on a rubric: these pile up at the extremes. An answer is grounded or it is fabricated. The judge says yes or no. When your scores clump at 0 and 1, the mean stops describing a typical case and starts describing a mixing ratio. A faithfulness of 0.75 does not mean "the average answer is 75% faithful." It means "roughly 75% of answers are faithful and the rest are not," and those two readings call for completely different actions. The first says polish. The second says find the 25% and figure out what they have in common.

The mean is a fine summary when the distribution is unimodal and roughly symmetric. It is an actively misleading summary when the distribution is bimodal, because the center of mass lands where no data is. And clumped-at-0-and-1 eval scores are bimodal by construction.

Criterion one: look at the distribution before you report the mean

The cheapest possible defense is to plot the scores before you quote a single number. Not summary stats. The histogram. Thirty seconds tells you whether you are looking at one mode or two.

import numpy as np

rng = np.random.default_rng(0)

# A model that is great on 'in-domain' queries and terrible on 'out-of-domain',
# 60/40 split. Beta draws keep scores in [0,1] and clumped near the extremes.
in_domain  = rng.beta(12, 1.2, size=600)   # clustered high, near 0.9-1.0
out_domain = rng.beta(1.2, 12, size=400)   # clustered low, near 0.0-0.1
scores = np.concatenate([in_domain, out_domain])

print(f"mean faithfulness: {scores.mean():.3f}")   # ~0.58, down in the near-empty valley

# a text histogram so this runs with no plotting deps
edges = np.linspace(0, 1, 11)
counts, _ = np.histogram(scores, bins=edges)
for lo, hi, c in zip(edges[:-1], edges[1:], counts):
    print(f"{lo:.1f}-{hi:.1f} | {'#' * (c // 8)} {c}")

Run that and the bars tell the story the mean hid: two towers, one low, one high, and a hollow middle where 0.58 is pointing. The number says "average student." The shape says "two students, one acing, one failing, sitting at the same desk."

Criterion two: report percentiles, and specifically look for the hollow middle

If you must reduce the distribution to numbers, percentiles carry the shape that the mean throws away. A unimodal distribution has its mass near the median. A bimodal one has a median stranded in the gap, with the 25th and 75th percentiles pulled toward the two separate clumps.

qs = [10, 25, 50, 75, 90]
pct = np.percentile(scores, qs)
for q, v in zip(qs, pct):
    print(f"p{q}: {v:.3f}")

# a crude bimodality flag: how much mass sits in the middle third [0.33, 0.66]
# vs the outer thirds. A hollow middle is the tell.
middle = ((scores >= 0.33) & (scores <= 0.66)).mean()
outer  = 1 - middle
print(f"mass in middle third: {middle:.2f}   outer thirds: {outer:.2f}")

When the middle third holds far less mass than the outer thirds, your average is describing a place your data avoids. That is the signal to stop reporting one number and start splitting the data.

Criterion three: break it down by slice, because the mean is a fossil of a real boundary

Bimodality is rarely random. It almost always means there is a variable you have not conditioned on: query type, source document, language, prompt template, tenant. The two spikes are two populations wearing one average. The fix is to find the axis that separates them and report per-slice means.

# reconstruct the slice we secretly built above and report per-slice
labels = np.array(["in_domain"] * 600 + ["out_domain"] * 400)

for slice_name in np.unique(labels):
    s = scores[labels == slice_name]
    print(f"{slice_name:11s} n={len(s):4d}  "
          f"mean={s.mean():.3f}  p50={np.median(s):.3f}")
# in_domain   n= 600  mean~0.92
# out_domain  n= 400  mean~0.09

Now the numbers are actionable. The aggregate 0.72 told me to tune. The per-slice split told me the model is fine and my retrieval falls off a cliff on out-of-domain queries, which is a different bug in a different subsystem. I had been editing prompts to fix a retrieval gap. The average sent me to the wrong file.

FAQ

Is the median a safe replacement for the mean here?

No, and that is the trap. On a 60/40 bimodal split the median just lands inside whichever clump holds more than half the mass. In the code above it comes back around 0.82, sitting up in the high clump and hiding the low population entirely. The median is not more honest than the mean on bimodal data. It is dishonest in a different direction. The distribution or a per-slice breakdown is the fix, not a different single number.

How do I detect bimodality automatically instead of eyeballing histograms?

For a quick automated flag, the dip test (Hartigan and Hartigan, 1985) tests unimodality directly and is a reasonable gate in CI. A cruder version is the middle-mass check above: if the central third holds much less mass than the outer thirds, flag it for a human to look at the plot. Automated flags are for triage. The plot is still where you understand it.

My scores are continuous and unimodal. Do I still need this?

Less urgently. If you have plotted the distribution and it is a single hump, the mean is a defensible summary and you can report it with a standard deviation. This whole problem is specific to distributions that clump, which is most rubric-graded and binary eval scores and fewer of your latency-style continuous metrics.

Open question

Per-slice reporting works when you already know the slicing variable. The hard case is when you do not: the distribution is clearly bimodal, but none of your logged metadata separates the two clumps cleanly. You know two populations exist, you just cannot name the boundary. Clustering the failures on their embeddings or their input features is the obvious move, but I have found the clusters are often not the ones a human would draw, and naming them post hoc invites the same cherry-picking these methods are supposed to prevent. What is the honest workflow for discovering the hidden slice when the data insists there is one but will not tell you what it is?