How to know your LLM app works: building an eval harness

You change one line in a prompt. You try three questions, the answers look better, you ship. A week later a customer gets told their non-refundable order will be refunded. The change helped the three questions you tried and broke a kind of question you didn't.

An eval (evaluation) is a test suite for behaviour that isn't exact. Ordinary unit tests check add(2, 2) == 4. An LLM's answer can be worded a hundred correct ways, can be right 90% of the time, and changes when you touch a prompt, a model version or the retrieval step. An eval harness is the small program that runs your system over a fixed set of cases, scores every answer, and tells you, with honest uncertainty, whether a change made things better or worse.

Part I · The pieces

Chapter 1: Cases are frozen questions with known answers

The problem

To measure anything you need a fixed set of inputs and some notion of a right answer. Two terms:

• Case: one input (a user question, plus any context) and what a good answer must satisfy.
• Eval set: the fixed list of cases. It's frozen: you don't change it while comparing two versions, or the comparison means nothing.

Details, built on the model

Cases are just data. Tags let us later see which kind of question got better or worse.

cases = [
  {"id": "r1", "tag": "refund", "q": "My blender arrived broken",
   "must": ["refund"], "must_not": []},
  {"id": "r2", "tag": "refund", "q": "Item damaged, what now?",
   "must": ["refund", "7 days"], "must_not": []},
  {"id": "p1", "tag": "policy", "q": "Can I return spoiled milk?",
   "must": ["cannot"], "must_not": ["refund"]},
  {"id": "d1", "tag": "delivery", "q": "Parcel never came",
   "must": ["investigat"], "must_not": []},
  {"id": "d2", "tag": "delivery", "q": "Order is late",
   "must": ["voucher"], "must_not": []},
]
print(len(cases), "cases;", sorted({c["tag"] for c in cases}))

5 cases; ['delivery', 'policy', 'refund']

Good cases come from real traffic and real failures: every bug report becomes a case, so the same bug can't come back silently. Real eval sets have hundreds of cases; we use 5 here, and then 200 synthetic ones, to keep the output readable.

Chapter 2: The runner and the scorers

The problem

We need to run a system over every case and mark each answer.

Details, built on the model

Two fake "systems" stand in for two prompt versions, so the article runs offline:

def system_a(q):
    q = q.lower()
    if "milk" in q:
        return "Sorry, perishables cannot be returned."
    if "broken" in q or "damaged" in q:
        return "We will refund you."
    if "never came" in q:
        return "We will investigate with the courier."
    return "Please wait a little longer."

def system_b(q):
    q = q.lower()
    if "milk" in q:
        return "Perishables cannot be returned, but a refund is ok."
    if "broken" in q or "damaged" in q:
        return "We will refund you within 7 days."
    if "never came" in q:
        return "We will investigate with the courier."
    return "Sorry it's late, here is a voucher."

def run(system, cases):
    sheets = {}
    for c in cases:
        try:
            sheets[c["id"]] = system(c["q"])
        except Exception as e:           # a crash is an answer too
            sheets[c["id"]] = f"ERROR: {e}"
    return sheets

def score(case, answer):
    a = answer.lower()
    hits = [m in a for m in case["must"]]
    if any(bad in a for bad in case["must_not"]):
        return 0.0                       # a forbidden claim fails it
    return sum(hits) / len(hits)

for name, sysf in [("A", system_a), ("B", system_b)]:
    sheets = run(sysf, cases)
    marks = {c["id"]: score(c, sheets[c["id"]]) for c in cases}
    print(name, marks, "mean", round(sum(marks.values()) / 5, 2))

A {'r1': 1.0, 'r2': 0.5, 'p1': 1.0, 'd1': 1.0, 'd2': 0.0} mean 0.7
B {'r1': 1.0, 'r2': 1.0, 'p1': 0.0, 'd1': 1.0, 'd2': 1.0} mean 0.8

The scorer here is a rule-based check: fast, free, deterministic. For open-ended answers you'd use an LLM-as-judge (another model given the question, answer and a rubric, asked for a score). The model still holds: it's just a different marker reading the same saved sheets. Judges need their own checking against a few human-marked answers, because a marker that is wrong 20% of the time makes every score fuzzy.

Part II · Trusting the numbers

Chapter 3: Averages hide regressions

The problem

B's mean is higher. Ship it? Look at case p1.

Details, built on the model

from collections import defaultdict

def marks_for(system):
    sheets = run(system, cases)
    return {c["id"]: score(c, sheets[c["id"]]) for c in cases}

ma, mb = marks_for(system_a), marks_for(system_b)
for c in cases:
    a, b = ma[c["id"]], mb[c["id"]]
    if a != b:
        word = "FIXED" if b > a else "BROKE"
        print(f"{word} {c['id']} ({c['tag']}): {a} -> {b}")

by_tag = defaultdict(list)
for c in cases:
    by_tag[c["tag"]].append(mb[c["id"]] - ma[c["id"]])
for tag, diffs in sorted(by_tag.items()):
    print(f"{tag:9} change {sum(diffs) / len(diffs):+.2f}")

FIXED r2 (refund): 0.5 -> 1.0
BROKE p1 (policy): 1.0 -> 0.0
FIXED d2 (delivery): 0.0 -> 1.0
delivery  change +0.50
policy    change -1.00
refund    change +0.25

The mean went up, and B now tells customers spoiled milk can be refunded. That's the exact story from the introduction, caught before shipping.

Chapter 4: Is the difference real? The paired bootstrap

The problem

On a real eval set, B scores 0.81 and A scores 0.78. Is that improvement, or luck in which cases happened to be in the set? With few cases, small differences are often noise.

Two terms:

• Paired: both versions answered the same cases, so we compare them case by case (B − A per case), not as two separate averages. This removes the noise from some cases just being harder.
• Bootstrap: estimate uncertainty by re-sampling. Draw a new "pretend eval set" of the same size from your cases with replacement (some cases picked twice, some not at all), compute the mean difference, repeat thousands of times. The spread of those results shows how much the number could wobble.

Details, built on the model

We simulate 200 cases where B is genuinely better, then bootstrap:

import numpy as np

rng = np.random.default_rng(0)
n = 200
a = (rng.random(n) < 0.78).astype(float)       # pass/fail per case
flip = rng.random(n)
b = a.copy()
b[(a == 0) & (flip < 0.45)] = 1                # B fixes some
b[(a == 1) & (flip > 0.97)] = 0                # and breaks some

def paired_bootstrap(a, b, reps=10_000, seed=1):
    r = np.random.default_rng(seed)
    d = b - a                                  # per-case difference
    idx = r.integers(0, len(d), size=(reps, len(d)))
    means = d[idx].mean(axis=1)
    return d.mean(), np.percentile(means, [2.5, 97.5])

diff, (lo, hi) = paired_bootstrap(a, b)
print(f"A {a.mean():.3f}  B {b.mean():.3f}  diff {diff:+.3f}")
print(f"95% interval [{lo:+.3f}, {hi:+.3f}]")

small = slice(0, 20)
diff, (lo, hi) = paired_bootstrap(a[small], b[small])
print(f"first 20 only: diff {diff:+.3f}, "
      f"interval [{lo:+.3f}, {hi:+.3f}]")

A 0.710  B 0.805  diff +0.095
95% interval [+0.045, +0.145]
first 20 only: diff +0.100, interval [+0.000, +0.250]

The model explains both lines. With 200 cases, the interval [+0.045, +0.145] stays above 0: almost every reshuffled paper still says B is better. With only the first 20 cases, the observed gain is about the same (+0.10), but the interval stretches from +0.25 all the way down to 0, so we can't rule out "no real change". That's why "it looked better on a few examples" isn't evidence, and why eval sets need size.

Chapter 5: Gate releases on it

The problem

An eval nobody runs protects nothing. The final step is making it run automatically, in CI (continuous integration: the checks that run on every pull request), and fail the build when it should.

Details, built on the model

CRITICAL = {"policy"}

def gate(cases, old, new):
    broke = [c["id"] for c in cases
             if c["tag"] in CRITICAL and new[c["id"]] < old[c["id"]]]
    d = np.array([new[c["id"]] - old[c["id"]] for c in cases])
    if broke:
        return False, f"critical cases broke: {broke}"
    if d.mean() < 0:
        return False, f"mean dropped by {d.mean():.2f}"
    return True, f"ok, mean change {d.mean():+.2f}"

print(gate(cases, ma, mb))
print(gate(cases, ma, ma))

(False, "critical cases broke: ['p1']")
(True, 'ok, mean change +0.00')

In CI, a False becomes a non-zero exit code, and the pull request shows exactly which case broke. For a real set you'd swap the plain mean check for the bootstrap interval from Chapter 4, cache answer sheets so unchanged parts don't cost API calls again, and pin the model version so the exam isn't changing under you.

All the mental models, in one place