Evaluating LLM systems like production software

Most teams evaluate models. Very few evaluate systems. The gap between the two is where production failures live — and it is wider than the benchmark numbers suggest.

Benchmarks measure the wrong unit

A model scoring well on a public benchmark tells you almost nothing about how your retrieval layer, prompt assembly, tool routing and fallback logic behave together under real traffic. The unit of evaluation has to be the system, not the weights.

Build the harness before the feature

We write the evaluation harness first. It runs on every pull request, against a frozen set of real cases, and it reports per-stage attribution: how often retrieval surfaced the right document, how often the model used it, how often the guardrail fired correctly.

Regression is the metric that matters

Absolute scores drift with every model release. What you actually need to know is whether this change made the system worse for cases it previously handled. Track regressions per case, not averages across the set.

Ship the harness to the client

The harness is part of the deliverable. A team that cannot evaluate the system cannot safely change it, and a system nobody can safely change stops improving the day we leave.