What I Mean by Evaluation

When I talk about evaluation, I mean the work of figuring out whether an AI system is actually good enough to trust. A model can look impressive in a demo and still be unreliable in practice. The interesting question is not whether it can produce a strong example, but whether the evidence is strong enough to support a decision.

I care about making that gap visible: define what good means, measure behavior, understand uncertainty, and be willing to say the result is not ready yet. Evaluation is not a final checkbox after the system is built. It is part of how I decide what the system is for and whether it is doing that job.

The clearest public example is my LLM Judge Eval Harness. It calibrates a quality threshold, measures held-out precision, estimates uncertainty with bootstrap confidence intervals, and intentionally fails when a promising score is not supported by enough evidence. The point is not just to produce a score. It is to make the decision boundary visible and know when the score is not enough.

That is the kind of AI work I want to keep doing: systems where trustworthiness depends on evidence, not on how polished the demo looks.