Evaluation Set
Also called: eval set, evals, golden dataset
A collection of real inputs with known-correct outputs, used to score an AI system objectively and catch regressions before they reach users.
A few hundred real inputs, each with the correct answer agreed by someone who knows the domain, stored alongside your code and run automatically on every change.
It is the single most valuable artefact in an AI project and the one most often missing.
Why it decides everything else
Without it you cannot answer whether the system is good enough to ship, whether yesterday's prompt change helped, or whether someone else can safely modify it. Every discussion becomes a matter of impressions.
With it, a pull request that drops accuracy fails the build. Prompt engineering becomes normal engineering with a red or green result.
What good looks like
Weighted towards awkward cases rather than the happy path, because the happy path is not where systems fail. Versioned with the code so it evolves with the system. Run in CI, not manually before releases.
The uncomfortable part
Agreeing the correct answers surfaces disagreements about what the system is actually for. Two domain experts will label the same ambiguous case differently. Resolving that is the valuable work, and it is much cheaper in week one than in month six.
Related terms