Build the evaluation set before the prototype

The easiest prototype cases are usually the least useful ones. They have clean inputs, an obvious answer, and enough context for the model to look capable. A dozen good responses can create a great demo while leaving the project team unable to answer a basic question: what would make this system good enough to use?

Build the evaluation set before the prototype. The set does not need to be large or automated on day one. It needs to describe the work honestly and make success concrete enough that two reviewers can reach roughly the same judgment.

Treat the set as an executable specification

A conventional product requirement may say, “Summarize the case and recommend a next action.” That leaves several decisions hidden. Which facts must appear? Which source wins when two records disagree? Is a recommendation allowed when required evidence is missing? When should the system ask for help?

An evaluation case forces those decisions into the open. Give the system an input and the context it would have in production. Record the acceptable outcome and how it will be graded. Add any action or claim that should cause an automatic failure.

OpenAI’s evaluation guidance recommends task-specific tests that reflect the real distribution of work, early evaluation, and continuous additions from logs. A versioned bank of cases describes what the product must handle.

Start by defining the unit under test. For a classifier, one case might be a message and its correct route. For a retrieval workflow, it may include the question, permitted source documents, and facts that the answer must support. For an agent, the case may require a simulated environment, tool responses, permission limits, and a verifiable final state.

Do not test only the final prose when the system takes actions. A confident message that says a refund was issued is wrong if the order system contains no refund. The outcome in the environment is the thing to grade.

Find cases in the existing work

Historical cases are useful because they contain details a workshop will miss: incomplete forms, old product names, conflicting notes, strange attachments, and policy exceptions that everyone has forgotten until they recur.

Ask the workflow owner for a bounded period of completed work and the records needed to reconstruct each decision. Do not assume the final status is correct. Some records encode a workaround rather than the desired outcome; others were reopened after the export was produced. A domain reviewer should verify the source material and the accepted result.

Build a case inventory before choosing the evaluation sample. Useful attributes might include case type, source channel, language, document format, customer segment, policy version, consequence of an error, and whether a person had to intervene. Choose attributes that change the work. A diversity column with no operational meaning will not improve the set.

The initial suite should contain common cases in roughly realistic proportions, plus separate coverage for rare cases that carry high cost. Add known past failures. Include inputs that lack enough information for a safe answer, since abstaining or escalating may be the correct result. If users can enter free text, include misspellings, terse requests, irrelevant context, and attempts to override the system’s instructions.

Synthetic cases can fill a specific gap, but label them. They are helpful for controlled variations, such as changing one field while keeping the rest of a document fixed. They should not quietly replace the messy historical records that the production system will face.

Handle source data under the same privacy and access rules that apply to the live workflow. Remove identifiers when the evaluation does not need them. An evaluation repository is still a data store.

Give every case enough context to be audited

Field	What belongs there
Case ID	A durable identifier that does not reveal sensitive data
Source and date	Where the case came from and which policy version applied
Inputs	The exact content supplied to the system
Available context	Records, tools, permissions, and state present at the start
Accepted outcome	A reference answer, valid range, required facts, or final system state
Failure conditions	Claims, omissions, tool calls, or actions that make the result unsafe
Grader	Code check, rubric, domain review, or a combination
Difficulty notes	Why the case matters and what ambiguity was resolved

Keep the expected result separate from an example response. A single polished response can cause reviewers to reject other valid wording. Specify required facts and allowed variation instead. When there is exactly one valid value, use it. When several resolutions are acceptable, record the set or the conditions that make each one valid.

NIST’s AI Risk Management Framework Measure function calls for teams to document test sets, metrics, and evaluation tools, and to demonstrate performance under conditions similar to deployment. It also lets someone reproduce a surprising score.

Resolve disagreement before scoring models

Two experienced reviewers may choose different next actions for the same case. That disagreement often exposes an ambiguity in the operating policy.

Have reviewers label a sample independently, then compare their decisions. For a disagreement, ask which source supports each answer and whether both are acceptable. Update the case, rubric, or policy before using it as a score. Preserve the adjudication note so the same argument does not happen during every model review.

Some qualities cannot be reduced to one exact answer. A case summary may need factual coverage, clear uncertainty, appropriate tone, and a useful next step. Write a rubric with observable descriptions for each score. Include examples near the pass boundary. “Good writing” is not a criterion a reviewer can apply consistently.

Human review remains part of the measurement system. OpenAI advises calibrating automated grading against human feedback, while Anthropic’s guide to agent evaluations separates code-based, model-based, and human graders. Use the cheapest reliable grader for each property. Code can verify a schema, a database state, a required citation, or a prohibited tool call. A domain reviewer is better suited to a judgment that policy has not made mechanical.

If a model grades subjective outputs, test the grader too. Compare its decisions with blinded human labels, inspect disagreements, and watch for preferences that have nothing to do with the task, such as longer answers. A numerical score does not become objective because software produced it.

Keep a set the builder cannot tune against

Prompt changes will improve the cases the builder can see. That is useful during development, but it also creates overfitting. Split the cases before serious tuning begins.

Use a working set for debugging. Builders should read those cases, inspect traces, and understand each failure. Keep a separate acceptance set under the control of the product or workflow owner. It can share the same categories and grading rules, but its exact cases should not drive daily prompt edits. A later regression set can preserve cases the system has already learned to handle.

Do not report one blended score and hide the slices. Show results for routine work, exceptions, high-consequence cases, languages or formats that matter, and any subgroup where the experience may differ. Set hard failure rules for prohibited outcomes. A high average should not compensate for an unauthorized write or an invented policy claim.

For a variable model, run important cases more than once. Anthropic distinguishes a test case from a trial for this reason: repeated trials reveal whether a success is dependable or occasional. Record the model, prompt, tool versions, parameters, and environment so the comparison can be reproduced.

A worked example

Imagine a hypothetical support workflow that drafts replies about subscription cancellations. This is an example, not a client case.

One historical request says the customer was charged after a cancellation attempt. The account record shows an active subscription, the chat transcript contains the attempted cancellation, and the billing policy grants agents discretion to refund within a defined window. A weak evaluation would compare the draft with one reference paragraph and score textual similarity.

A useful case describes the required outcome instead. The draft must acknowledge the disputed charge, identify the relevant account and chat facts, avoid stating that a refund has already been issued, and route the case to a person with refund authority. If the system has a read-only billing tool, any attempted write is an automatic failure. A code grader can inspect tool calls and required fields. A reviewer can judge whether the explanation represents the policy accurately.

Now vary one fact at a time. Remove the chat evidence. Move the charge outside the policy window. Add a second account with a similar name. Present an outdated policy excerpt next to the current one. Each variation tests a decision boundary, not a preference for wording.

The resulting failures tell the team what to change. Missing account facts may point to retrieval. An invented refund may require a clearer permission boundary. Inconsistent handling across repeated trials may mean the task needs a fixed workflow rather than more prompt language. The evaluation is doing product design work before production users have to discover the gaps.

Make the build earn its next step

Set the acceptance rule while the team is still willing to hear “no.” Define the minimum pass rate, the cases that must all pass, the allowed cost and latency, and the failures that stop the pilot. Record who can waive a criterion and why. A waiver is a risk decision, so it should not hide inside a prompt review meeting.

When the prototype fails, add diagnosis rather than adjectives. Classify the failure by the part of the system that could fix it: missing context, wrong retrieval, unsupported reasoning, tool misuse, poor policy, grader error, or an ambiguous case. Change one part, rerun the working set, then check the acceptance set for regressions.

After launch, production failures become candidate cases after review and removal of sensitive data. Keep old cases tied to the policy version they test. Retire a case when the underlying work no longer exists, not because it makes the score inconvenient.

This is the evidence step in the A.G.E.N.T. delivery framework: agree on the job, assemble cases from the real workflow, and make the smallest system pass them. A prototype is worth building when the team knows which cases it must handle, what a pass looks like, and what result would end the experiment. Without those answers, prompt changes cannot show whether the system is getting closer to an agreed result.