Check What Actually Happened

Did Cancellation Really Happen?

The assistant says, “Order R17 is canceled.” What evidence would make that true?

A clear sentence is only an output. If the stored status remains active, the requested outcome did not happen, so task completion fails.

A product eval repeatedly checks behavior on a user-relevant task. It asks whether the application completed the work, including any required state change.

Check the Evidence Chain

For R17, inspect four pieces of evidence: the identity requested, the identity selected, the cancellation result, and the final stored status.

Cancellation checks should examine the selected order and actual result.[[1]](https://hamel.dev/blog/posts/evals-faq/)

Accept success only when those pieces agree: R17 was requested, R17 was selected, cancellation succeeded, and R17 ended canceled. Polished wording cannot repair a mismatch.

Follow the Trace

A trace records the interaction and intermediate evidence, showing what the application attempted and what its tools returned.

A trace view from Rechat’s assistant, shown in Husain’s 2024 article. · Hamel Husain, Your AI Product Needs Evals (2024); Rechat trace in LangSmith by LangChain; cropped.

Rechat’s assistant takes instructions from real estate agents. One feature adds contacts (people’s names, phone numbers, emails, and similar details) to the agent’s CRM, the database where those contacts are stored. Its test sends two requests. The first creates a contact, such as “Create a contact for Tom Williams… job title Manager.” The second looks that contact up, such as “What’s Tom Williams’ job title?” If the lookup finds no match, or several, the contact was not created or fetched correctly. So the test passes only when the lookup returns exactly one contact.

In Rechat’s contact test, the system creates a contact, retrieves it, and checks for exactly one result.[[2]](https://hamel.dev/blog/posts/evals/) Like the R17 check, it looks past the confirmation to the retrieved result.

A failed trace is the trace of a case that did not meet its criterion. The R17 trace is one: the reply says canceled, but the stored status is still active. An open-ended note is a short comment in your own words about what went wrong, written without a fixed list of labels. An example is “reply claims canceled; status still active.”

Keep a notebook entry with the trace and a short description of the failure: “reply claims canceled; status still active.” Save the actual evidence alongside the note. A later reviewer can check what happened instead of relying on your label alone.

Separate Success Dimensions

Suppose R17 ends canceled, but the reply prints an internal identifier. Task completion passes; privacy fails.

Success can include separate dimensions such as task fidelity, privacy, response time, and cost.[[3]](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests)

Each needs its own criterion. Correct state does not prove safe disclosure, and a quick response does not prove cancellation. One overall impression of quality hides these distinctions.

Design Realistic Cases

Start with a realistic request: “Cancel order R17.” Then define observable criteria before judging the reply.

For completion, require the requested identity, selected identity, successful result, and canceled final status. For privacy, forbid exposed internal information. Measure response time separately.

Match evidence to the task. A summary may require no state change.

Start with rules you can already state: “the requested order ends canceled” can become a check now.

Other requests need clarification. Suppose your team asks for a “helpful” reply but disagrees about what helpful means. Read actual replies together and describe the difference between an acceptable reply and an unacceptable one. Then write a rule for that difference. Reviewing outputs helps you discover requirements; it does not postpone checks for requirements you already understand.

Benchmarks Do Not Check This Task

A model benchmark is a general test that ranks models on its own fixed set of tasks. A better benchmark score for a new model is not the same as passing your own realistic cases, because the benchmark does not run your application’s complete workflow.

Inspect changed traces, preserve failed cases, and ask: what actually happened?

In an invented test, an assistant moves the requested appointment to Tuesday but includes a private staff note in its reply. Which result preserves the useful distinction?

• Mark the whole case successful because the appointment moved.
• Record the schedule change as successful and the disclosure as a separate failure.
• Record the schedule change as failed because the reply exposed private information.
• Use the response time to decide which result to keep.

Answer: Record the schedule change as successful and the disclosure as a separate failure.

The appointment reached the requested state, so that task criterion passes. The exposed staff note fails a different criterion. Collapsing both into one task result hides what improved or failed; response time does not resolve either judgment.

A team asks for a “helpful” cancellation reply but disagrees about what helpful means. The required order-state change is already clear. What should it do next?

• Postpone every check until the team agrees about helpfulness.
• Check the known state requirement now, and review actual replies to clarify the disputed requirement.
• Treat the state change as proof that every reply is helpful.
• Use the most fluent reply as the fixed rule for all future cases.

Answer: Check the known state requirement now, and review actual replies to clarify the disputed requirement.

The state requirement is explicit and can be checked immediately. Reviewing actual outputs helps the team turn its vague helpfulness request into a criterion. Neither fluency nor the state change settles all other dimensions.

In an invented cancellation test, the request names order Q8. The tool reports success for Q9, and Q8 still shows active. The reply says Q8 was canceled. Which conclusion does the trace support?

• Completion passes because the tool returned success.
• Completion fails because the requested order did not reach the required state.
• Completion is unknown until the model’s benchmark score is checked.
• Completion passes if the reply contains the requested order number.

Answer: Completion fails because the requested order did not reach the required state.

The tool succeeded on a different order. The requested order is still active, so completion fails. A benchmark does not inspect this workflow, and a correct-looking sentence is not evidence of the required state change.