Evaluating a run

By the end, you can

• Judge a run's evidence packet against its task's acceptance criteria, and state which criteria are met, unmet or unsettled by the packet's evidence (LA-5).
• Decide and justify a disposition — accept, revise or roll back — for a completed run, using the packet's evidence rather than an overall impression (LA-5).

Before you start

This is Module 4, Lesson 3, and the last lesson before Module 5's work on rollback and handoff. It assumes this module's evidence types (Lesson 1) and the evidence packet built as a run happens (Lesson 2), and Module 2's acceptance criteria — a checkable claim, a pass/fail line and a named check, decided before the run. This lesson is where those threads meet: using a packet to judge a run against its criteria, and deciding what happens next.

Evaluating a run means checking the packet against the criteria, one line at a time

Evaluating a run is not re-running the task, and it is not re-forming an impression from glancing at the output. It is going through the acceptance criteria decided in Module 2, one at a time, and for each one asking whether the packet contains evidence that settles it favorably, evidence that settles it unfavorably, or no matching evidence at all.

The Scrum Guide's Definition of Done frames the same discipline for a different kind of work: it is "a formal description of the state of the Increment when it meets the quality measures required for the product," and "work cannot be considered part of an Increment unless it meets the Definition of Done." Applied here: a run does not count as accepted because it looks finished. It counts as accepted because its packet demonstrates, criterion by criterion, that the standard decided in advance was actually met.

Each criterion lands in one of three states:

• Met — the packet contains evidence of a type that can settle this criterion, and it settles it favorably.
• Unmet — the packet contains evidence of a type that can settle this criterion, and it settles it unfavorably.
• Unsettled — no evidence in the packet of a type that could settle this criterion exists yet, per Lesson 1's distinctions. An unsettled criterion is not the same as a failed one, and it should not quietly be treated as passing either.

From criteria to a disposition: accept, revise or roll back

Google's code-review practice offers a close model for this decision, built for a different kind of change but transferable: reviewers should favor approving a change "once it is in a state where it definitely improves the overall code health of the system being worked on, even if [it] isn't perfect" — the same reasoning supports an accept decision even when a minor, explicitly named and owned gap remains. On the other side, "nothing... justifies checking in [a change] that definitely worsen[s] the overall code health of the system" — the same reasoning behind a roll-back decision when a run's evidence shows real harm, not merely imperfection.

The disposition decision is the same human-review pause this course has described since Module 1: OpenAI's guardrails guidance describes it as pausing the run "so a person or policy can approve or reject a sensitive action" before it takes effect. Deciding a run's disposition using its evidence packet is that pause, applied after the run rather than before a single action — with a concrete artifact in front of the reviewer instead of a bare description to take on trust.

• Accept — every acceptance criterion is met by evidence in the packet, and any unresolved risk recorded in Lesson 2 is judged acceptable to carry forward — named and owned, not hidden.
• Revise — one or more criteria are unmet or unsettled, but the run's changes are safe to leave in place while the specific gap is addressed. The fix targets the gap, not the whole task from scratch.
• Roll back — one or more criteria are unmet in a way that makes it unsafe or wrong to leave the current change in place, or an unresolved risk is judged unacceptable to carry forward. How to execute a rollback safely is Module 5's subject; this lesson covers only deciding that rollback is the right call.

Write the disposition down, with reasoning

A disposition is not just a word. For each one, record: which specific criteria were checked and against what evidence; how any unresolved risk from the packet was judged; and who decided, and when. NIST's Appendix C states the underlying requirement plainly: "Human roles and responsibilities in decision making and overseeing AI systems need to be clearly defined and differentiated" — a disposition record with reasoning attached is where that definition actually lives on paper, not merely as a stated principle.

A worked example: disposition for the dependency-bump run

Building on Lesson 2's dependency-bump agent. Acceptance criteria, decided in advance: (1) the lockfile pins the new version and no other dependency changed, checked by the diff; (2) the full test suite passes, checked by the test result; (3) no test was skipped or excluded to make the suite pass, checked by the test result plus the command log.

The packet shows: the diff confirms only the target dependency changed (criterion 1: met). The test result shows the suite passing (criterion 2: met). The command log shows no skip flag was used, but the packet's honest unresolved-risk line, from Lesson 2, notes the suite does not cover the project's slowest integration path, which this run never exercised at all (criterion 3: unsettled for that specific path, not unmet).

Disposition: accept, with the unresolved risk named and owned — someone is assigned to run the integration path manually before the next release, not silently dropped from the record. Contrast: if the test result had instead shown a failure on a safety-relevant path, the same evidence-based process would support roll back instead, because leaving that specific change in place would be unsafe, not merely imperfect.

Accessibility notes

This lesson is text-first, with no images, audio, video or downloadable artifacts. The practice exercise's model answer sits behind a native disclosure control that is reachable and operable by keyboard and correctly announced by screen readers. The knowledge check uses native radio-button inputs with a visible question and options, and posts its result to a live status region so assistive technology announces the outcome without a page reload.

Sources and limits

This lesson synthesises the sources below into a practical learning model. It is not a security standard, legal advice or a guarantee that any particular agent design is safe.

1. The 2020 Scrum Guide — Scrum.org and Scrum Inc.. Defines the Definition of Done as the shared standard a completed increment must meet before it counts as finished — the same standard this lesson applies to judging a run's packet against its acceptance criteria.
2. The Standard of Code Review — Google Engineering Practices Documentation. States the standard a reviewer's approve, request-changes or reject decision rests on — the model this lesson applies to an accept, revise or roll-back disposition.
3. Guardrails and human review — OpenAI API documentation. Documents human review pausing a run so a person can approve or reject a sensitive action before it takes effect — the mechanism a disposition decision hands control to.
4. Appendix C: AI Risk Management and Human-AI Interaction — NIST AI Resource Center. Calls for explicit human roles and responsibilities in AI oversight — the accountability a disposition decision and its recorded reasoning exist to satisfy.