Capturing run evidence

By the end, you can

• Build an evidence packet during a run rather than reconstructing one after it, and state what capturing evidence after the fact costs (LA-4).
• Record an unresolved risk or known gap as a first-class piece of evidence, rather than omitting it because the task is not fully verified (LA-4).

Before you start

This is Module 4, Lesson 2 of the Local AI Agents course. It assumes Lesson 1's five evidence types and Module 3's dedicated, disposable workspace. This lesson is not about what evidence is — that was Lesson 1 — it is about when it gets captured, and it argues that the answer changes what the evidence is actually worth.

Evidence captured after the fact is a reconstruction, not a record

Building an evidence packet once a run has already finished relies on memory of what happened, on logs that may have rotated or been overwritten, and on a diff computed against a state the workspace has since moved past. None of that is dishonest, but all of it is weaker than the alternative: capturing each artifact at the moment the run produces it.

OpenAI's guidance on evaluating agent workflows describes what a trace actually is: "the end-to-end record of model calls, tool calls, guardrails, and handoffs for one run." That record exists because the run generated it as it happened — it is not a summary written afterward from someone's recollection of what the run probably did. The same logic applies at the level of an individual local task: a command log captured as each command executes, and a diff captured at the point the evidence is assembled, are the run's own record. A description written up later, after the workspace has moved on, is a reconstruction of that record, not the record itself.

What "changed files with no evidence" actually costs

A set of changed files with no accompanying evidence packet is, to a later reviewer, indistinguishable from an untested guess that happened to touch the right files. The reviewer cannot tell whether tests were run, what was actually checked, or whether an earlier failure was silently retried until it passed. Two options are left: trust the change without checking, or redo the verification work from scratch. Building the packet as the run happens is what saves a later reviewer that redo — it is the specific cost this lesson is naming, not an abstract principle about diligence.

Build the packet as the run happens

Each evidence type from Lesson 1 has its own moment to be captured, and the moment is always at or near the point the run produces it, not after the fact:

• Command log — captured as each command executes, not reconstructed from a shell history that may not retain everything.
• Diff — recomputed at the point the evidence is assembled, against the actual prior state, not "eventually" once the task feels finished.
• Test result — captured immediately after running a check, recording the specific test and its exact outcome, not a later impression of "the tests were fine."
• Failure trace — kept for anything that errored during the run, even if it was later resolved. A resolved failure with no trace of what went wrong is a weaker record than a resolved failure with one.
• Review note — added once a person reviews the above; this is usually the one piece of the packet that genuinely happens after the run, because the reviewer was not there while it ran.

Unresolved risk is evidence, not a gap to hide

The honest version of a packet sometimes has to say "this was not fully verified," and that line belongs in the packet as a first-class piece of evidence, not as an omission that would make the packet look more complete.

Google's SRE practice built a related discipline for a different kind of write-up — a review after an incident, not a routine run — but the underlying habit transfers directly. A postmortem is "a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root cause(s), and the follow-up actions to prevent the incident from recurring," and for it to be genuinely useful it "must focus on identifying the contributing causes of the incident without indicting any individual or team for bad or inappropriate behavior." Applied to a run's evidence packet: an honestly recorded unresolved risk is not evidence of failure to be hidden. It is exactly the kind of evidence a later reviewer needs to make a real decision, and naming it plainly costs nothing compared to a reviewer discovering it was quietly left out.

Examples of a legitimate unresolved-risk line: "the change was tested against a sample of the data, not the full production shape"; "one of three target files could not be verified because the test suite does not cover it"; "the fix addresses the reported symptom, but the underlying cause is still unconfirmed." Each of these is a specific, checkable statement about what is not yet known — the same discipline Module 2's acceptance criteria applied to what is claimed to be true, applied here to what is honestly still unsettled. None of them disqualify the packet; each one is exactly what the next lesson's disposition decision needs to see.

There is an accountability reason behind this honesty, not just a quality one. NIST's Appendix C observes that "Human roles and responsibilities in decision making and overseeing AI systems need to be clearly defined and differentiated" — and the person responsible for overseeing a run can only actually carry that responsibility if the packet tells them what remains unresolved, rather than leaving them to discover it after they have already signed off.

A worked example: an evidence packet built during a config-timeout run

Returning to the earlier request-timeout task: as the agent edits the config file, the diff is captured immediately, showing only the timeout constant changed. As the start-up test runs, its outcome is recorded right away, not summarized later from memory. The command log accumulates as each command executes. One honest line is added to the packet before it is considered finished: "tested against the local development server only; not exercised under production load." That line does not undermine the other evidence — it tells a later reviewer exactly what the packet does, and does not, cover.

Accessibility notes

This lesson is text-first, with no images, audio, video or downloadable artifacts. The practice exercise's model answer sits behind a native disclosure control that is reachable and operable by keyboard and correctly announced by screen readers. The knowledge check uses native radio-button inputs with a visible question and options, and posts its result to a live status region so assistive technology announces the outcome without a page reload.

Sources and limits

This lesson synthesises the sources below into a practical learning model. It is not a security standard, legal advice or a guarantee that any particular agent design is safe.

1. Evaluate agent workflows — OpenAI API documentation. Defines a trace as the record generated for one run — the artifact this lesson says to keep as the run happens, not reconstruct afterward.
2. Site Reliability Engineering, Chapter 15: Postmortem Culture: Learning from Failure — Google SRE Book. Models a written, blameless record of what happened and what remains unresolved — the same honesty this lesson asks of a run's evidence packet.
3. Appendix C: AI Risk Management and Human-AI Interaction — NIST AI Resource Center. Calls for explicit human roles and responsibilities in AI oversight — the accountability an unresolved-risk record exists to support.