Evidence types

By the end, you can

• Distinguish command logs, diffs, test results, review notes and failure traces as evidence types, and state what each can and cannot prove about a run (LA-4).
• Match an evidence type to the specific claim it is capable of supporting, and recognize when a claim has no matching evidence yet (LA-4).

Before you start

This is Module 4, Lesson 1 of the Local AI Agents course. It assumes Module 2's acceptance criteria — a checkable claim, a pass/fail line and a named check — and the boundaries and grants named across Modules 1 and 3, which a run's evidence will eventually need to be checked against. Module 2 Lesson 2 named the check; this lesson names what a check is actually allowed to point at: five distinct evidence types, each capable of settling a different kind of claim, and none of them interchangeable with the others.

Evidence is not one thing

"We have evidence the run worked" is not specific enough to act on, because "evidence" covers several genuinely different kinds of artifact, each answering a different question. Treating them as interchangeable is how a claim ends up resting on evidence that never actually addressed it.

OpenAI's guidance on evaluating agent workflows describes the run-level version of this directly: a trace records "the end-to-end record of model calls, tool calls, guardrails, and handoffs for one run." That is a command log and a diff's worth of material bundled into one artifact — the point stands regardless of the exact tooling: a run either produces this record as it happens, or nobody can reconstruct what actually occurred.

• Command log — the record of what commands ran, in what order, with what output. It answers: what did the agent actually do?
• Diff — the literal before/after text of a changed file or set of files. It answers: exactly what changed, and nowhere else?
• Test result — the output of a predefined, repeatable check run against the changed system. It answers: does the system behave the way that specific check defines as correct?
• Review note — a person's (or a named process's) judgement, recorded against the other evidence they actually looked at. It answers: did someone with the authority to decide look at this, and what did they decide?
• Failure trace — the record of what happened when something did not work: error output, a stack trace, a reproduction path. It answers: what specifically went wrong, not merely that something did.

What each type proves, and what it does not

None of the five settle more than the specific question they were built to answer — treating one as proof of a wider claim is where a packet quietly stops being evidence and starts being an assumption.

• A command log proves the agent ran certain commands. It does not prove the output of those commands was correct, wanted, or safe — only that they ran.
• A diff proves exactly what changed, character for character. It does not prove the change was intended, that it was reviewed, or that nothing outside the diffed files was also affected.
• A test result proves the specific behaviors that test exercises, and no others. A passing suite proves the tests that exist still pass — it says nothing about behavior the suite never checked.
• A review note proves that a named person or process inspected specific evidence and reached a decision. It is only as strong as what was actually inspected: "approved" with no record of what was checked barely counts as evidence at all.
• A failure trace proves what specifically went wrong. Its absence after a report of "it failed" is itself a gap — a failure with no trace is a claim with no evidence behind it.

Matching evidence to the claim it can actually support

The discipline this lesson teaches is asking, for any claim about a run: which of these five evidence types, if any, could actually settle it? A claim with no matching evidence type available should be treated as unsettled — not quietly assumed true because the run otherwise looks fine, and not treated as false either.

Google's code-review practice makes the review-note type concrete: a reviewer's decision is meant to rest on a stated standard, not an impression — favoring approval "once [a change] is in a state where it definitely improves the overall code health of the system being worked on, even if [it] isn't perfect." A review note that just says "looks good" has not actually recorded that standard being applied; a review note that says what was checked and against what standard has.

A worked example: raising a request-timeout value

A local agent is asked to raise a service's request-timeout constant from 5 seconds to 15. Four claims about the run, and the evidence type that actually settles each:

• "The timeout value changed from 5 to 15." Settled by the diff.
• "Nothing else in the config file changed." Also settled by the diff — the full diff, not a description of it.
• "The service starts successfully with the new value." Settled by a test result (or a command log of a start attempt with its outcome recorded) — not by the diff, which says nothing about runtime behavior.
• "Raising the limit to 15 seconds is a safe change for this service." No single automated artifact settles this on its own — it needs a review note: a person weighing the diff, the test result, and their own knowledge of the service, and recording that judgement.

Accessibility notes

This lesson is text-first, with no images, audio, video or downloadable artifacts. The practice exercise's model answer sits behind a native disclosure control that is reachable and operable by keyboard and correctly announced by screen readers. The knowledge check uses native radio-button inputs with a visible question and options, and posts its result to a live status region so assistive technology announces the outcome without a page reload.

Sources and limits

This lesson synthesises the sources below into a practical learning model. It is not a security standard, legal advice or a guarantee that any particular agent design is safe.

1. Evaluate agent workflows — OpenAI API documentation. Defines a trace as the end-to-end record of a run's model calls, tool calls, guardrails and handoffs — the concrete shape a command log or run trace actually takes.
2. The Standard of Code Review — Google Engineering Practices Documentation. States the standard a reviewer's decision rests on — what a review note, as an evidence type, actually has to record.