‘Looks Good’ Is Not Verification: Give Coding Agents an Evidence Matrix

Part 3 of “Designing Coding Agents That Earn Write Access.”

A global verification skill in my setup contains a neat table.

See package.json? Try the JavaScript scripts.

See pubspec.yaml? Consider Dart formatting, analysis, and tests.

See Cargo.toml? Reach for Cargo.

It looks portable. It looks systematic. It also becomes wrong the moment the repository has a wrapper command, a monorepo boundary, two competing manifests, or a project rule that says the obvious command is not the real gate.

This is the final trap in agent verification: command discovery can look like proof.

It is not proof. It is only the beginning of the investigation.

I know this because of a number, and the number is nine.

That is how many files a JSON sorter rewrote in a single pass, having been invoked bare in a repository root with no path or glob argument. It was not running to change anything. It was running during a verification pass.

Nothing in that sequence was a false negative. The step ran, the step succeeded, and the step was not read-only — which was a category I had not been tracking at all. I had been sorting verification steps into “passed” and “failed,” when some of them belong in “rewrote your files.”

The rule I wrote afterward is boring enough to fit on one line: run formatters and rewriters only with an explicit path or glob, never bare in a repository root, including during verification steps.

The clause that earns its keep is the last one. “Including during verification” is the part I would have assumed was implied, right up until nine files disagreed with me.

That is when I stopped asking an agent “did the checks pass?” and started asking “which evidence did you produce, and what does it actually cover?”

The Table That Wants to Become Authority

A manifest-to-command table is useful because it gives an agent candidate commands instead of letting it improvise. That is already better than hardcoding one ecosystem into a global tool.

But candidate discovery is not repository policy.

Detecting Flutter can suggest that flutter analyze may be relevant. It cannot establish that the repository uses flutter analyze directly, that it should run before tests, or that the changed package even owns the pubspec.yaml you found.

The evidence order needs to be explicit:

1. Read applicable repository instructions and contribution guidance.
2. Prefer the project’s declared scripts or command catalog.
3. Inspect manifests, lockfiles, workspace configuration, and tool configuration.
4. Use a conservative fallback only when repository evidence supports it.

The order matters. If a project defines a Makefile target, a justfile target, a package script, or its own wrapper, that interface carries more local intent than a generic ecosystem guess.

My earlier instinct was to celebrate successful detection. I am remarkably easy to impress when a tool prints a green checkmark (there, I admitted it). The repository, unimpressed, kept asking whether I had detected the right scope (repositories are rude like that, but often correct).

Lockfiles Are Evidence Too

Even a seemingly simple JavaScript repository can punish a premature assumption.

package.json tells you that a package exists. It does not tell you whether the project uses npm, pnpm, Yarn, or Bun.

The verification skill therefore checks the lockfile before choosing the package manager. In a workspace, it also scopes the command to the changed package through the package manager’s own filtering mechanism.

That still does not solve every case. A mixed repository may contain several applicable manifests. A wrapper command may own the sequence. A workspace root may not own the package being changed.

When those signals conflict and repository policy does not identify the controlling package, the correct result is not a creative command. It is [UNKNOWN].

I know that looks anticlimactic. No spinner. No green check. No triumphant “all tests passed.”

But an honest unknown is stronger evidence than a command that exits zero in the wrong directory.

Verification Depends on the Change, Not Only the Stack

After command discovery comes classification.

One Flutter repository in the source material encoded its verification policy as a matrix keyed by change type rather than one universal command. The shape was roughly this:

documentation only
→ run the documentation formatting gate

non-generated Dart source
→ format, then run the project test command

code-generation or localization input
→ run the repository bootstrap or regeneration command

new native dependency
→ run the native dependency setup path

This is more precise than “run everything.” It is also more precise than “run the usual Flutter checks.”

A documentation edit does not need to boot the entire mobile toolchain. A generated output change cannot be verified by formatting the output if the authoritative input was never regenerated. A native dependency change needs native setup evidence that a Dart-only test cannot provide.

The matrix makes the agent classify the change before selecting the proof.

That small pause prevents two expensive habits:

• Running a giant suite for every trivial edit.
• Running a cheap but irrelevant check for a risky change.

Verification should be proportional, but “proportional” must not mean “whatever feels sufficient today.” The matrix turns proportionality into policy.

A Successful Build Is a Very Specific Fact

The same repository policy explicitly rejected a generic build as its verification step.

That boundary is useful far beyond Flutter.

A successful build proves that the build system completed for that configuration. It does not automatically prove that behavior is correct, tests pass, generated sources are current, formatting matches policy, or the changed runtime path works.

Agents love broad green signals because they compress the story. “Build passed” sounds much more satisfying than “the build command completed, but the behavior was not exercised.”

The operating contract from Part 1 does not allow that compression. A completion claim must remain the same size as its evidence.

Evidence: formatter completed
Allowed claim: formatting gate passed

Evidence: test suite completed
Allowed claim: that test suite passed

Evidence: application built
Allowed claim: that build completed

Evidence: none
Allowed claim: verification was not run

This may be the least exciting table in the series. It may also be the one that prevents the most exciting failures.

Formatters Complicate the Story

Verification commands are not always read-only.

A formatter may rewrite files. If changes were already staged, the working tree and the index can diverge after formatting. The verification process must either use a check-only mode or re-stage the affected files and inspect the staged patch again.

This detail is easy to miss because “format” sounds like validation. Sometimes it is mutation wearing a validator’s lanyard.

The nine-file sorter was a blast-radius problem: the right kind of tool, aimed at everything. There is a second axis underneath it, which is that a mutating step can succeed completely and still be the wrong evidence for the file it was aimed at.

The clearest case in my own tooling is why dotenv-linter is barred from ios/.xcode.env. The linter does exactly what it was built to do — it rewrites export VAR=VALUE into VAR=VALUE — and the file that comes out still looks entirely valid. Xcode’s build system, however, requires that export for the variables to propagate into build phases. One keyword removed per line is a very quiet kind of breakage: the file is rewritten, the commit passes, and the damage surfaces downstream at build or runtime, in a way the diff does not explain.

That is why the set of file-mutating linters is handled by an explicit allowlist — which of them may run against which targets — rather than by trusting each one to be locally harmless. A tool whose job is to mutate can produce output that is syntactically valid and semantically broken in whatever consumes it, and no amount of downstream validation is as cheap as not making the edit.

The same principle applies to code generation, lockfile regeneration, and bootstrap commands. An agent should know whether a verification step observes state or changes it. The operating contract must account for the difference.

Build the Evidence Matrix in Three Layers

I now think of agent verification as three connected layers.

Layer 1: Discover the repository interface

Read instructions, declared scripts, command catalogs, manifests, lockfiles, and CI configuration. Do not begin with the agent’s favorite ecosystem command.

Layer 2: Classify the change

Identify whether the task changes documentation, source code, generated inputs, dependencies, native integration, or another risk class defined by the repository.

Layer 3: Match claims to executed evidence

Record what actually ran, its result, and what that result proves. Do not upgrade “one command exited zero” into “the feature works.”

A compact policy can look like this:

verification:
  discovery:
    - repository_instructions
    - declared_scripts
    - manifests_and_lockfiles
    - conservative_fallback
  classify_by:
    - changed_paths
    - generated_inputs
    - dependency_scope
    - native_impact
  report:
    - commands_run
    - observed_results
    - unverified_requirements

The YAML is illustrative, not a standard. Please do not publish a package named evidence-matrix before finishing this article (actually, I cannot stop you).

No Verification Setup Is a Result

Some repositories genuinely have no relevant lint or test setup for a change.

The wrong response is to invent a familiar command and hope it means something. The correct response is to report the absence.

This is where source-verified output becomes operational rather than philosophical. The question “which command verifies this repository?” is itself a factual question. The answer must come from repository evidence.

If the evidence does not define a command, say so. Then identify the smallest manual check the human should perform, if the task permits one.

An agent that admits “this repository does not declare a test for this path” is more useful than one that runs a generic test command and presents the exit code as closure.

Lessons Learned

The first lesson was that ecosystem detection is candidate discovery, not authority. A manifest can point toward a toolchain without defining the repository’s verification policy.

The second lesson was that verification must be indexed by change class. Different changes create different failure modes, so they need different evidence.

The third lesson was that every green result has a boundary. A formatter proves formatting. A test suite proves the cases it ran. A build proves that build.

The final lesson was that [UNKNOWN] can be a successful verification outcome. If the repository does not contain enough evidence to select a real command, refusing to invent one is the contract working as designed.

Across this series, the progression is now complete:

operating contract
→ evidence-bounded roles
→ change-bounded verification

The contract defines what the agent may do. The roles define where it may learn from. The evidence matrix defines what it may claim afterward.

Only then does “write access” start to mean something.

Take the last change an agent marked complete in your repository and reconstruct the evidence chain. Which instruction selected the command? Which change class did the command verify? Which final claims were actually supported by its output?

If those answers are fuzzy, the problem may not be the agent’s intelligence.

It may be the missing matrix.

Previous: Part 2 — “Stop Hiring One Super-Agent.”