Verification Loops: Never Trust an Unrun Edit

The route

4 steps to Done

1. 01

   Verify at the write boundary

   The cheapest catch-point is the moment of the edit.

   Preview prompt + verify gate ▾ Hide ▴

   Implement write-time verification. On every file write or edit: run the appropriate fast check for the file type (python: compile/ruff; js/ts: esbuild parse or tsc; json/yaml: parse) within a second-scale budget; on failure, revert the write (or never apply it - validate a staged copy) and return the exact error with line numbers to the model as the tool result; on success, return a confirmation including the check that ran. Wire this into the edit tool itself so no write path skips it. Test with a deliberately broken edit and confirm the revert-and-teach cycle produces a corrected second attempt.

   • ✓ Checks are file-type appropriate and fast
   • ✓ Failed writes never persist on disk
   • ✓ Errors return with line numbers
2. 02

   Add the post-change verify step

   Syntax says it parses; only execution says it works.

   Preview prompt + verify gate ▾ Hide ▴

   Implement behavioral verification after each logical change. Mechanics: after the model completes a coherent unit of work (an edit batch, a todo item), the harness runs the verify step - the project's relevant tests (targeted by changed paths where a mapping exists, full quick suite otherwise), or for untested projects a smoke script (start the service, hit the key endpoint, check the response). The verdict plus the failing output (bounded, tail-biased) goes back to the model. The loop treats a red verify as the task NOT advancing: the todo item stays open. Configure per-project verify commands in the harness config rather than hardcoding.

   • ✓ Verify triggers per logical unit, not per keystroke
   • ✓ Red verify keeps the todo item open
   • ✓ Verify commands configurable per project
3. 03

   Gate completion on evidence

   Done is a property of the evidence, not the prose.

   Preview prompt + verify gate ▾ Hide ▴

   Implement evidence-based completion. Build the evidence record as the task runs: every verification event appends {check, command, verdict, output-digest, timestamp}. At completion time, the harness gates: all todo items closed, latest verify green, and every requirement in the task mapped to at least one piece of passing evidence (the model produces the mapping; the harness checks the referenced evidence exists and passed). If gating fails, completion is refused with the gaps listed. The final output to the user includes the evidence table. Test with a multi-requirement task and confirm an unverified requirement blocks completion.

   • ✓ Evidence record built automatically
   • ✓ Requirement-to-evidence mapping checked
   • ✓ Final output shows the table
4. 04

   Red-team your own verification

   Verification you have not tried to fool is decoration.

   Preview prompt + verify gate ▾ Hide ▴

   Attack the pipeline. Attempt the classic evasions: (1) an edit that parses but breaks behavior - the write check should pass and the verify step must catch it; (2) a 'fix' that makes the failing test pass by weakening the assertion - review the diff-of-tests in verify output and add a harness rule flagging test-file modifications during fix tasks for explicit approval; (3) completion claimed with a stale green from before the last edit - the gate must require evidence newer than the newest change; (4) verify command sabotage (echo passing) - verify commands come from config, not model output. Fix every hole found and document the pipeline's guarantees honestly, including what it still cannot catch.

   • ✓ All four evasions attempted
   • ✓ Test-weakening flagged for approval
   • ✓ Evidence freshness enforced

Research-backed

Sources behind this flow