Version Your Harness and Track Behavior

The route

4 steps to Done

1. 01

   Extract behavior into versioned artifacts

   You can only track what has an identity.

   Preview prompt + verify gate ▾ Hide ▴

   Extract every behavior-bearing artifact into versioned files. Inventory: system prompt sections, tool descriptions and schemas, permission/danger rules, routing tables, reminder templates, repair prompts, sub-agent prompts. Move each into a dedicated files structure (prompts/, tools/, rules/ or equivalent) loaded at startup - no behavioral string literals left in code. Define the harness version: a semver bumped on behavioral change, or the git SHA of the artifacts tree; implement version resolution at startup and stamp it into every session record and log line header. Verify by changing one file and seeing the new version in the next session's logs.

   • ✓ Inventory covers all behavior-bearing strings
   • ✓ Artifacts load from files at startup
   • ✓ Version stamps appear in session logs
2. 02

   Start the behavior changelog

   Write the diff a user would notice, not the diff git shows.

   Preview prompt + verify gate ▾ Hide ▴

   Institute the behavior changelog. Format per entry: version, date, author, WHAT CHANGED (artifact-level), EXPECTED BEHAVIOR DIFF (the observable difference: 'agent now asks approval before installs', 'summaries shorter and structured'), RISK NOTES (what could regress), and eval results reference (next step). Backfill entries for the last 3 known changes as practice. Enforce via process: PRs touching artifact files require a changelog entry (CI check on the diff). The changelog is user-facing documentation - people running agents track behavior changes the way API users track breaking changes.

   • ✓ Entries describe observable behavior
   • ✓ Risk notes included
   • ✓ CI blocks artifact changes without entries
3. 03

   Build the smoke eval and behavioral diffing

   Ten fixed tasks make drift visible in minutes.

   Preview prompt + verify gate ▾ Hide ▴

   Build the smoke-eval suite: 10-12 fixed tasks exercising the behaviors that matter (a multi-step plan task, a verification-gated fix, a permission-boundary probe, a summarization, a tool-routing case, a refusal case). For each, define what to record: key decisions (tools chosen, approvals requested), output shape, and pass/fail against criteria. Implement the runner: execute the suite against the current harness, store results keyed by harness version; a diff command compares two versions' results side by side, highlighting decision changes. Run it on the current version as the first baseline. Process rule: artifact PRs run the suite and include the diff in review.

   • ✓ Suite covers plans, gates, permissions, and refusals
   • ✓ Results stored per harness version
   • ✓ Diff output reviewable in PRs
4. 04

   Prove the loop with a real change

   One tracked change end to end validates the whole system.

   Preview prompt + verify gate ▾ Hide ▴

   Run a full tracked change. Pick a real pending improvement (e.g. tightening a tool description). Execute the process: branch, edit the artifact, write the changelog entry with expected behavioral diff, run the smoke suite, review the actual diff against expectations (surprises mean either the change or the suite needs work), merge with version bump, and confirm new sessions log the new version. Then prove rollback: reproduce the previous version from history and run one suite task on it. Document the loop briefly so the next contributor follows it without asking.

   • ✓ Expected vs actual behavioral diff compared
   • ✓ Version bump visible in new sessions
   • ✓ Prior version reproduced and runnable

Research-backed

Sources behind this flow