Evals for Your Harness

The route

4 steps to Done

1. 01

   Design the task set

   Representative, tiered, and partially held out.

   Preview prompt + verify gate ▾ Hide ▴

   Build the eval task set from real usage. Mine transcripts/issues for 15-25 tasks across the classes that matter: multi-step builds, bug fixes with verification, research/navigation questions, permission-sensitive operations, summarizations, and refusal-appropriate requests. Tier by difficulty (smoke: must always pass; standard: the working band; stretch: currently unreliable). For each task write: the exact starting state (repo fixture, files), the request, and success criteria (machine-checkable where possible: tests to pass, artifacts to exist, strings to appear). Hold out 20% of tasks - never used when tuning prompts, only for validation. Package fixtures reproducibly.

   • ✓ Tasks mined from real usage
   • ✓ Success criteria machine-checkable where possible
   • ✓ Holdout set marked and respected
2. 02

   Implement graders

   Automate what can be checked; rubric what must be judged.

   Preview prompt + verify gate ▾ Hide ▴

   Build the grading layer. AUTOMATIC: per task, run the machine checks (test suite passes, endpoint responds, output validates against schema, forbidden actions absent from the decision log) - these produce hard pass/fail plus sub-scores. RUBRIC (for quality dimensions: plan sensibility, summary fidelity, refusal quality): write 3-5 criterion rubrics with anchored scores (what a 2 vs 4 looks like), graded by an LLM judge with the rubric and the artifacts - calibrate the judge against 10 human-graded samples and report agreement before trusting it. Version the rubrics. Every grade records its evidence (which check, which output) for auditability.

   • ✓ Machine checks cover the checkable
   • ✓ Judge calibrated against human grades
   • ✓ Every grade carries evidence
3. 03

   Run honestly: variance and the results store

   One run is an anecdote; the distribution is the datum.

   Preview prompt + verify gate ▾ Hide ▴

   Build the runner and store. RUNNER: one command executes the suite (or a tier) against the current harness; each task runs N times (default 3; smoke tier can be 1); results capture scores, decisions, durations, and costs per run. STORE: results keyed by harness version + timestamp; the report shows per-task mean and spread, per-tier aggregates, and deltas versus the previous version with variance-aware flagging (a delta inside run-to-run noise is 'no signal', not a win). Run the first full baseline now and sanity-check: smoke tier at or near 100%, stretch tier imperfect - a suite everything passes measures nothing.

   • ✓ N runs per task with spread reported
   • ✓ Deltas flagged against noise bands
   • ✓ Baseline shows a discriminating suite
4. 04

   Gate changes and close the loop

   The suite earns its cost when it starts saying no.

   Preview prompt + verify gate ▾ Hide ▴

   Operationalize gating. Process: every harness-artifact PR runs the smoke tier automatically (CI) and standard tier on demand; the PR description includes the eval delta table; regressions beyond the noise band require justification or fixes before merge; the holdout set runs weekly, not per-change, to detect overfitting to the tuned set. Validate the whole system with a sabotage test: deliberately degrade the harness (remove the verification step or gut a key prompt section), run the suite, and confirm scores drop clearly and diagnosably. Then restore, re-run, and confirm recovery. Document the gating rules alongside the changelog process.

   • ✓ Smoke tier in CI, deltas in PRs
   • ✓ Holdout runs on a schedule
   • ✓ Sabotage produced clear, diagnosable drops

Research-backed

Sources behind this flow