Step 1 of 4
Design the task set
Representative, tiered, and partially held out.
Build the eval task set from real usage. Mine transcripts/issues for 15-25 tasks across the classes that matter: multi-step builds, bug fixes with verification, research/navigation questions, permission-sensitive operations, summarizations, and refusal-appropriate requests. Tier by difficulty (smoke: must always pass; standard: the working band; stretch: currently unreliable). For each task write: the exact starting state (repo fixture, files), the request, and success criteria (machine-checkable where possible: tests to pass, artifacts to exist, strings to appear). Hold out 20% of tasks - never used when tuning prompts, only for validation. Package fixtures reproducibly.
Expected after this step
A tiered, fixtured task set with held-out validation tasks.
Should not happen
- ✕Evals that test toy tasks nothing like real usage
- ✕Single-run scores treated as truth despite run-to-run variance
- ✕Grading by the same model that did the work, uncalibrated
- ✕A suite that exists but never gates anything
Verify before continuing
Do not move on until every check is true. The complete button stays locked until then.
Do not continue if…
- !Evals that test toy tasks nothing like real usage
- !Single-run scores treated as truth despite run-to-run variance
- !Grading by the same model that did the work, uncalibrated
- !A suite that exists but never gates anything
If the AI messes this up
Use this when the AI fakes progress or breaks the feature. It forces a real fix.
Tasks are toy-sized. Take three real sessions from history and convert them into fixtures verbatim - real mess (ambiguity, partial context) is exactly what the suite must contain.