Rule-File Code Review Gates
Run deterministic policy checks on everything your agent produces: rule files reviewing plans, diffs, and evidence before anything merges.
The route
4 steps to Done
- 01
Write the initial rule set
Codify the review comments you keep repeating.
Preview prompt + verify gate ▾ Hide ▴
Author the first rules. Mine sources: recurring review comments on agent output, incident history, and the project's conventions doc. Rule types to implement: PATTERN rules (regex over diff text: no hardcoded credentials/URLs, no TODO-without-issue, no console.log in production paths), AST rules (structural: no bare except, no mutable default args, exported functions need types - use the language's parser), and DIFF-SHAPE rules (churn thresholds per file type, test files must change when src changes beyond a threshold, lockfile changes require dependency-change declarations). Each rule: id, severity (block/warn/info), rationale, matcher, fix guidance, and 2+ test cases (one firing, one passing). Start with 8-12 rules that would have caught real past issues.
- ✓ Every rule has rationale and fix guidance
- ✓ AST rules cover structural conventions
- ✓ Test cases fire and pass as designed
- 02
Build the evaluation engine
Fast, deterministic, and readable verdicts.
Preview prompt + verify gate ▾ Hide ▴
Implement the engine. Input: a diff (or workspace state) plus optionally the agent's plan; execution: run all applicable rules (pattern rules over diff hunks, AST rules over changed files' parsed trees, shape rules over diff metadata); output: a structured verdict - overall pass/block, plus findings [{rule_id, severity, file, line, message, guidance}]. Performance: full evaluation in seconds (parse changed files only). Determinism: same input, same verdict, always - property-test this. Provide two frontends: a CLI (evaluate a diff, exit code by severity) and a harness API (the loop calls it directly). Run it against three historical agent diffs and review the findings for sanity.
- ✓ Findings carry id, location, and guidance
- ✓ Determinism property-tested
- ✓ Evaluation fast enough for the loop
- 03
Wire the gates into the loop
Catch at plan time when cheap; enforce at diff time when final.
Preview prompt + verify gate ▾ Hide ▴
Integrate both gate points. PLAN GATE: when the agent produces its plan/todo, evaluate plan-applicable rules (touching protected areas, missing test steps for src changes, dependency additions without declaration); violations return to the model as corrective feedback before execution starts. DIFF GATE: before task completion (and in CI before merge), evaluate the full diff; BLOCK findings prevent completion - the model receives the findings verbatim and iterates; WARN findings attach to the completion report for human review; INFO logs. The model's fix iterations re-run only affected rules for speed. Test the loop: seed a task that will violate a blocking rule and verify the catch-fix-pass cycle in the transcript.
- ✓ Plan gate blocks before wasted execution
- ✓ Block findings gate completion until fixed
- ✓ Warns surface in the completion report
- 04
Institute the incident-to-rule ritual
The gate compounds when every escape becomes a rule.
Preview prompt + verify gate ▾ Hide ▴
Operationalize rule growth. RITUAL: every incident or bad-merge postmortem answers 'what rule would have caught this?' - the answer becomes a rule PR with the incident as its firing test case; rules ship within a week of the incident while context is fresh. HYGIENE: quarterly rule review - false-positive rates per rule from gate logs; noisy rules get tuned or demoted to warn; dead rules (never fired in 6 months) get reviewed for relevance. GOVERNANCE: blocking-severity changes require review; the rule file carries a changelog. Prove the ritual now: take the most recent real incident, write its rule, test it against the original offending diff (must fire) and current head (should pass), and merge it through the process.
- ✓ Incident rule fires on the historical diff
- ✓ False-positive review scheduled with data
- ✓ Blocking changes governed and changelogged
Research-backed