Cost Control: Model Routing and Effort Tiers

The route

4 steps to Done

1. 01

   Profile spend by call class

   Find where the money actually goes before moving anything.

   Preview prompt + verify gate ▾ Hide ▴

   Profile my agent's model spend. Instrument every model call with: call class (main-loop reasoning, compaction summary, memory extraction, classification/routing, repair, sub-agent, embedding), model, tokens in/out, and computed cost. Run a representative day of tasks (or replay recent transcripts through the accounting). Produce the spend breakdown: cost share per class, average call size per class, and the top-3 cost concentrations. Typical finding worth checking: auxiliary calls (summaries, classifications) making up a large cost share while needing far less capability than they get.

   • ✓ Every call site tagged with a class
   • ✓ Costs computed from real token counts
   • ✓ Top concentrations quantified
2. 02

   Build the routing table and wire it

   One table, consulted by every call.

   Preview prompt + verify gate ▾ Hide ▴

   Implement routing. Create the routing config: per call class -> {model, effort/reasoning setting where the API supports it, max output tokens, temperature}; initial assignments by judgment (main reasoning stays premium; summaries/classifications/extractions go economy; repair mid-tier). All model calls flow through the client wrapper, which resolves settings from the table by class - remove any hardcoded model names at call sites. Support per-session overrides (a --premium flag forcing everything up, for debugging). Log the resolved route on every call. Verify by running a task and checking the log shows each class on its assigned tier.

   • ✓ No hardcoded models remain at call sites
   • ✓ Effort settings applied where supported
   • ✓ Route logged per call
3. 03

   Add escalation on failure

   Cheap first, strong on evidence of need.

   Preview prompt + verify gate ▾ Hide ▴

   Implement escalation. Per class, define the failure signals that justify retrying stronger: validation failure after repair (structured outputs), verification red twice on the same unit (main loop), sub-agent reports marked truncated/inadequate, or explicit low-confidence markers. On signal: retry the call/unit on the next tier up (economy -> mid -> premium), log the escalation with its trigger, and cap escalations per task to keep worst-case cost bounded. Also support de-escalation experiments: a config flag routing a sample of a premium class to mid-tier with outcomes tracked (feeds the eval in the next step). Test escalation by forcing a cheap-tier failure and watching the ladder climb in logs.

   • ✓ Failure signals defined per class
   • ✓ Escalations logged with triggers
   • ✓ Per-task escalation cap enforced
4. 04

   Guard quality and report the savings

   The eval is what makes cost optimization safe.

   Preview prompt + verify gate ▾ Hide ▴

   Prove it safe and worth it. QUALITY GUARD: build a fixed eval set (10-15 tasks spanning the routed classes: summaries judged for fact retention, classifications against labels, main tasks against completion criteria); run it under premium-everything and under the routing table; compare per class with an agreed regression threshold (e.g. no class drops more than 2 points or 5%). Any class over threshold rolls back a tier and gets re-run. REPORTING: add cost-by-class to session summaries and a periodic report tracking spend and eval scores over time. Deliver the final table: cost reduction achieved and quality deltas per class, both from measurements.

   • ✓ Eval covers every routed class
   • ✓ Over-threshold classes rolled back
   • ✓ Savings and deltas reported from data

Research-backed

Sources behind this flow