Observability for Agent Runs
Instrument your agent like production software: traces, session replay, cost tracking, and failure analysis on open standards.
The route
4 steps to Done
- 01
Instrument spans across the loop
The trace tree is the run's skeleton - build it first.
Preview prompt + verify gate ▾ Hide ▴
Add structured tracing to the agent. Span hierarchy: session -> task -> turn -> {model_call, tool_execution, hook_check, compaction, memory_op}. Attributes per span type: model calls get model, tier, tokens in/out, cost, latency, stop reason; tool spans get tool name, argument digest, result size, verdict; hook spans get rule and verdict. Use OpenTelemetry SDK conventions (GenAI semantic conventions where they fit) so exporters and existing backends work. Export to your available backend (an OTLP collector, or a local trace store if self-contained). Add a scrubber on attribute capture: known secret patterns and configured sensitive fields are redacted before export. Verify one session renders as a correct, complete tree.
- ✓ Hierarchy covers all loop operations
- ✓ Token/cost attributes on model spans
- ✓ Scrubbing verified on a seeded secret
- 02
Build session replay
The debugging superpower: watch the run again, slowly.
Preview prompt + verify gate ▾ Hide ▴
Implement replay on top of the trace/event data. Store per session the ordered event stream (spans plus payload records: full prompts, responses, tool results - scrubbed, with large payloads referenced to an artifact store). Build the replay view: step through the session event by event; at each step show the model's exact input context, its output, the tool calls and results, and running token/cost totals; support jump-to-error and filtering by span type. This can be a simple web view or rich CLI - navigability matters more than polish. Verify by replaying a recent session and confirming the replay matches the live experience.
- ✓ Exact model inputs visible per step
- ✓ Jump-to-error works
- ✓ Large payloads stored by reference
- 03
Track the economics
Cost per session shapes design decisions - surface it everywhere.
Preview prompt + verify gate ▾ Hide ▴
Build the metrics layer. Per session compute and store: total cost, cost by call class, tokens by direction, tool-call counts, duration, and outcome (completed/escalated/cancelled). Surface: a per-session cost line in the session summary shown to the user; a periodic aggregate report (cost per task class over time, expensive-session outliers listed); alerts on runaway sessions (cost exceeding a threshold mid-run triggers a pause-and-confirm). Backfill metrics from stored traces for recent history. Review the first aggregate report and note one design implication (e.g. a class worth routing cheaper - feeds the cost-routing work).
- ✓ Cost line in every session summary
- ✓ Aggregates trend over time
- ✓ Runaway alert tested mid-run
- 04
Run the failure-analysis drill
Observability proves itself on the next bad day - rehearse it.
Preview prompt + verify gate ▾ Hide ▴
Establish and drill the failure workflow. WORKFLOW: failure reported -> find the session (by ID/time/user) -> open the trace tree -> jump to the failing span -> inspect the exact context and payloads via replay -> classify the root cause (bad tool result, prompt gap, permission block, model error, harness bug) -> file the fix with the trace linked. DRILL: take two real recent failures (or inject one: a tool returning garbage mid-task); run the workflow end to end, timing it; compare against log-diving honestly. Then wire anomaly capture: sessions ending in escalation/error auto-flag with their traces linked in the report channel. Document the workflow one page or less.
- ✓ Two failures root-caused via traces
- ✓ Time-to-cause compared to log-diving
- ✓ Escalated sessions auto-flag with links
Research-backed
Sources behind this flow
Tier 3 · memory
MemoryLens
Observability and debugging for AI agent memory: instruments the memory pipeline (write, read, compress, update) to show exactly why an agent 'forgot' something. pip-installable with framework integrations.
Tier 6 · agent-research
AgentOps
The observability and DevTool platform for AI agents: session replays, cost and token tracking, and failure analysis via a lightweight SDK spanning the major agent frameworks.
Tier 6 · agent-research
OpenLIT
OpenTelemetry-native observability for LLM apps and agents: one-line instrumentation for traces, metrics, cost tracking, plus prompt management and a playground - GenAI observability on open standards.
Tier 6 · agent-research
RagaAI Catalyst
An agent observability, evaluation, and monitoring platform: trace agent executions, run evals, and manage datasets/experiments across the agent lifecycle.
Tier 1 · claude-code-internals
Deep Dive: Agent Orchestration Patterns
Follows a dispatched agent through its full lifecycle - foreground vs background execution, auto-backgrounding of long tasks, sidechain transcripts for resume, layered tool filtering, and result handoff - then catalogs six reusable orchestration patterns: worker pool, pipeline, supervisor-worker, adversarial verification, fork-join, and resume chain. The adversarial verification agent, which must run real commands and issue a PASS/FAIL verdict while being forbidden from editing the project, gets special attention as a defense against LLM self-verification failure modes.