Step 1 of 4
Instrument spans across the loop
The trace tree is the run's skeleton - build it first.
Add structured tracing to the agent. Span hierarchy: session -> task -> turn -> {model_call, tool_execution, hook_check, compaction, memory_op}. Attributes per span type: model calls get model, tier, tokens in/out, cost, latency, stop reason; tool spans get tool name, argument digest, result size, verdict; hook spans get rule and verdict. Use OpenTelemetry SDK conventions (GenAI semantic conventions where they fit) so exporters and existing backends work. Export to your available backend (an OTLP collector, or a local trace store if self-contained). Add a scrubber on attribute capture: known secret patterns and configured sensitive fields are redacted before export. Verify one session renders as a correct, complete tree.
Expected after this step
A complete scrubbed trace tree for a real session.
Should not happen
- ✕Unstructured print-logging that cannot reconstruct a run
- ✕Traces without token/cost attributes, blinding the economics
- ✕Secrets in span payloads, turning observability into a breach
- ✕Dashboards built, never used - observability without a debugging workflow
Verify before continuing
Do not move on until every check is true. The complete button stays locked until then.
Do not continue if…
- !Unstructured print-logging that cannot reconstruct a run
- !Traces without token/cost attributes, blinding the economics
- !Secrets in span payloads, turning observability into a breach
- !Dashboards built, never used - observability without a debugging workflow
If the AI messes this up
Use this when the AI fakes progress or breaks the feature. It forces a real fix.
The tree has gaps (orphan spans, missing turns). Check context propagation - every operation must open its span inside its parent's context; async boundaries are the usual leak point.