Observing a Fleet of Coding Agents: Observability Is a Surface, Not a Feature

We run AI coding agents the way you'd run a small engineering team. They pick up issues, scope them, write code, review each other's pull requests, and merge when the work converges. Some run in a sandboxed VM, some run as GitHub Actions jobs, and one runs a long autonomous loop from a laptop. On a busy day there are dozens of agent turns in flight.

The obvious question — the one that decides whether any of this is sustainable — is simple to ask and historically painful to answer: what are they actually doing, and what is it costing us? This post is about the answer we built, and the principle behind it.

The principle: observability is a surface, not a feature

It is tempting to treat observability as a feature you bolt on — a dashboard here, a log line there, a cost number you remember to print. That approach fails the moment the system grows, because the gaps are invisible until you need the data that isn't there.

We hold a stricter line, written into our architecture decisions as a binding contract:

If you find yourself piping ad-hoc commands together to check state, a subcommand is missing — add it rather than scripting around the gap.

Observability is a surface you query, not a feature you remember to use. The practical form of that contract is a table that every engineer can recite: to see X, run Y. Is the world healthy? Run one command. Recent events? One command. What did an agent cost? One query against a known stream.

When the answer to "how do I see X" is "well, you'd have to grep three log files and join them by hand," that is not an inconvenience. It is a bug in the observability surface, and we fix it like any other bug.

This reframing does real work. It turns "we should probably add some metrics" into "this domain object ships without its query surface, so it isn't done." It turns a missing number into a failing acceptance criterion.

What we want to be able to answer

Before any schema, the design starts from the questions. A coding-agent fleet is worth observing only if you can answer, on demand and without spelunking:

• Cost. What did this run cost? Which agent role is the most expensive? How does cost-per-iteration trend over a long autonomous loop?
• Tool use. Which tools does an agent reach for? How often do tool calls fail? Where is time spent?
• Latency. Which operation in a run was the slow one? Is the model thinking, or is a tool blocking?
• Session shape. How many turns did a task take? Did a multi-step agent hand-off chain actually complete, or die halfway?
• Attribution. Every one of the above, sliceable by which agent, which role, which run — not just fleet-wide totals.

Notice these are all delta questions: more or less, faster or slower, cheaper or pricier than last time. That framing is deliberate. A telemetry surface that can only show you "the current value" is a status light; one that lets you compare across runs and roles is a tool for actually improving the system.

The shape of the answer

Three ideas carry the whole design.

1. Agents emit standard telemetry; we don't instrument them by hand. The agents speak OpenTelemetry — the open standard for traces, metrics, and logs. We don't wrap their internals in bespoke timing code. Instead we let each agent emit its own signal and collect it centrally. "Observability follows the agent" — wherever a turn runs, its telemetry carries the context of what spawned it.

2. Three pillars, one backend. Every agent action lands as one of three kinds of signal:

• Logs — discrete events. This prompt was sent. This tool was called and rejected. This API request used N tokens and cost $X.
• Traces — spans with timing and parent/child structure. This whole turn took 12s; the Bash call inside it took 4s.
• Metrics — cumulative counters and histograms. Total tokens by model. Tool-call counts. Session counts.

The art is knowing which pillar answers which question. Cost-and-tokens → metrics. "Did this specific turn fire?" → logs. "Why was this slow?" → traces. A single tool-using turn produces signal in all three, joined by a shared session key.

3. One query surface, not three dashboards. All of it flows into a single queryable store. The interface is SQL (with PromQL where Prometheus-native operators help). That is the "surface" the principle demands: a place you go to ask, not a fixed dashboard you hope already shows what you need.

What this buys you

Once the surface exists, the payoff compounds:

• Cost discipline becomes routine, not a fire drill. "What did the agents cost yesterday, broken down by role?" is a query, not a project.
• Regressions are visible. If a change to a prompt doubles the tokens per task, the metric moves and you see it.
• Gaps become honest. Because the surface is queryable, you can ask "what isn't here?" and get a real answer — which is the subject of the next section.

The honest part: what we can't see yet

A credible observability story names its blind spots. Ours, as of this writing:

• Some agent runtimes are quieter than others. Not every agent we run emits all three pillars. One emits rich logs but no traces (so no latency profiling for it); another emits traces and metrics but no event-level logs. We document which runtime emits what in a coverage matrix, so a missing row is a known gap, not a surprise.
• Orchestration decisions live outside the telemetry. The logic that decides retry vs. escalate, or flips a workflow label, runs in the orchestrator itself — which deliberately opens no spans. Those decisions are recorded to a local event log, but they aren't yet in the central surface. The time an agent spends waiting (for CI, for a human, for a poll) shows up only as a gap between spans.
• External calls are uninstrumented. When an agent shells out to call an external API, that HTTP round-trip isn't captured as a span.

None of these are secret, and that is the point. The coverage matrix and the "still NOT instrumented (by design)" list are first-class documents. The surface tells you the truth about its own edges.

The takeaway

If you take one thing from this: don't treat observability as something you add to a system — treat it as a surface the system is required to expose. Start from the questions you must be able to answer, make those answers a single query away, and treat a missing answer as a defect rather than a nice-to-have. For a fleet of autonomous coding agents, that discipline is the difference between a system you operate and a system that operates you.

---

Appendix: how it actually works here

The narrative above is deliberately runtime-agnostic. For the engineers and operators in the arena, here is the concrete wiring. The binding contract is the Agent Observability Contract (ADR-0002); the shared telemetry stack is ADR-0030. The day-to-day query path is the telemetry-query skill — use its bin/oo helper rather than hand-rolling curl.

Three lanes, one OpenObserve backend

Lane	What runs	Exports to
castra	the work-issue-runtime agent fleet inside the b4castra Lima VM	OpenObserve in-VM
fullsend	GitHub Actions coding-agent personas on the ganymede self-hosted runners	OpenObserve @ganymede
ralph	the phandelver autonomous loop, exported tailnet-direct from the operator laptop	OpenObserve @ganymede, tagged castra_lane='ralph'

All three write the same streams with the same resource attributes (castra_persona, castra_work_id, castra_lane, github_run_id, …), so a single castra_work_id filter walks every layer of a run.

The three pillars, by stream

Logs — agent_events_raw (one row per event; mainly claude-code + codex):

event_name	Carries
user_prompt	prompt_length, prompt (when OTEL_LOG_USER_PROMPTS=1)
api_request	input_tokens, output_tokens, cache_*_tokens, cost_usd, duration_ms, effort, attempt
tool_decision / tool_result	tool_name, decision (accept/reject), success, duration_ms, output snippet
plugin_loaded, hook_execution_*	lifecycle

Traces — agent_traces_raw (spans; claude-code + pi):

• claude: claude_code.interaction (whole turn root), tool.bash, claude_code.tool.execution, plus collector-derived tool_args_summary.
• pi @the-agency/pi-observability (fullsend): GenAI-semconv spans (gen_ai chat <model>, agent.turn, tool.<name>) carrying cost_total_usd, gen_ai_usage_* tokens, and pi_session_skill_names (skill inventory, not invocation). This is pi's only cost source.

Metrics — per-metric streams:

• claude_code_*: token_usage, cost_usage, session_count, active_time_total (labels: model, effort, token_type).
• pi_*: pi_turns, pi_prompts, pi_tokens_input/output, pi_tool_calls, pi_tool_errors, pi_tool_duration (+ histogram derivatives).

Workflow-level events (CI lane only) — emitted by scripts/ci/workflow-event.sh: run.start, run.complete (outcome), persona.output, persona.comment_posted.

Per-driver coverage (the "quieter runtimes" from the body)

Driver	Activation	Logs	Traces	Metrics
claude-cli / claude-agent-sdk	env (CLAUDE_CODE_ENABLE_TELEMETRY=1 + OTEL_*)	✅	✅	claude_code_*
codex-cli	~/.codex/config.toml [otel] block	✅	❌	❌
pi-coding-agent	PI_OTEL_ENABLED=true + pi extension	❌	✅	pi_*

• codex emits only logs — no traces, no metrics. No latency profiling.
• pi emits no agent_events_raw logs (no event hook) — turn-level evidence is in traces/metrics only.

Cross-pillar joins

session_id is the universal key. OpenObserve queries one stream per request, so cross-pillar joins are done client-side after two queries — pivot "this span took 4s" → "what was the agent doing" via agent_traces_raw ⋈ agent_events_raw on session_id.

Not instrumented (by design)

What	Why it's invisible
Time between node spawns (waiting on CI / GitHub polls)	gaps between spans, not attributable
Orchestrator decisions (retry vs. escalate, label flips)	live in events.jsonl, never opened as spans
gh / external HTTP shell-outs	no HTTP-client instrumentation

Closing these would need either an @opentelemetry/api import in the orchestrator (to open node-start/finish spans) or a file-tailer forwarding events.jsonl into OpenObserve. Both are deferrable; per-spawn resource attrs already give ~80% of the practical "which role/run produced this" visibility.

Starter queries

The telemetry-query skill ships canned queries — e.g. ralph-cost-per-iteration.sql, plus cost/latency/tool-use starters in its README.md. Numeric fields arrive as strings; CAST(... AS DOUBLE) before aggregating.