feat(capsule): astrid-capsule-metrics — runtime telemetry rollup and exposition

[joshuajbouw]

Summary

Add astrid-capsule-metrics — a first-party capsule that aggregates kernel and capsule telemetry into structured rollups, exposes them over IPC, and optionally publishes Prometheus/OpenTelemetry-shaped exports. The current visibility story is tracing to stderr, kernel logs, and astrid events tail (#685) — adequate for live debugging, terrible for "how is the host actually behaving over the last hour."

Why

• Operators need rollups, not streams. astrid events tail shows what just happened; nobody wants to wc -l a tail to know how many tool calls succeeded today.
• Capsule authors need cost visibility. LLM calls, HTTP egress, KV writes — capsule authors should be able to ask "how much did my capsule do" without each capsule rolling its own counters.
• Approval and capability fatigue tracking. Deny rates per principal/action are a security signal. We have the data on the bus; nothing aggregates it.
• Sysadmin punch-list dependency. Bus tail (feat(cli,kernel): astrid events tail — live event-bus subscriber for IPC debugging #685), caps trace (feat(cli,capabilities): astrid caps trace <agent> <action> — strace for capability resolution #691), receipts (feat(cli,audit): astrid receipt — cryptographically signed compliance receipt for a session #693), usage report (feat(cli): astrid usage report — aggregated per-agent telemetry rollup #687) all benefit from a canonical metrics store rather than each re-deriving counters off raw events.
• Kernel-is-dumb compliance. Aggregation does not belong in the kernel. It belongs in a capsule that subscribes to the bus like any other consumer.

Scope

The capsule subscribes to standard topics on the event bus and maintains in-memory + persisted counters/histograms keyed by (metric, principal, capsule, outcome).

Metric families (initial):

• astrid_tool_calls_total{capsule, tool, outcome} — every tool invocation
• astrid_tool_call_duration_seconds{capsule, tool} — histogram per tool
• astrid_llm_tokens_total{capsule, provider, model, direction} — direction ∈ {input, output, cache_read, cache_write}
• astrid_llm_cost_usd_total{capsule, provider, model} — derived from provider pricing tables (config-driven, not hardcoded)
• astrid_http_egress_bytes_total{capsule, host} — outbound network volume
• astrid_capability_decisions_total{principal, action, outcome} — outcome ∈ {allow, deny}
• astrid_approval_prompts_total{principal, action, outcome} — outcome ∈ {approve, deny, expire}
• astrid_kv_ops_total{capsule, op} — op ∈ {get, put, delete}
• astrid_event_bus_publishes_total{topic, principal} — bus throughput
• astrid_kernel_uptime_seconds — gauge

Surfaces:

• IPC query topic metrics.v1.query — request/response for {metric, filters, range, resolution} → series. The primary consumer is astrid usage report (feat(cli): astrid usage report — aggregated per-agent telemetry rollup #687) and any future TUI dashboard.
• Prometheus exposition — optional. Capsule binds a localhost listener (capability-gated, off by default) on configurable port and serves /metrics in standard exposition format. Off by default because exposing a local port is a capability decision, not a freebie.
• OpenTelemetry push — optional, behind a config flag. Pushes OTLP-formatted metrics to a configured collector. For shops that already run an observability stack.
• Audit-anchored snapshots — periodic snapshot published to a topic so receipts (feat(cli,audit): astrid receipt — cryptographically signed compliance receipt for a session #693) can include "metrics at time T" as a signed property.

Storage:

• Hot window in-memory (configurable, default 24h, 1m resolution).
• Cold storage in capsule-local KV. Default retention configurable, default 30 days at 5m resolution downsampled.
• Both bounded — the capsule must never grow unbounded. Eviction is loud (logs warning if a cardinality budget is hit).

Capabilities required:

• Subscribe to a curated set of astrid.v1.* and *.v1.* topics (read-only).
• kv:put/kv:get for its own namespace.
• net:listen:127.0.0.1:<port> only if Prometheus exposition is enabled.
• No fs_write outside its capsule sandbox.

Why uniquely Astrid

Most metrics stacks are agent-side scrapes that bypass authorization. Here, the metrics capsule is itself a capability-restricted principal — it only sees topics it's allowed to subscribe to. Sensitive payloads (credentials, secrets-bearing topics) are aggregated by metadata only, never by value. This is consistent with the bus tail design notes in #685.

Out of scope

• Distributed metrics — single-host only. Multi-host aggregation belongs to a downstream collector via OTLP.
• Per-event log retention — that's the audit chain's job. Metrics is counters/histograms, not events.
• Alerting rules — out-of-band; consumers can poll the IPC interface or scrape Prom.
• Cost pricing tables for every model — start with a pluggable provider pricing config; we are not maintaining the world's pricing.

Design questions

• Cardinality bounds. What's the max series count before we shed labels? Hard-code a default; configurable. Need a clear behaviour when a capsule misbehaves and emits high-cardinality tool names.
• Topic taxonomy. We're betting the bus already publishes enough signals. Audit which topics need to exist; some may need to be added/standardised before this capsule is useful. Likely overlaps with the bus-tail topic taxonomy work in feat(cli,kernel): astrid events tail — live event-bus subscriber for IPC debugging #685.
• Sensitive payload redaction. Credential-bearing topics already redact payloads (per feat(cli,kernel): astrid events tail — live event-bus subscriber for IPC debugging #685). Metrics should additionally avoid recording label values from sensitive topics — confirm the redaction policy carries across both consumers.
• Time source. Wall-clock from the host. The capsule cannot influence kernel time; just records it.

Dependencies

• Bus-tail / topic-tap admin path (feat(cli,kernel): astrid events tail — live event-bus subscriber for IPC debugging #685) — overlaps on "who can subscribe to what." Implement the capability story once.
• Per-agent telemetry rollup (feat(cli): astrid usage report — aggregated per-agent telemetry rollup #687) — direct consumer of this capsule's IPC interface.
• Audit receipts (feat(cli,audit): astrid receipt — cryptographically signed compliance receipt for a session #693) — wants signed snapshots of metric state at audit-chain positions.

Acceptance

• Capsule loads as a stock first-party capsule under the standard manifest.
• astrid usage report --capsule <name> --range 24h returns aggregated data without scraping logs.
• Disabled cleanly: removing the capsule causes no kernel-side errors; everything still routes.
• Cardinality budget enforced — load test with a misbehaving emitter shows graceful shedding, not OOM.

[joshuajbouw]

Design input: host-ops vs capsule-telemetry division (+ catalogue contribution)

I ran a scoping pass over the whole metrics surface and want to land the results here, because this issue is the right home for the capsule-telemetry half. Two concrete outcomes:

1. The two layers, and where the line falls

There are really two metrics layers, and keeping them separate avoids duplicated/competing endpoints:

	Host-internal operational (gateway /metrics, #791)	Capsule telemetry rollup (this issue)
Answers	How is the runtime itself behaving?	What is the workload doing / costing?
Examples	HTTP RED, event-bus saturation, sandbox traps/fuel, daemon connections, process_* CPU/mem	LLM tokens & cost, tool calls, capability/approval decisions per principal
Source	Direct counter!/gauge!/histogram! from daemon-process code	Bus subscription (tool.v1, llm.v1, capability, approval) — exactly as scoped here
Endpoint	Unauthenticated, network-ACL'd	Capability-gated, off by default (as scoped here)
Contract	None (kernel-internal)	Rides existing event contracts — no new transport

The host-ops layer is now tracked separately in #791 (foundation slice — process_* + astrid_build_info — already up on feat/host-operational-metrics). The key point: a WASM capsule structurally cannot see bus depth / socket accepts / sandbox fuel, so those belong in #791; and per-principal cost rollups don't belong in the kernel, so they belong here. The two don't overlap.

2. The bus-subscription architecture here is the right call

The scoping pass briefly evaluated an alternative — a new astrid.v1.metrics.emit IPC push transport so capsules could emit into the host recorder. I'm withdrawing that idea in favour of what this issue already proposes: the capsule subscribes to events it already receives and aggregates them itself. No new host ABI, no new topic, no kernel passthrough — strictly more kernel-is-dumb-compliant. No metrics RFC is needed.

3. Design constraints worth baking in (the capsule exposes Prometheus too)

Since the optional exposition here serves a /metrics scrape, it should inherit the same rules the host-ops doc (core/docs/metrics.md, on the #791 branch) spells out:

• Identity must be kernel-stamped, never payload-derived. Namespace/attribute a metric by the bus event's kernel-verified sender (the principal-attribution variant in host-ABI RFC RFC: Host ABI rfcs#22 distinguishes kernel-verified from uplink-claimed). A capsule must not be able to spoof another's series.
• The aggregator is the trust boundary. It ingests arbitrary bus payloads, so it must enforce per-source label-tuple budgets, a per-source distinct-metric-name cap, and label-value length caps — and drop on violation (never panic, never unbounded growth). The eviction-is-loud behaviour already in scope is the right instinct; make it apply to cardinality too.
• Collapse caller/remote-influenced labels to allow-lists. model, tool, and host are the dangerous axes: model strings are env/caller text, tool/host names can be remote-advertised. Map each through a registered-set *_static(x) -> &'static str with an other fallback. Never label by raw URL/host/tool/model.
• principal as a label is acceptable here but not on the host-ops endpoint. Because this endpoint is capability-gated and off-by-default, per-principal series (needed for feat(cli): astrid usage report — aggregated per-agent telemetry rollup #687/tracking: production multi-tenancy — complete user isolation and per-principal resource quotas #653) are fine. The host-ops /metrics is unauthenticated, so it forbids principal labels entirely — another reason the layers are separate.

4. Catalogue contribution (complements the families already listed)

Beyond the families in the issue body, the per-capsule loop internals are high-value and only visible from capsule-space. All names namespaced astrid_capsule_<name>_*, all labels bounded:

Metric	Type	Labels (bounded)	What it answers
astrid_capsule_react_turns_total	counter	outcome {completed,tool_dispatch_failed,llm_error,timeout,max_iterations,cancelled}	Agent control-loop turns by terminal state
astrid_capsule_react_loop_iterations	histogram(count)	—	Tool-call rounds per turn (thrash near max_iterations vs single-shot)
astrid_capsule_react_phase_duration_seconds	histogram	phase {awaiting_identity,awaiting_prompt_build,streaming,awaiting_tools}	Latency per agent-turn stage
astrid_capsule_router_tool_dispatches_total	counter	outcome {forwarded,invalid_name,publish_failed}	Tool dispatch; invalid_name = topic-injection probing. Never label by tool name
astrid_capsule_openai_llm_requests_total	counter	model*, outcome {ok,api_error,no_api_key,stream_error}	LLM requests; model via allow-list collapse
astrid_capsule_openai_request_duration_seconds	histogram	model*	Dominant agent-turn latency per model
astrid_capsule_openai_tokens_total	counter	model*, direction {input,output}	Cost accounting (pairs with the llm_cost_usd already scoped)
astrid_capsule_session_writes_total	counter	operation {append,get_messages,clear}, outcome {ok,cas_contended}	Rising cas_contended = single-writer invariant violated
astrid_capsule_http_fetch_requests_total	counter	method*, status_class, outcome	Agent egress; method collapsed capsule-side, never raw URL/host
astrid_capsule_react_context_compactions_total	counter	result {compacted,unchanged,skipped_no_budget,timeout}	Compaction behaviour
astrid_capsule_memory_injections_total	counter	source {personal,project}, truncated {true,false}	High truncated = memory exceeds the cap

* = mandatory allow-list collapse at the emit/aggregate site.

The full conventions, security deny-list, and cardinality-budget rules these follow are in core/docs/metrics.md (#791). Happy to help shape the aggregator's cardinality-enforcement layer when this moves from needs-design into build.