signals — trajectory triage infrastructure for agentic interactions

[syedhashmi]

Title

epic: signals — trajectory triage infrastructure for agentic interactions

Overview

Plano is implementing the signal-based triage framework described in
Signals: Trajectory Sampling and Triage for Agentic Interactions
(arXiv:2604.00356, Chen et al., Apr 2026).

The paper proposes attaching lightweight, model-free signals to live agent
trajectories so that a small fraction of high-information interactions can be
sampled for review, preference-data construction, and post-deployment
optimization. The reported result: 82% informativeness at 1.52× efficiency
over heuristic baselines on τ-bench, without affecting online agent behavior.

This epic tracks the end-to-end delivery of that framework inside
brightstaff: detectors → attributes → triage sampler → feedback loop.

Signal taxonomy (paper §3)

Three coarse-grained categories, seven mid-level groups, 20 leaf signal types:

Layer	Category	Intent
Interaction	misalignment	semantic/intent mismatch between user and agent
	stagnation	discourse continues without visible progress
	disengagement	withdrawal of cooperative intent
	satisfaction	explicit stabilization and completion
Execution	failure	action attempts that don't yield usable outcomes
	loops	repetitive execution patterns without progress
Environment	exhaustion	boundary and infrastructure conditions

All detectors are computed without model calls; all are attached as structured
OTel span attributes + events so they surface in existing observability
pipelines without a separate sidecar.

Phases

Phase 1 — Analyzer + observability integration ✅ DONE

Delivered by #903.

• Rust port of the Python reference (katanemo/signals) into
  crates/brightstaff/src/signals/, aligned with the paper's three-layer
  taxonomy and 20 leaf SignalTypes
• Strongly-typed SignalReport exposing per-layer, per-category counts
  and severity (no stringly-keyed maps)
• Dual-emit OTel attributes:
  - New layered keys (signals.interaction.misalignment.count, …)
  - Legacy aggregate keys kept for backward compatibility
• Per-instance span events (signal.<type> with confidence, snippet, metadata)
• Summary string grouped by taxonomy
• Parity harness (tests/parity/signals/) — 100% agreement with Python
  reference on 2,000 lmsys/lmsys-chat-1m samples (seed 42)

Phase 2 — Parity, coverage, documentation 🔜 IN PROGRESS

Tracked in the follow-up issue (child of this epic). Briefly:

• Documentation for the new OTel attribute set; deprecation notice for legacy keys
• Synthetic tool-trace dataset for execution.* + environment.* parity
  (lmsys covers only interaction — 10/20 leaf types fire naturally there)
• Pin the HF dataset revision in the parity workflow for bit-reproducibility
• Upstream the grouped-summary format to katanemo/signals (optional)
• Legacy OTel key sunset in a subsequent release

References

• Paper: arXiv:2604.00356
• Python reference implementation: katanemo/signals
• Plano Phase 1 PR: signals: feature parity with the latest Signals paper. Porting logic from python repo #903
• Phase 2 follow-ups:
• Related observability surface: Add Prometheus metrics endpoint and Grafana dashboard for brightstaff #904 (brightstaff Prometheus metrics)

Authors / DRIs

• Paper: @nehcgs, @adilhafeez, @salmanap
• Plano implementation DRI: @syedhashmi