Skip to content

signals: follow-ups deferred from PR #903 #909

@syedhashmi

Description

@syedhashmi

Title

signals: follow-ups deferred from PR #903

Summary

PR #903 landed the Rust port of the Signals paper taxonomy into brightstaff
with 100% parity against the Python reference on 2,000 lmsys-chat-1m samples.
Three items came up during review that were intentionally scoped out of that
PR. This issue tracks them so they don't drop on the floor.

Follow-ups

1. Documentation for the new OTel attribute set — P1

The layered attribute keys (signals.interaction.misalignment.count,
signals.execution.failure.severity, etc.) and per-instance span events
(signal.<type> with confidence/snippet/metadata) added in #903 are not yet
documented. Legacy keys (signals.frustration.*, signals.escalation.requested,
etc.) are being kept in dual-emit for backward compatibility and should be
flagged as deprecated in the docs.

Acceptance:

  • docs/source/guides/observability/ has a signals attributes reference
    listing all layered keys with descriptions and example values
  • Legacy keys section marks each as deprecated with a migration pointer
  • docs/source/observability/monitoring.rst lead example updated to show
    layered keys

2. Upstream Python reference: grouped summary format — P3, optional

In #903 the Rust generate_summary was refactored to produce a
taxonomy-grouped output. The Python reference at katanemo/signals
still emits the flat format. The parity comparator classifies summary
as Tier-C so this is not a correctness issue, but it's the one known
intentional divergence and worth aligning upstream.

Acceptance:

  • Open a PR against katanemo/signals porting the grouped-summary
    format (same layer headers, same severity notation, same alert tail)
  • Re-run the parity harness and confirm Tier-C divergence drops to zero

3. Legacy OTel key sunset — P3

Decided offline during #903 review to keep signals.frustration.*,
signals.escalation.requested, signals.follow_up.repair.*,
signals.positive_feedback.count, signals.repetition.count for one
release for backward compatibility. This issue tracks the eventual removal.

Acceptance:

  • Identify all internal consumers of legacy keys (dashboards, alerts,
    samplers, any downstream ingest)
  • Migrate each consumer to the layered keys
  • Remove emit_legacy_attributes from crates/brightstaff/src/signals/otel.rs
  • Remove legacy constants from crates/brightstaff/src/tracing/constants.rs
  • Bump version per the release process in CLAUDE.md

Context

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions