Skip to content

✨ Expose the knowledge advisor rank-versus-holdout summary through a read-only dashboard route #18897

Closed

Description

@clubanderson

Follow-up from the audit of #17054 (rank-versus-holdout comparison, owner condition 6).

Problem

retro.Lane.AdvisorArmSummary() computes the comparison the RFC needs (per arm: bead count, mean FixAttempts, mean CIFailureCount, mean KicksReceived; deliver versus holdout), but it is called only from tests and docs. No dashboard route, CLI or API exposes it, and the raw decisions live in /data/knowledge-advisor-decisions.jsonl on the spoke's PVC, so publishing the comparison on #17054 needs a manual exec or a throwaway program. The advisor settings are not exposed in the dashboard API either.

Wanted

  1. A read-only dashboard route, for example GET /api/knowledge/advisor/summary?since=<ts>, that returns:
    • the window (first and last decision timestamps, rank mode start if recorded, holdout percent),
    • decision counts by mode and arm (shadow, deliver, holdout) and by outcome (delivered, held out, rate-limited with no arm, invalid, timeout, over budget),
    • the AdvisorArmSummary() per-arm bead counts and means (mixed and none excluded or listed separately),
    • cost and latency from the decision records where present (mean latency, prompt and completion tokens).
      Never return slugs, refs or any content, only counts and means. Roles: read or higher, like other read-only knowledge endpoints.
  2. A small "Advisor" card or section in the Knowledge area of the dashboard showing the same, with a one-line caveat when bead counts are small.
  3. Expose the advisor settings read-only (knowledge.advisor*: mode, holdout_percent, model, budget) so the page can show what produced the numbers.
  4. OpenAPI/api-reference entry, docs (retro-lane.md, knowledge docs), tests (empty store, mixed arms, small-n caveat, role gating), changelog fragment.

Acceptance

The comparison for #17054 can be read from the dashboard (or curl with a token) with no exec, and the numbers match Lane.AdvisorArmSummary().

Activity

  1. self-assigned this
    on Nov 18, 2026
  2. clubanderson commented on Nov 18, 2026

    @clubanderson
    MemberAuthor

    Claimed by the operator's dispatcher; fix in progress.

  3. added
    stage/020-claimedClaimed by the scanner or a relay contributor
    and removed
    stage/020-claimedClaimed by the scanner or a relay contributor
    on Nov 18, 2026
  4. clubanderson commented on Nov 18, 2026

    @clubanderson
    MemberAuthor

    Delivered by #18904 (merged to v5): GET /api/knowledge/advisor/summary, the Knowledge-section Advisor card, OpenAPI/api-reference entries, docs and tests. The per-arm bead means are not narrowed by ?since (Lane.AdvisorArmSummary has no window); latency and token means populate from records written after this change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions