Skip to content

✨ retro: record the knowledge-advisor arm per bead for the holdout outcome comparison (advisor phase 2, slice 4) #17156

Closed
1 of 1 issue completed

Description

@clubanderson

Part of #17054 (approved direction: #17054 (comment)). RFC: docs/rfc-17054-llm-ranked-knowledge-advisor.md § "Measurement". Land on v5.

Readiness: needs the ranking decorator (#17155) merged, so decisions exist to attribute.

Why

The approved measurement is a holdout comparison using counters the retro lane already reconstructs (RetroRecord in src/pkg/retro/retro.go: KicksReceived, CIFailureCount, FixAttempts, drift pauses). Adding each bead's advisor arm to that record answers "did delivered advice help" without a new model call, a "did this help" line, or searching PRs for slugs — all three were rejected in the RFC and the owner decision.

Scope

  • src/pkg/retro/retro.go: add to RetroRecord — AdvisorArm string (none|shadow|deliver|holdout|mixed), AdvisorDecisions int, AdvisorDelivered int, AdvisorHeldOut int.
  • Source of truth: the decision records written by the ranking decorator (prompt-history entries with trigger knowledge-advisor, plus the decision log line). Prefer reading the prompt-history store the retro lane already reads over parsing logs; if a kick-id to bead mapping is needed, derive it from the kick's issue/PR ref the same way RetroRecord.IssueRef/PRRef are filled today.
  • src/pkg/retro/analysis.go: include the arm fields in the compact record handed to the analysis model as counters only (no reasons, no pane text).
  • Expose a per-arm summary (counts and the mean of FixAttempts, CIFailureCount, KicksReceived per arm) on the retro lane's existing report surface so an operator can compare arms without running a model.

Acceptance criteria

  • A bead whose kicks were all in the holdout arm yields AdvisorArm: holdout; mixed kicks yield mixed; no decisions yield none.
  • The analysis prompt payload contains the new counters and no advisor reason text (test asserts absence).
  • The per-arm summary is produced with zero model calls (retro.analysis_model empty in the test).
  • Existing retro tests unchanged in outcome.

Tests expected

src/pkg/retro/retro_test.go additions with a fixture of decision records. CI is the verdict.

Housekeeping

  • Changelog fragment: changelog.d/added-retro-advisor-arm.md, single - bullet.
  • Docs: retro section of src/docs/agent-knowledge-and-memory.md (or the retro doc if separate): what the arm means and how to read the comparison.

Activity

  1. added
    help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.
    on Nov 8, 2026
  2. hivecommons-hive commented on Nov 8, 2026

    @hivecommons-hive
    Contributor

    Deferred this cycle: readiness requires the ranking decorator (#17155) merged so decisions exist to attribute. Not starting until then.


    🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

    — hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

  3. hivecommons-hive commented on Nov 8, 2026

    @hivecommons-hive
    Contributor

    🔒 Claimed by agent scanner (agent on hive hosted-available-oke-11-placeholder-r05x). The claim lasts 2h0m0s without activity and renews while the work continues. Reply /unclaim to release, or /claim to take it over if you outrank the holder.

    Claim history
    • 2026-11-08 06:36 UTC — claimed by scanner (agent on hive hosted-available-oke-11-placeholder-r05x)
  4. hivecommons-hive commented on Nov 8, 2026

    @hivecommons-hive
    Contributor

    PR #17207 covers this issue: #17207

    The PR body says this is partial/non-closing work (Refs/Part of or leave-open wording), so Hive will not add a closing keyword. GitHub will not show its issue-list PR icon without a closing keyword; the stage comment is the link.

  5. hivecommons-hive commented on Nov 8, 2026

    @hivecommons-hive
    Contributor

    All children delivered: #17211 → PR #17218.

    Every tracked child of this issue is closed, so the hive is closing it as completed. Reopen it if something it asked for is still missing.

  6. github-actions commented on Nov 8, 2026

    @github-actions
    Contributor

    @clubanderson this issue was closed by @hivecommons-hive[bot] as completed.

    If this was closed by a merged PR, the fix only takes effect once it ships in a tagged release and your deployment updates to it — check your running version against the fix before concluding the fix failed.

    If you are still seeing the problem, comment /reopen and it will be reopened for you — you do not need write access. Please add what you are still observing (running version, logs, steps) so it is not closed the same way again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/schedulerIssues or PRs related to edge workspaceenhancementNew feature or requesthelp wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.hive/partially-addressedA merged PR addressed only part of this issue; remainder still actionable

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions