Repository navigation
✨ knowledge: LLM ranking decorator over the mid-task adviser — rank-only, fail-closed, shadow/rank/holdout - #17202
Conversation
…ly, fail-closed, shadow/rank/holdout) Phase 1 types the matcher's top hit on the first sight of a scope, which fires on passing reads as often as on real work. Letting a small model choose among the matcher's candidates, or choose nothing, cuts that noise without giving up phase 1's guarantee: the agent only ever receives a stored, reviewable knowledge entry, so a hostile pane can at most make the advisor pick a weaker candidate or stay silent, never type attacker text. The pass fails closed: invalid answers count as null, null never falls back, and budgets (per agent-hour, per UTC day) bound hive spend before any request is made. Shadow mode and a hashed holdout arm exist so the change can be measured on a live hive before it changes delivery, and one decision log line per call carries what that measurement needs. The scheduler now keeps a short per-agent kick summary with an identity that moves on every primed kick, so the model sees what the agent was asked to do and the holdout arm is stable within a kick. Closes #17155 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com> Co-authored-by: clubanderson <407614+clubanderson@users.noreply.github.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor. hive/awaiting-ci: CI pending — sweep will merge when green. 🐝 Hive Agent: — hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88 Review findings from Conclusion: the production change reads correct to me; the PR is blocked by its own new test, which fails deterministically (3/3 re-runs in The failing test
The real runtime path does more than the fixture: What I verified in the production code (no findings)
Once the test is fixed and the shard is green this looks ready to me. Evidence bundle `#172028676fa2` sha256:ae24d22a297d7fc943555bd0b4936ec166fbff4f01f418434160b132ae6eac1a — download from the dashboard or `hivectl review evidence #17202`. |
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of No conflicting open PRs or same-file overlaps were found. |
There was a problem hiding this comment.
Conclusion: the decorator itself reads correctly — rank-only, fail-closed, the model's answer is validated against the candidate set and the reason never reaches the pane. But the PR's own test is red deterministically, and the kick identity the decorator keys on is not the one the runtime uses. Both are small, fixable in this PR.
1. TestKnowledgeAdviserLLM_Prompt cannot pass (CI test (rest 4/4) is red on this) — medium
src/cmd/hive/knowledge_adviser_llm_test.go:521 asserts !strings.Contains(prompt, "ghp_") after feeding the tail through logscrub.RedactPaneTokens. That scrubber keeps the first seven characters by design — crux/logscrub/pane.go:37-40 returns m[:7] + "***", i.e. ghp_aaa*** — so the scrubbed tail always contains ghp_ and the prompt carries it verbatim (knowledge_adviser_llm.go, buildKnowledgeAdviserLLMPrompt writes scope.PaneTail unchanged, which is the stated design). This is a test defect, not a code defect: the production path is fine. Assert on the full literal instead, e.g. strings.Contains(prompt, "ghp_"+strings.Repeat("a", 36)) must be false and strings.Contains(prompt, "ghp_aaa***") true. The failing job log shows only this assertion firing.
2. Decorator kick identity ≠ runtime kick identity — low
scheduler.KickSummary.ID advances in recordKickSummary (src/pkg/scheduler/kick_priming.go, called from primeKnowledgeStats → templates.go:362) every time a kick message is rendered. The runtime, however, gates advice on kickAt from agent.LastKick (crux/agent/knowledge_advisor.go:353-369), i.e. on dispatch. The two diverge in both directions:
- Rendered-but-withheld kicks bump the ID without a dispatch:
kickmessage.go:333("kick message withheld"), and the freshness/provider-backoff drops incmd/hive/main_eval_cycle.goafterBuildKickMessagesWithSkips. - Review-swarm kicks (
main_eval_cycle.go:1015-1023) dispatch without going throughprimeKnowledgeStats, so the ID does not move.
Consequences, all within the LLM path only: knowledgeAdvisorHoldout(agent, kick.ID, …) can flip arm mid-kick, contrary to src/docs/agent-knowledge-and-memory.md:173-174 ("every scope in one kick lands in the same arm"); freshCandidates forgets the kick's delivered slugs, so the model may spend a budgeted call re-picking a slug the runtime already typed — the runtime drops it silently at knowledge_advisor.go:365 while the decision log says outcome=delivered; and the in-Advise "kick moved" check (knowledgeOutcomeGated) can fire on a render that never dispatched. Not a blocker for shadow rollout, but the measurement claims in the docs rest on arm stability, so either key on the runtime's kick (expose LastKick through the seam/scope) or document this as a known gap.
Checked and fine
- Pick validation: an unknown slug or malformed JSON →
invalid→ silence, never fallback (parseKnowledgeAdviserLLMAnswer). Pane text is only ever a prompt input. - Budget:
takeis pre-call and refuses without spending; UTC day rollover and rolling hour window read correctly. mode: off/ unresolved block falls straight through toinner.Advise, so phase 1 is byte-identical.- API key only in the
Authorizationheader; the decision log carrieserr.Error()but no key. Response body capped at 1 MiB. primeKnowledgeStatschange:issues[:limit]withlimit=0on an empty/nil slice is safe;len(issues)==0still returns an empty injection after recording the summary.- Changelog fragment, docs and
#17155scope line up with the diff.
I have not approved, closed, or changed anything.
— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
Review follow-ups (below
|
|
Triage of the red Cause is in the new test, not the adviser. The fixture pre-scrubs the pane tail with Suggested fix (test only): // 36 'a's is the raw secret; the scrubber leaves "ghp_aaa***" behind by design.
if strings.Contains(prompt, "ghp_"+strings.Repeat("a", 36)) {
t.Error("prompt carries an unredacted token")
}
if !strings.Contains(prompt, "ghp_aaa***") {
t.Error("prompt lost the scrubbed token marker")
}That keeps the intent (the secret never reaches the ranker) while asserting the scrubber's real contract, and it still fails if Filed by quality agent (ACMM L4/L6 — full mode) 🐝 Hive Agent: — hive: agent=quality backend=copilot model=claude-fable-5.1 copilot=1.0.88 |
RedactPaneTokens keeps a 7-char prefix (ghp_aaa***), so asserting the prompt contains no "ghp_" contradicted the verbatim-pane-tail check. Assert on the raw secret and the masked form instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
|
Thank you for your contribution! Your PR has been merged. We'd love to hear how your experience was: share feedback |
Why
Phase 1 of the mid-task advisor types the matcher's top hit the first time a scope appears, including on passing reads. RFC
docs/rfc-17054-llm-ranked-knowledge-advisor.md(§ Pipeline, § Ranker vs. generator, § Budget, § Measurement) proposes a ranking pass: a small model sees the kick summary, the scrubbed pane tail and the matcher's candidates, and picks one candidate slug ornull. Because the model only ranks, the agent still receives only stored knowledge entries. A hostile pane can at most cause a weaker pick or silence. It can never get text typed into a session.This is slice 3 of #17054. It builds on the async seam (#17153) and the
knowledge.advisor_llmconfig block (#17154).What
src/cmd/hive/knowledge_adviser_llm.go: a decorator around the phase-1knowledgeAdviser, installed whenKnowledgeAdvisorLLM()resolves at boot. It re-reads the config on every call, andmode: offgives exactly the phase-1 behaviour.WantsPaneTailandPaneLines.adviseUpTo), not by changing the global. Kick-primed and already-delivered slugs are removed first. If no candidates remain, no model call is made.temperature: 0, max 120 tokens,user: hive-knowledge-advisor:<agent>andtimeout_s. The answer is parsed withjsonextract.Object. A slug outside the candidates, or a malformed answer, counts asinvalidand is treated asnull. There is no retry.nullgives silence. Errors, timeouts and spent budgets give silence or the phase-1 top-1, depending onon_error.shadowdelivers the phase-1 top-1 and logs the model's pick.rankwithholdout_percenthashes the kick identity into a delivery arm or a holdout arm; holdout kicks log the pick and deliver nothing.knowledge advisor: decisionline with agent, kick, scope kind/key/action, candidates, pick, reason, verdict, outcome, latency and token counts. Thereasonis never typed into the pane.KickSummary(an ID that changes on every primed kick, plus primed issue refs, titles and labels, ≤600 chars).httptestmodel server and a fake clock. They cover each acceptance criterion: off behaves byte-identically to phase 1, invalid slug, null never falls back, timeout with matcher vs silence, the 7th per-agent call and 301st fleet call, shadow, rank with 100% holdout, and prompt contents.src/docs/agent-knowledge-and-memory.mdnow covers the pipeline, modes, budgets, the log line and rollout. Also adds the changelog fragment.Closes #17155
— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88 requested_by=@clubanderson