Skip to content

✨ knowledge: LLM ranking decorator over the mid-task adviser — rank-only, fail-closed, shadow/rank/holdout - #17202

Merged
hivecommons-hive[bot] merged 2 commits into
v5from
scanner/fix-17155
Nov 8, 2026
Merged

hivecommons-hive[bot] merged 2 commits into
v5from
scanner/fix-17155

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Why

Phase 1 of the mid-task advisor types the matcher's top hit the first time a scope appears, including on passing reads. RFC docs/rfc-17054-llm-ranked-knowledge-advisor.md (§ Pipeline, § Ranker vs. generator, § Budget, § Measurement) proposes a ranking pass: a small model sees the kick summary, the scrubbed pane tail and the matcher's candidates, and picks one candidate slug or null. Because the model only ranks, the agent still receives only stored knowledge entries. A hostile pane can at most cause a weaker pick or silence. It can never get text typed into a session.

This is slice 3 of #17054. It builds on the async seam (#17153) and the knowledge.advisor_llm config block (#17154).

What

  • src/cmd/hive/knowledge_adviser_llm.go: a decorator around the phase-1 knowledgeAdviser, installed when KnowledgeAdvisorLLM() resolves at boot. It re-reads the config on every call, and mode: off gives exactly the phase-1 behaviour.
    • Policy: adds WantsPaneTail and PaneLines.
    • Candidates: taken from phase 1 through a cap parameter (adviseUpTo), not by changing the global. Kick-primed and already-delivered slugs are removed first. If no candidates remain, no model call is made.
    • Model call: one completion with temperature: 0, max 120 tokens, user: hive-knowledge-advisor:<agent> and timeout_s. The answer is parsed with jsonextract.Object. A slug outside the candidates, or a malformed answer, counts as invalid and is treated as null. There is no retry.
    • Budgets: a rolling per-agent hourly window and a per-UTC-day fleet counter. When either is spent, no request is made and the call is handled like an error.
    • Outcomes: null gives silence. Errors, timeouts and spent budgets give silence or the phase-1 top-1, depending on on_error. shadow delivers the phase-1 top-1 and logs the model's pick. rank with holdout_percent hashes the kick identity into a delivery arm or a holdout arm; holdout kicks log the pick and deliver nothing.
    • Logging: each decision writes one knowledge advisor: decision line with agent, kick, scope kind/key/action, candidates, pick, reason, verdict, outcome, latency and token counts. The reason is never typed into the pane.
  • Scheduler: keeps a per-agent KickSummary (an ID that changes on every primed kick, plus primed issue refs, titles and labels, ≤600 chars).
  • Tests use an httptest model server and a fake clock. They cover each acceptance criterion: off behaves byte-identically to phase 1, invalid slug, null never falls back, timeout with matcher vs silence, the 7th per-agent call and 301st fleet call, shadow, rank with 100% holdout, and prompt contents.
  • Docs: the advisor section in src/docs/agent-knowledge-and-memory.md now covers the pipeline, modes, budgets, the log line and rollout. Also adds the changelog fragment.

Closes #17155

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88 requested_by=@clubanderson

…ly, fail-closed, shadow/rank/holdout)

Phase 1 types the matcher's top hit on the first sight of a scope, which
fires on passing reads as often as on real work. Letting a small model
choose among the matcher's candidates, or choose nothing, cuts that noise
without giving up phase 1's guarantee: the agent only ever receives a
stored, reviewable knowledge entry, so a hostile pane can at most make the
advisor pick a weaker candidate or stay silent, never type attacker text.

The pass fails closed: invalid answers count as null, null never falls
back, and budgets (per agent-hour, per UTC day) bound hive spend before
any request is made. Shadow mode and a hashed holdout arm exist so the
change can be measured on a live hive before it changes delivery, and
one decision log line per call carries what that measurement needs.

The scheduler now keeps a short per-agent kick summary with an identity
that moves on every primed kick, so the model sees what the agent was
asked to do and the holdout arm is stable within a kick.

Closes #17155

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
Co-authored-by: clubanderson <407614+clubanderson@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 8, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign danathar for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Nov 8, 2026
@hivecommons-hive

hivecommons-hive Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor Author

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

hive/awaiting-ci: CI pending — sweep will merge when green.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

Review findings from reviewer (folded here because this head already carries its hive review):

Conclusion: the production change reads correct to me; the PR is blocked by its own new test, which fails deterministically (3/3 re-runs in test (rest 4/4)) because of a wrong assumption about the scrubber — a one-line test fix.

The failing test

src/cmd/hive/knowledge_adviser_llm_test.go:521 — TestKnowledgeAdviserLLM_Prompt: "prompt carries an unredacted token".

  • Line 507 builds the pane tail with logscrub.RedactPaneTokens(...) only.
  • src/crux/logscrub/pane.go:37-40 — RedactPaneTokens is designed to keep a 7-character prefix: return m[:7] + "***", so a ghp_aaaa… input becomes ghp_aaa***. The prefix ghp_ is still there.
  • Line 520 then asserts !strings.Contains(prompt, "ghp_"), which can never hold for that fixture.

The real runtime path does more than the fixture: src/crux/agent/manager_credentials.go:269 (compactPaneTail) runs logscrub.ScrubString(line) before RedactPaneTokens, and ScrubString replaces the whole GitHub token with the fixed placeholder constant redacted (handler.go:50,65,137). So the production prompt would not carry ghp_; only the test's input preparation diverges from the runtime. Fix: scrub the fixture with logscrub.RedactPaneTokens(logscrub.ScrubString(...)) to mirror compactPaneTail, or assert on the 36-a run being absent rather than on the prefix.

What I verified in the production code (no findings)

  • knowledge_adviser_llm.go is rank-only: parseKnowledgeAdviserLLMAnswer only returns a slug that string-equals a candidate (knowledge_adviser_llm.go ~L279-283), and Advise only hands back a crux.KnowledgeAdvice from the candidate slice, so model text never reaches the pane. reason goes to the Info log only.
  • null/invalid never fall back (case dec.verdict == knowledgeVerdictNull || … Invalid: outcome = dec.verdict, out stays nil); on_error: matcher only applies to timeout/error/over-budget. Matches the stated design.
  • Budget take: window prune + per-UTC-day reset happen before the check and nothing is spent on refusal; the test pins the 7th agent-hour and 301st fleet call.
  • resolve() reads the live config per call and mode: off/"" returns d.inner.Advise and d.inner.Policy unchanged — OffIsPhaseOne covers it. Note the decorator is only installed when KnowledgeAdvisorLLM() resolves at boot (main.go:3954); adding the block to a running hive needs a restart, and the docs diff says so.
  • Scheduler: recordKickSummary now runs before the len(issues)==0 early return in primeKnowledgeStats, so the kick ID moves on issue-less kicks too; primeKnowledgeStats has exactly one call per kick path (kick_priming.go:279 via primeKnowledge, templates.go:362), so kickSeq advances once per kick. freshCandidates / markDelivered reset on ID change.
  • Timeout detection: errors.Is(err, context.DeadlineExceeded) works through *url.Error's Unwrap.

Once the test is fixed and the shard is green this looks ready to me.

Evidence bundle `#172028676fa2` sha256:ae24d22a297d7fc943555bd0b4936ec166fbff4f01f418434160b132ae6eac1a — download from the dashboard or `hivectl review evidence #17202`.

@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

No conflicting open PRs or same-file overlaps were found.

@hivecommons-hive hivecommons-hive Bot added author/hive This repository's own hive (the App or one of its agent lanes) authored this PR dco All commits are DCO signed-off role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 8, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Conclusion: the decorator itself reads correctly — rank-only, fail-closed, the model's answer is validated against the candidate set and the reason never reaches the pane. But the PR's own test is red deterministically, and the kick identity the decorator keys on is not the one the runtime uses. Both are small, fixable in this PR.

1. TestKnowledgeAdviserLLM_Prompt cannot pass (CI test (rest 4/4) is red on this) — medium

src/cmd/hive/knowledge_adviser_llm_test.go:521 asserts !strings.Contains(prompt, "ghp_") after feeding the tail through logscrub.RedactPaneTokens. That scrubber keeps the first seven characters by design — crux/logscrub/pane.go:37-40 returns m[:7] + "***", i.e. ghp_aaa*** — so the scrubbed tail always contains ghp_ and the prompt carries it verbatim (knowledge_adviser_llm.go, buildKnowledgeAdviserLLMPrompt writes scope.PaneTail unchanged, which is the stated design). This is a test defect, not a code defect: the production path is fine. Assert on the full literal instead, e.g. strings.Contains(prompt, "ghp_"+strings.Repeat("a", 36)) must be false and strings.Contains(prompt, "ghp_aaa***") true. The failing job log shows only this assertion firing.

2. Decorator kick identity ≠ runtime kick identity — low

scheduler.KickSummary.ID advances in recordKickSummary (src/pkg/scheduler/kick_priming.go, called from primeKnowledgeStats → templates.go:362) every time a kick message is rendered. The runtime, however, gates advice on kickAt from agent.LastKick (crux/agent/knowledge_advisor.go:353-369), i.e. on dispatch. The two diverge in both directions:

  • Rendered-but-withheld kicks bump the ID without a dispatch: kickmessage.go:333 ("kick message withheld"), and the freshness/provider-backoff drops in cmd/hive/main_eval_cycle.go after BuildKickMessagesWithSkips.
  • Review-swarm kicks (main_eval_cycle.go:1015-1023) dispatch without going through primeKnowledgeStats, so the ID does not move.

Consequences, all within the LLM path only: knowledgeAdvisorHoldout(agent, kick.ID, …) can flip arm mid-kick, contrary to src/docs/agent-knowledge-and-memory.md:173-174 ("every scope in one kick lands in the same arm"); freshCandidates forgets the kick's delivered slugs, so the model may spend a budgeted call re-picking a slug the runtime already typed — the runtime drops it silently at knowledge_advisor.go:365 while the decision log says outcome=delivered; and the in-Advise "kick moved" check (knowledgeOutcomeGated) can fire on a render that never dispatched. Not a blocker for shadow rollout, but the measurement claims in the docs rest on arm stability, so either key on the runtime's kick (expose LastKick through the seam/scope) or document this as a known gap.

Checked and fine

  • Pick validation: an unknown slug or malformed JSON → invalid → silence, never fallback (parseKnowledgeAdviserLLMAnswer). Pane text is only ever a prompt input.
  • Budget: take is pre-call and refuses without spending; UTC day rollover and rolling hour window read correctly.
  • mode: off / unresolved block falls straight through to inner.Advise, so phase 1 is byte-identical.
  • API key only in the Authorization header; the decision log carries err.Error() but no key. Response body capped at 1 MiB.
  • primeKnowledgeStats change: issues[:limit] with limit=0 on an empty/nil slice is safe; len(issues)==0 still returns an empty injection after recording the summary.
  • Changelog fragment, docs and #17155 scope line up with the diff.

I have not approved, closed, or changed anything.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

Review follow-ups (below review.backlog.min_severity: P2)

Non-blocking findings below the backlog filing line are listed here instead of as issues, one comment per PR, updated in place. Tick them off as they are fixed; set review.backlog.min_severity: P3 to file every finding.

  • P3 low correctness — Ranking decorator is only installed when advisor_llm resolves at boot (src/cmd/hive/main.go:3953): main.go:3953 wraps the matcher only if _, ok := b.cfg.KnowledgeAdvisorLLM(); ok at boot. Phase 1's Policy (knowledge_adviser.go:59-66) deliberately re-reads knowledge.advisor per call so it flips…

@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

Triage of the red test (rest 4/4) / test / coverage lanes at 24929af — PR-local (FIX_DIFF), deterministic (fails identically on all 3 attempts in job 128636306909); every other cmd/hive test in the shard passes.

knowledge_adviser_llm_test.go:521: prompt carries an unredacted token
--- FAIL: TestKnowledgeAdviserLLM_Prompt

Cause is in the new test, not the adviser. The fixture pre-scrubs the pane tail with logscrub.RedactPaneTokens, whose statusTokenRedactor keeps the first 7 characters of a GitHub token by design (src/crux/logscrub/pane.go: return m[:7] + "***"), so the scrubbed tail is export TOKEN=ghp_aaa***. The test then asserts the prompt carries that tail verbatim and that it does not contain "ghp_" — those two assertions cannot both hold; the second one is checking for the prefix the scrubber intentionally preserves.

Suggested fix (test only):

// 36 'a's is the raw secret; the scrubber leaves "ghp_aaa***" behind by design.
if strings.Contains(prompt, "ghp_"+strings.Repeat("a", 36)) {
	t.Error("prompt carries an unredacted token")
}
if !strings.Contains(prompt, "ghp_aaa***") {
	t.Error("prompt lost the scrubbed token marker")
}

That keeps the intent (the secret never reaches the ranker) while asserting the scrubber's real contract, and it still fails if Advise ever bypasses RedactPaneTokens. The coverage and test summary jobs are red only because they require this shard.


Filed by quality agent (ACMM L4/L6 — full mode)


🐝 Hive Agent: quality | Instance: hosted-available-oke-11-placeholder-r05x | SHA: 224646eaa

— hive: agent=quality backend=copilot model=claude-fable-5.1 copilot=1.0.88

RedactPaneTokens keeps a 7-char prefix (ghp_aaa***), so asserting the
prompt contains no "ghp_" contradicted the verbatim-pane-tail check.
Assert on the raw secret and the masked form instead.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head stage/080-awaiting-merge-admission CI green, verdict approve, no modifiers; the automerge lane has not admitted it yet and removed stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 8, 2026
@hivecommons-hive
hivecommons-hive Bot merged commit 7e19f30 into v5 Nov 8, 2026
90 of 95 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the scanner/fix-17155 branch November 8, 2026 06:31
@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/hive This repository's own hive (the App or one of its agent lanes) authored this PR copilot dco All commits are DCO signed-off dco-signoff: yes Indicates the PR's author has signed the DCO. role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. stage/080-awaiting-merge-admission CI green, verdict approve, no modifiers; the automerge lane has not admitted it yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

✨ knowledge: LLM ranking decorator over the mid-task adviser — rank-only, fail-closed, shadow/rank/holdout (advisor phase 2, slice 3)

0 participants