Skip to content

✨ Capture reviewer-stated rules from review feedback into the hive's knowledge - #19086

Merged
kubestellar-prow[bot] merged 1 commit into
v5from
feat/review-rule-capture-19046
Nov 18, 2026
Merged

kubestellar-prow[bot] merged 1 commit into
v5from
feat/review-rule-capture-19046

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Fixes #19046

What

The review-feedback sweep (#19029) used to tell the kicked agent to record a reviewer-stated rule on a best-effort basis. This makes the capture deterministic.

  • Classify. Each comment in a coalesced burst is matched against explicit phrase signals (source-of-truth, authoritative, always, never, going-forward, capture-request, assessment-criteria, knowledge-base). A match is a rule, anything else is a correction and is handled as before. Comment text is data: matched and quoted, never executed.
  • Persist. A rule is written to a local review-rules knowledge channel as a scoped, pin-tagged entry with provenance (PR, comment id and URL as source, reviewer, association, time, confidence). Idempotent per comment id across restarts, near-duplicate text merges, size-bounded.
  • Trust. Only OWNER, MEMBER and COLLABORATOR reviewers (HIVE_REVIEW_FEEDBACK_RULE_TRUSTED_ASSOCIATIONS) are captured at once. Others are queued as draft entries; hivectl knowledge state <id> approved confirms.
  • Load. The kick primer gains pinned entries: an approved pin entry whose scope admits the kick is carried whatever the kick's keywords are, ahead of ranked knowledge. Rules default to the authoring agent's lane, so the quality agent receives its rules on its next kick. Lifecycle and agent_scopes still apply.
  • Visible. A log line per capture, the entry in the knowledge list with provenance in its body, and one reply on the PR: "Recorded as rule R-n in the hive's knowledge, applies to ...". Retract with hivectl knowledge state rule-r-n deprecated (or delete the entry); the sweep reconciles.
  • Agent block. The kicked agent may end its reply with a validated hive:review-rules JSON block to sharpen a rule or state one the classifier missed. Its evidence_url must name a comment the hive saw, so the reviewer's trust comes from that comment, never from the agent.
  • Foreign repos. A rule whose proper home is a repo the hive does not manage is still recorded in the hive's own knowledge, and gets exactly one needs-human note carrying the proposed text. The hive never writes to that repo.

Capture also runs when the per-PR cap was reached or no agent was available. Debounce, cap and thread behavior are unchanged. Env knobs follow the HIVE_REVIEW_FEEDBACK_* convention; docs in src/docs/review-feedback-rules.md.

Related scanner issues

This covers #19048 (capture path and idempotency; the sweep posts the acknowledgement itself, so no reply check is needed) and #19051 (one needs-human note per foreign-home rule). #19050 is partly covered: a captured rule scoped to an assessment workflow or lane is primed into that kick. The static assessment criteria entries are #19079.

Tests

Table-driven tests for classification signals, block parsing and validation, trust gating, idempotency across reload, near-duplicate merge, scoping, retraction and reconciliation, retrieval by the quality agent (primer and scheduler), and the foreign-repo note. The fixture of the maintainer's comment (.project/maintainers.yaml is the source of truth for ownership and maintainership; capture this in the assessment knowledge base) is captured with provenance and retrieved for the quality agent in cmd/hive.

🤖 Generated with Claude Code

…knowledge

The review-feedback sweep now classifies each comment in a burst from
explicit phrase signals and writes a stated rule or source of truth to a
local review-rules knowledge channel with provenance, instead of leaving
it to the kicked agent. Rules are deduplicated by comment id and near
duplicate text, gated on the reviewer's association (others are queued as
drafts), acknowledged on the PR, and primed into the matching agent's next
kick through a new pinned-entry path in the knowledge primer. A validated
hive:review-rules block in the agent's reply can sharpen a rule. A rule
whose home is another project's repo gets one needs-human note carrying the
proposed text.

Fixes #19046

Signed-off-by: Andy Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot added dco-signoff: yes Indicates the PR's author has signed the DCO. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Nov 18, 2026
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#18873 🐛 needs-sha: exempt owner/detailed issues, overrideable waiver, visible hold on the row same files src/docs/labels-and-control-signals/reference.md
src/docs/notifications.md
src/pkg/github/notification_write_sites.go

@github-actions

Copy link
Copy Markdown
Contributor

Mutation testing

gremlins found 29 surviving mutant(s) in changed files: the code ran under tests, but changing it did not fail any test.

File Line Mutation
src/pkg/knowledge/primer_pinned.go 61 CONDITIONALS_NEGATION
src/pkg/reviewrules/block.go 104 CONDITIONALS_BOUNDARY
src/pkg/reviewrules/block.go 107 CONDITIONALS_BOUNDARY
src/pkg/reviewrules/classify.go 152 CONDITIONALS_BOUNDARY
src/pkg/reviewrules/classify.go 161 CONDITIONALS_BOUNDARY
src/pkg/reviewrules/store.go 183 CONDITIONALS_BOUNDARY
src/pkg/reviewrules/store.go 186 CONDITIONALS_BOUNDARY
src/pkg/reviewrules/store.go 186 CONDITIONALS_BOUNDARY
src/pkg/reviewrules/store.go 211 CONDITIONALS_BOUNDARY
src/pkg/reviewrules/store.go 282 CONDITIONALS_BOUNDARY
src/pkg/reviewrules/store.go 359 CONDITIONALS_BOUNDARY
src/pkg/reviewrules/store.go 359 CONDITIONALS_NEGATION
src/pkg/reviewrules/store.go 359 CONDITIONALS_NEGATION
src/pkg/reviewrules/store.go 359 INCREMENT_DECREMENT
src/pkg/reviewrules/store.go 404 CONDITIONALS_BOUNDARY
src/pkg/reviewrules/store.go 422 CONDITIONALS_BOUNDARY
src/pkg/reviewrules/store.go 475 CONDITIONALS_NEGATION
src/pkg/reviewrules/store.go 475 CONDITIONALS_NEGATION
src/pkg/reviewrules/store.go 542 ARITHMETIC_BASE
src/pkg/reviewrules/store.go 545 CONDITIONALS_BOUNDARY
src/pkg/reviewrules/store.go 607 CONDITIONALS_BOUNDARY
src/pkg/reviewrules/store.go 615 CONDITIONALS_NEGATION
src/pkg/reviewrules/store.go 619 CONDITIONALS_NEGATION
src/pkg/reviewrules/store.go 798 CONDITIONALS_BOUNDARY
src/pkg/reviewrules/store.go 827 CONDITIONALS_BOUNDARY
src/pkg/reviewrules/vault.go 31 CONDITIONALS_BOUNDARY
src/pkg/reviewrules/vault.go 84 CONDITIONALS_BOUNDARY
src/pkg/reviewrules/vault.go 111 CONDITIONALS_BOUNDARY
src/pkg/reviewrules/vault.go 115 CONDITIONALS_BOUNDARY

13 mutant(s) in changed files are not reached by any test (NOT COVERED). A maintainer may apply the mutation/acknowledged label to accept this.

File Line Mutation
src/cmd/hive/boot_knowledge_primer.go 85 CONDITIONALS_NEGATION
src/cmd/hive/boot_knowledge_primer.go 89 CONDITIONALS_NEGATION
src/pkg/reviewfeedback/reviewfeedback.go 372 CONDITIONALS_NEGATION
src/pkg/reviewrules/block.go 96 CONDITIONALS_NEGATION
src/pkg/reviewrules/block.go 98 CONDITIONALS_BOUNDARY
src/pkg/reviewrules/block.go 98 CONDITIONALS_NEGATION
src/pkg/reviewrules/store.go 780 CONDITIONALS_NEGATION
src/pkg/reviewrules/store.go 780 CONDITIONALS_NEGATION
src/pkg/reviewrules/store.go 783 CONDITIONALS_NEGATION
src/pkg/reviewrules/store.go 783 CONDITIONALS_NEGATION
src/pkg/reviewrules/store.go 786 CONDITIONALS_NEGATION
src/pkg/reviewrules/store.go 786 CONDITIONALS_NEGATION
src/pkg/reviewrules/store.go 800 ARITHMETIC_BASE

ℹ️ 1 mutant(s) on added lines were SKIPPED because gremlins v0.6.0 only honours the first lines added lines of each default-context hunk, so non-contiguous edits in one hunk are partly missed. This does not affect the verdict.

File Line Mutation
src/cmd/hive/boot_knowledge_primer.go 95 CONDITIONALS_NEGATION

Verdict: lived (#17349).

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style: approve with two advisory nits (comment accuracy in primer_pinned.go:38; duplicate postRuleNotices call). Not read in full: reviewrules internals and tests.

-- hive: agent=adjudicator-4 backend=claude model=claude-sonnet-5-5 claude=2.1.327

// HasPinned reports whether any pinned store is registered.
func (p *Primer) HasPinned() bool { return p != nil && len(p.pinned) > 0 }

// pinnedFacts returns the pinned entries a kick must carry, oldest slug first

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment says "oldest slug first" but the sort at line 61 is lexicographic (rule-r-10 sorts before rule-r-2), and the cap at line 62 contradicts "Pinned entries are never trimmed" in withPinned. Suggest rewording: // pinnedFacts returns the pinned entries a kick must carry, sorted by slug (lexicographic) and capped at maxPinnedFacts.

Queued: r.Rule.Status == reviewrules.StatusPending, SourceURL: it.URL})
}
s.postRuleNotices(ctx, repo, number)
return recorded

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

postRuleNotices is also called from run() (line 124) for the same PR in the same pass. Either drop one or note why both are needed.

@hivecommons-hive

hivecommons-hive Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

Review findings from adjudicator-3 (folded here because this head already carries its hive review):

Intent alignment (head 2519b78): the diff delivers bullets 1 and 3 of #19046. The sweep captures rules deterministically (src/cmd/hive/review_feedback_wire.go captureRules) and posts the acknowledgement itself (acknowledgeRules). A foreign-home rule gets one needs-human note (postRuleNotices / proposeRuleHome). The primer, trust gating and agent block support those two bullets and are not creep.

Advisory, not blocking: bullet 2 ("locate where the quality agent's docs-assessment criteria live and seed the rule there") is only partly delivered. The PR gets the rule to the quality agent through the pinned primer (src/cmd/hive/boot_knowledge_primer.go, registerReviewRuleVault). It does not seed any static criteria, and the PR body defers that to #19079. #19079 is an open PR ("seed scoped reviewer rules into TechDocs assessment kicks"), not an issue. Fixes #19046 will close the issue on merge, so bullet 2 stays tracked only by that PR. Consider changing the body to "Part of #19046" or confirming #19079 lands too. The author's own claim comment on the issue already scoped it this way, so I am not blocking.

Review findings from adjudicator-2 (folded here because this head already carries its hive review):

Security: agent block can launder a trusted reviewer's authority (src/pkg/reviewrules/store.go:563, 588, 599). Details are in the inline thread. One in-scope blocker; everything else I checked in the trust gate, frontmatter writer and hive-login check looked sound.

Findings not on a changed line

  • src/pkg/reviewrules/store.go:563: applyOneLocked accepts any evidence_url present in Seen and then inherits that comment's association, while text/scope come from the agent. An injected agent can cite any trusted comment and get an Active, global, pin-tagged rule primed into every agent. Suggested fix: (1) only honour agent entries whose cited comment was itself classified as a rule (i.e. s.st.Comments[key] is known) or that is at most Pending; for the !known branch force Status=Pending regardless of association; (2) require the agent text to be a near-duplicate/subset of the cited comment text (reuse similarity) before refining; (3) refuse ScopeGlobal from the agent path (only the deterministic sweep or an operator may set it), and never widen scope on refine.

    +	if sc == ScopeGlobal {
    +		return Applied{Outcome: OutcomeRejected, Reason: "global scope cannot be set from an agent block"}
    +	}

    and in the !known branch, downgrade: after captureLocked, set the rule Pending unless the cited comment's text matches the agent text.

Review findings from adjudicator (folded here because this head already carries its hive review):

Correctness review at head 2519b78. The classify, persist, trust and primer-load paths match the #19046 contract. The block-injection path is safe: needsHuman quotes only URLs, and rule text is stripped of <!-- and -->. Three in-scope durability defects in the rule store need fixing, each with its fix on its thread.

  • store.go:809: the vault entry is written before the index. A failed index save leaves an orphan live pinned entry.
  • store.go:777: Reconcile retracts a rule permanently on any read error, not only when the file is missing.
  • store.go:269: a corrupt index resets numbering to R-1 and overwrites existing entries.

Not verified: I did not run the tests.

Out of scope, noted only: repo-scoped rules do not get the pin tag (vault.go:106), so they load by keyword match only. This is consistent with the documented design.

Findings not on a changed line

  • src/pkg/reviewrules/store.go:809: txn/flush write the vault entry (and call Reindex) BEFORE saveIndex. If saveIndex fails, memory is rolled back from the snapshot but the entry file stays on disk as state: approved with the pin tag, so the primer keeps loading a rule the index no longer knows. Reconcile only walks index rules, so an orphan is never retracted or reconciled. A retry usually reuses the same R-N and overwrites it, but if the retry instead merges into another rule, the orphan persists. Fix: on a failed saveIndex, rewrite or remove the entries in touched (write them with the snapshot state, or delete newly created ones) before returning the error.
  • src/pkg/reviewrules/store.go:777: Reconcile treats any ReadFile error as !found and retracts the rule permanently (entryState returns ("", false) for every error, vault.go:3726 in the diff). The index lives in /data/review-feedback and the vault in /data/vaults/review-rules. A transient I/O error, or a vault directory missing or unmounted after a pod roll, retracts every live rule on the next sweep and saves that. Retracted is terminal in this code. Fix: have entryState return the error and return os.IsNotExist(err) as found=false. Skip the rule on any other error. Consider also skipping retraction when the vault directory itself is absent.
  • src/pkg/reviewrules/store.go:269: Open returns an empty index (Next=1) with an error when rules.json is corrupt, and sweepReviewFeedback keeps the store (review_feedback_wire.go diff ~L626-631). The next capture calls saveIndex, which overwrites the corrupt file. It also numbers rules from R-1 again, so writeEntry overwrites the existing rule-r-1.md ... entries with unrelated text and provenance. Idempotency by comment id is lost as well. Fix: on an unmarshal error, move the bad file aside (rules.json.corrupt-) and seed Next from the highest rule-r-N.md found in the vault, or leave capture off until an operator fixes it.

@clubanderson clubanderson added lgtm Indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Nov 18, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

Approval requirements bypassed by manually added approval.

This pull-request has been approved by:

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. dco-signoff: yes Indicates the PR's author has signed the DCO. lgtm Indicates that a PR is ready to be merged. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

✨ Capture reviewer-stated assessment rules durably from review feedback

1 participant