Skip to content

🐛 Reviewer-stated rules are skipped once feedback is handled, in plain PR comments, and their preferences have no Settings home #19135

Closed
3 of 3 issues completed

Description

@clubanderson

Problem

A maintainer reviewing a hive-authored assessment PR asked that rules stated in review ("X is the source of truth", "we should consider this in future") land in the hive's knowledge base. This request comes from @nate-double-u (cncf/techdocs#424). #19086 and #19095 added deterministic capture into the review-rules knowledge channel, but once deployed three gaps keep the concrete example from ever being captured:

  1. Capture is gated on the feedback-handled state. captureRules only sees d.Items, which come from UnansweredFeedback (src/pkg/github/pr_feedback.go:236). That drops every comment named in a hive:feedback-handled marker (line 261), every inline comment older than the hive's last reply in its thread or in a resolved thread (lines 279, 303), and every id the sweep store already recorded (src/pkg/reviewfeedback/reviewfeedback.go:215). A rule a reviewer stated before the capture code was deployed, or in a thread the agent already answered, is never classified. The inline comment carrying the maintainers.yaml rule on Add draft documentation analysis for Hive Commons (pilot) cncf/techdocs#424 is already in such a marker.
  2. Issue-level PR comments only count when they address the hive. A conversation comment is fed to capture only when it mentions the App bot or the configured AI author, or starts with /hive (pr_feedback.go:187, :269). A maintainer stating a rule in a plain PR comment, or addressing the operator, is never classified.
  3. Quoted text and questions are classified as the reviewer's rule. Classify matches the whole body (src/pkg/reviewrules/classify.go:97). A comment that quotes the hive's own reply (a > block) and then asks "is there a way to add to the knowledge-base?" would be recorded as a rule attributed to the reviewer, with the hive's words as its text.

Smaller: future-looking phrasings such as "future assessments should ..." or "moving forward" do not match (classify.go:70 requires "for future ..."), a rule sentence that only says "we should consider this in future" is recorded without the text it refers to, and which phrases count and which associations are trusted are env-only knobs with no Settings home.

Fix (one PR)

  • Rule capture reads its own candidate set: every human comment on the hive's open PR (inline, review body, conversation) whatever its handled, resolved or addressed state. The rule index tracks which comments it has evaluated, separately from the sweep's handled ids, so the first pass after deploy is a one-time backfill over the comments still visible on hive-authored PRs in the attribution window, and later passes classify only new comments. Routing kicks is unchanged.
  • The classifier ignores quoted (>) lines, fenced code and questions, and falls back to the whole comment when the rule sentence is too short to stand alone.
  • Broaden going-forward to "future assessments/analyses/reviews/work", "for the future", "moving forward".
  • Preferences become review_feedback.rules.* keys in hive.yaml (enabled, trusted associations, extra phrases, disabled signals), with env vars still honoured, and an owner-only Settings card (Features, Review) to edit them in place without a restart.
  • Tests for each case, docs, changelog fragment.

Beads or the review-rules channel?

The operator asked whether the beads ledger (bd remember, bd create) would be enough. Compared:

beads (bd remember / advisory bead) review-rules channel (#19086)
Reaches the authoring agent on its next run No direct path. Open beads appear as work or advisory findings; a fact reaches kicks only after it is closed, synthesized into bead-synth-wiki, and ranked by keyword match Yes. Pinned entries are primed into every matching kick ahead of ranked knowledge, keyword-less kicks included (#19095)
Scope per agent, workflow or repo Per-agent store only agent, workflow, repo, global scopes
Provenance Title and free-form metadata; bd remember authors as system PR, comment URL, reviewer, association, time, source
Dedupe Title upsert only Per comment id plus near-duplicate text merge
Retraction Close the bead, but advisory beads are re-marked stale or re-filed by the advisory loop hivectl knowledge state rule-r-N deprecated, reconciled by the sweep
Owner visibility Bead ledger view Dashboard knowledge list, plus the "Recorded as rule R-n" reply on the PR
New code needed Priming, scoping, provenance and retraction would all have to be built None for storage; this issue only fixes what gets classified

Choice: keep the review-rules channel and do not add a beads path. Beads would duplicate the store without reaching the agent any better. This issue stays scoped to the capture bugs and the settings.

Acceptance

  • Once deployed, the maintainers.yaml inline comment on Add draft documentation analysis for Hive Commons (pilot) cncf/techdocs#424 is captured as a rule on the next sweep pass, even though it is already in a handled marker.
  • A trusted reviewer's plain PR comment stating a rule is captured; one that only quotes the hive and asks a question is not.
  • The four preferences are editable by the owner in Settings and take effect without a restart.

Activity

  1. added
    help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.
    on Nov 18, 2026
  2. hivecommons-hive commented on Nov 18, 2026

    @hivecommons-hive
    Contributor

    Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

    • scanner (2026-11-18 21:10 UTC): Decomposed into three child sub-issues (capture candidate set, classifier, settings): the work spans several independent concerns. The next kick picks them up.
  3. added a commit that references this issue on Nov 18, 2026
    5750061
  4. added a commit that references this issue on Nov 18, 2026
    8b397ba
  5. clubanderson commented on Nov 18, 2026

    @clubanderson
    MemberAuthor

    The capture half of this issue (rule candidates from every human comment, an evaluated-id index separate from the handled ids, one-time backfill) landed in #19146 via the duplicate #19136. #19143 is rebased on it and carries the rest: the classifier fixes for quotes, code, questions and future-looking phrasings, and the review_feedback.rules.* preferences with their Settings card.

  6. hivecommons-hive commented on Nov 19, 2026

    @hivecommons-hive
    Contributor

    All children delivered: #19136 → PR #19146, #19137 → PR #19144, #19138.

    Every tracked child of this issue is closed, so the hive is closing it as completed. Reopen it if something it asked for is still missing.

  7. added a commit that references this issue on Nov 19, 2026
    317b388
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.triage/acceptedIndicates an issue or PR is ready to be actively worked on.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions