Skip to content

πŸ› Reviewer-stated rules: ignore quotes and questions, match more future phrasings, and give the preferences a Settings card - #19143

Merged
kubestellar-prow[bot] merged 7 commits into
v5from
fix/review-rules-capture-backfill
Nov 19, 2026
Merged

kubestellar-prow[bot] merged 7 commits into
v5from
fix/review-rules-capture-backfill

Conversation

@clubanderson

@clubanderson clubanderson commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Fixes #19135

A maintainer asked that rules reviewers state on the hive's assessment PRs ("X is the source of truth", "we should consider this in future") land in the hive's knowledge base. Thanks to @nate-double-u for the request and the concrete example on cncf/techdocs#424. #19086 and #19095 added the capture; this PR fixes why that example would never be captured once deployed, and gives the preferences a Settings home.

What changes

The capture half of #19135 (every human comment is a rule candidate, an evaluated-id index separate from the sweep's handled ids, one-time backfill) landed in #19146 while this PR was in CI; this PR is rebased on it and keeps the rest.

  • Classifier fixes. Block quotes (>), fenced code and questions never count, so a comment that quotes the hive and asks "is there a way to add to the knowledge-base?" is not recorded as the reviewer's rule (with ✨ Review-rule capture reads its own candidate setΒ #19146 such plain PR comments are now candidates, so this matters). A short rule sentence that refers back ("we should consider this in future") keeps the rest of the comment. going-forward also matches "future assessments", "moving forward" and "for the future".
  • Preferences are config with a Settings card. review_feedback.rules.enabled, trusted_associations, extra_phrases and disabled_signals in hive.yaml (the env vars still work; a saved value wins). Owner-only GET/PUT /api/config/review-feedback/rules, validated, audited as config_review_feedback_rules, applied on the sweep's next pass without a restart (Store.SetConfig). The card sits under Settings, Features, Review, with a one-line explanation, the default, and where each current value comes from. Defaults are unchanged: OWNER,MEMBER,COLLABORATOR, all built-in signals, no extra phrases. The vocabulary reaches the dashboard through pkg/config, so pkg/dashboard gains no import.

Beads

The issue compares the beads ledger with the review-rules channel. Beads do not reach the authoring agent's next kick directly, have no per-scope pinning, provenance or retraction semantics, so this PR keeps the channel and adds no beads path.

Tests

  • pkg/reviewrules: future-looking phrasings, quotes, code and questions, the real inline comment from Add draft documentation analysis for Hive Commons (pilot)Β cncf/techdocs#424, extra phrases and disabled signals, phrase validation, live config change, env parsing.
  • cmd/hive: a conversation comment that quotes the hive and asks is not captured while one stating a rule is; settings win over env; capture off yields no store.
  • pkg/config and pkg/dashboard: round trip, owner-only, validation, audit, clear.

πŸ€– Generated with Claude Code

@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 18, 2026
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#19199 ✨ ask: hub backend for optional Ask AI help (phase 1) same files src/pkg/config/config.go
#19201 πŸ“– Dashboard UI map, docs-site help links and a Settings entry per tab same files src/docs/settings.md
src/pkg/dashboard/static/index.html
#19192 πŸ“– docs: prepare Ask AI dashboard help coverage same files src/docs/settings.md
#19102 πŸ› Let the agent that opened a PR reply on its review threads CONFLICT src/cmd/hive/review_feedback_wire.go (conflict)

@kubestellar-prow kubestellar-prow Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 18, 2026
@kubestellar-prow kubestellar-prow Bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Nov 18, 2026
@clubanderson
clubanderson force-pushed the fix/review-rules-capture-backfill branch from a5308df to 5750061 Compare November 18, 2026 21:18
@kubestellar-prow kubestellar-prow Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 18, 2026
@github-actions github-actions Bot added the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Nov 18, 2026
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

UI render evidence

Render evidence captured.

Built from b5af845e4747.
Screenshots: download the artifact (sign-in to GitHub required).
Run: https://github.com/hivecommons/hive/actions/runs/44441950419

Page Section Viewport Theme File Size
hive overview desktop light hive-overview-desktop-light.png 312 KB
hive overview desktop dark hive-overview-desktop-dark.png 306 KB
hive overview phone light hive-overview-phone-light.png 466 KB
hive overview phone dark hive-overview-phone-dark.png 464 KB
hub-fleet overview desktop light hub-fleet-overview-desktop-light.png 113 KB
hub-fleet overview desktop dark hub-fleet-overview-desktop-dark.png 115 KB
hub-fleet overview phone light hub-fleet-overview-phone-light.png 265 KB
hub-fleet overview phone dark hub-fleet-overview-phone-dark.png 270 KB
hub-dashboard overview desktop light hub-dashboard-overview-desktop-light.png 338 KB
hub-dashboard overview desktop dark hub-dashboard-overview-desktop-dark.png 344 KB
hub-dashboard overview phone light hub-dashboard-overview-phone-light.png 624 KB
hub-dashboard overview phone dark hub-dashboard-overview-phone-dark.png 656 KB

These are fixture renders, not live data: the page was built from the PR head and filled with synthetic data from src/scripts/fixtures/render-evidence/. They show layout and theming, not how real hives behave. How to read them: src/docs/ui-render-evidence.md.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness: no in-scope blocker found at 5750061. Two low out-of-scope notes, which don't block this PR:

  • src/cmd/hive/review_feedback_wire.go:258: the debounce uses CreatedAt and Evaluated is keyed by comment id, so a comment edited into a rule after it was first classified is never re-read.
  • src/cmd/hive/review_feedback_wire.go:268: refused comments (rule limit reached) are retried and warn on every sweep pass.

-- hive: agent=adjudicator backend=claude model=claude-sonnet-5-5 claude=2.1.327

@hivecommons-hive

hivecommons-hive Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Hive status β€” updated in place; edits do not notify. Full history is in the Activity Monitor.

Review follow-ups: 3 open of 3 (below review.backlog.min_severity: P2)

Non-blocking findings below the backlog filing line, listed here instead of as issues. Tick them off as they are fixed; set review.backlog.min_severity: P3 to file every finding.

  • P3 low correctness β€” Comment edited after the debounce is never re-read (src/cmd/hive/review_feedback_wire.go:258): The debounce compares s.Now to it.CreatedAt and Evaluated is keyed by comment id (store.go CaptureComment marks every non-refused outcome). A comment that read as a non-rule when first classified and…
  • P3 low correctness β€” Comments classified as not-a-rule are never re-read after a Settings change (src/pkg/reviewrules/store.go:551): CaptureComment marks a comment evaluated on OutcomeNotRule (store.go:551). Adding an extra phrase or re-enabling a disabled signal in the new Settings card (applied via SetConfig) therefore never re-…
  • P3 low correctness β€” Refused comments warn on every sweep pass (src/cmd/hive/review_feedback_wire.go:268): OutcomeRefused (rule limit reached or failed write) deliberately does not mark the comment evaluated (store.go CaptureComment). Each later pass therefore re-runs Remember and CaptureComment and logs …

@clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Why:

  • the branch conflicts with the base and needs a rebase

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

Review findings from adjudicator-4 (folded here because this head already carries its hive review):

Correctness: one low-severity finding on the anaphor fallback in ExtractRuleText (see inline). The rest of the classifier, SetConfig locking and the settings API read correct. Tests not run.

Findings not on a changed line

  • src/pkg/reviewrules/classify.go:342: The fallback uses own, which still contains question sentences, so 'We should consider this in future. Is there a way to add to the knowledge base?' records the question as the rule. Build the fallback from the non-question sentences instead:
    -		text = NormalizeText(own)
    +		var parts []string
    +		for _, s := range sentences(own) {
    +			if !s.question {
    +				parts = append(parts, s.text)
    +			}
    +		}
    +		text = NormalizeText(strings.Join(parts, ". "))

stage 080-awaiting-merge-admission β†’ 071-needs-author: conflicting

Review findings from ci-maintainer (folded here because this head already carries its hive review):

Correctness pass at d87faeb: no blocking issues. The settings path, locking in CaptureComment/SetConfig and the PUT rollback hold up.

Two low, advisory edge cases in src/pkg/reviewrules/classify.go:

  1. The anaphor fallback in ExtractRuleText (text = NormalizeText(own)) uses the unquoted body, so question sentences are recorded despite the docs saying they never are. Suggested fix:
-	if len(strings.Fields(text)) < minStandaloneWords && anaphor.MatchString(text) {
-		text = NormalizeText(own)
-	}
+	if len(strings.Fields(text)) < minStandaloneWords && anaphor.MatchString(text) {
+		var rest []string
+		for _, s := range sentences(own) {
+			if !s.question {
+				rest = append(rest, s.text)
+			}
+		}
+		if len(rest) > 0 {
+			text = NormalizeText(strings.Join(rest, " "))
+		}
+	}
  1. sentences() splits on newlines, so a hard-wrapped question ("Could we capture this\nin the knowledge base?") leaves the first line unflagged and it is captured as a rule.

@github-actions github-actions Bot removed the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Nov 18, 2026
@clubanderson clubanderson added lgtm Indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Nov 18, 2026
…re phrasings, and give the preferences a Settings card

The rule classifier read the whole comment, so a reply that quotes the
hive and asks whether the hive can add something to the knowledge base
would be recorded as the reviewer's rule with the hive's words as its
text. Block quotes, fenced code and questions no longer count, a short
rule sentence that refers back keeps the rest of the comment, and
going-forward also matches future assessments, moving forward and for
the future.

Which phrases count and whose rules are recorded at once are the owner's
preferences: review_feedback.rules.enabled, trusted_associations,
extra_phrases and disabled_signals in hive.yaml, with the env vars still
honoured and a saved value winning. An owner-only Settings card under
Features, Review edits them in place, validated, audited as
config_review_feedback_rules, and applied on the sweep's next pass.

The candidate set and evaluated-id backfill this issue also asked for
landed in #19146.

Fixes #19135

Signed-off-by: Andy Anderson <andy@clubanderson.com>
@clubanderson
clubanderson force-pushed the fix/review-rules-capture-backfill branch from ee59ad0 to 8b397ba Compare November 18, 2026 21:53
@clubanderson clubanderson changed the title πŸ› Read reviewer-stated rules from every PR comment once, and give their preferences a Settings card πŸ› Reviewer-stated rules: ignore quotes and questions, match more future phrasings, and give the preferences a Settings card Nov 18, 2026
@clubanderson clubanderson removed lgtm Indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Nov 18, 2026
@github-actions github-actions Bot added the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Nov 18, 2026
@hivecommons-hive

hivecommons-hive Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

hive/scanner: this PR has merge conflicts with v5 (mergeable_state=dirty). It is not a hive-authored branch, so I am not pushing to it; please rebase onto origin/v5 and force-push, and the sweep will pick it up once CI is green.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

@github-actions github-actions Bot removed the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Nov 18, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

Merge blocked: this PR now has merge conflicts with the base branch. Please rebase or merge the base in (signed off); it was otherwise CI-green and approved.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

Resolve conflicts with #19144: keep the Classifier/sentence-based quote and
question handling from this branch, take v5 doc line numbers, and keep the
review-feedback/rules API rows.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@github-actions github-actions Bot added the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Nov 19, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI is red on this head for a docs reason, and the fix is mechanical. I did not trace the classifier logic in pkg/reviewrules in depth, so this is not a correctness sign-off.

api-reference citations drifted (changelog-fragment-guard, relative links and anchors resolve). The PR adds /api/config/review-feedback/rules and edits dashboard/openapi.json plus route-registering and static page files, so the guard reports: "api-reference citations must be resynced". Run bash src/scripts/check-api-reference-citations.sh --fix and commit the result. If static pages under src/pkg/dashboard/ changed, the same job asks for the matching baseline in src/pkg/dashboard/webstatic/style_ratchet_test.go (styleRatchetBaselines) to be lowered.

native-queue-policy is also failing on exact-head evidence ("Admission blocked"); I did not read the provider verdict it names, so that one needs the author's eye.

The conflicts-with-open-pr label is set; I did not compare it against the other PR.

-- hive: agent=reviewer-2 backend=claude model=claude-sonnet-5-5 claude=2.1.327

…le is too short to stand alone

Signed-off-by: Andy Anderson <andy@clubanderson.com>
Signed-off-by: Andy Anderson <andy@clubanderson.com>
Signed-off-by: Andy Anderson <andy@clubanderson.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness review at head 56fb518: no blockers. The classifier now ignores quotes, fenced code and questions, and short back-referring rule sentences keep the surrounding comment. Settings merge over env as documented, SetConfig and CaptureComment take the store lock, and the owner-only API validates input and rolls back when the save fails. One low edge case is on the line comment (hard-wrapped questions). I did not run the tests.

-- hive: agent=ci-maintainer backend=claude model=claude-sonnet-5-5 claude=2.1.327

@@ -85,45 +199,69 @@ var metaClause = regexp.MustCompile(`(?i)^\s*(?:please\s+|and\s+)?(?:capture|rec

var sentenceSplit = regexp.MustCompile(`[.!?]+\s+|\n+`)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low: every newline is a sentence boundary here, and a sentence is a question only if it ends in '?' (line 256). A hard-wrapped question such as "Could we document this\nin future?" leaves its first line as a non-question that matches capture-request, so it is recorded as a rule. An optional fix is to join single newlines within a paragraph before splitting, keeping blank lines, list items and headings as boundaries.

@clubanderson clubanderson added lgtm Indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Nov 19, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

Approval requirements bypassed by manually added approval.

This pull-request has been approved by:

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@github-actions github-actions Bot removed the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Nov 19, 2026
Signed-off-by: Andy Anderson <andy@clubanderson.com>

# Conflicts:
#	src/docs/api-reference/auth.md
Signed-off-by: Andy Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot removed the lgtm Indicates that a PR is ready to be merged. label Nov 19, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

New changes are detected. LGTM label has been removed.

@github-actions github-actions Bot added the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Nov 19, 2026
@clubanderson clubanderson added the lgtm Indicates that a PR is ready to be merged. label Nov 19, 2026
@kubestellar-prow
kubestellar-prow Bot merged commit 317b388 into v5 Nov 19, 2026
102 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the fix/review-rules-capture-backfill branch November 19, 2026 02:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. conflicts-with-open-pr Open PR has a merge conflict with another open PR dco-signoff: yes Indicates the PR's author has signed the DCO. lgtm Indicates that a PR is ready to be merged. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

πŸ› Reviewer-stated rules are skipped once feedback is handled, in plain PR comments, and their preferences have no Settings home

1 participant