Skip to content

✨ feat: bounded scrub-drop Matrix line source behind dashboard.matrix - #15967

Merged
hivecommons-hive[bot] merged 2 commits into
v5from
scanner/fix-15934
Oct 31, 2026
Merged

hivecommons-hive[bot] merged 2 commits into
v5from
scanner/fix-15934

Conversation

@hivecommons-hive

@hivecommons-hive hivecommons-hive Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Why

Matrix mode (RFC #15862, @hanthor) needs a stream of real pushed lines, but those lines come from contributor patches and could carry credentials. This adds the in-memory source that turns sentinel patch observations into displayable lines, and it is built so that it can only drop content, never reveal it. The source has no HTTP surface yet; the endpoint and SSE nudge follow in #15935.

Safety invariants

  • Default off. sentinelSweepOptions sets PatchObserver only when dashboard.matrix.Active() and a dashboard server exists. Otherwise no observer is registered and nothing is created or retained. If the flag is turned off later, the existing source stops admitting observations and clears its backlog. A cmd test asserts that PatchObserver is nil when the flag is off.
  • Uses the existing fetch only. The only input is SentinelPatchObservation.Files[].Patch. There are no new GitHub calls.
  • Hive-identified authors only. Observations with HiveIdentified=false are rejected in Observe, before anything is queued.
  • Public allowlisted repos only. A repo must satisfy !Private && cfg.Dashboard.Matrix.RepoAllowed(repo).
  • Drop, never mask. If logscrub.ScrubString or logscrub.RedactPaneTokens would change a line, the line is dropped. So is any line that already has a redaction marker, contains a token-alphabet run of 32 or more characters, or is longer than 160 runes. Long lines are dropped rather than truncated, so a cut can never expose part of a token. BEGIN/END armour blocks such as PEM keys are dropped whole. Denied paths (.env*, keys/certs, id_*, secret/credential/token segments, lockfiles, vendor/node_modules, minified/generated files, exclude_paths) are skipped before the patch is parsed.
  • Bounded. Observe sends to a 64-slot channel without blocking and counts drops when it is full. A single goroutine does the processing and recovers from panics. Limits: 200 lines per PR head, a 2,000-line ring, dedup sets with a 24h TTL capped at 500 PRs plus a global hash cap, 600 lines per minute, and per-observation file and patch-size caps.

Tests cover the fixture-secret patch, each gate, path deny-list, dedup across re-observe and rebase, and every bound under synthetic load. They also check that a full channel never blocks, and test the wiring when the flag is off.

New dependency: pkg/dashboard now imports crux/logscrub (added to internal/testutil/dashboard_import_allowlist.txt) because the drop-never-mask invariant requires the scrubber itself (ScrubString, RedactPaneTokens, ContainsRedactionMarker), and no existing seam exposes it to pkg/dashboard.

Closes #15934
Part of #15862 (Matrix mode RFC tracker; endpoint + SSE nudge follow in #15935)

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88 requested_by=@hanthor

Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Oct 31, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign danathar for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Oct 31, 2026
@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#15849 ✨ Show online ClankeR IDs on the Contributors count pill CONFLICT src/docs/api-reference/dashboard-misc.md
#15977 ✨ feat: serve Matrix lines via authorized /api/matrix/lines and content-free code-lines SSE nudge CONFLICT src/docs/api-reference/auth.md
src/docs/api-reference/dashboard-misc.md
src/docs/api-reference/health-status-events.md
src/docs/api-reference/snapshots-and-style.md
src/pkg/dashboard/matrix_source.go
src/pkg/dashboard/server.go

@github-actions github-actions Bot added the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Oct 31, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

hive/awaiting-ci: CI pending — sweep will merge when green.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot added author/hive This repository's own hive (the App or one of its agent lanes) authored this PR dco All commits are DCO signed-off role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 31, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security perspective — approve (advisory only). Read head d923601 plus pkg/github/sentinel_sweep.go, pkg/config/dashboard_matrix.go, crux/logscrub/{handler,pane}.go, and cmd/hive/main_sweeps.go. Trust boundary holds: HiveIdentified/Private are set by the sentinel (sentinel_sweep.go:102-103), not the observer; admits re-checks Active(), allowlist and both flags in Observe and again in ingest (matrix_source.go:107-111,163-165); sentinelSweepOptions is re-evaluated per sweep (main_sweeps.go:1333,1355) so flag-off reset is live. Fixture test covers every acceptance-criteria secret shape.

Two low notes, neither blocking:

  1. Armour tracking only sees + lines — matrix_source_filter.go:101-113. If a PEM -----BEGIN header is a context (unchanged) line and only the body is re-added (rebase/rotation), inArmour never flips. Body lines are 64 base64 chars and die on the entropy gate, but the final base64 line of a PEM is often <32 chars and would be emitted. Cheap fix: track armour state across all lines of the hunk (not just +), and/or also drop any line matching ^[A-Za-z0-9+/]+={0,2}$ of length ≥16.

  2. No short-secret heuristic (by design per #15934, out of scope): matrixAcceptLine (matrix_source_filter.go:120-141) only drops on logscrub patterns, ≥32-char runs, length and markers. password: hunter2 in a non-denied path (e.g. config/app.yaml) is admitted. Lines are already public on GitHub so this is amplification, not disclosure; worth a (?i)(password|passwd|secret|api[_-]?key)\s*[:=] drop rule in a follow-up.

— hive: agent=adjudicator-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

Hive filed review findings that do not block this PR as backlog items:

Cap: 10 backlog item(s) per PR per day.

@hivecommons-hive

hivecommons-hive Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor Author

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

Review findings from reviewer (folded here because this head already carries its hive review):

Read the two new source files against the v5 tree, the cmd/hive wiring, and the existing SentinelPatchObservation/PatchObserver plumbing in pkg/github. The gating, drop-only filtering, dedup and bounds all do what the body says; go build ./pkg/dashboard/ ./cmd/hive/ is clean. One correctness issue worth fixing before merge.

Panic recovery leaves m.mu locked forever — a panic becomes a permanent, silent hang (medium).

  • src/pkg/dashboard/matrix_source.go:150-158 — safeIngest wraps ingest in recover() so the processing goroutine survives a panic.
  • src/pkg/dashboard/matrix_source.go:173 / :223 — ingest takes m.mu.Lock() and releases it with an explicit m.mu.Unlock(), not a defer. Everything between (pruneLocked, matrixAddedLines, matrixAcceptLine, appendLocked, evictLocked) runs under the lock.

Mechanism: if anything in that region panics, recover fires in safeIngest but m.mu is never released. The run loop (:144) happily takes the next observation, calls ingest, and blocks on Lock() forever. From then on the 64-slot channel fills, Observe drops every observation and bumps dropped, and — once #15935 lands — Since/Latest (:298, :313) deadlock the HTTP handler that calls them. So the recover guards against exactly one failure mode and converts it into a worse one: no log line after the first warning, no crash, just a dead Matrix source until restart.

TestMatrixSourceRunNotifiesAndRecovers (matrix_source_test.go:398-427) does not catch this because it panics inside notify, which ingest deliberately calls after m.mu.Unlock() (:223-227).

Suggested fix: move the locked section into a helper that defers the unlock and returns (added, seq, notify), then call notify from ingest after it returns — keeps the "notify outside the lock" property and makes the recover actually recover. A test that injects a panic under the lock (e.g. via a test-only hook in appendLocked, or by making m.now panic once — it is called before Lock() today, so it would need to move or a different hook used) and then asserts a subsequent Observe is ingested would fail without the fix.

Everything else I checked looks correct to me: ring iteration order in Since is right for both the growing and wrapped cases; rebase dedup across heads works because hashes survives a head change while headCount resets; evictLocked accounting of totalHashes is consistent with removeLocked; sentinelSweepOptions is called per sweep (cmd/hive/main_sweeps.go:1333,1355), so MatrixObserver refreshing the allowlist on every call is live, and the nil-when-inactive path really does register nothing. Docs line-number churn in docs/api-reference/*.md is the expected citation shift from the five new Server fields.

Review findings from reviewer-2 (folded here because this head already carries its hive review):

Conclusion: the source itself reads correct to me, but the PR fails its own CI on a structural guard and cannot merge as-is: test (rest 2/4) is red because pkg/dashboard gained a direct import that the dashboard import ratchet forbids.

Blocking (correctness — CI red, caused by this diff)

src/pkg/dashboard/matrix_source_filter.go:9 imports github.com/hivecommons/hive/crux/logscrub. src/internal/testutil/dashboard_import_ratchet_test.go:59 fails on this head (run 41170770479, job test (rest 2/4)):

pkg/dashboard gained a new dependency on crux/logscrub — route it through an existing seam or add it to the allowlist with justification in the PR.

crux/logscrub is not in src/internal/testutil/dashboard_import_allowlist.txt (only crux, crux/agentparse, crux/claude, crux/kubejob, crux/paneclass, crux/sandbox, crux/skillreg, crux/toolapprove, crux/watchdog are). The coverage job then fails only as a consequence (rest=failure).

The repo already has the seam pattern for exactly this: src/pkg/dashboard/status_builder.go:3388-3393 (redactTokens → crux.RedactPaneTokens, "so dashboard applies the runtime's identical pane redaction without importing pkg/logscrub directly"), backed by src/crux/observation.go:74. But crux currently exposes only RedactPaneTokens; matrix_source_filter.go:124-128 also needs logscrub.ScrubString and logscrub.ContainsRedactionMarker. Two ways to go green, either is fine by me:

  1. Add ScrubString/ContainsRedactionMarker seams next to crux.RedactPaneTokens in src/crux/observation.go and call those from matrix_source_filter.go (keeps the ratchet tight — matches the existing pattern).
  2. Add github.com/hivecommons/hive/crux/logscrub to the allowlist (sorted) with the justification in the PR body.

What I verified and found sound

  • Gates are re-checked inside the worker: matrix_source.go:111 (admits) runs in Observe and again at the top of ingest (:160), so an observation already queued when MatrixObserver is called with an inactive config (:344-349, setConfig + reset) is discarded rather than ingested. Good.
  • Observe is non-blocking (:120-126) and the sentinel already wraps the call in its own goroutine + 2s bound with panic recovery (src/pkg/github/sentinel_sweep.go:107-125), so the sweep cannot stall on this source.
  • Ring iteration in Since (:296-304) starts at m.next, which is 0 until the ring fills and the oldest slot afterwards — order is correct in both regimes; reset (:316-322) clears next too.
  • evictLocked (:254-271) terminates: when only the just-ingested PR remains and its hash count alone exceeds the cap, len(m.prs) > 1 is false so it becomes evictable and totalHashes drops to 0.
  • Dedup-TTL expiry path (:191-199) refreshes the timestamp without double-counting totalHashes; TestMatrixSourceExpiredHashInLivePR covers it.
  • MatrixConfig.Active/RepoAllowed and SentinelPatchObservation{HiveIdentified,Private} already exist on v5 (src/pkg/config/dashboard_matrix.go:21-33, src/pkg/github/sentinel_sweep.go:82-91); no new GitHub calls are introduced, matching the body.
  • The src/docs/api-reference/*.md changes are purely the server.go line-citation shift from the 5 added struct fields.

No security, scope, or docs concerns beyond the above. Once the import is routed/allowlisted this looks ready for a maintainer.

Review findings from adjudicator-3 (folded here because this head already carries its hive review):

Security perspective (head ebb006f). Read the diff plus pkg/github/sentinel_sweep.go:70-124, pkg/config/dashboard_matrix.go, and crux/logscrub/{handler,pane}.go. The trust gates hold: admits (matrix_source.go:104-107) requires Active() && HiveIdentified && !Private && RepoAllowed and is re-evaluated inside the worker (matrix_source.go:171), so an observation queued before the flag flips off is rejected; every filter rule drops rather than rewrites (matrix_source_filter.go:112-135); PatchObserver is nil when inactive (main_sweeps.go:1289-1294, asserted by main_sweeps_matrix_test.go). No blocker. Two advisory notes:

  1. Low — malformed exclude_paths glob fails open silently. matrixPathDenied discards the path.Match error (matrix_source_filter.go:62,65), and MatrixConfig.Validate only rejects empty entries (pkg/config/dashboard_matrix.go:58-62). An operator exclusion like internal/[secret (ErrBadPattern) never matches, with no log or config error, so the operator believes a path is excluded when it is not. Suggested fix: in Validate, reject entries where path.Match(p, "") returns ErrBadPattern.

  2. Info — "forgets its backlog" is partial. reset() does m.ring = m.ring[:0] (matrix_source.go:303), which keeps the old matrixLine strings alive in the backing array until overwritten (clear(m.ring) first would release them). Separately, an ingest already past admits when MatrixObserver is called with an inactive config appends after reset() (matrix_source.go:264-267,339-345); the next inactive call clears it. Both are in-process-memory only and there is no read surface yet, so no exposure today, but worth tightening before [scanner] Serve Matrix lines via authorized /api/matrix/lines and content-free code-lines SSE nudge #15935 adds the endpoint.

Out of scope (pre-existing in #15933, noted for backlog): Private is pr.GetBase().GetRepo().GetPrivate() (sentinel_sweep.go:103), a nil-safe getter that reads as public when base.repo is absent from the PR payload. The explicit Repos allowlist is the real guard, so this only matters if an operator allowlists a private repo.

Review findings from adjudicator-2 (folded here because this head already carries its hive review):

Correctness review @ ebb006f — no blockers; one advisory.

Flag-off reset has a one-observation race with an in-flight ingest (src/pkg/dashboard/matrix_source.go:163-168 vs :310-318). ingest evaluates admits(obs) before taking m.mu; MatrixObserver with an inactive config does setConfig + reset() under m.mu. If an ingest passes admits on the old active config and then acquires the lock after reset(), its lines land in the ring and prs while the flag is off, until the next sweep's MatrixObserver call resets again. Window is a single observation and there is no reader yet (#15935), so this is advisory, but it slightly undercuts the "disabled ⇒ nothing retained" invariant in #15934. Fix: re-check m.cfg.Load().Active() after m.mu.Lock() (or move the admits call under the lock).

Everything else checked out: gates match the contract (admits at :106-109), dedup hash excludes head (:197), rate/head caps break before storing hashes so dropped lines are re-admissible later, Since ordering is correct for both partial and wrapped rings (:273-285), wiring is re-evaluated per sweep (src/cmd/hive/main_sweeps.go:1333,1355), and the fixture/gate/bound tests in matrix_source_test.go cover the acceptance criteria.

…ine source

Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness review (head d923601)

The source logic matches #15934: gates in admits (src/pkg/dashboard/matrix_source.go:107-110), re-checked in ingest (:166), drop-only line filter (matrix_source_filter.go:120-141), ring/dedup/rate/per-head bounds all hold as written, and sentinelSweepOptions is re-evaluated per sweep (src/cmd/hive/main_sweeps.go:1333,1355) so the nil-when-off wiring is live. go build ./pkg/dashboard/ ./cmd/hive/ is clean at head.

1. CI test (rest 2/4) fails: TestDashboardImportRatchet (blocking)
src/pkg/dashboard/matrix_source_filter.go:9 adds github.com/hivecommons/hive/crux/logscrub to pkg/dashboard, which is not in src/internal/testutil/dashboard_import_allowlist.txt (crux entries at lines 10-18 stop at kubejob/paneclass). The ratchet (dashboard_import_ratchet_test.go:59) fails the shard, and coverage fails downstream ("rest=failure"). Fix: add github.com/hivecommons/hive/crux/logscrub to the allowlist between crux/kubejob and crux/paneclass (entries are sorted, line 9) and note in the PR body that #15934 mandates logscrub.ScrubString/RedactPaneTokens as the drop oracle — there is no existing seam in pkg/dashboard that exposes them.

2. Lines can land in the backlog after the flag is turned off (low, non-blocking)
ingest checks admits before taking m.mu (matrix_source.go:166-176). If MatrixObserver runs setConfig(off) + reset() (matrix_source.go:345-349) between that check and the lock, the in-flight observation is still appended and survives until the next sweep calls MatrixObserver again. Cheap fix: re-check m.cfg.Load().Active() after m.mu.Lock() and return 0 if inactive.

3. Server.Close never stops the source goroutine (low, non-blocking)
contribute_ws.go:5072-5077 closes only the contribute hub; matrixSource.close() (matrix_source.go:134) is only called from tests. Harmless for the process-lifetime singleton, but worth wiring in with the endpoint child (#15935).

— hive: agent=adjudicator backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it and removed stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 31, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author
ebb006f

ℹ️ Sentinel notice — informational; author is trusted (hive agent), merge is not blocked

Hive flagged this PR (author @hivecommons-hive[bot], head ebb006f66879) because it matches behaviors that can override security controls, escalate privileges or damage the codebase. This is a heuristic, not an accusation — a maintainer should confirm the change is intended before it merges.

  • remote_code_execution — adds a remote-fetch-and-execute, decoded payload, reverse shell or embedded credential (Adds curl|sh, eval of encoded payloads, reverse shells or embedded private keys)
    • src/pkg/dashboard/matrix_source_test.go

Hive did not add the sentinel-alert label because this author is trusted. The finding remains recorded for audit and the dashboard Security tab; set sentinel.trusted_authors_block: true to require the blocking label for trusted authors too.

@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it labels Oct 31, 2026
@hivecommons-hive
hivecommons-hive Bot merged commit c1975c2 into v5 Oct 31, 2026
89 of 98 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the scanner/fix-15934 branch October 31, 2026 05:52
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author
Evidence bundle `#15967ebb006f` sha256:473e1a102388117a35b693849e3506cd530c3141e87adbe289098c9a986210d6 — download from the dashboard or `hivectl review evidence #15967`.

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/hive This repository's own hive (the App or one of its agent lanes) authored this PR conflicts-with-open-pr Open PR has a merge conflict with another open PR copilot dco All commits are DCO signed-off dco-signoff: yes Indicates the PR's author has signed the DCO. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[scanner] Build Matrix line source: scrub-drop safety pipeline, dedup and bounded backlog

0 participants