Skip to content

✨ Add Crux-owned durable event and work lanes - #16373

Merged
hivecommons-hive[bot] merged 10 commits into
v5from
feat/crux-durable-lanes-jetstream-20261102
Nov 3, 2026
Merged

hivecommons-hive[bot] merged 10 commits into
v5from
feat/crux-durable-lanes-jetstream-20261102

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Summary

Adds forge-neutral Crux-owned event/work lanes backed by NATS JetStream, with Hive as an opt-in peer participant. Work uses independent consumer groups, revision-fenced leases, replay, explicit expiration recovery, fair retries, and bounded durable acknowledgement history.

Includes namespace isolation, duplicate-publish identity verification, authorized foreign-expiration recovery, persisted retry rotation, and physical completion/recovery reservations with reusable nonshrinking slots.

Validation

Independent precommit review cleared staged diff SHA-256 31b7856138377b09c621cd46270ffc29a83f5b73cf29dc5b6361c703c365c2ea. Embedded-broker and API/serve/config regressions cover restart, lost responses, stale receipts, retry starvation, expiration recovery, and near-byte-capacity operation. Formatting and whitespace checks passed. Executable Go validation is pending canonical CI, as required by repository policy; no passing build/test claim is made yet.

Rollout

Opt-in only. No live NATS deployment or production enrollment is enabled by this publication.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 3, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign nicholasjackson for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Nov 3, 2026
@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Docs map: this PR adds page(s) under src/docs/ that
src/docs/documentation-map.md does not link:

  • crux-lanes.md

A page missing from the map is one readers do not find. Please add a
line in the section a reader would actually look in, matching the
format of its neighbours.

If the page is indexed from a subdirectory README, or is deliberately
unlisted for now, ignore this. It is a reminder, not a gate; it never
blocks a merge.

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

No conflicting open PRs or same-file overlaps were found.

@github-actions github-actions Bot added the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Nov 3, 2026
@hivecommons-hive

hivecommons-hive Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Why:

  • the review swarm requested changes

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

Review findings from adjudicator-4 (folded here because this head already carries its hive review):

Style re-review at 509abf8 (delta vs b1041e5: 6 files under src/crux/lanes/, all tests plus the encodeGroup validate-before-marshal reorder).

The delta is clean from a style perspective — processOutcome split into processResponse/waitPublisherProcess (publish_generation_test.go:174-199) is a readability win, and the encodeGroup reorder (work_budget.go:103-105) is pinned by two new tests. No new findings.

The three style findings from the previous head are still present and unchanged:

  1. src/crux/lanes/config.go:181-194 — the Recoverers loop is still nested inside the Groups loop, shadowing group/m and re-validating every recoverer once per group (and never when Groups is empty). Suggested fix: hoist it to a sibling loop:
 		for group, m := range l.Groups {
 			if !ValidID(group) || len(m) == 0 || len(m) > DefaultMaxMembers {
 				return fmt.Errorf("%w: delivery group", ErrInvalid)
 			}
-			for group, m := range l.Recoverers {
-				if !ValidID(group) || len(m) == 0 || len(m) > DefaultMaxMembers {
-					return fmt.Errorf("%w: recovery capability", ErrInvalid)
-				}
-				for _, p := range m {
-					if !allowed(l.Groups[group], p) {
-						return fmt.Errorf("%w: recoverer must be a group member", ErrInvalid)
-					}
-				}
-			}
 			members = append(members, m...)
 		}
+		for group, recoverers := range l.Recoverers {
+			if !ValidID(group) || len(recoverers) == 0 || len(recoverers) > DefaultMaxMembers {
+				return fmt.Errorf("%w: recovery capability", ErrInvalid)
+			}
+			for _, p := range recoverers {
+				if !allowed(l.Groups[group], p) {
+					return fmt.Errorf("%w: recoverer must be a group member", ErrInvalid)
+				}
+			}
+		}
  1. src/crux/serve/lanes.go:68,73,185-202 — lane error codes ("lane_invalid", "lane_conflict", …) remain bare string literals while the same file uses api.CodeUnauthorized at line 49. Add CodeLane* constants to src/crux/api and reference them here.
  2. src/crux/api/lanes.go:32-60 — exported wire types LaneEnvelope, LanePublished, LaneReplay, LaneRecord, LaneClaim still lack doc comments (LanePublish has one; the comment above LaneReplay describes the cursor, not the type).

Review findings from reviewer-2 (folded here because this head already carries its hive review):

Read the trust boundaries and wiring at 24a9ffc, not the full 7.5k-line diff. What I verified looks correct; what I did not read is named below so a human can scope their own pass.

Verified

  • Workflow edits (v2-ci.yml:200-207, integration-candidate.yml:258-261, integration-technical-v2-ci.yml:124-127) only add go test -race steps for the new packages. No permission, trigger or secret changes.
  • Auth split is real, not nominal. serve/serve.go:173-176 rejects a project token on every agent RPC; serve/lanes.go:46-51 requires a project token on every /v1/lanes path, and lanes/auth.go:23 refuses to bind a project token equal to the admin token or another project's, so the admin token cannot reach lanes and vice versa. Project() is nil-safe (auth.go:37) and scans every entry with subtle.ConstantTimeCompare, so an unconfigured server (LaneAuth == nil) falls through to the legacy mux untouched (lanes.go:26).
  • Every broker entry point authorizes against the registry before touching state: Publish (broker.go:192), Replay (:341), Claim (work.go:113), receipt ack/nack/renew (work.go:239), and receipts additionally bind l.Project != project → ErrStale (work.go:255), so a leaked receipt is useless to another principal.
  • Hub side is opt-in and non-fatal: cmd/hive/lanes.go:17-19 returns when HIVE_LANES_ADDRESS is unset; every config error logs and returns rather than aborting boot. governor.AddModeChangeObserver (governor.go:383-396) wraps rather than replaces the hook observer, and the read at governor.go:465 is still under g.mu with dispatch after release, so the existing lock discipline holds.
  • api/client.go:282: the 204 short-circuit is needed by ClaimLane on ErrEmpty (serve/lanes.go:120-122) and ack/nack (:180); without it the decoder would fail on an empty body.
  • nats-server/v2 lands in src/go.mod:63 as a direct dep, but it is imported only from _test.go files (serve/lanes_test.go, lanes/broker_test.go, lanes/work_slots_test.go, pkg/laneparticipant/participant_test.go) — the hub binary does not embed a broker.
  • Docs state the retry policy the code implements: no dead-letter, retry until ack or retention expiry (docs/crux-lanes.md:166-168; work.go:310-319 backoff capped by max_retry_delay).

Not read in depth: the Claim/recovery state machine in work.go:117-231 and work_recovery.go (CAS loop, slot reservation, retry rotation). The tests for it are extensive (recovery_test.go, retry_*_test.go, work_slots_test.go) and the body's description matches the code paths I did read, but I have not traced the fairness/expiry invariants end to end; build-go and patch-coverage were still running when I looked.

Info: Registry() (lanes/config.go:263-275) returns the full api.Lane (publishers, subscribers, group membership of other projects) to any member of the lane. That is configuration not credentials, and it is intentional per the docs table at crux-lanes.md:69, so just noting it is a deliberate disclosure surface.

Nothing blocking from the parts I read.

Review findings from adjudicator-3 (folded here because this head already carries its hive review):

HUMAN DECISION NEEDED — @clubanderson: accept a new Crux subsystem (NATS JetStream broker + nats-server/v2 dependency, src/crux/lanes/, src/pkg/laneparticipant/) with no linked issue/spec defining its intended scope.

Delta re-review (3c77f34 → 24a9ffc): the only new commit is Merge branch 'v5' (12 commits, 0 conflict hunks per git show --cc 24a9ffc). Every PR-owned file is byte-identical to the previous head; the 122 "changed" files are v5 work already landed (#16377, #16351, #16384, …). Nothing new to judge in the delta.

Prior findings at this head:

  • intent-alignment src/docs/crux-lanes.md:1 — no linked issue/spec: still present (PR body has no Fixes/Closes; body is the only contract).
  • correctness src/crux/lanes/broker.go:1 — broker/work-recovery not independently verified: still present (file unchanged).
  • style src/go.mod:63 — nats-server/v2 inside the // indirect block: still present.

Advisory (not blocking, author's call): changelog.d/fixed-crux-lane-diagnostics.md records a "fixed" entry for behaviour introduced by this same PR; no release ever had the bug, so readers will hunt for a regression that doesn't exist. Suggest folding it into changelog.d/added-crux-durable-lanes.md and deleting the fixed- fragment.

Review findings from adjudicator-5 (folded here because this head already carries its hive review):

docs-currency — delta re-review at 509abf8 (prior head b1041e5)

The delta (src/crux/lanes/{failure,publish_generation,publish_response,work_budget_shape,work_budget}_test.go, work_budget.go) is test-only plus an internal error-ordering swap in encodeGroup (work_budget.go:103-112); no operator-facing text changes with it. Both prior docs-currency findings are still present and still in scope:

  1. src/docs/env-vars.md has no rows for the new HIVE_LANES_* lookups. src/pkg/laneparticipant/participant.go adds os.Getenv for HIVE_LANES_ADDRESS, HIVE_LANES_LANE, HIVE_LANES_GROUP, HIVE_LANES_TOKEN, HIVE_LANES_SNAPSHOT_PROJECTS; grep LANES src/docs/env-vars.md at head is empty. The file's own rule (env-vars.md:598-600, "If your change adds … an environment variable lookup, update this file in the same PR") and its note that the converse direction is unenforced by CI (:611-613) make this a hand-maintained gap. Suggested rows, after the HIVE_DASHBOARD_COOKIE row in "Core hive runtime" (env-vars.md:56):
| `HIVE_DASHBOARD_COOKIE` | No | none | **Client-side only** - read by `hivectl tui`, never by the server. Cookie header value (e.g. `hive_session=...`) carrying a per-user session, for hives that do not accept the shared token: hub-hosted ones, and spokes with an `authorized_users` allowlist. See [hivectl.md, Credentials](hivectl.md#credentials). |
| `HIVE_LANES_ADDRESS` | No | none (lane participation disabled) | Opt-in gate for Hive joining a Crux durable lane as an ordinary project member; read by `pkg/laneparticipant` at boot. See [Crux durable lanes, Hive adapter](crux-lanes.md#hive-adapter). |
| `HIVE_LANES_TOKEN` | Required when `HIVE_LANES_ADDRESS` is set | none | Crux lane bearer token for the Hive participant. |
| `HIVE_LANES_LANE` | Required when `HIVE_LANES_ADDRESS` is set | none | Lane ID (`^[a-z][a-z0-9_-]*$`) Hive publishes to and subscribes from. |
| `HIVE_LANES_GROUP` | Required when `HIVE_LANES_ADDRESS` is set | none | Work group ID Hive leases from on that lane. |
| `HIVE_LANES_SNAPSHOT_PROJECTS` | No | none | Optional comma-separated explicit publisher project IDs whose lane events Hive mirrors into snapshots. |

(Adjust Required/Default to match participant.go exactly if any of these has a fallback I did not see.)

  1. src/docs/documentation-map.md does not link the new src/docs/crux-lanes.md. The map lists every other crux page (documentation-map.md:124-126, :233); the new page is unreachable from the index. Insert after line 125:
- [`crux serve`: the runtime over a local API](crux-serve.md) — **Developer:** operate the runtime server through its authenticated API, event stream, CLI, or Go client.
- [Crux durable lanes](crux-lanes.md) — **Developer/Operator:** Crux-owned durable event and work lanes: enabling them in `crux serve`, the public HTTP/Go API, cursors and leases, byte/slot bounds, the `HIVE_LANES_*` opt-in adapter, and deployment verification.

Limits: GitHub API hourly cap was reached mid-review, so I re-read participant.go via the full PR diff rather than at the file level; the five variable names above are copied from that diff.

Hive filed review findings that do not block this PR as backlog items:

Cap: 10 backlog item(s) per PR per day.

Review findings from ci-maintainer (folded here because this head already carries its hive review):

Docs-currency review (head c93b25b; read the diff plus src/docs/crux-serve.md, src/docs/crux-lanes.md, src/crux/lanes/config.go, src/crux/serve/lanes.go, src/crux/api/lanes.go, src/pkg/laneparticipant/participant.go, .github/workflows/v2-tests.yml, CONTRIBUTING.md).

The new crux-lanes.md checks out against the code: the bounds table matches lanes/config.go constants and Validate, the route/status table matches serve/lanes.go (400/401/403/409/429/503/504), the client method names match api/lanes.go, the hive-governor-*-v1 schemas match participant.go:28-30, and the "normal 90% coverage gate" claim matches v2-tests.yml (THRESHOLD=90, floor_key maps crux/<path>). Three places where prose did not move with behavior:

  1. src/docs/crux-lanes.md ("Deployment and verification", last two sentences) tells operators "Repository policy prohibits local Go test/build/vet; the executable verdict is CI, not a fake-store-only claim." That is false for readers of this page — CONTRIBUTING.md:64-65 instructs contributors to run go build ./... ./crux/... and go test ./... ./crux/... locally — and the rest of that paragraph is a PR-validation inventory (which tests exist and what they exercise) rather than operator guidance. Suggest trimming the paragraph to what an operator can act on:

    The named CI integration step tests real embedded JetStream lanes, API and
    serving in the standalone Crux module and the Hive participant through the
    root module; the normal affected-package coverage gate also covers
    `crux/lanes` and `pkg/laneparticipant`. Locally, `go test ./crux/lanes
    ./crux/api ./crux/serve ./pkg/laneparticipant` from `src/` runs the same
    suites (they start and remove their own embedded brokers).
    

    (replacing everything from "The named CI integration step…" to the end of the file).

  2. src/docs/crux-serve.md "## What this is not" (unchanged, final section) still says the server "is not tenancy, quotas or hive-as-a-client: the server trusts whoever can reach the socket or holds the token". With this PR the lane routes have per-project bearer credentials and capability ACLs (crux/lanes/auth.go, serve/lanes.go 401/403 paths), per-lane max_messages/max_bytes quotas (lanes/config.go), and Hive is a client of them (cmd/hive/lanes.go). The PR updated the intro paragraph on this page but not this section. Suggest:

    -It is not tenancy, quotas or hive-as-a-client: the server trusts whoever can
    -reach the socket or holds the token, and hive still constructs
    -`*agent.Manager` in-process. Those are the rest of Stage 3 in
    -[design/crux.md](design/crux.md#after-the-cut).
    +For agent control it is not tenancy, quotas or hive-as-a-client: those
    +routes trust whoever can reach the socket or holds the token, and hive still
    +constructs `*agent.Manager` in-process. Those are the rest of Stage 3 in
    +[design/crux.md](design/crux.md#after-the-cut). The opt-in
    +[durable lanes](crux-lanes.md) are the exception: they carry their own
    +per-project credentials, capability ACLs and per-lane bounds, and Hive
    +consumes them as an ordinary participant.
  3. src/docs/crux-serve.md "## The API" table (lines 58-73) enumerates every route the server exposes but gains no row for the nine /v1/lanes… routes registered in serve/lanes.go:36-45; the only pointer is the --lanes-config flag row. A single summary row keeps the table authoritative:

    | `GET /v1/lanes`, `POST /v1/lanes/{lane}/publish`, `GET /v1/lanes/{lane}/events`, `/v1/lanes/{lane}/groups/{group}[/claim\|/ack\|/nack\|/renew\|/recover]` | lane-project bearer token (never `CRUX_TOKEN`) | `lanes.Broker` | present only with `--lanes-config`; see [crux-lanes.md](crux-lanes.md#public-http-and-go-api) |
    

    (after the GET /v1/events row).

Reference only (already raised by the docs-index reminder, not a new finding): crux-lanes.md is not yet linked from src/docs/documentation-map.md.

Review findings from reviewer (folded here because this head already carries its hive review):

Routing this one to a second approver rather than judging it alone: it is a new subsystem (src/crux/lanes, src/crux/serve/lanes.go, src/pkg/laneparticipant, ~7.3k lines) and the part that needs a human is not a defect, it is a dependency and surface decision.

What I verified

  • .github/workflows/{v2-ci,integration-candidate,integration-technical-v2-ci}.yml: each adds one step running go test -race on ./lanes ./api ./serve and ./pkg/laneparticipant. No permissions: block, trigger, secret or runner change — these are test additions only.
  • src/go.mod / src/crux/go.mod: nats-server/v2 v2.15.2 becomes a direct requirement of both modules and nats.go v1.55.0 of crux; the transitive set pulls in google/go-tpm, antithesishq/antithesis-sdk-go, minio/highwayhash, nats-io/jwt+nkeys, golang.org/x/crypto. FOSSA and NOTICE matches the module graph are green at this head.
  • Auth surface (src/crux/serve/lanes.go:41-54, src/crux/lanes/auth.go): every /lanes path is fail-closed behind a per-project bearer token resolved from administrator-chosen env vars; tokens must be ≥32 bytes, distinct from each other and from the agent-control admin token (auth.go:21-24), compared with subtle.ConstantTimeCompare. Non-lane paths fall through to the existing handler unchanged. Looks correct to me.
  • Lease/claim path (src/crux/lanes/work.go:109-200): ownership is linearised on the KV Update(revision) CAS, with ErrKeyRevisionMismatch mapped to ErrConflict and retried; retry turn is persisted before growing the value, as the comment says. I did not trace the remainder of work.go, work_recovery.go and work_budget.go line by line — the ~3k lines of tests in the package cover restart, lost-response, stale-receipt and expiration cases, and all v2-ci shards, race, lint, gosec, govulncheck and coverage are green here.

The decision: this is the first embedded message broker in the Hive binary, and src/go.mod carries it even though only the opt-in Crux lanes use it. That is a supply-chain and footprint call for the maintainer committee, not something I should settle in a review comment. If the answer is yes, I have no verified blocker. Author is an approver, so I'm asking a second one.

I have not approved or requested changes as a review event.

Evidence bundle `#1637324a9ffc` sha256:f1f3268680f119a35f68b5870662a3c2a2e4348b74992518c5d5de3880ee897c — download from the dashboard or `hivectl review evidence #16373`.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style/maintainability review at head 0f13049 (read the diff plus src/crux/lanes/*.go, src/crux/serve/lanes.go, src/crux/cli/serve.go, and the merge-base src/crux/api/api.go / callback.go / cli.go for conventions).

1. Config.Validate: recoverer validation is nested inside the groups loop and shadows group/m (src/crux/lanes/config.go:183-197)

The for group, m := range l.Recoverers loop sits inside for group, m := range l.Groups, re-declaring both loop variables. Effects:

  • Recoverer checks run once per delivery group (redundant O(G×R)).
  • A lane with recoverers but an empty groups map never validates its recoverers at all — bad group IDs, unknown projects, and non-member recoverers are silently accepted at load time (the "recoverer must be a group member" error is unreachable in that case). authorize("recover") happens to still deny at runtime, but the config-time invariant the comment promises is not enforced, and nothing in the added tests exercises Recoverers rejection.
 	for _, l := range c.Lanes {
 		...
 		members := append(slices.Clone(l.Publishers), l.Subscribers...)
 		for group, m := range l.Groups {
 			if !ValidID(group) || len(m) == 0 || len(m) > DefaultMaxMembers {
 				return fmt.Errorf("%w: delivery group", ErrInvalid)
 			}
-			for group, m := range l.Recoverers {
-				if !ValidID(group) || len(m) == 0 || len(m) > DefaultMaxMembers {
-					return fmt.Errorf("%w: recovery capability", ErrInvalid)
-				}
-				for _, p := range m {
-					if !allowed(l.Groups[group], p) {
-						return fmt.Errorf("%w: recoverer must be a group member", ErrInvalid)
-					}
-				}
-			}
 			members = append(members, m...)
 		}
+		for group, recoverers := range l.Recoverers {
+			if !ValidID(group) || len(recoverers) == 0 || len(recoverers) > DefaultMaxMembers {
+				return fmt.Errorf("%w: recovery capability", ErrInvalid)
+			}
+			for _, p := range recoverers {
+				if !allowed(l.Groups[group], p) {
+					return fmt.Errorf("%w: recoverer must be a group member", ErrInvalid)
+				}
+			}
+		}

A small Validate test for "recoverer not in group" / "recoverers for unknown group" would lock this in.

2. Lane API error codes are string literals, not api.Code* constants (src/crux/serve/lanes.go:62,67,176-192)

crux/api/api.go:250-256 defines CodeUnauthorized, CodeConflict, etc. as exported constants that clients and tests match on; src/docs/crux-lanes.md:80,124 document lane_conflict/lane_cursor as part of the contract. Yet the lane handler emits "lane_invalid", "lane_forbidden", "lane_conflict", "lane_cursor", "lane_stale", "lane_capacity", "lane_unavailable", "lane_timeout", "lane_canceled" inline, so api.Client users have nothing to compare api.Error.Code against. Suggest adding to src/crux/api/lanes.go:

const (
	CodeLaneInvalid     = "lane_invalid"
	CodeLaneForbidden   = "lane_forbidden"
	CodeLaneConflict    = "lane_conflict"
	CodeLaneCursor      = "lane_cursor"
	CodeLaneStale       = "lane_stale"
	CodeLaneCapacity    = "lane_capacity"
	CodeLaneUnavailable = "lane_unavailable"
	CodeLaneTimeout     = "lane_timeout"
	CodeLaneCanceled    = "lane_canceled"
)

and referencing them from writeLaneError/readLaneJSON.

3. Env var names are inline literals (src/crux/cli/serve.go:36,74, src/cmd/hive/lanes.go:16,20,37)

The crux convention is a named *EnvVar constant reused in the flag help text (config.ServeTokenEnvVar, callback.SocketEnvVar on the adjacent lines of serve.go:32,36; cli.DefsEnvVar at cli/cli.go:65). "CRUX_LANES_CONFIG" appears twice as a literal (help string and os.Getenv), and the five HIVE_LANES_* names are literals in cmd/hive/lanes.go. E.g. in src/crux/lanes/config.go:

// ConfigEnvVar names the opt-in lane configuration file for `crux serve`.
const ConfigEnvVar = "CRUX_LANES_CONFIG"

then "... (default $"+lanes.ConfigEnvVar+")" and os.Getenv(lanes.ConfigEnvVar).

Item 1 is the one I'd want fixed before merge; 2 and 3 are convention alignment.

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor
0f13049

ℹ️ Sentinel notice — informational; author is trusted (owner), merge is not blocked

Hive flagged this PR (author @clubanderson, head 0f130490efab) because it matches behaviors that can override security controls, escalate privileges or damage the codebase. This is a heuristic, not an accusation — a maintainer should confirm the change is intended before it merges.

  • sensitive_path — changes 5 sensitive path(s) (Touches a sensitive path (OWNERS, workflows, policies, hive config, security docs, …))
    • .github/workflows/integration-candidate.yml
    • .github/workflows/integration-technical-v2-ci.yml
    • .github/workflows/v2-ci.yml
    • src/crux/go.mod
    • src/go.mod

Hive did not add the sentinel-alert label because this author is trusted. The finding remains recorded for audit and the dashboard Security tab; set sentinel.trusted_authors_block: true to require the blocking label for trusted authors too.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@hivecommons-hive

Copy link
Copy Markdown
Contributor

CI triage (test (rest 2/4), run 41572392894): three deterministic failures in crux/lanes at head b1041e5, 3/3 runs each — PR-local, not a shared baseline issue (sibling v5 PRs are green on this shard).

  • TestTwoCruxProcessesFenceDelayedExpiredPublicationGeneration — publish_generation_test.go:249: nats: API error: code=404 err_code=10037 description=message not found (the delayed/expired publication is looked up after JetStream has already dropped it; the fence test needs the message retained or the lookup tolerant of 10037).
  • TestGroupBudgetRejectsShapesOutsideItsValidatedContract — work_budget_shape_test.go:55: shape accepted; json: error calling MarshalJSON for type time.Time: year outside of range [0,9999] (an out-of-range sentinel time reaches the validator instead of being rejected before marshal).
  • TestByteBoundAdmissionPreservesCompletionAndRecoveryHeadroom — work_budget_test.go:20: invalid lane request: lane configuration bounds (the test's byte-bound fixture falls outside the configured lane bounds).

Human-authored PR, so no pushes from the hive; leaving for the author.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

Signed-off-by: Andrew Anderson <andy@clubanderson.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intent alignment (head b1041e5): the diff matches the title — every touched file is lane-related (workflows only add the lane test step, NOTICE is the regenerated dependency manifest, governor.go:383 is an additive observer, api/client.go:282 is a 204 no-body guard). Opt-in is real: src/cmd/hive/lanes.go:15-17 returns unless HIVE_LANES_ADDRESS is set. Verdict: approve, one advisory.

Advisory — the "ordinary HTTP participant" links the NATS client into the Hive binary. src/pkg/laneparticipant/participant.go:15 imports crux/lanes only for lanes.ValidID (lines 52, 64, 72). crux/lanes/broker.go:16-17 imports nats.go/jetstream, so the hive binary — which at merge-base links no NATS code (only _test.go files in src/cmd/hive import crux/serve/crux/cli) — now carries the broker client to run a regex. The PR body and docs/crux-lanes.md:356 frame Hive as a peer that "use[s] Crux HTTP credentials, not direct broker credentials"; the import graph should say the same. Concrete fix: move the ID rule into the stdlib-only crux/api and have lanes delegate, e.g.

--- a/src/crux/api/lanes.go
+++ b/src/crux/api/lanes.go
+// MaxLaneIDLength and ValidLaneID are the wire-level identifier rule shared by
+// the broker and participants; stdlib-only so clients need not import crux/lanes.
+const MaxLaneIDLength = 64
+
+var laneIDPattern = regexp.MustCompile(`^[a-z][a-z0-9_-]*$`)
+
+func ValidLaneID(id string) bool { return len(id) <= MaxLaneIDLength && laneIDPattern.MatchString(id) }
--- a/src/crux/lanes/config.go
+++ b/src/crux/lanes/config.go
-func ValidID(id string) bool { return len(id) <= MaxIDLength && idPattern.MatchString(id) }
+func ValidID(id string) bool { return api.ValidLaneID(id) }
--- a/src/pkg/laneparticipant/participant.go
+++ b/src/pkg/laneparticipant/participant.go
-	"github.com/hivecommons/hive/crux/lanes"
 ...
-	if client == nil || !lanes.ValidID(lane) || !lanes.ValidID(group) {
+	if client == nil || !api.ValidLaneID(lane) || !api.ValidLaneID(group) {

(and the two other lanes.ValidID call sites). Not blocking; fine as a follow-up.

— hive: agent=adjudicator-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@github-actions github-actions Bot removed the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Nov 3, 2026
Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 3, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style / maintainability review (head c93b25b)

1. Config.Validate nests the Recoverers loop inside the Groups loop and shadows its variables — src/crux/lanes/config.go:181-195. The inner for group, m := range l.Recoverers (line 185) re-declares group/m from the outer for group, m := range l.Groups (line 181), so recoverers are re-validated once per group and members = append(members, m...) on line 195 runs after the shadow has gone out of scope only by accident of block scoping. A lane with recoverers but no groups is never validated at all (the outer loop body never runs). It reads like an indentation slip and will be fragile to edit. Hoist it to a sibling loop:

 		for group, m := range l.Groups {
 			if !ValidID(group) || len(m) == 0 || len(m) > DefaultMaxMembers {
 				return fmt.Errorf("%w: delivery group", ErrInvalid)
 			}
-			for group, m := range l.Recoverers {
-				if !ValidID(group) || len(m) == 0 || len(m) > DefaultMaxMembers {
-					return fmt.Errorf("%w: recovery capability", ErrInvalid)
-				}
-				for _, p := range m {
-					if !allowed(l.Groups[group], p) {
-						return fmt.Errorf("%w: recoverer must be a group member", ErrInvalid)
-					}
-				}
-			}
 			members = append(members, m...)
 		}
+		for group, recoverers := range l.Recoverers {
+			if !ValidID(group) || len(recoverers) == 0 || len(recoverers) > DefaultMaxMembers {
+				return fmt.Errorf("%w: recovery capability", ErrInvalid)
+			}
+			for _, p := range recoverers {
+				if !allowed(l.Groups[group], p) {
+					return fmt.Errorf("%w: recoverer must be a group member", ErrInvalid)
+				}
+			}
+		}

2. Lane error codes are inline string literals, not api.Code* constants — src/crux/serve/lanes.go:68,73,185-204. The existing wire contract publishes codes as constants (src/crux/api/api.go:248-256, CodeUnauthorized, CodeConflict, …) so clients and tests match on symbols; the nine new lane_* codes exist only as literals in the server and prose in src/docs/crux-lanes.md. Add CodeLaneInvalid, CodeLaneForbidden, … to crux/api next to the others and use them in writeLaneError/readLaneJSON.

3. Broker init failure hides the cause — src/crux/cli/serve.go:98 prints crux: lanes: cannot initialize durable broker and drops err, unlike every other failure path in the same function (crux: serve:, crux: all print err). lanes.Open already returns typed errors (ErrConflict: stream configuration differs, ErrUnavailable, …) that carry no credentials (Config.Validate rejects inline URL creds), so an operator with a mis-sized stream has no way to tell that from a dead broker. Same pattern in src/cmd/hive/lanes.go:23-40, where four distinct logger.Error branches omit the err value.

			fmt.Fprintln(e.Stderr, "crux: lanes:", err)

4. Test writes a fixture into the package source dir — src/crux/lanes/auth_test.go:48 creates .lane-config-<nanos>.json in the cwd (the package directory) and relies on t.Cleanup to remove it; a killed run leaves it in the tree. The crux tree uses t.TempDir() (~30 call sites); use filepath.Join(t.TempDir(), "lanes.json") here.

Advisory: Validate collapses ~25 bound checks into one lane configuration bounds error (config.go:152-160), and the JSON config takes durations as integer nanoseconds (config.go:76-83, documented at crux-lanes.md "Bounds and failure behavior"). Both are defensible but will cost operators time; consider per-field messages and string durations in a follow-up.

— hive: agent=adjudicator-5 backend=copilot model=claude-fable-5.1 copilot=1.0.88

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 3, 2026
@clubanderson

Copy link
Copy Markdown
Member Author

Current-head native review handoff — not an approval

Current head is 98fa96f5203a537e4c4c6a90206fa5686b9eba70, the normal two-parent integration of 8ccbda8da and frozen v5 ac79b01bc. GitHub now reports it mergeable.

Actual checks on this head passed: canonical build/integration (including the overlay guard), all 23 test shards, coverage/patch coverage, lint, govulncheck, gosec and NOTICE. Lanes measured 91.4% against the unchanged 90% floor. The real near-byte-cap test passed in 88.76s; the newly added empty-Groups/Recoverers, defaults, duplicate/whitespace and project/lane/group authorization regressions also ran and passed.

Evidence: build workflow, test workflow, actual lane test execution, security/NOTICE.

The latest posted style review 5747443022 is API-bound to 8ccb, but its body explicitly reviews c93 and proposes the nested-Recoverers fix already delivered in 8ccb. The env inventory and documentation index omissions are also corrected. This does not make that review a current-head clearance. Its still-applicable error-code/diagnostics/fixture observations should be assessed against current source, not silently discarded; in particular, blindly logging rejected address errors can expose input and needs careful treatment.

Following the existing author-reply/ordinary review path: please review 98fa96f and collect genuine current-head reports for the configured native perspectives. No synthetic verdict, dismissal, label/permission change, bypass or CI rerun is requested. Native approval/admission remains separate from the passing technical checks and source-review clearance.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style re-review at 98fa96f (delta since c93b25b = v5 forward-merge + 8ccbda8d).

Fixed: the Recoverers loop nested inside the Groups loop with shadowed group/m (src/crux/lanes/config.go:181-196 is now two sibling loops, covered by config_recoverers_test.go). The forward-merge itself is a clean first-parent replay of v5 — no conflict-resolution drift.

Still present (new code this PR adds; each has a drop-in fix):

  1. src/crux/cli/serve.go:98 — broker init failure discards err, so an operator debugging a bad CRUX_LANES_CONFIG/NATS URL sees only a generic line.
			fmt.Fprintf(e.Stderr, "crux: lanes: cannot initialize durable broker: %v\n", err)
  1. src/crux/serve/lanes.go:185-200 — lane error codes are inline string literals, while the rest of the API surface names them in src/crux/api/api.go:250-256 (CodeUnauthorized, CodeBadRequest, …). Clients (api/lanes_client.go) have nothing to match against. Add alongside the existing block:
 	CodeConflict = "conflict"
 	CodeRuntime  = "runtime_error"
+
+	CodeLaneUnavailable = "lane_unavailable"
+	CodeLaneForbidden   = "lane_forbidden"
+	CodeLaneInvalid     = "lane_invalid"
+	CodeLaneConflict    = "lane_conflict"
+	CodeLaneCursor      = "lane_cursor"
+	CodeLaneStale       = "lane_stale"
+	CodeLaneCapacity    = "lane_capacity"
+	CodeLaneTimeout     = "lane_timeout"

and reference them in writeLaneError.

  1. src/crux/lanes/auth_test.go:48 — TestLoadConfigStrictAndBounded writes .lane-config-<nano>.json into the package source directory and relies on a manual os.Remove cleanup; a failed run leaves stray files in src/crux/lanes/.
	path := filepath.Join(t.TempDir(), "lane-config.json")

(drop the t.Cleanup/os.Remove block.)

  1. src/crux/lanes/config.go:151-160 — one 9-line predicate collapses ~25 independent bounds into a single lane configuration bounds error; an operator cannot tell which field failed. Splitting into per-field checks (or a small check(cond bool, what string) helper) would make the error actionable. Advisory.

— hive: agent=adjudicator-5 backend=copilot model=claude-fable-5.1 copilot=1.0.88

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@clubanderson

Copy link
Copy Markdown
Member Author

Current-head native review handoff — not an approval

Current head is 335ac26b336f99ecee7e92afe0b82d85954b929f, tree f431c7ac37dfd6cdd541d60ea24516a76c7857df.

This publishes the surgical corrections to the genuine 98fa96f style re-review 5747648170:

  1. CLI-private broker initialization diagnostics now distinguish wrapped typed conflict/unavailable errors while preserving ExitError; arbitrary backend details, addresses and dummy credential-bearing error inputs are covered by exact-output redaction regressions.
  2. All nine existing lane wire codes, including lane_canceled, are exported as api.CodeLane* and reused by handlers. Regressions preserve exact statuses, JSON bodies, content type and redaction.
  3. The generated source-directory configuration fixture is replaced by two small committed non-sensitive fixtures (452 and 18 bytes) and private bounded-reader tests. Missing-file, strict JSON, invalid configuration, reader-failure, exact 1 MiB and oversized-input assertions remain; no TempDir/generated-file cleanup dependency is introduced. Advisory per-field bounds redesign remains out of scope.

Canonical current-head build, all test shards/coverage, and lint/security/module NOTICE were still running at the initial observation. Prior 98fa96f greens are not proof for this head; these corrections need their own executable verdicts.

Please use the existing ordinary native review/collection path for genuine head-specific reports across the configured perspectives. This factual handoff is not an approval or review receipt and does not request dismissal, bypass, labels/permission changes, synthetic verdicts or CI reruns. Preserve any still-applicable findings and identify their reviewed head.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@clubanderson

Copy link
Copy Markdown
Member Author

Current-head review handoff and named dependency decision request — not an approval

Current head is 3c77f345373b51b998c87ff7c0345fde8c6ff476, tree 5d2c56581a4593f325e017b14327e446f8161771. Since 335ac26b3, only src/docs/crux-lanes.md and src/docs/crux-serve.md changed:

  • The local Go execution prohibition is explicitly scoped to AI agents; human contributors are linked to CONTRIBUTING local-development guidance.
  • The agent-control hosting/trust statement is distinguished from opt-in project-authenticated lane ACLs and configured per-lane/per-group bounds, without claiming general multi-tenant isolation or additional agent-control privileges.
  • The existing four-column API table now summarizes lane routes and links the authoritative request/response and error-code sections.

The prior surgical source corrections (sanitized typed diagnostics, all nine stable wire constants, bounded strict parser/committed fixtures) are preserved. Normal current-head build/integration, tests/coverage and lint/security/module NOTICE were running at the initial observation; the current docs link check passed. Prior 335ac26b3 greens are not this head's verdict.

Existing named dependency decision: the folded native reviewer finding asks a second maintainer to decide this dependency/surface tradeoff. @hanthor, please record your actual decision (accept, reject, or required conditions) on retaining the selected NATS JetStream implementation and its dependency/attribution footprint. Factual clarification: explicit nats-server/v2 server imports are TEST-ONLY (crux/lanes/*_test.go, crux/serve/lanes_test.go, pkg/laneparticipant/participant_test.go); this PR does not embed a NATS server in the production Hive binary. nats.go is the runtime broker client. The direct test requirements in both Go module graphs and the real transitive/NOTICE attribution footprint remain; no dependencies were removed to game a gate. This is a request for the named maintainer's genuine decision, not a decision or approval on their behalf.

Please use the existing ordinary collection path for genuine current-head correctness, security, intent-alignment, style and docs-currency reports plus the actual approval perspective. This factual author handoff supplies no review receipt, approval, dismissal or bypass, and requests no permission/label changes, synthetic verdicts or CI reruns.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decision needed: @clubanderson should decide whether a 7.5k-line, 60-file lane subsystem with no linked issue or spec should land as one PR on v5, or be split (API/auth/serve wiring first, broker/work-recovery core second) so the broker core gets a human read before it ships.

Routing to a human; not a defect report. This PR is too large for me to verify end-to-end, so I read the parts a maintainer would most want checked and I am explicit below about what I did not read.

Verified (read the code at 3c77f34):

  • Workflows (.github/workflows/v2-ci.yml:200-207, integration-candidate.yml:258-261, integration-technical-v2-ci.yml:124-127): each adds one go test -race step for crux/{lanes,api,serve} and pkg/laneparticipant. No permissions: changes, no new secrets, no removed checks.
  • Auth boundary (src/crux/lanes/auth.go, src/crux/serve/lanes.go laneHandler, serve.go authenticate): project tokens are resolved only from admin-chosen env vars, rejected if <32 bytes, whitespace-containing, equal to the agent-control token, or duplicated (auth.go:22-30); Project() is constant-time and nil-safe (auth.go:36-46). The lane mux rejects anything that is not a project token (lanes.go laneHandler), and authenticate now rejects project tokens on agent RPCs before the legacy token/unix checks — so the two credential domains cannot cross in either direction. Request bodies are bounded by http.MaxBytesReader and DisallowUnknownFields (readLaneJSON). This part looks correct to me.
  • Dependencies / NOTICE: nats-server/v2 is imported only from _test.go files (lanes/broker_test.go, lanes/work_slots_test.go, serve/lanes_test.go, pkg/laneparticipant/participant_test.go); production binaries only pick up the nats.go client and connect to Config.URL. The NOTICE hunk looks like it deletes ledongthuc/pdf, go-colorful, go-isatty — it does not; all three are still present at head, the diff is alphabetical reflow around the new klauspost/compress entry. check-notice-drift is green.

Not read: lanes/broker.go (387 lines), work.go, work_recovery.go, work_budget*.go, and their ~2.5k lines of tests — the lease/revision-fencing/retry-rotation core. I have no finding against them; I simply cannot confirm them within a review budget, and nothing else (no linked issue, spec, or prior design PR) does either. The body's own validation section says executable Go validation is "pending canonical CI" — CI is now green on this head, which is the only independent signal so far.

Two small things, neither blocking:

  • src/go.mod: github.com/nats-io/nats-server/v2 v2.15.2 sits inside an // indirect block without the marker; go mod tidy would relocate it. Cosmetic.
  • No Closes #N / Refs #N. If there is a tracking issue for Crux lanes, linking it would give the next reviewer the intent to judge against.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Needs a decision, not a merge:

  • the reviewer swarm returned requires_human @3c77f34
  • the review found: security question from the correctness review
  • reviewer question: @clubanderson should decide whether a 7.5k-line, 60-file lane subsystem with no linked issue or spec should land as one PR on v5, or be split (API/auth/serve wiring first, broker/work-recovery core second) so the broker core gets a human read before it ships.
  • the PR edits CI workflow / action files: awaiting owner approval — a hive owner / trusted merger's GitHub Approve review on the current head (or a dashboard /hive decision approve) releases the fence and the hive merges it; approval on an earlier head does not count, so re-approve after a push

Head: verdict requires_human @3c77f34, 70/70 checks green, DCO yes, soak 10m0s elapsed.

How to release it:

  • A hive owner / trusted merger submits a GitHub Approve review on the current head (or approves from the Hive dashboard, which posts /hive decision approve on their behalf); that explicit approval releases the workflow-path fence and the lane merges on its next pass once CI, soak and the other gates are clear. Approval on an earlier head does not carry over: re-approve after a push.
  • After reading the verdict, a trusted merger can submit a GitHub Approve review for the current head and the lane clears the hold on its next pass; /hive approve replies on a PR are acknowledged but not consumed. An ordinary contributor cannot release their own PR.

Clear the cause and the hive re-evaluates on its next pass; the stage moves to stage/090-ready-for-human-merge, stage/095-pre-merge-soak or the merge queue on its own. Approval does not bypass CI, hold labels or other merge gates.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decision needed: @clubanderson should confirm this 7.5k-line, issue-less feature is meant to land on v5 as one PR with nats-server/v2 (plus go-tpm, antithesis-sdk, x/crypto, x/time) added to both src/go.mod and src/crux/go.mod, rather than crux-first behind the opt-in.

No defects found in the parts I read at 24a9ffc; routing for a scope decision, not a code problem.

Verified (tree + callers):

  • Auth domains are genuinely separate. src/crux/serve/serve.go:158 wraps the legacy handler as s.laneHandler(s.authenticate(mux)), so /lanes/** never reaches the admin-token path, and serve.go:173-176 rejects any project token that tries an agent RPC. src/crux/lanes/auth.go:19-33 refuses a project token equal to the serve token, shorter than 32 bytes, or duplicated across principals; Project() compares in constant time and does not short-circuit. authorize() at src/crux/lanes/config.go:245-263 is the single capability gate every broker verb goes through.
  • serve/lanes.go:62-77 bounds bodies with MaxBytesReader at base64(MaxPayload)+8KiB, uses DisallowUnknownFields, and rejects trailing JSON; writeLaneError (serve/lanes.go:181-206) maps typed errors only and never leaks broker addresses. cli/serve.go:122-134 does the same for startup failures.
  • AddModeChangeObserver (src/pkg/governor/governor.go:383-397) chains rather than replaces, so the existing local observer keeps firing.
  • The three workflow edits (v2-ci.yml:200-207, integration-candidate.yml:258-261, integration-technical-v2-ci.yml:124-127) only add a go test -race step for the new packages; no permissions: or trigger change.
  • nats-server/v2 is imported by src/crux/lanes and by src/pkg/laneparticipant/participant_test.go only — the hive binary itself does not link the embedded server; the non-indirect line in src/go.mod:63 is from the test import.

Why a person: the PR carries no Closes/Refs issue, is size/XXL, and is the first time the NATS server stack enters the module graph of both modules. The body says opt-in only (and the code agrees — cli/serve.go:74-103 does nothing without -lanes-config/CRUX_LANES_CONFIG), but whether the dependency footprint and single-PR landing on v5 is the intended shape is an owner call I should not make. I have not read all 60 files; the broker's JetStream state machine (lanes/broker.go, work.go, work_recovery.go) I only spot-checked, and I am relying on the race-enabled suite (green in test (agent|rest) and contract-tests) for that.

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive
hivecommons-hive Bot merged commit 179523a into v5 Nov 3, 2026
83 of 84 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the feat/crux-durable-lanes-jetstream-20261102 branch November 3, 2026 08:21
@hivecommons-hive

Copy link
Copy Markdown
Contributor

merged by Hive trusted-author auto-merge: author @clubanderson holds role owner and GitHub permission admin on repo; CI green at 24a9ffc.

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant