Skip to content

🌱 Top up v6 with v5 (ETag cache size setting and 7 more) - #17290

Merged
clubanderson merged 9 commits into
v6from
topup/v5-to-v6-20261108b
Nov 8, 2026
Merged

clubanderson merged 9 commits into
v6from
topup/v5-to-v6-20261108b

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Fifth top-up after #17278: forward-merge of v5 (d8b1d3a78) into v6. Plain merge commit β€” merge with --merge, not squash β€” so v6 keeps v5's SHAs and sign-offs. Supersedes the automated sync/v5-to-v6 PR #17282.

Commits carried (8)

changelog.d/added-etag-cache-size-setting.md and added-crux-s7-saved-run-results.md are kept for the next v6 release (not compiled on v5 yet).

Conflict resolutions

File Resolution
src/pkg/github/etagcache.go v6 (#17172) and v5 (#17286) both added a gzip-compressed, byte-bounded cache. v5's version taken for the shared code (config-driven defaults 128 MiB / 8192 entries, SetETagCacheLimits, ETagCacheUsage, gzipped/size entry fields, body-only byte accounting); v6-only pieces kept on top: evictions counter + ETagCacheSnapshot (now also reporting MaxEntries), cacheableSince, githubAppJWTTransportChain. v6's in-package HIVE_GITHUB_ETAG_CACHE_MAX_MB read dropped β€” the same env var is now handled by config.applyBootstrapEnv and applied via SetETagCacheLimits at boot. TestETagCache_MaxBytesFromEnv removed accordingly; etagcache_workingset_test.go updated to the v5 field names.
src/pkg/github/client.go ETagCacheInfo is the union (bytes, max_bytes, max_entries, evictions); etagCacheInfo() delegates to ETagCacheSnapshot().
src/pkg/dashboard/status_builder.go, src/pkg/hub/github_http.go union of fields / v5 accessor.
src/pkg/config/config.go v6 split this file; v5's hunks applied to the v6 homes: GitHubConfig fields + DefaultGitHubETagCacheMax* consts + Effective*/ValidateETagCacheLimits β†’ forge_config.go, env override β†’ load.go; validate.go auto-merged.
CHANGELOG.md both; v5.204.1 / v5.204.0 placed after v6.0.1 (v6.3.0 already carries the agent-step entry).
src/docs/scale-envelope.md v6 citation, re-anchored to client.go:5489-5499; check-docs-citations.py clean (2031 verified).

src/pkg/dashboard/static/index.html regenerated via go generate ./pkg/dashboard/... (-check passes). src/scripts unchanged, no TRUSTED_SCRIPTS_TREE re-pin needed. go build ./... passes.

DCO/native-queue-policy reds are expected on forward-merges (others' v5 commits); the merge commit itself is signed off. To be admin-merged with a merge commit like #17265.

Karibhou and others added 9 commits November 8, 2026 19:17
The Copilot usage sniff (crux/proxy/github_proxy_copilot.go) logs
"copilot sniff: upstream dial failed" at ERROR when it cannot reach the
*.githubcopilot.com host it is MITMing - e.g. a dial timeout to
telemetry.enterprise.githubcopilot.com. No log-fed class recognised the
line, so five in ten minutes opened an unclassified_error_repeated
self-report (#17262).

It is the same symptom as the opaque-tunnel "proxy: CONNECT dial
failed" (spoke cannot reach an agent-requested host), so count it under
proxy_connect_dial_failed (github-proxy, medium) with that class's
remedy. The regression row in TestRecurringRuntimeMessagesAreClassified
fails on the parent (LogFedClass returns "").

Fixes #17262

Hive-Run: #17262
Hive-Plan: selfreport-unclassified-error-classification
Hive-Spec: fleet-report#log-fed-classes

Signed-off-by: Karibhou <Karibhou@users.noreply.github.com>
Co-authored-by: Karibhou <Karibhou@users.noreply.github.com>
Co-authored-by: hivecommons-hive[bot] <280983584+hivecommons-hive[bot]@users.noreply.github.com>
The one document a new driver wires up from (RFC crux#114 order-of-work
item 7): S1β†’S7 composed end to end with curl and crux CLI forms, server
config keys and flags, control-token vs lane-principal auth, the
failure drills from the acceptance sections, a symptom β†’ saved-record
map, and what hive uses today vs. after the 2026-11-16 cutover. Linked
from the design record, the documentation map and the v5 integration
guide.

Part of hivecommons/crux#114

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
Automated release commit. Compiles changelog.d/ fragments and moves
the CHANGELOG.md Unreleased section into a dated v5.204.0 entry.
See src/docs/releases.md.

Signed-off-by: hive-release-bot <actions@github.com>
* ✨ crux: saved run results (S7 complete)

Completes RFC crux#114 S7 beyond the minimum that shipped with S2:

- POST /v1/runs/{id}/restart re-runs the saved request as the original
  owner, re-validated against the current configuration; the two
  records link both ways; 409 conflict while the old run is live or
  when its record predates the saved request; run.restart audit.
- GET /v1/runs lists runs newest first, owner-scoped unless operator,
  with ?status= and ?limit=; output is stripped from the list.
- GET /v1/runs/{id}/history is the record's status transitions. The
  runstate index now logs transitions only (payload-only updates are
  not transitions).
- Reconcile at serve restart marks running steps of an unfinished run
  unfinished with the same reason.
- crux runs list|restart|history.

Part of hivecommons/crux#114

Signed-off-by: Andrew Anderson <andy@clubanderson.com>

* πŸ“– crux serve: S7 routes in the route table

Signed-off-by: Andrew Anderson <andy@clubanderson.com>

* πŸ“– crux design: drop forward link to the integration guide

Signed-off-by: Andrew Anderson <andy@clubanderson.com>

* crux/api: cover the saved-run client and list query

Signed-off-by: Andrew Anderson <andy@clubanderson.com>

* πŸ“– crux design: link the integration guide

Signed-off-by: Andrew Anderson <andy@clubanderson.com>

* crux/api: run paths do not escape ids

Signed-off-by: Andrew Anderson <andy@clubanderson.com>

---------

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
Automated release commit. Compiles changelog.d/ fragments and moves
the CHANGELOG.md Unreleased section into a dated v5.204.1 entry.
See src/docs/releases.md.

Signed-off-by: hive-release-bot <actions@github.com>
… default to 128 MB (#17286)

* ✨ Make the GitHub ETag response cache size configurable and raise the default to 128 MB

The conditional-request (ETag / 304) cache was hard-coded at 4096 entries /
64 MiB. A busy multi-repo hive filled it within hours; from then on every
store evicted a body another poller was about to revalidate, so free 304s
became charged fetches and hourly REST usage crept back toward the
installation limit. The operator reached for HIVE_GITHUB_ETAG_CACHE_MAX_MB,
which did not exist.

- github.etag_cache_max_mb / github.etag_cache_max_entries (YAML, JSON, env
  HIVE_GITHUB_ETAG_CACHE_MAX_MB / _MAX_ENTRIES), validated 0..2048 MiB and
  0..1Mi entries with 0 meaning the default.
- Defaults raised to 128 MiB / 8192 entries; bodies above 512 B are stored
  gzip-compressed so the byte cap holds several times as many responses.
- github.SetETagCacheLimits resizes the shared cache live (evicting LRU at
  once when shrinking); called at boot and from PUT /api/config/github.
- ETagCacheInfo / /api/gh-rate-limits / status etag_cache report bytes,
  max_bytes and max_entries; the Debug GitHub API budget panel shows an
  'ETag cache: N entries Β· X / Y MB' row.
- Settings β†’ Forge App gains 'ETag cache size (MB)' and 'ETag cache entries'
  with help marks; docs, hive.yaml.example and env-vars reference updated.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>

* 🌱 Use the shared formatNumber helper for the ETag cache entry count

Signed-off-by: Andrew Anderson <andy@clubanderson.com>

---------

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
Brings v5.204.0/v5.204.1 and the configurable GitHub ETag cache size
(#17286) onto v6. ETag cache: v5's config-driven bounds and gzip store are
taken for the shared code, with v6's eviction counter, ETagCacheSnapshot,
cacheableSince and App-JWT transport chain kept on top.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 8, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign clubanderson for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Nov 8, 2026
@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v6

Conflicts are checked with every PR merged onto the current tip of v6, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#17282 🌱 Forward-merge v5 into v6 (2 commits) same files src/docs/crux-integration-guide.md
src/docs/design/crux-work-services.md
src/docs/documentation-map.md
src/docs/fleet-report.md
src/docs/integration-guide.md
src/pkg/fleetreport/runtime.go
src/pkg/fleetreport/runtime_test.go

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the merged src/pkg/github/etagcache.go at 985424a end to end plus its callers (cmd/hive/github_client_config.go:126, pkg/dashboard/api_config.go:357, pkg/config/load.go:450-459, forge_config.go:216-241). The cache conflict resolution looks correct to me: byte accounting uses len(e.body) consistently in get/put/drop/evictLocked (etagcache.go:340-399), the v6-only evictions counter is incremented at :396 and surfaced through ETagCacheSnapshot (:184-198), and the dropped in-package env read is genuinely replaced β€” load.go:450 reads HIVE_GITHUB_ETAG_CACHE_MAX_MB into GitHub.ETagCacheMaxMB, validate.go:162 bounds it, and boot applies it via SetETagCacheLimits at github_client_config.go:126.

One merge residue to fix before this lands (docs-currency, low):

src/docs/env-vars.md now has two rows for HIVE_GITHUB_ETAG_CACHE_MAX_MB. Line 60 is v5's (default 128, 1–2048, applies from Settings) and is correct for the merged code. Line 119 is v6's pre-#17286 row and is now wrong on three counts: it says the default is 64 (code: forge_config.go:208 = 128), that non-positive values "fall back to the default" (code: validate.go:162 rejects < 0 at boot), and "read once at process start" (code: api_config.go:357-358 re-applies on a Settings save). Neither branch had both rows (v5: 1, v6: 1, this head: 2), so it's an auto-merge artifact; env_vars_doc_parity_test.go only checks presence, which is why CI is green. Deleting line 119 is the whole fix.

Info, not a defect: v6's etagCacheMaxBytesFromEnv silently accepted any value up to 1<<20 MiB; the merged code refuses > 2048 at Validate (forge_config.go:239). That is v5's documented choice (env-vars.md:60), just noting it in case a v6 operator has an oversized value set β€” boot will now fail loudly instead of honouring it.

Checked nothing else in the 54 files beyond the conflict-table entries; DCO/native-queue-policy reds match the stated forward-merge pattern.

β€” hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

hivecommons-hive Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Hive status β€” updated in place; edits do not notify. Full history is in the Activity Monitor.

Review findings from reviewer-2 (folded here because this head already carries its hive review):

Read the merged tree at 985424a against both parents (origin/v6 @ a11e466, origin/v5 @ d8b1d3a), not just the diff. The etagcache.go resolution is clean: the file is v5's #17286 verbatim plus exactly the v6-only pieces the body lists (evictions counter, ETagCacheSnapshot now with MaxEntries, cacheableSince, githubAppJWTTransportChain); client.go / status_builder.go / github_http.go are the straightforward union. Two things the resolution leaves behind, one behavioural and one documentary:

1. Hub mode loses HIVE_GITHUB_ETAG_CACHE_MAX_MB (medium, correctness). On v6 before this merge the bound was read inside the package (etagCacheMaxBytesFromEnv, v6 src/pkg/github/etagcache.go:179), so every binary sharing sharedETagCache honoured it β€” including the hub, which goes through hgithub.NewHTTPTransport at src/pkg/hub/github_http.go:35 and src/pkg/hub/commit_behind.go:42. After the merge the env var is parsed into config at src/pkg/config/load.go:450-453 and only applied by github.SetETagCacheLimits from two places: spoke boot (src/cmd/hive/github_client_config.go:126, inside configureGitHubClient) and the dashboard Forge-App save (src/pkg/dashboard/api_config.go:358). HIVE_MODE=hub returns via deps.runHub at src/cmd/hive/main.go:623 before configureGitHubClient runs, and runHubWithDeps (src/cmd/hive/hub_boot_seams.go:68) loads config but never calls SetETagCacheLimits β€” grep SetETagCacheLimits src/pkg/hub src/cmd/hive/hub_boot_seams.go is empty. Net effect on a v6 hub: the cap goes from "env or 64 MiB" to a fixed 128 MiB and the documented env var is silently ignored. The same gap exists on v5 (it is how #17286 was built), so this is a v5 bug the merge imports into v6 as a regression rather than a resolution mistake; the fix is one github.SetETagCacheLimits(cfg.GitHub.EffectiveETagCacheMaxBytes(), cfg.GitHub.EffectiveETagCacheMaxEntries()) after the config load in runHubWithDeps, on both lines. Flagging so it is not lost behind the "merge with --merge" admin step β€” a follow-up PR rather than a reason to hold this one is a reasonable call.

2. src/docs/env-vars.md now documents the variable twice, contradictorily (low, docs). Line 60 is v5's row (default github.etag_cache_max_mb, else 128; 1–2048; 0 = default). Line 119 is v6's pre-merge row β€” default 64, "Unset, non-numeric, or non-positive values fall back to the default", "Read once at process start" β€” which describes the in-package reader this merge deletes. -1 now fails ValidateETagCacheLimits (src/pkg/config/forge_config.go:240) instead of falling back, and the dashboard save re-applies the bound at runtime, so line 119 is wrong on default, range and lifetime. It should go.

Everything else I checked holds: plainBody (etagcache.go:124) keeps the decode-error-as-miss behaviour v6's rawBody had (etagcache.go:415 drops the entry on nil), and the Go/dashboard/hub test shards are green at this head. Not approving β€” that is a human's signature β€” but apart from the two items above the resolution looks correct to me.

@clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Why:

  • the review swarm requested changes

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Diagnosis (hive scanner, read-only β€” maintainer branch, nothing pushed):

The only real red is native-queue-policy, and it fails on DCO: action_required (native-queue-policy.cjs:122) β€” i.e. it is mirroring the probot DCO app. Everything else on 985424a is green: 70+ checks, prow's dco status ("All commits have Signed-off-by"), Sign-off survives the squash, changelog-fragment-guard, release-lines-in-sync. The two CANCELLED Sign-off survives the squash / verify PR contents entries are cancel-in-progress casualties with a SUCCESS duplicate each.

Why probot DCO is red: it flags carried v5 squash commits whose author identity β‰  Signed-off-by (same artifacts as on #17265 and #17282 β€” e.g. c05ba93 Karibhou noreply-email mismatch, 155d51f / d8b1d3a "Andy" vs "Andrew Anderson"). They are already v5 history; re-signing them here would diverge this branch from v5, so it is not something a push to this PR can fix.

Why the policy did not substitute prow's dco as it did on #17282: Decision B on #17030 (FORWARD_MERGE in .github/scripts/native-queue-policy.cjs, src/docs/native-merge-queue.md) allows the prow path only for head == sync/v5-to-v6, base == v6, author github-actions[bot]. A maintainer-opened topup/v5-to-v6-20261108b is deliberately outside that shape ("any doubt keeps probot DCO required"), so it falls back to probot DCO.

Options, all maintainer-side:

  • A. Set the probot DCO check to pass on this PR (the app's "Set DCO to pass" action), then /lgtm + /approve β€” native-queue-policy re-evaluates from the DCO check-run and should go green without a new push.
  • B. Let the v6-topup.yml workflow refresh sync/v5-to-v6 to the current v5 tip and merge 🌱 Forward-merge v5 into v6 (2 commits)Β #17282 instead; that shape takes the prow dco path automatically. 🌱 Forward-merge v5 into v6 (2 commits)Β #17282 currently carries 2 of these 8 commits, and the overlap bot already lists it as touching the same files, so one of the two should be closed when the other lands.
  • C. Widen Decision B to accept a maintainer-opened top-up branch that is a verified v5β†’v6 --no-ff merge (the policy already does the parent-reachability proof) β€” a follow-up policy change, not a fix for this PR.

Tide is otherwise waiting only on approved + lgtm.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

β€” hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

@clubanderson
clubanderson merged commit 44ed958 into v6 Nov 8, 2026
85 of 93 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the topup/v5-to-v6-20261108b branch November 8, 2026 20:27
@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants