Skip to content

Bound live observation work across requests and polling - #367

Open
jhgaylor wants to merge 1 commit into
mainfrom
fix/shared-live-read-budget
Open

Bound live observation work across requests and polling#367
jhgaylor wants to merge 1 commit into
mainfrom
fix/shared-live-read-budget

Conversation

@jhgaylor

@jhgaylor jhgaylor commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

An 11-member Kubernetes estate took 153s to observe, then 147s on a repeated read with the source cache warmed. A single member made 217 API requests in 28s. Polling and HTTP requests launched duplicate live scans, while one browser change event scheduled four estate refreshes within 15 seconds. The existing per-estate pool limited each request separately.

This change bounds observation work across the Behold process and removes the browser amplification:

  • Share a FIFO budget across graph reads, component status, lifecycle diff/plan, Helm reads, and run/operator status reads. Default to two active subprocesses (one on a one-CPU host), honor BEHOLD_ESTATE_CONCURRENCY globally, and bound the waiting queue to 64 distinct reads.
  • Share matching in-flight work by project, resolved Chant, source stamp/generation, exact argv, and effective environment. Each caller owns its result; completed live results are never cached. Source invalidation prevents a new request from joining pre-edit work.
  • Track cancellation per GET/poll subscriber. Stop the worker only when no subscribers remain; impose a configurable 180s running-read deadline. On Unix, terminate the npm/tsx/Node process group, escalate to KILL, and retain the slot through cleanup. Delegated mutations bypass scheduling and deduplication.
  • Serialize browser refreshes, retain one pending refresh, suppress superseded responses, and remove the unconditional 3/8/15s follow-ups.
  • Resolve poll namespace bindings through the same estate helper as HTTP overlays; reuse the primary poll result for lanes capture rather than observing it again.

The source-stamp walker moves to a dependency-neutral module so both the existing source cache and the new scheduler use the same identity rules.

Validation: VITEST_MAX_WORKERS=2 just check passed: typecheck, 85 test files / 1,638 passed, 1 existing skipped, and production bundle build. New tests exercise shared concurrency, in-flight-only reuse, bounded queues, failure/deadline recovery, source/env/namespace isolation, HTTP cancellation with another subscriber still present, and termination of a real descendant that ignores TERM. A browser burst of 20 notifications produces one follow-up with peak concurrency one.

Preserved full-suite timeout evidence (related to #334)

The default parallel local run timed out at 5s in these existing tests:

  • src/carve-route.test.ts: “finds the manifests in the project's carveout and publishes what they record”
  • src/ops-route.test.ts: “keeps a step's entities on the card and degrades the estate link to unresolved when no member can be graphed here (chant#2022)”

The same two failures reproduced on unchanged main (1a00dfd) using the same dependency installation. Both files passed in isolation (29 tests), and the complete required gate passed with two test workers. First-failure JSON and terminal logs were preserved during investigation. No test timeout was increased and no existing test was disabled.

This addresses duplicate work and resource spikes; it does not remove Chant's per-read Kubernetes discovery cost. No production latency improvement is claimed yet. Windows cancellation terminates the direct child only. Queued reads wait outside the running-read deadline. Removing timed browser follow-ups means eventual post-operation changes are picked up by subsequent drift events or manual refresh.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant