Skip to content

RFC-64 M1 4/7: add adaptive capacity primitives - #2015

Open
branarakic wants to merge 9 commits into
codex/rfc64-m1-core-priority-schedulerfrom
codex/rfc64-m1-adaptive-primitives
Open

RFC-64 M1 4/7: add adaptive capacity primitives#2015
branarakic wants to merge 9 commits into
codex/rfc64-m1-core-priority-schedulerfrom
codex/rfc64-m1-adaptive-primitives

Conversation

@branarakic

@branarakic branarakic commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

M1 stack

This is PR 4 of 7 in the RFC-64 M1 stack.

  1. RFC-64 M1 1/2: make Edge CG sync explicitly on-demand #2011 — explicit Edge synchronization lifetime
  2. RFC-64 M1 2/7: expose verified VM and SWM convergence #2012 — truthful selected-CG VM/SWM convergence
  3. RFC-64 M1 3/7: bound Core public CG coverage #2013 — bounded Core public-CG coverage
  4. This PR — adaptive-capacity primitives
  5. Role-aware adaptive Core activation, public configuration, and status
  6. Bounded operator-only runtime evidence hooks
  7. Fail-closed M1 acceptance verifier and real-runtime release launcher

Exact stack provenance

  • Base: 235c8071a098782d0aa454ec2eededd95280376b
  • Head: 43768351aee0d4c93e86bddf065197ed6aced496
  • Range-diff against the preceding reviewed stack: 9 of 9 patches identical (eight existing commits plus the controller-state review fix)
  • Focused validation: 6 files / 92 tests passed
  • Agent dependency/build chain: TypeScript, public type tests, and package-root test passed

Summary

This PR adds the internal primitives required for adaptive synchronization capacity:

  • an optional policy-owned live limit for global sync admission;
  • a fast-down/slow-up adaptive controller expressed as a pure state transition;
  • host and store-pressure sampling;
  • an optional live cap for automatic Core public-CG coverage.

This slice is intentionally dormant. DKGAgent does not instantiate the controller, attach a live admission provider, or alter the live Core coverage batch until PR 5.

User and operator impact

There is no synchronization behavior change and no new operator configuration surface in this PR.

  • Edge and Core nodes keep their existing static requester limits.
  • Existing Core coverage continues using its configured fixed batch.
  • Existing callers of withGlobalSyncBackpressure retain the same call shape and behavior.
  • Adaptive-capacity configuration is intentionally withheld until PR 5, where the agent will own and enforce it atomically with runtime activation.

This split lets reviewers validate admission safety, controller policy, sampling, and scheduler seams independently from runtime activation.

Before

sequenceDiagram
    participant J as Sync job
    participant A as Global admission
    participant C as Core coverage planner
    participant S as Triple store

    J->>A: Request admission
    A->>A: Apply static configured limit
    A-->>J: Start or queue
    C->>C: Apply fixed configured batch
    J->>S: Execute sync work
Loading

After this PR

sequenceDiagram
    participant J as Sync job
    participant P as Admission policy
    participant A as Global admission
    participant C as Core coverage planner
    participant S as Triple store

    J->>A: Request admission with existing static policy
    A->>P: Resolve effective limit
    P-->>A: Static limit
    A-->>J: Start or queue unchanged
    C->>C: Apply fixed configured batch unchanged
    J->>S: Execute sync work
    Note over P,C: Dormant live-limit and effective-batch seams are activated only in PR 5
Loading

Compatibility and safety

  • Static admission is unchanged when no live-capacity provider is attached.
  • Dynamic capacity is policy-owned and node-wide rather than mutable per admission.
  • A live limit can never exceed the static policy ceiling.
  • Invalid or throwing providers retain the last valid safe limit.
  • Capacity reductions are drain-only; running work is never cancelled.
  • Capacity increases explicitly wake queued work.
  • Explicitly selected CGs are never capped by the automatic-coverage batch.
  • syncCorePublicBatchSize: 0 still disables automatic Core coverage.
  • The absolute adaptive requester ceiling is 8, with lower operator, hardware, and store ceilings available to the activation layer.
  • Missing store-pressure telemetry prevents controller growth above the conservative initial concurrency of 2.
  • Controller decisions are calculated as one pure transition and then applied once.
  • Operator-facing controller state is derived from action, reason, cooldown, and telemetry facts rather than stored as a parallel label.

Validation

  • Focused controller, sampler, backpressure, scheduler, priority, and coalescing suites: 6 files / 92 tests passed
  • Agent dependency/build chain passed, including TypeScript, public type tests, and package-root test
  • git diff --check

Rollout risk

Low. This PR adds internal extension points and policy logic but does not connect them to live agent execution or expose an operator-facing adaptive setting.

Review focus

  1. Static callers retain exact behavior and call shapes.
  2. Dynamic capacity is one policy-level invariant beneath the static ceiling.
  3. Capacity downshifts never cancel running work.
  4. The reducer preserves the intended AIMD outputs and cooldown/counter invariants.
  5. This dormant slice exposes no configuration that operators could mistake for active behavior.

Comment thread packages/agent/src/dkg-agent-types.ts Outdated
Comment thread packages/agent/src/sync/backpressure.ts Outdated
Comment thread packages/agent/src/sync/adaptive-capacity.ts
@branarakic branarakic changed the title RFC-64 M1 4/7: add adaptive capacity primitives and config RFC-64 M1 4/7: add adaptive capacity primitives Aug 2, 2026
@branarakic
branarakic force-pushed the codex/rfc64-m1-core-priority-scheduler branch 2 times, most recently from e628fe3 to 695cb97 Compare August 2, 2026 02:57
Comment thread packages/agent/src/dkg-agent-lifecycle.ts Outdated
Comment thread packages/agent/src/sync/adaptive-capacity.ts
Comment thread packages/cli/src/daemon/context-graph-catchup-coordinator.ts Outdated
Comment thread packages/agent/src/sync/core-public-coverage-scheduler.ts Outdated
Comment thread packages/agent/src/dkg-agent.ts
@branarakic
branarakic force-pushed the codex/rfc64-m1-core-priority-scheduler branch 6 times, most recently from d0dcb18 to fde07fa Compare August 2, 2026 12:04
@branarakic
branarakic force-pushed the codex/rfc64-m1-adaptive-primitives branch from 9c848db to c5b9ec1 Compare August 2, 2026 12:10
Comment thread packages/agent/test/adaptive-capacity-sampler.test.ts
Comment thread packages/agent/src/sync/core-public-coverage-scheduler.ts Outdated
@branarakic
branarakic force-pushed the codex/rfc64-m1-adaptive-primitives branch from c5b9ec1 to 5685115 Compare August 2, 2026 12:24
Comment thread packages/agent/src/sync/adaptive-capacity-sampler.ts
@branarakic
branarakic force-pushed the codex/rfc64-m1-core-priority-scheduler branch from acfee2b to 9896d6b Compare August 2, 2026 12:52
Comment thread packages/agent/src/sync/adaptive-capacity-sampler.ts
@branarakic
branarakic force-pushed the codex/rfc64-m1-adaptive-primitives branch from 4b83ea1 to e7bbbac Compare August 2, 2026 12:53
@branarakic
branarakic force-pushed the codex/rfc64-m1-core-priority-scheduler branch from 9896d6b to b0d93ed Compare August 2, 2026 14:11
@branarakic
branarakic force-pushed the codex/rfc64-m1-adaptive-primitives branch from a6fa200 to 264d92d Compare August 2, 2026 14:13
@branarakic
branarakic force-pushed the codex/rfc64-m1-core-priority-scheduler branch from b0d93ed to 34d3cf0 Compare August 2, 2026 15:12
@branarakic
branarakic force-pushed the codex/rfc64-m1-adaptive-primitives branch from 264d92d to aa41669 Compare August 2, 2026 15:16

@otReviewAgent otReviewAgent left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Operational Notice: Review Agent could not complete this review.

Verification reviewer failed: retry_exhausted

@otReviewAgent otReviewAgent left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Operational Notice: Review Agent could not complete this review.

Business logic reviewer failed: retry_exhausted

@branarakic
branarakic force-pushed the codex/rfc64-m1-core-priority-scheduler branch from 34d3cf0 to 235c807 Compare August 2, 2026 15:44

@otReviewAgent otReviewAgent left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Operational Notice: Review Agent could not complete this review.

Business logic reviewer failed: retry_exhausted

@branarakic
branarakic force-pushed the codex/rfc64-m1-adaptive-primitives branch from aa41669 to 4376835 Compare August 2, 2026 15:46
}

/** Resolve one node-wide policy; the optional live limit is shared by every admission. */
export function resolveSyncGlobalBackpressure(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Bug: Adaptive capacity is not wired into production admissions

What's wrong
The changed backpressure API accepts a live capacity resolver, but this diff does not connect the new adaptive controller/sampler to any production call path. As a result, the feature does not actually throttle requester concurrency or Core automatic coverage under CPU, heap, event-loop, or store pressure.

Example
On a core node with heap pressure above criticalHeapRatio, the new controller would halve its current inflight value. In the submitted diff, no production code creates that controller, samples pressure, passes controller.getCurrentInflight() into resolveSyncGlobalBackpressure, calls notifyGlobalSyncBackpressureCapacityChanged, or passes the adaptive coverage batch into the Core coverage scheduler. Admissions therefore continue using the static configured/global default limit and automatic coverage continues at the configured batch size.

Suggested direction
Wire the new adaptive capacity runtime into the existing DKG agent sync lifecycle instead of only exposing helper APIs; production admission and coverage planning need to consume the controller’s current limits.

For Agents
Add a runtime owner in the agent lifecycle/base layer that resolves adaptive bounds, constructs AdaptiveCapacityController and AdaptiveCapacitySampler, periodically samples demand/pressure, calls notifyGlobalSyncBackpressureCapacityChanged() after capacity changes, passes the live currentLimit into the global backpressure policy used by withGlobalSyncBackpressure, and uses getEffectiveCoverageBatch() when planning Core public coverage. Add an integration test proving pressure reduces both requester admissions and automatic coverage breadth.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants