Enforce resource limits and qualify connector capacity

Parent: #970

Deliverable

Enforced resource limits plus a repeatable benchmark report establishing supported connector counts and throughput for each transport.

Dependencies

#982, #983, #991.

Galactic Edge integration (#992)

Implement this milestone as a consumer of the Galactic Edge provider contract. #992 owns provider authorization, host attachment lifecycle, shared-state fencing and maintenance, independently enforced forwarding expiry, prefix conflict arbitration/publication, and gVPC traffic policy. MASQUE owns client authentication, connector /96 assignment and retention, session packet checks, transport behavior and its provider adapter. Use the foundation's versioned contract and existing provider helpers; do not introduce a second host-networking controller, lease store or BGP publisher. Any required foundation change must be tracked and qualified under #992 before this milestone claims integrated acceptance.

Work

Implement or complete the design's connection/request, buffer, queue and resource limits, including admission and overload behavior.
Build a repeatable load harness with documented hardware, versions, packet sizes, active connector counts and traffic mix.
Measure HTTP/3 and HTTP/2 separately, including slow readers, packet loss, request churn and overload while existing connectors carry traffic.
Check that expiry, revocation, routing and tenant isolation remain correct under load; tune the implementation to meet the agreed targets.

Acceptance criteria

The report includes throughput, latency, CPU/memory use, accepted/rejected work and the tested supported connector count for each transport.
Overload causes bounded, documented rejection/backpressure rather than unbounded memory growth or stalled authority expiry.
Resources recover after clients disconnect or requests fail; a slow connector cannot consume unbounded shared capacity.
The capacity benchmark against the agreed targets runs on staging and is tracked in MASQUE: Run the capacity benchmark on staging #1242; this issue closes on the loopback harness, the enforced limits and the 6.25 s slow-client bound.

Additional integration acceptance

Budget provider instances, attachments, EdgeRoutes, map capacity/headroom, API operations and lease-renewal work alongside transport sessions/buffers. Account for one renewed lease per instance and multiple connector routes.
Preserve renewal/expiry progress under request floods, stalled admission/API calls, reconnect storms and slow packet devices; one connector cannot monopolize shared locks or queues.
Verify CONNECT-IP server lacks overload limits and tenant isolation checks #991 limits before client admission and queued-byte limits for both transports. Saturation must reject bounded work without reviving expired sessions or dropping unrelated tenants.
Reconcile results with the foundation capacity/admission limits and the Define connector admission, addressing and ownership contracts #972 workload; report packet, API and foundation resource budgets as well as protocol throughput. Measured on staging under MASQUE: Run the capacity benchmark on staging #1242.

Scope follows #970: CONNECT-IP with HTTP/2 fallback; no public API development, plugin/portal integration, product-client development or production rollout.

fix(connectip): Bound CONNECT-IP work before admission

Verdict: valid. Checked against 66eb2bb3, the top of the open CONNECT-IP stack (#1205, through #1208), since #982 and #991 land there.

Most of the contract's pre-admission and session limits are in place: the connection, per-source, stream, header, handshake, admission, per-credential and session caps from #991 and #1202, the 128-packet / 256 KiB per-session queue toward the client, the bounded capsule reader, and EAGAIN on the device counted as a drop. Three gaps remain against the resource table in docs/designs/masque/contract.md:

Blocked delivery over HTTP/3 has no 5 s timeout. A minimal test opens an HTTP/3 session on loopback, stops the client reading its UDP socket and keeps injecting packets toward it. The server's packet writer blocks in SendDatagram once the QUIC datagram queue fills (32 frames). The session ends only when the QUIC idle timeout fires, 30 s later. HTTP/2 already ends a session whose capsule write is blocked for 5 s.
A full per-session queue toward the client drops packets without counting them. TUNBackend.readLoop ignores the result of enqueue. The contract says a queue drop is a counted packet drop.
There is no load harness or capacity report. Nothing in the repository measures goodput, latency, CPU, memory, or accepted and rejected work per transport. Nothing exercises overload, slow readers, churn or loss while other connectors carry traffic.

The physical two-node benchmark the contract specifies cannot run from code changes alone. A loopback harness can still measure each transport, exercise overload and recovery, and give the target-environment run a repeatable tool.

Approved plan. Two PRs appended to stack #1205 on top of #1208. Both say "Related to #984", and this issue stays open for the two-node benchmark the contract specifies.

fix/issue-984-limits
- proxySession ends a session when one write toward the client stays blocked for 5 s, on both transports. It counts the end in galactic_connectip_packets_denied_total with reason slow_client. Only that session ends.
- TUNBackend.readLoop counts a packet dropped on a full session queue with reason queue_full.
- Tests: a stalled HTTP/3 client (30 s today, 5 s after the fix), a stalled HTTP/2 client, and the queue-full counter.
test/issue-984-load
- An opt-in loopback load harness, task test:connectip-load, that runs the real server for HTTP/3 and HTTP/2 separately. It covers 1, 8 and 32 connectors, 1280-byte and 64-byte packets, an overload flood while connectors carry traffic, a slow reader, request churn, HTTP/3 packet loss, and lease expiry under load.
- It reports goodput, p50/p99 RTT, loss, CPU, RSS, goroutines, and accepted and rejected work, and checks that resources return to baseline after clients leave.
- docs/edge/masque-capacity.md gives the method and the loopback results, marked as not the contract's target environment.

Risk: a client that cannot take traffic for 5 s now loses its session over HTTP/3 instead of holding it for up to 30 s. That is the contract's stated behavior, and HTTP/2 already does it.

fix(connectip): Time a run of slow writes toward a stalled client

fix(connectip): Keep a slow-write run across probe-freed writes

test(connectip): Pin the prompt streak that ends a slow-write run

docs(edge): Mark the blocked-delivery timeout as missed on both trans…

docs(edge): Mark the 5 s delivery timeout missed on both transports

Two PRs in stack #1205 cover the code side of this issue. Both say "Related to", so this issue stays open.

fix(connectip): End a session whose client stops receiving for 5 s #1210 ends a session whose client stops receiving on both transports, and counts drops on a full queue toward a client. A harness run showed the first version let a stalled HTTP/3 client hold its session for 13.3 s, because QUIC probe timeouts keep draining the datagram queue. The PR now also ends the session when slow writes keep recurring.
test(connectip): Add a CONNECT-IP load harness and capacity report #1220 adds task test:connectip-load and docs/edge/masque-capacity.md with the first loopback run, 3 runs of every scenario, all passing. At 32 connectors, HTTP/3 reached 2403 Mbit/s and HTTP/2 2971 Mbit/s, and peak memory was 136 MiB. A stalled client ended after 5.5 s on HTTP/2 and 6.8 to 8.0 s on HTTP/3, so neither meets the 5 s target exactly. The report leaves that choice to the target review.

Closing this issue still needs the two-node benchmark the contract specifies, and review of any target change.

fix(connectip): End a session whose client stops receiving for 5 s

fix(connectip): Time a run of slow writes toward a stalled client

fix(connectip): Keep a slow-write run across probe-freed writes

test(connectip): Pin the prompt streak that ends a slow-write run

test(connectip): Add a CONNECT-IP load harness and capacity report

docs(edge): Mark the 5 s delivery timeout missed on both transports

docs(masque): Keep the isolation floor and name the pending capacity run

Closing this on the loopback harness, the enforced limits and the 6.25 s slow-client bound. The capacity benchmark on staging, and the budget reconciliation it measures, continue in #1242. The limits and the stalled-client end merged in #1210. The load harness and its report merged in #1220. The 6.25 s HTTP/3 fix merged in #1230. The results are in the MASQUE capacity guide.

Criterion	Evidence	Status
The report gives throughput, latency, CPU and memory, accepted and rejected work, and the tested connector count for each transport	The loopback report, run 3 times per scenario: HTTP/3 and HTTP/2 at 1, 8 and 32 connectors with 1280-byte and 64-byte packets. At 32 connectors, 2403 Mbit/s on HTTP/3 and 2971 Mbit/s on HTTP/2, with RTT p50/p99, CPU cores and peak RSS for each row. The overload rows count refused requests, about 18,000 to 20,000 per run	Met on loopback
Overload gives bounded, documented rejection, with no unbounded memory growth or stalled expiry	256 flood clients, twice the connection cap, were refused with 503 or a closed connection. Session queues held at 128 packets. Leases revoked mid-flood ended their sessions in 1.0 to 3.9 ms	Met
Resources recover after clients leave, and a slow connector can't take unbounded shared capacity	Every run returned to zero sessions, backend sessions and QUIC connections, with goroutines back to baseline. A stalled client's session ended after 5.5 s on both transports in all 9 runs each, and no other connector starved	Met
Staging benchmark; close on the loopback harness, limits and the 6.25 s bound	5.5 s against the 6.25 s bound on HTTP/3 and HTTP/2 (#1230)	Met; benchmark in #1242
Budget instances, attachments, EdgeRoutes, map headroom, API and lease-renewal work	The contract's resource budget table sets each one: one renewed lease per instance, at most 128 connector routes per cell, 25% map headroom, and the EdgeService and Lease write rates	Budgeted; measured in #1242
Renewal and expiry hold under floods, stalled admission, reconnect storms and slow devices; no connector monopolizes shared queues	Overload ran with a 2 s admission stall and revoked leases mid-flood. Churn ran 10 new requests a second beside 8 connectors. TUN session queues cap each connector at 128 packets and 256 KiB. The slow-reader runs left the other connectors receiving	Met on loopback
#991 limits apply before admission, with queued-byte limits on both transports, and saturation never revives expired sessions or drops unrelated tenants	Unit tests cover connection, per-source, header, handshake, stream, admission and session limits on both transports, and the TUN queue byte and packet limits. No probe runs before admission. In the overload runs no connector starved and revoked sessions stayed ended	Met
Reconcile results with the foundation limits and the #972 workload	Needs staging measurements	Moved to #1242

These numbers come from one host, with the clients sharing the server's CPUs. They show the limits hold. They don't show the capacity of an edge node, which is what #1242 measures.

Navigation Menu

Repository navigation

Enforce resource limits and qualify connector capacity #984

Deliverable

Dependencies

Galactic Edge integration (#992)

Work

Acceptance criteria

Additional integration acceptance

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Enforce resource limits and qualify connector capacity #984

Description

Deliverable

Dependencies

Galactic Edge integration (#992)

Work

Acceptance criteria

Additional integration acceptance

Activity

privateip commented on Oct 28, 2026

privateip commented on Oct 28, 2026

privateip commented on Oct 28, 2026

privateip commented on Oct 29, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions