sandbox: contain the interface the packets actually leave by, and put it in the launch path - #996
Merged
pmilic021 merged 58 commits intoSep 18, 2026
Merged
Conversation
|
Someone is attempting to deploy a commit to the MakePrisms Team on Vercel. A member of the Team first needs to authorize it. |
added 20 commits
September 14, 2026 10:59
…ctually leave by `sandbox_net`'s rules sit on the host kernel's OUTPUT chain. A gVisor payload never traverses it: runsc runs its own netstack and hands finished packets to the namespace's veth, so the denied destinations stay reachable while the readback proves the rules are installed. Measured on this repo's fixtures, both families, over TCP. Adds `sandbox_iface`: the same rendered `NetPolicy`, translated into clsact egress `flower` filters on that veth. It holds no policy of its own — every prefix, every exception and their order are derived from `NetPolicy::rules`, so a range added there appears here without anyone remembering to, and the parity tests fail if a translation is ever dropped. The drops carry no protocol match, deliberately: the demonstrated leak was TCP, and a TCP-only filter passes the very fixture that found the hole. Ships `tc`/`ip` in the netfilter sidecar behind `test -x` assertions, and a second dumb applier (`apply-iface`) that may run nothing but `tc` — the interface, the prefixes and the order are chosen in Rust and read back out of the kernel by a different container. Not wired into `establish` yet; that is the next commit.
`sandbox_iface` rendered, applied and verified a plan nothing called. This wires it into `sandbox_netns::establish`, after the iptables plan and its readback and before the holder is handed back, so every contained job gets both layers or gets no launch. The device is measured, never assumed: an unprivileged probe container enumerates the namespace's links and `select_egress_link` refuses anything that is not a job's own namespace — a bridge among the links, a missing loopback, more than one candidate. The one container that holds `NET_ADMIN` therefore does not also choose what to filter. The link that was filtered is carried out on `Containment::egress_dev` so a caller names the interface that is actually filtered rather than the one everybody assumes is `eth0`. Failure discipline is the chain's: the applier's count is cross-checked against the rendered step count (a truncated stdin applies perfectly and exits 0), the filters are read back by a second container running a different verb, and every `?` leaves through the holder guard, which destroys the namespace. Exit 6 (an image built without iproute2) and exit 3 (a partially filtered interface) are named in the error, because "which refusal" is what the operator needs. Unconditional, not runsc-only: `establish` is not told which runtime the caller will launch under, and "contained under one runtime" is the state being closed. Tests, in the live file that executes plans against a real kernel rather than asserting a render: * `establish_filters_the_veth_the_packets_actually_leave_by` — the shipped entry point, then the kernel is asked what that namespace holds. * `a_namespace_missing_one_egress_filter_is_refused` — the red-prove: remove one filter from a live namespace and the readback must name it. A verifier that cannot fail is worth nothing, and every other case here would stay green. * `the_output_chain_alone_lets_a_runsc_job_out_and_the_veth_filters_stop_it` — the regression gate. Leg 1 asserts the LEAK as a success: with the iptables policy installed and verified, a gVisor job reaches a denied destination that a `runc` job in the same namespace cannot. Leg 2 installs the veth plan and the same connection is refused, with an allowed destination still reachable from inside and the denied listener still answering from outside. The runtime is taken from `MAXPLAYER_RUNSC_RUNTIME` and not defaulted, for the reason the image is not defaulted: a guess would measure `runc` on a host without gVisor and report the leak closed by rules that never had to stop anything.
Three corrections, each one measured on the gvisor-repro VM rather than reasoned
about — the first draft of these tests passed type-check and failed the machine.
1. `tc filter del` does not take the `clsact` keyword that `filter show` takes:
iproute2 answers `Unknown filter "clsact", hence option "egress" is
unparsable`. The red-prove now deletes through the same hook spelling the
daemon's own argv uses.
2. **One namespace per gVisor payload.** `runsc` claims the namespace's links
when it starts, so a SECOND joiner gets ENETUNREACH — a refusal no rule
caused, which reads exactly like containment. It cost this gate a *control*
before any rule existed. Every probe now builds its own holder, which is also
what production does per job.
3. **One network, because a job holder has one.** The canary fixture attaches
its holder to three networks; `select_egress_link` then refuses to pick one
of three links rather than leave two open, and the gate failed with "expected
exactly one non-loopback link, found 3". That is the product being right and
the fixture being unrealistic. The topology is now the one a contained job
gets: one network, one veth, and the denied destination is a second address
on the same listener inside 198.18.0.0/15, routed on-link into each namespace
so a refusal is a drop and never a missing route.
Measured on the VM, `runsc` release-20260817.0 / iproute2 in the first-party
sidecar image, all three live tests green:
establish_filters_the_veth_the_packets_actually_leave_by ... ok
a_namespace_missing_one_egress_filter_is_refused ... ok
the_output_chain_alone_lets_a_runsc_job_out_and_the_veth_filters_stop_it ... ok
The last one is the regression gate, and its leg 1 asserts the LEAK as a
success: with the iptables policy installed and verified, a gVisor job reaches
198.18.7.2 while a runc job in an identically prepared namespace cannot. Leg 2
installs the veth plan and the same connection is refused, with the allowed
destination still reachable from inside and the listener still answering from
outside.
…y, owned fixtures, integrated launch matrix Round 2/3 against the advisor verdict 11a5a8fb. F1/F4/F5 complete and gated; F2 authored but never executed (live runtime held); F3 module written but NOT registered in lib.rs and with no acceptance script yet. Committed to preserve the lane at close. Not a claim of completeness.
…and that can fail
…ng are accounted for
The live matrix measured it: with ff00::/8 dropped, a job reaches NO v6 address. Neighbour Solicitation goes to a solicited-node MULTICAST address, so that one rule starves ND and every v6 destination fails alike — denied and allowed. A refusal measured on a dead stack proves nothing about fc00::/7. Both enforcement hooks now permit exactly two ICMPv6 control messages and nothing else: solicitation to ff02::1:ff00:0/104 and advertisement, each at hop limit 255 (RFC 4861 §11 — a router decrements, so 255 can only come from this link). Every destination denial is unchanged. The narrowing is enforceable at both hooks, measured not assumed: iptables takes --icmpv6-type with -m hl --hl-eq, tc flower takes ip_proto icmpv6 with type and ip_ttl, and both were installed and read back on a live kernel. A payload cannot use the exception: types 135/136 need a raw socket, jobs run --cap-drop ALL (CapEff 0), and a ping socket sends only echo, which stays dropped. Restores integrated.allowed.v6 to Connected and adds the counter-control the lazy fix would fail: integrated.denied.v6-link-local, where the listener's own link-local answers an unfiltered joiner and must refuse a contained job.
The partial-install test cut a fixed four lines off the end of the plan. The plan is every v4 rule then every v6 rule, so that meant 'drop the three v6 rules and one v4 rule' only while v6 had three rules. With the two neighbour-discovery exceptions v6 has five, the same cut removed four v6 lines and no v4 line, the v4 readback verified, and the test that exists to catch a partial install passed against a complete v4 policy. Cut is now v4_rules - 1, so the shortfall cannot be absorbed by rules that follow it, and both families are asserted to refuse: v4 short by exactly one rule, v6 never reached.
Rebasing onto 995 replaced the readback model this control was written against. The ND exceptions are deliberately routed into 995's `matched` list so its ordering check governs them rather than a second, parallel check of my own -- which is what the control wants, but it means the refusal is now worded by that check: the rule, its index, and the drop that precedes it. The behaviour never regressed. An ND ACCEPT appended below the range drops is still refused; only the adjective in the message changed, and the assertion was pinned to the adjective. Pinned instead to the same wording its sibling ordering tests assert on, and additionally to the advertisement role and its inertness, so an unrelated refusal cannot satisfy it.
Advisor R2, F3 evidence fidelity: `parse_case` assigned each field by overwriting, so a repeated field let the last word win. `outcome=refused outcome=connected` parsed as Connected -- a record could carry a refusal and the pass contradicting it, and the pass is what got counted. The same held for `log`, which decides which file a reader would open to check the claim. Each field is now assigned once and a repeat is refused, naming both values. There is no honest reading of a case that states two outcomes, and refusing is the only answer that cannot be gamed by field order. Test asserts the outcome and the log spelling. Verified meaningful: with last-wins restored it fails, and the log leg passes validation outright.
Advisor R2, F4: `SidecarGuard::drop` deregistered unconditionally, so cancellation itself destroyed the record cleanup depends on. The future is dropped mid-command, that drops the guard, the guard strikes the name from the registry, and the holder's `Drop` -- reading that registry a moment later -- finds nothing to remove. The container the blocking docker client had already created stays joined to the namespace with no guard and no remover. The doc comment claimed the name stayed with the holder on cancellation; the code did the opposite, and the comment was the only place it was true. The guard now deregisters only a command that returned, marked on the far side of the await where the command is provably no longer in flight. A cancelled command leaves its name deliberately: keeping a name whose container was never created costs one `docker rm` answering "No such container", which is already treated as success, while dropping a name whose container does exist costs a pinned namespace nothing cleans up. Adds the cancellation witness that was missing -- the future is polled once so the registration exists and the command is in flight, then dropped, which is how tokio cancels. No runtime and no docker involved, so it measures the custody rule itself. Verified meaningful: restoring unconditional deregistration turns it red. The sibling finishing test now says finishing explicitly.
Advisor R2, F4: `force_remove` used a blocking `output()`, which has no timeout, under a comment promising a bounded removal. This runs inside `Drop`, so a docker client talking to a daemon that has stopped answering held the teardown thread for as long as the daemon stayed wedged -- and teardown is the path a panicking or aborted job takes. Removal now spawns and waits against a deadline. On expiry the child is killed and reaped, and the wait is reported as a failure naming the container as possibly still present, rather than as a removal that worked. Twenty seconds: far longer than a real removal, which finishes in well under a second, and short enough that a wedged daemon ends the job instead of pinning the caller forever. Two tests, because a deadline that fires unconditionally would pass the first alone: a child guaranteed not to exit is abandoned, reported as a possible leak, and proven killed rather than left running; a child that exits at once is waited on normally.
Advisor R2, F2: the oracle turned every nonzero payload status into `Refused`. An image that lost `nc` exits 127 through `sh`, 126 says not executable, 2 says the tool rejected its own arguments, 128+n says a signal killed it -- and each of those scored as containment. That is the most flattering way this matrix could be wrong: a fixture that silently lost its payload would have reported perfect containment on every denied leg, and the record would have looked its best at the moment it measured nothing. Only exit 1 now means refused, which is what BusyBox `nc` -- the Alpine fixture's tool -- returns for a refused connection and for the `-w` timeout. Everything else nonzero becomes `ToolFailed(code)`, which is never containment evidence and is printed loudly, because unlike `NeverStarted` such a leg did start and so looks like it measured something. Also asserts the neighbouring-port leg as Connected, which is what its own comment and the saved record claim. It asserted only `!= NeverStarted`, so Refused passed -- meaning a filter that had silently become port-scoped, the one failure the leg exists to rule out, would have satisfied it. The classifier's witness is offline: it is a property of the classifier, not of any network, and every denied row in the record is its verdict.
Advisor R2, F1: `check_counter_line` scanned for recognised predicates rather than reading the line. Anything absent from both token lists rode through untouched, and `src_ip` is absent from both -- so appending `src_ip 192.0.2.123` to a faithful `Sent` line left the parsed rule and the verification unchanged, on a line the parser skips, using a key that narrows the rule to a single source. A list of what is forbidden cannot refuse what nobody thought to forbid. Statistics lines are now read to the end against the shape iproute2 prints -- `Sent` in its eleven tokens, `backlog` in its five, the `Action statistics:` header with nothing after it -- with counts checked as counts. Anything not positively read is refused, predicate or not. Four counterexamples added to the existing strict-accounting table: the advisor's `src_ip` in v4 and v6 spellings, arbitrary text no list will ever contain, and a counter replaced by something that is not a count. Verified meaningful: restoring the blacklist turns the first red.
Two legs failed against the rebased head for reasons the merge introduced, not for reasons the policy changed. `the_pinhole_production_installs_is_the_one_the_policy_names` asserted that every `pass` rule on the veth carried the configured proxy range. After 995, production also installs the resolver pinholes the config asks for, so the leg read `["49200-49299", "53", "53"]` and called the configuration a hole. Widening it to "ignore port 53" would have retired the assertion: a port-53 rule to *any* destination is exactly the wide-open case the leg exists to catch. The recorder now pairs each pass rule's port with its `dst_ip`, and a rule is admissible only if it is the proxy range, or port 53 to the resolver the gate itself named. A port-53 rule to anywhere else still fails, and a missing resolver pinhole now fails too — a job that cannot reach its resolver resolves nothing. `a_job_prepared_and_launched_by_production_is_contained_on_its_veth` asserts the allowed destination stays reachable on the neighbouring port, so that a port-scoped denial cannot hide. `RunscNet`'s listener ran a single `nc -l` on PORT, so nothing was listening on OTHER_PORT and the leg measured an absent listener rather than policy. The listener now binds both ports, as the `Canary` fixture already did, and readiness covers both: an unchecked second `nc -l` would let the control pass for the wrong reason. `dns_servers` is now a named constant shared by the gate configs, documented as TEST-NET-1 and explicitly not left empty — empty falls back to the host's resolv.conf and resolvectl, which would make these legs depend on the DNS configuration of whichever machine ran them.
maxy-player
force-pushed
the
w-gvisor-interface-impl2
branch
from
September 14, 2026 18:17
7d14c51 to
c01e35a
Compare
added 7 commits
September 14, 2026 11:27
`GATE_DNS_RESOLVER` is 192.0.2.53 — TEST-NET-1, routed nowhere, answering nothing. It exists to make the rendered rule set deterministic so the pinhole leg can check the SHAPE of what production installed, and that is all it can do: a rendered `--dport 53 -j ACCEPT` and an enforced one produce the same green in this file. The previous commit message here said this file was made to "measure the merged DNS seam". That overstates it. It measures the seam's rule shape — that the resolver exceptions exist, go to the configured resolver and nowhere else, and that nothing wider was installed. It does not, and cannot, show that a contained job resolves a name. Functioning resolution is measured against a resolver that actually answers, in `sandbox_dns_live`: real dnsmasq fixture through the same production path, v4/v6 resolution through the written resolv.conf, UDP truncation falling back to TCP 53, host-stub discovery, and the denied-neighbour controls (another private address, and a non-53 port on the resolver itself). Both doc comments now say so, so neither this constant nor this file's readback can be cited as evidence of working DNS. No assertion is changed and no control is weakened; this commit is comments only.
… and refuse a readback this parser cannot account for Two residual defects from the same review, both of the same shape: a thing that was not established being treated as a thing that was. **Sidecar custody (F4).** `run_sidecar` called `registration.completed()` after every returned result, reasoning that `docker run --rm` has removed the container by then. That is true of a client REAPED with a status, including a nonzero one. It is not true of a client killed on our own 120s deadline, killed by a signal, or failed before the wait — a stdin write error, a wait error, a panicked task. In each of those the daemon may still be creating, running or removing the container, and striking the name from the registry removed the one cleanup target for a container that outlived its client. The cleanup path was producing the orphan the registry exists to prevent. `run_bounded_tracked` now reports whether the child was reaped, and custody ends only on that. Both halves are asserted: a nonzero exit DOES end custody (else every sidecar reports a phantom leak), a deadline kill does NOT. This does not claim to close the whole cancellation/daemon-cleanup bound. A delayed blocking create can still land after cleanup has run, and one attempted `docker rm` is still not a completion fence. Those remain open. **Readback accounting (ND parser).** `ReadbackRule::parse_all` silently dropped an unflagged token appearing before any flag, and discarded a trailing `!` when the line ended. So `-A OUTPUT garbage -p ipv6-icmp …` and a line ending in `!` parsed to predicates IDENTICAL to their canonical twins and were accepted as those rules. That is static malformed-readback acceptance — not a demonstrated kernel-emittable bypass, and no packet escape is claimed — but a readback that is not read whole is not evidence about the namespace. Malformed lines are now recorded and refuse the whole readback. They are kept in the parsed list rather than dropped: dropping would shrink the rule count and hide an unexpected rule from the check that exists to notice one. `value()` refuses to interpret them, so one can never satisfy a required rule. Tests use the reviewer's own counterexamples and assert the retained predicates are identical to the canonical form — which is why this was invisible.
…not only the flag that feeds it A negative control on the previous commit found its own test too weak. Reverting `run_sidecar` to the defective `registration.completed()`-on-every-result left `only_a_reaped_client_may_end_a_sidecars_custody` GREEN: that test calls `run_bounded_tracked` directly, so it proves the child_exited flag is computed correctly and proves nothing about what the caller does with it. The defect lives in the caller. `run_sidecar` now delegates to `run_sidecar_with_deadline`, whose bound is a parameter purely so the rule can be measured offline -- a test cannot wait out the production 120s deadline, and the assertion has to be about the registry after a kill, not about a boolean. The new test drives `run_sidecar` itself with two temp scripts and asserts the holder's registry: a client killed on the deadline keeps its name as a cleanup target, a client reaped normally has its name struck. Both halves again, so "never deregister" cannot pass. No docker and no daemon are involved. Verified by reverting the fix: the new test fails on the registry assertion (sandbox_netns.rs:1799) while the flag test still passes. Fix restored, both green.
…easure the host leg that nothing measured F3 saved-row fidelity. The validator checked that a case's log field was nonempty TEXT. It never opened the log. So a complete, attributable record could cite a real green run that did not contain the test the row claims — and the map from test function to case id was mine, by hand, stated in the record itself as not machine-checked. Case records now carry `test=`, the function that asserts the case, and the new `sandbox_evidence::corroborate` opens each named log and requires that test to be recorded `ok` there. It matches the `test <name> ... ok` line rather than the bare name, so cargo's `failures:` block cannot make a red run look green, and it refuses absolute or `..` log paths — evidence reaching outside the run is not attributable to it. Wired into the acceptance gate's own leg, so the one command checks it. What corroboration establishes: the named log exists, names that test, and shows it passing. What it does NOT establish: that the test's assertions are the right ones for that case id. Reading the body is still the only thing that settles that, and the doc comment says so. **And it immediately caught a false row of my own.** `host.unaffected.during-cleanup` is REQUIRED by the matrix, and grepping the whole repo finds it in exactly one place: the requirement list. No test asserted it. The round-2 record scored it `connected` anyway, on the strength of the sibling cleanup test running nearby — but that test probes only from inside containers, so it cannot observe the host's egress in either direction. The row was a claim, not a measurement. So the leg is now authored. `host_can_reach` connects from the test process itself, on the VM host, in no namespace — the only instrument that can see host-global mutation, which every container-side probe here is blind to by construction. It asserts the host reaches the canary before a contained job exists, while one is contained (with that job asserted Refused, so the host leg sits beside real containment), and after teardown. A port nobody listens on must come back unreachable first, or all three legs would pass on a probe that cannot say no. The old record does not validate under this gate, which is correct: it carries no test= fields and one row nothing measured. A fresh live run at this head replaces it.
…r is confirmed gone, refuse an ND line this parser cannot consume Three defects the R2 final named, each with a control that reproduces the failure rather than exercising the flag beside it. 1. A create that outlives its future was never fenced. Adoption supplies a NAME to remove; it says nothing about WHEN the container appears. The create runs on a blocking thread, cancelling the future above it does not stop that thread, and cleanup ran immediately: it asked docker to remove a container that did not exist YET, was told "No such container" -- treated as benign, correctly -- and returned satisfied. The create then landed and left a container nobody held. No ordering of removes fixes this, so CreationFence moves the removes to the far side of the create settling. The ticket lives inside the blocking closure, never in the future, so a cancellation cannot release it early. 2. Custody ended on a reaped client. `--rm` is a request to the daemon, not a receipt from it: a nonzero exit, a deadline kill, a stdin write that failed before the wait all return from the same call, and none of them is the container being gone. Custody now ends only on CONFIRMED absence, and a confirmer that cannot answer keeps the name a cleanup target -- the cost being one `docker rm` that says "No such container", which cleanup already treats as success. 3. The ND readback tolerated tokens it could not consume. A canonical ND ACCEPT carrying a leading unflagged token after OUTPUT, or a dangling final `!`, verified with the same accepted predicates. The whole readback is now refused, and the malformed line is kept rather than dropped so the rule count stays honest. The custody test was also made hermetic: it reached a live `docker inspect` from an offline unit test and passed only because a daemon happened to answer. Controls, each reverted and observed RED before restoring: - remove the fence wait -> cleanup_does_not_remove_ahead_of_a_create... fails - release on client exit -> a_reaped_client_whose_container_is_still_there... fails - stop marking malformed -> 4 readback tests fail, incl. the canonical ND one Offline: 1641 passed, 0 failed.
… kill, confine evidence paths, and gate establish itself The R3 verdict's remaining production-path faults, each closed with a test whose negative control was observed RED before it was observed green. Creation fencing reached only the holder. `establish`'s sidecar creates went through an unfenced path, so cleanup's settle-wait could pass at count zero and issue removes for a container that did not exist yet; the container then arrived with nobody holding it, pinning the namespace. The ticket now travels into the blocking closure on that path too, so a cancelled future cannot release it while the create is still running. Custody could still end on the client alone. A SIGKILLed client is reaped with a status and no exit code, and the old rule let that flag release the name even though a killed client proves nothing about what the daemon did. Custody now ends only on daemon-side absence. Evidence path confinement was lexical. `raw/x.log` is neither absolute nor `..` and could still be a symlink to another run's green log, so `corroborate` now resolves both sides and requires the result to stay under the record's own directory, reading the path it just checked. `establish` itself had no test at all, which is the fault behind the proxy and cancellation findings: a fixture stood in for the production path and could agree with a bug the production path does not survive. Both gates now drive the real function through a spawn-site seam, so the argv every plan test asserts is the argv production runs. One proves the address `establish` measures is the address its rendered plan pinholes; the other cancels mid-create and proves cleanup outlasts the create and removes the holder it made. The seam is named outside the `MAXPLAYER_` prefix on purpose: that prefix is reserved for config and refuses unknown variables fail-closed, which the full suite caught by refusing to start 14 unrelated tests. Offline suite: 1646 passed, 0 failed, 13 ignored.
The seam that lets the two new gates drive the real `establish` without a daemon was selected by an environment variable. That is a privilege boundary, not a convenience knob: anyone able to set a variable on the process could redirect every containment command -- create, inspect, remove -- at a binary of their choosing, in a shipped build. Production selection is now the constant `docker` under `cfg(not(test))`, with no environment read on the path at all. The override lives in a process-local static compiled only under `cfg(test)`, restored on drop so a panicking test cannot leave it set for whatever runs next. Both gates were re-observed RED against this mechanism before being accepted green, since the mechanism they run through changed: - plan pinholes a literal instead of the measured address -> FAILED - fence released before the create -> "cancellation returned in 173.3ms" and the source was restored byte-identical after each. Offline suite 1646 passed / 0 failed / 13 ignored.
added 19 commits
September 14, 2026 19:24
…rop a live obligation The callback existed; its termination paths did not hold. Three orderings ended with cleanup silently doing nothing, and each is closed here at the point where ownership actually ends. REGISTRATION IS NOW ATOMIC WITH THE ZERO TRANSITION. `register` checks `in_flight` and installs the job under one guard, and the last ticket drop takes the slot while still holding that same guard, in the same lock order. The ordering that used to lose the job outright -- wait times out, last ticket drops and finds an empty slot, cleanup installs a callback afterwards -- is now exhaustive instead: either a drop is still coming and the fence keeps the job, or none is, and `Registration::AlreadySettled` hands it back and it RUNS. A registration on an already-idle fence can no longer be a parked closure nothing will ever fire. The earlier test could not see this because it held the last ticket alive across registration, which is the one ordering where the bug cannot happen. A FENCE BEING DESTROYED RUNS WHAT IT STILL HOLDS. The slot holds a `FnOnce`, and a `FnOnce` that is merely dropped does nothing at all, so a fence that died still holding cleanup discarded it in silence -- indistinguishable downstream from never having owned the container. The job deliberately keeps no `Arc` back to its own fence, because that is a cycle and the fence would never be destroyed at all; `Drop for CreationFence` is what makes that safe. A FAILED CONFIRMATION STAYS OWED. The job now returns `Custody`, so a removal that was issued and never confirmed cannot end by returning quietly: it hands back the work over exactly the names still outstanding, and the runner puts it back in the slot. A later settlement runs it again; if none comes, destruction does. Retries are bounded by the runner, not by the job, and nothing new waits -- each attempt is the existing removal and confirmation under `bounds.confirm`. No registry, no journal, no proxy, no extra wait window, no log-only remedy. One slot, one job, as before. Three gates, each observed RED against the code it replaces: the settle-before-register race, loss of every `Arc` to the fence, and a failed confirmation followed by the later real removal. They assert the CONTAINER and the daemon's own log -- presence markers gone, removals issued, the owner looking again -- not `holds_retained_owner`, which is only the code's opinion of itself. The four round-2 controls were re-run at this head and are still RED. Source restored byte-identical after every control. Offline lib suite green at default threads: 1663 passed, 0 failed. What this still does not claim: tickets track this process's work, not daemon settlement, so nothing here proves a deferred create landing after destruction is observed. Once the last reference to a fence is gone with a name still unconfirmed, in-process ownership genuinely ends and the boot reaper is the backstop -- now said out loud, with the names, rather than by returning quietly.
Bob-renewal round 1 of 3 for PR 996, answering the R3 verdict's three open findings on cleanup custody termination. 1. Finite destructor retries no longer terminate custody. A fence being destroyed runs its owed jobs RETAINED_FINAL_RUNS times and then TRANSFERS whatever is still owed to a process-lifetime CleanupSupervisor (one in-memory queue, one thread, woken by its own schedule and by every adoption). Bounded work per attempt, doubling backoff capped at 8x `reschedule`. Nothing is released by a clock. 2. Local completion is no longer taken for daemon completion. A create client that is killed at its bound, signalled, or loses the daemon mid-request is recorded on its fence as UNANSWERED, and at settlement becomes a watch-for-landing obligation owned by the supervisor. An absent inspect does not discharge it; only a daemon observation does: the container seen present and then removed and confirmed, or the daemon's event log since the request showing the exact name. Query failure never counts as absence. The API has no "never applied" observation, so a request that was never delivered stays watched at bounded cost for the process lifetime; that limitation is documented. 3. Collisions preserve every obligation. The fence holds a Vec of owners; registration pushes, and a still-owed job is pushed back next to the others rather than dropped or assigned over. RetainedOwner is now data (names, client, bounds, what is owed) rather than a FnOnce, so any owner can run one attempt and report exactly what is still outstanding. Gates (stand-in daemon, unit evidence): >3 refused removals with every original reference gone then recovery; unanswered holder AND joiner landing after local completion and initial absence; daemon unreachable keeps both kinds of obligation; two jobs on one fence both kept and discharged; landed-and-gone discharged on the event log. Two lib tests marked `#[ignore]` run the same paths against the real daemon on the VM.
…ad-free scheduling, supervised fast Drop
Round 2 of the Bob-renewal cycle on PR 996; the four failures of the round-1 verdict, nothing wider.
F1 A watch that saw a name absent no longer discharges on "an event under this name". The daemon's
event log is read as id/name/action, paired per container id, and the name is discharged only
when every container created under it since the request was destroyed AND a fresh inspect says
absent. A created id with no destroy is LANDED and stays owed. Deterministic race test: the
stand-in lands the container the instant after the absent inspect answers, before the event query.
F2 Whether the daemon ANSWERED a client that ended is decided from positive proof only — exit 0, its
own refusal text in a stderr read to EOF, or a `run` exit that is the contained command's — and
decided BEFORE the pending-drain return. Unknown stderr, early drain returns and read errors all
record the create as unanswered. The eight-substring whitelist is gone.
F3 The supervisor worker is started at fence creation and persists; a refused spawn no longer waits
for a future adoption — every cleanup event (ticket release, fence or holder destruction, adoption)
retries the thread and otherwise runs one due attempt inline. The event-log reader is bounded by
the owner's confirm bound, so a descendant holding that pipe costs one name one attempt, not the
whole queue. Tests: spawn refused throughout; a held event pipe with another owed name.
F4 The ordinary, settled `NetnsHolder::drop` hands every refused holder/joiner removal to the process
supervisor instead of sweeping, logging and returning. Gated on the actual `Drop` (unit and live)
and on the production fence reporting to the process supervisor.
Continues the stood-down lane's uncommitted work at 93bc9a9 on its merits: per-job deadline+grace label written at create, seat/role labels, listing and parse, expired_owned predicate, bounded sweep_expired, and the deadline traced from prepare_launch into the holder create. Checkpoint: not yet wired to a periodic caller, helpers not yet labelled.
…s their holder Helpers are labelled at the single run_sidecar funnel, not per argv builder, so a helper added later cannot be born unsweepable. Helper job id is a distinct label from HOLDER_LABEL so the boot reaper's selection still sees holders only.
First tick fires at startup, so a seat that was killed mid-job rediscovers its leftovers from their own labels. Best-effort: a failed pass is retried by the next tick, never in place.
Boundary at the stamp, per-job deadlines judged separately, unreadable/foreign stamps never a permission, production deadline traced into the label and out to the sweep, helper stamp parity, and against a stand-in daemon: seat scoping, refused removal retried by the NEXT sweep, unread listing is an error not an empty sweep, late appearance caught with no registry, bounded per pass.
…ontainers Petar's revision of PR 996 replaces continuous custody with a bounded periodic sweep keyed on each job's own deadline, so the machinery that kept obligations alive in memory for the life of the process is removed: - CleanupSupervisor (its thread, backoff schedule, adoption and outstanding reporting), SupervisorState, ScheduledOwner, Outstanding, BACKOFF_CAP_MULTIPLE and FenceBounds::reschedule. - The watch-for-landing obligation: UnansweredCreate, Owed (now the single remove-and-confirm job), watch_for_landing, note_unanswered, watch_unanswered, and the docker `events --since` reader (Lifecycle, lifecycle_since, lifecycle_from_events) with daemon_answered/container_named_by. - The fence's supervisor/unanswered fields and both drop-site handoffs; the refused-removal adoption in the holder's ordinary teardown. What stays: best-effort removal and confirmation, the retained-owner handoff across a create that is still in flight, RETAINED_FINAL_RUNS rounds at fence destruction, and the names printed when that is not enough. Those names are now left to sweep_expired, which is why nothing has to outlive the fence: every container carries its job's cleanup-after stamp. Tests for the retired paths are deleted with it; sandbox_netns:: is 64 passed, 0 failed under --features acp,wallet.
`cargo test -p maxplayer-core --locked` and `cargo test -p maxplayer --locked` (CI's two default-feature rows) did not COMPILE on this branch before this commit: `NetnsHolder::drop` and `RetainedOwner::remove_and_confirm` are not feature-gated, but `HolderCleanup` and `container_is_absent` were `#[cfg(feature = "acp")]`, so the names were out of scope with default features. Verified present at 93bc9a9, this lane's base, so it is not from the custody removal. Removing those three gates is the minimal fix: nothing in either item needs a feature-gated dependency. Both rows now pass — core 493 passed, maxplayer 162 + 2 + 3 + 3 + 6 passed, 0 failed.
Maxie's acceptance point: 32 removals is a bound on COUNT, not on duration. The removals are serial and each carries SWEEP_DOCKER_DEADLINE, so a daemon that accepts the connection and then hangs makes one pass 20s + 32x20s = 660s -- past two further ticks -- and the old select! arm awaited that pass inline, so the seller loop served no offer, award, drain or shutdown while it ran. - SWEEP_PASS_BUDGET (240s, under the 300s interval) bounds the whole pass, listing included. Checked before STARTING each removal, never mid-call. - ReapReport::deferred reports what the pass selected and never asked docker about -- the count-bound remainder and whatever the budget stopped -- in selection order, budget-stopped names first, so the next pass reaches the oldest leftovers first. The run loop logs the count. - The tick SPAWNS the pass (spawn_local) instead of awaiting it, and an in-flight flag drops a tick that lands while a pass is still running, so passes cannot stack. A Drop guard clears the flag, so a panicking pass does not silence the sweep for the life of the process. Tests: a spent budget starts no removal and defers all six (deterministic, no sleep); the following pass removes them; the deferred backlog keeps selection order; the count-bound test now asserts deferred and that docker was never asked about those names. sandbox_netns:: 66 passed, 0 failed.
16f9032 says it continues "the stood-down lane's uncommitted work at 93bc9a9". That is wrong, and this commit is the correction, since the branch is published and its history is not being rewritten. The predecessor lane w-gvisor-interface-impl3 is at 8c2a8ee, clean, with four commits above 93bc9a9 -- e643ebb, 565cb72, b940bc3, 8c2a8ee -- and that work is published on refs/heads/w-gvisor-interface-impl3. Verified read-only at 2026-09-15T14:26Z: rev-parse 8c2a8ee, empty status, and git ls-remote returning the same object for that ref. The description was stale, not invented: it came from my brief, which was accurate when written and overtaken when that lane committed. Nothing about the code in 16f9032 changes -- it was adopted on its merits, reviewed line by line, and every gate reported for this branch ran on heads containing it. Only the sentence about the predecessor's state was false.
Re-cut unadapted from 8c2a8ee (impl3), which wrote them against the same public API: a_real_holder_carries_its_own_expiry_stamp_on_the_daemon and the_sweep_removes_an_expired_holder_and_leaves_one_inside_its_deadline. The merged tree BUILDS -- impl3 never built it -- cargo test -p maxplayer-core --features acp,wallet --locked --test sandbox_netns_live --no-run produced target/debug/deps/sandbox_netns_live-0d2739f0d283af32, 0 errors. New row neither branch had: a_production_stamped_helper_past_its_expiry_is_ swept_against_the_real_daemon. It takes its labels from production's own helper_label_args -- not from strings the test spells out, which is the difference that matters -- puts them on a real container joined to the job's namespace with no --rm, reads all four labels back off the daemon, and then requires the sweep to select the job's whole expired set, holder and helper, 2 selected, 0 failed, 0 deferred. Still #[ignore]d, as every row in this file is. Live run in gvisor-repro follows.
A red that says only 'still listed' cannot distinguish the daemon hiding the helper from the sweep seeing it and leaving it -- and that distinction is the whole row. The failure message now carries three things captured before the cleanup destroys them: the ReapReport, production's OWN listing argv (list_owned_argv) run against the daemon before the sweep, and what the seat still holds after it. Earned, not decorative: this is what identified a stale-binary run as stale rather than as a flake. cargo judges freshness by mtime, and a source restored with git archive carries the COMMIT's mtime -- older than the artifacts -- so the rebuild was skipped and a mutated-control binary answered three further runs. The listing showed both containers present and stamped while the report removed one, which is only possible if the binary is not the source.
…g it The gate reported red on an independent machine at 65 passed / 1 failed. Two constants in the fixture, not the production drain, decided that result. The 2s deadline was a guess about how fast a host spawns `#!/bin/sh exit 0`. Where it was wrong the deadline killed the client before it exited, the run took the deadline-kill path, and the post-exit drain the gate exists for never ran. The budget is now eight times a spawn measured on THIS host, seconds earlier, through this same function. The descendant held the pipe for a fixed `sleep 6`. That is shorter than the four-deadline bound asserted below it, so an unbounded drain would have returned inside the bound and the gate would have passed through its own regression. The descendant now holds until this test writes a release file, with a two minute safety cap so a panic upstream leaves nothing holding a pipe. Verified on this host at this tree: 8 isolated runs and 4 runs under eight busy loops, all green; module suite 66 passed / 0 failed under the same load. Negative control — the drain's `recv_timeout(remaining())` swapped for a plain blocking `recv()` — turns it RED, naming a 421s wait against a 2s bound; the mutation was reverted and the file re-hashed to its pre-control content.
Three defects from the MakePrisms#996 round-2 review, each with an acceptance test that was shown to go red when the fix alone is reverted. D1 — a container the sweep cannot validate was silently filtered, so a leaked container and a clean host produced the same operator log. `partition_owned` now gates removal on four conditions (seat, role, job, readable stamp) and returns what it refused via `ReapReport.skipped`, which the seller's sweep tick logs per container. Another seat's containers stay neither removed nor reported: they are not this seat's business either way. D2 — the pass took the first MAX_SWEEP_REMOVALS candidates every tick, so a head that permanently failed removal was re-attempted forever and the tail was never asked about once. The pass now resumes after the last id it ATTEMPTED, wrapping at the end; advancing on attempt rather than success is what stops a permanently refused container from parking the cursor on itself. D3 — the cleanup stamp was rebuilt as `now + remaining` at the create, making it a fresh measurement of a job deadline that had already been decided. A clock that stepped backward in between therefore stamped a container EARLIER than the deadline its job was running under, and the sweep could remove it from under a job still inside its own time. `AgentRunTimeout::JobDeadline` now carries the absolute `deadline_unix` alongside the remaining duration, and `launch_cleanup_stamp` restates it without consulting the clock. Probes, which have no job deadline to carry, pass None and keep the window fallback. Test-only: the sweep cursor is process-global and keyed by seat, so sweep rows sharing one seat string shared a resume point and inherited each other's position. Each row now takes its own seat, as two seats on one host would.
The live 26-row matrix at 647c457, built and run inside gvisor-repro under runsc, failed two sweep rows with the D1 gate's own report naming the cause: skipped: [(84baae..., MissingJob)]. Production launches a holder with HOLDER_LABEL=<job id> and never with HELPER_JOB_LABEL; only helpers wear the latter. list_owned_argv asked the daemon for the helper label alone, so every real holder parsed as job: None and the D1 removal gate refused it as MissingJob on that pass and every pass after it. The holders this sweep exists to reclaim were the one class of container it could never remove. The listing now carries the holder's own job column and the parser reads the job per role, falling back to HOLDER_LABEL. Absence is still never permission: a container naming no job in either label is still refused and still reported. The unit fixtures could not have caught this — write_listing gives every row a helper job label, which production does not — so the regression row states the production-shaped listing in full. Mutation control: dropping the fallback turns a_holder_stamped_the_way_production_stamps_it_is_swept_not_refused red and nothing else; the file restores byte-identical (710a3894a8dd0b70).
…wins `parse_owned_listing` resolved the job id with `field(&mut fields).or_else(|| field(&mut fields))`. That is not per-role resolution: a present helper-job column short-circuited the holder column, so a container carrying two CONFLICTING job ids was ACCEPTED, and the sweep then reasoned about it under the wrong job. `OwnedContainer` now keeps `helper_job` and `holder_job` as separate fields, and `resolve_job` selects the column belonging to the row's own role. Columns that disagree, a job sitting only in the other role's column, and an unknown role are each SKIPPED and REPORTED through the existing skip channel rather than guessed at; agreeing columns are accepted. `list_owned_argv` still requests BOTH columns, because a conflict cannot be detected by declining to look at it. The test fixture had been hiding this. `write_listing` wrote every row's job into the helper column whatever the row's role — a shape no production launch emits — so no existing test ever presented a holder the way the daemon actually labels one. It now writes per role, which is what makes the three new cases meaningful: columns disagree, job in the wrong role's column, and each valid role still resolving from its own column.
…decides the sweep Item 2 of the PR996 restart round. The existing the_production_deadline_reaches_the_container_and_decides_the_sweep builds the cleanup stamp by hand, so it cannot observe the production path losing the deadline. This adds a SIBLING rather than rewriting it, and says so: the hand-built test still covers stamp parsing, this one covers provenance. The new test drives the production functions end to end against a real daemon: prepare_launch -> launch_cleanup_stamp -> container label -> list_owned_argv -> parse_owned_listing -> partition_owned. Labels are read back through the production listing argv, never a hand-written docker ps, so a change to the argv or the label schema is observable here. It is #[ignore]d rather than returning early when docker is absent: a test that returns early on a missing precondition reports as PASSED, which is the failure mode this round exists to remove. It is wired into the live VPS gate explicitly by name alongside the 26-row runsc matrix. Its mutation control is the point: dropping Some(deadline) at the sole production call site moves the recorded stamp by nearly a day and reds the stamp assertion. A green that cannot go red is decoration.
maxy-player
force-pushed
the
w-gvisor-interface-impl2
branch
2 times, most recently
from
September 16, 2026 13:51
78060b2 to
806d1d1
Compare
added 4 commits
September 17, 2026 04:19
…e run Round 2 items A and B. A. The existing live test enters at `prepare_launch` with a deadline it builds itself, so it guards the `launch_cleanup_stamp` argument and nothing upstream. It cannot see the forwarding edge break, because it supplies the value a mutation there would remove. Add a sibling that enters at `run_agent_job_with_env` — the function the orchestrator actually calls — carrying an `AgentRunTimeout::JobDeadline`, and read the daemon-written labels back through the production listing, parser and selection. Both arms are kept; they fail for different reasons. Correct the doc comment that called the old arm "the production call site", singular: that wording is how the bypassed caller went unnoticed. The launch future is not `Send` (`engine::run_job` takes `sink: &mut dyn FnMut(RunEvent<'_>)`), so it is driven on its own thread with a current-thread runtime rather than `tokio::spawn`. B. Every invocation used the same seat constant, so `list_owned_argv(&seat)` matched other concurrent runs and the pass path force-removed every listed container. Resources now hang off a per-invocation `LiveScope`: a seat unique to this run, and teardown in `Drop` so it still happens when an assertion panics — the path that most needs it. Teardown re-lists under this run's seat rather than reusing the asserted vector. Both tests remain `#[ignore]`d; they need a docker daemon.
Round 2 item C: the branch did not merge. Two content conflicts, both
mechanical unions rather than choices between the two sides.
seller_exec.rs, run_agent_job_with_env: main added the session MCP server
list and `launch_with_mounts`; this branch added the sixth `prepare_launch`
argument carrying `timeout.deadline_unix()`. Neither touched the other's
lines. Kept both.
seller_node/run.rs: main added two arguments to the job call; this branch
turned `AgentRunTimeout::JobDeadline` into a struct variant carrying the
absolute deadline. Kept both.
Main's new code then met this branch's changed signatures at six places, each
resolved to preserve main's behaviour exactly:
- `with_prepared_launch`'s `JobLaunch` gains main's `mcp_servers`.
- Two test `DockerPolicy` literals gain main's `mcp_tools`.
- The credential-bridge and held-tool `prepare_launch` calls pass `None`:
neither is a job, so neither has a deadline to stamp.
- Two live acceptance sites constructed `JobDeadline(420s)` as a tuple. They
stay `JobDeadline` — NOT `HarnessProbe` — because `cleanup_policy` reads
that variant as CaptureThenRemove, and a probe would silently downgrade it
to RemoveOnly. The absolute second is taken at construction, the one
instant where `now + remaining` is the deadline rather than a guess.
No behaviour of main's was dropped or rewritten.
…eft short The merge at a5fa204 brought main's new fields, but the r2 gate ran --lib only, so the integration-test target was never compiled against them and did not build: E0063 missing `held_tools` and `mcp_tools` in SandboxConfig (:2218) E0063 missing `mcp_servers` in JobLaunch (:3194) The 26-row runsc matrix lives in this target, so no row could run until it built. Empty is the correct value at both sites, not a placeholder: this matrix measures network containment only, declares no MCP or held tools, and the argv builder reads `mcp_servers` solely to decide whether a server needs a mount. Product code is untouched; the change is three fields in one test file. Verified on VPS 18.233.168.75 under runsc release-20260831.0 at this tree: build EXIT=0, test binary sha256 50fb5015477c1283e4036ba9f3d33cb29205c4de337458ea5f41fa61ead6b61a 26-row matrix ROWS=26 PASS=26 FAIL=0, permission-denied rows 0
CI was red at a5fa204 on three jobs — default features, money-path, and the shipped acp+wallet combo — with four instances of: error[E0425]: cannot find function `sweep_seat` in this scope at :4248 "prod-shape", :4300 "conflict", :4357 "misplaced", :4405 "per-role-ok" `fn sweep_seat` was defined under #[cfg(feature = "acp")] while those four call sites are plain #[test] fns with no gate, so the function vanished in any build without acp. With acp on it compiled; the --all-features gate is a SUPERSET and could not see the default build break. The helper is dropped from the gate rather than the four tests being gated: it is a pure string helper with no acp dependency, and those tests are item 1's D1 regression coverage, which must run in every configuration. The eight call sites from :4653 keep the gating they already had. Gates at this tree, both run locally: cargo test -p maxplayer-core --locked EXIT=0, 511 passed, 0 failed cargo test -p maxplayer-core --all-features --locked EXIT=0, 1768 passed, 0 failed, 46 ignored Before the fix, the default build failed to compile: EXIT=101, 4x E0425.
This was referenced Sep 17, 2026
added 2 commits
September 17, 2026 12:30
…t creates
Round 2/3 FAILING was the live test's own lifecycle, not the deadline path.
Test code only: no production cleanup redesign, no deadline rework, and item 1's
assertions and the caller proof are untouched.
F1 — ownership started too late. `docker network create` and its assert ran
before `create_dir_all` and before the `Self { .. }` value existed, so a panic in
that gap left a created network with no owner. The guard holding both names is
now constructed FIRST; every fallible step happens after it, where an unwind
reaches `Drop`.
F2 — the runner was never joined on a panic. It was a plain JoinHandle owned by
the test body with `join` after all the assertions; the mutation panic proves
that path is reached, so teardown could race a thread still creating containers.
The scope now owns the thread and a cancel channel, and cancels-and-joins it on
both success and unwind BEFORE removing anything. Drop also takes a second
listing pass, so a container that appeared during the first is still removed.
F3 — the name collapsed to the pid. `&seat[..24]` is 16 pid hex digits plus the
first 8 digits of `as_secs()`, which for a ~1.79e9 timestamp are all zero, so
nanoseconds and the sequence were discarded and pid reuse reused the name. The
name now carries its own token of low seconds, nanoseconds and sequence. The
comment that claimed the counter was mixed in is corrected rather than left to
mislead the next reader.
F4 — Drop could abort the process and hid its failures. The docker helper no
longer `.expect`s: a panic while unwinding aborts and destroys the failure the
test was reporting. Removal statuses are reported on stderr instead of being
discarded into `let _`.
Gates at this tree:
cargo test -p maxplayer-core --locked EXIT=0 511 passed, 0 failed
cargo test -p maxplayer-core --all-features --locked EXIT=0 1768 passed, 0 failed, 46 ignored
cargo test -p maxplayer-core --all-features --locked --no-run EXIT=0 15 executables, 0 errors
…absent listing failures
F1: a name is not ownership. LiveScope now records Held::{Proposed,Acquired,Uncertain}
per resource. The network is Acquired only after `docker network create` succeeds; the
workdir uses `create_dir` (not `create_dir_all`, which accepts a directory this invocation
did not create) so ownership is decided by this call. Drop removes ONLY Acquired resources
and reports Uncertain ones instead of deleting on a guess. Cleanup after "network created,
then mkdir failed" still runs.
F4: a failed observation is not absence. Teardown checks `listed.status.success()` before
parsing stdout, keeps exit status and stderr, and reports ownership as UNKNOWN rather than
breaking as if nothing were owned. Every eprintln! reachable from Drop or runner-panic
reporting is replaced by a fallible `writeln!(std::io::stderr(), ..)`, so a failed
diagnostic write can neither skip cleanup nor double-panic during an unwind.
Test code only: all hunks inside `mod tests`; zero production bytes changed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Job-local veth containment for gVisor jobs: the rendered policy is translated onto the interface the packets actually leave by, and applied in the launch path rather than beside it.
Why. Under
runscthe job's network stack is in userspace; the hostOUTPUTchain never sees its connections, so an iptables-only policy does not contain it.tc/floweron the job's own veth does, because every packet crosses that link whatever stack produced it.Head is
d6774e4e, on baseb9ee77f1(27 commits) — which is the currentmainof the base repo. The branch contains it, is 0 commits behind, andgit merge-treereports a clean merge. An earlier revision of this description said main had moved tob45f8651and that this branch was not rebased onto it; that was the fork holding the head branch, not the PR's base repo, and the statement was wrong.Renewed corrective round 1 — head
c57e6c2fFive residual production-path defects, each closed with a test whose negative control was observed RED before it was observed green, and whose file was then restored byte-identical.
establish's sidecar creates were unfenced, so cleanup's settle-wait could pass at count zero and issue removes for a container that did not exist yet; the container then arrived with nobody holding it, pinning the namespace. Control: cleanup returned in 302.6ms mid-create...and no leading/could still be a symlink to another run's green log.corroboratenow resolves both sides, requires containment, and reads the path it just checked.establishhad no test at all — the fault behind the proxy and cancellation findings, since a fixture can agree with a bug the production path does not survive. Two gates now drive the real function: one proves the addressestablishmeasures is the address its rendered plan pinholes; the other cancels mid-create and proves cleanup outlasts the create and removes the holder it made. They run through a spawn-site seam, so every rendered argv the plan tests assert is the argv production runs. The cancellation gate cancels on a marker written as the create starts, not on a stopwatch — a fixed deadline raced the probe and measured nothing.The seam is deliberately named outside the
MAXPLAYER_prefix, which is reserved for config and refuses unknown variables fail-closed; the first version was caught by the full suite refusing to start 14 unrelated tests.Correction at
d6774e4e: the test seam was a production boundaryAt
c57e6c2fthe docker client this crate spawns was selectable by an environment variable, in shipped builds. Anyone able to set a variable on the process could have redirected every containment command — create, inspect, remove — at a binary of their choosing: a security regression acquired through testing, which is the worst way to acquire one.Production selection is now the constant
dockerunder#[cfg(not(test))]with no environment read on the path at all; the override is a process-local static compiled only under#[cfg(test)]and restored on drop. Both gates that run through the seam were re-observed RED against the new mechanism before being accepted green — plan rendered from a literal instead of the measured address, and fence released before the create (cancellation returned in 173.3ms) — with the source restored byte-identical after each.What the two
establishgates do and do not proveThey drive the real
establish— its real ordering of probe, fence, create, render, apply, read back, and its real cancellation and cleanup — and they catch real defects in it, as the controls above show. That is offline unit credit for ordering, cancellation and plan-address propagation.They are not evidence of runtime proxy connectivity, of a real launch, or of cancellation against a real daemon: a stand-in client answers as it is scripted to and cannot prove a packet reached a proxy. Claims about actual launch/proxy/cancellation behaviour rest on the live matrix below, not on these.
Gates at this head (lima
gvisor-repro, 6.8.0-139-generic aarch64, runsc-release-20260817.0, docker-29.1.3): live matrix 20 passed / 0 failed (86.97s) · 995 DNS 9/0 (44.77s, functional linespayload runtime=runsc, resolvers172.29.7.53,fd00:d57::53,2001:db8:bb::53) · numeric egress 1/0 (17.54s) · offline suite 1646 passed / 0 failed / 13 ignored · acceptance ACCEPTED (offline), exit 0.Both test binaries were rebuilt at this head and their content hashes differ from the previous head's — matrix
e8bc681b…(wasb8136713…), lib0e7063b3…(was1e001c03…) — which is what rules out a stale artifact. The sidecar imagesha256:43d80f022dce…istc-capable (/sbin/tc,iproute2-v7.0.0). Running the matrix without--ignoredexits 0 printing1 passed; 20 ignored: a green that measures nothing.Sidecar image. The
tc-capable image is built fromdocker/maxplayer-netfilter/on this branch and is not published by this lane; a deploying owner builds or loads it under its own authority. Identity of the image these gates ran against:sha256:43d80f022dce98ccceb3d299ce92e3859279cb17d93e990927e74644c2fd2d6c, arm64/linux, Alpine,/sbin/tciproute2-v7.0.0. It has no registry digest because it was never pushed. Productionprepare_launchhardcodes…:v0.5.9, so that tag was applied to the local image inside the VM only, used, and then removed (0such tags remain).What is here
sandbox_iface— renders aNetPolicyinto a per-jobIfacePlan(clsact egress,flowerdrops, both families, ordered), with an argv builder for the sidecar and a readback verifier.sandbox_netns::establish— measures the job's link by unprivileged probe, derives the plan, applies it through the netfilter sidecar, cross-checks the applied count and then verifies by independent readback. Fail-closed: any?on the path destroys the namespace via the holder guard, so a job never starts with partial filters.docker/maxplayer-netfilter—tc/iproute2added, plusapply-iface.#[ignore]d, replayed on a real gVisor host.Gates that ran
Offline, this head:
Live, in a disposable lima VM (Docker 29.1.3,
runscregistered), with every binary and the sidecar image built from this head before the run:sandbox_netns_live)sandbox_dns_live)sandbox_egress_live)The branch's own acceptance gate,
scripts/sandbox-acceptance.sh, run against the saved record for this head: ACCEPTED (offline), exit 0, with 1676 passed / 0 failed in the offline suite it runs first. That gate now opens each log the record names and requires the named test to appear in it with anokresult.a4349e53b1c7c5ab9ad7e3c77ddaade176ba3697122a5b544430c4c4d30086d4930cd590f74fcd34a3bd721218dc71e04a206993554edd7bb436ad6053acc24643d80f022dce…, built from this branch'sdocker/maxplayer-netfilter/It can still fail
A green matrix is worth only the failure it can produce. The rule that admits the DNS pinholes was re-pointed at a resolver the gate does not configure, leaving production behaviour untouched, and the leg went red:
The matrix also carries a counter-control (
integrated.denied.v6-link-local) that fails if the ND exception is "fixed" by widening it.Relationship to the DNS lanes — corrected
An earlier revision of this description said this branch was "independent of the DNS lanes #991 / #995 — different layer, no shared files." That is no longer true and is withdrawn. #995 has merged, and this branch is now rebased across it and integrates with it:
IfacePlanderives frompolicy.rules(), so sandbox DNS: a resolver contained jobs can reach, with the live gVisor gates that prove it #995's DNS pinholes mirror onto the veth automatically — no separate translation.ReadbackRule/typedPredicate) rather than restored alongside it, because sandbox DNS: a resolver contained jobs can reach, with the live gVisor gates that prove it #995'sas_exceptionadmits only udp/tcp rules carrying-dand--dport; ND needs its own shape, which rejects negated predicates.passrule on the veth must be either the configured proxy range, or port 53 to the resolver the configuration named. A port-53 rule to any other destination still fails, and a missing resolver pinhole now fails too.Rule shape is not working DNS
The containment matrix configures
192.0.2.53— TEST-NET-1, routed nowhere, answering nothing. It exists to make the rendered rule set deterministic. A rendered--dport 53 -j ACCEPTand an enforced one produce the same green there, so nothing in that file should be cited as evidence that a contained job can resolve a name.f8b075efwrites that limit into the doc comments (comments only, no assertion changed).What establishes DNS is 995's own live suite, run on this head against a real dnsmasq fixture that answers — in-crate modules, so the integration-test binary never contained them:
resolv.confThese are gVisor results: the harness cross-checks docker's view of the runtime against the payload's own kernel banner and refuses to count a
runcrun.One note in the interest of not overclaiming: 995 marks
known_open_preexisting_defect_other_private_and_non53_must_be_deniedas KNOWN OPEN, failing underrunsc. It passes on this head. That is consistent with veth filters enforcing what an iptables-only policy cannot underrunsc, but I did not run the base baseline needed to attribute it, so it is recorded as observed and unattributed, and 995's ignore attribute is left exactly as written.Still independent of #991 (host plane); no host-plane work here.
Limits
runscbuild.sandbox_netns.rsbuilds it asconcat!("ghcr.io/makeprisms/maxplayer-netfilter:v", env!("CARGO_PKG_VERSION"))— so the v0.5.9 release cut moved it automatically.main'sdocker/maxplayer-netfilter/Dockerfilestill installs noiproute2, so the published v0.5.9 image has notc: until an image carrying this branch's Dockerfile is built and rolled, merging this protects nothing in production. No image was published by this branch; the gate ran against a VM-local tag that shadowed nothing (no image existed under it beforehand).75b17b5a…bit-for-bit, which is what makes the run attributable to committed source rather than a mutated tree. There is still no signed commit-to-artifact chain./32or/128rather than normalizing equivalent spellings;known_open_preexisting_defect_other_private_and_non53_must_be_deniedpasses here under provenrunscbut attribution to this branch is unestablished (it needs a baseline build ofb9ee77f1, and the VM had ~1.2G free). The VM-host before/during/after egress measurement is now closed — see below.seller_node::lock::tests::second_acquire_fails_closed_while_first_is_heldfailed once under a parallel full-suite run and passed in isolation and on one full re-run; provenance not established. An earlier revision called it a baseline flake — that attribution is withdrawn: one failure and two passes cannot separate a pre-existing flake from one this branch introduced. Repetitions to settle it were not authorized, so nothing is claimed about its cause.No merge, no tag, no main push, no deployment, no image publication.
Round-2 review fixes (
f8b075ef→3a43ad04→5d7837df→df93a558)Three commits, each a code fix with a control that fails without it.
1. A sidecar could lose custody of a client that was still alive.
run_boundedreported only the command'sResult, so a client that was killed or timed out looked like one that exited, andrun_sidecarderegistered custody on both — a live container could stop being a cleanup target.run_bounded_trackednow returns(Result, bool child_exited)and custody ends only on a reaped client.Worth stating: my first control for this stayed green when I reverted the fix — it tested the flag, not the decision that consumes it.
5d7837dfaddsrun_sidecar_with_deadline(deadline parameterised so the timeout path is reachable offline) and asserts at the decision site; reverting the fix now fails it atsandbox_netns.rs:1799.2. Malformed readback lines were silently tolerated. The parser accepted repeated
!flags, a dangling trailing!, and a bare value before any flag — so a rule whose shape it did not understand could read back as something benign.ReadbackRulenow carriesmalformed,value()returnsNonefor such a line, andverify_readbackrefuses the whole readback. Malformed lines are kept, not dropped: dropping them would shrink the rule count and hide an unexpected rule, which is the failure this is meant to catch. Four controls; reverting the parser change turns three of them red.3. A saved row named a log, and nothing checked the log. The record named a case id, an outcome and a log file, and the validator only checked that the log field was nonempty text — it never opened it.
SavedCasenow carries atestfield, andsandbox_evidence::corroborateopens each named log and requirestest <name> ... okto appear in it (refusing absolute or..paths). It is wired into the acceptance gate.This does not establish that a test's assertions are the right ones for its case id; only reading the test body settles that. Two mappings are named rather than buried:
integrated.denied.neighbour-portandintegrated.exception.proxy-pinholeare both asserted by a test that installs the full plan through the sidecar against the canary rather than through a production launch, so theintegrated.prefix overstates the path.What fix 3 immediately caught: a false row
host.unaffected.during-cleanupis required byREQUIRED_CASES, and that id appeared in exactly one place in the repo — the requirement list. No test asserted it, and the round-2 record scored itconnectedanyway, on the strength of the sibling cleanup test running nearby. That test probes only from inside containers, and a container-side probe cannot observe the host's own egress. It was a claim, not a measurement.The leg is now authored:
the_hosts_own_egress_is_unaffected_before_during_and_after_a_jobs_cleanupconnects withTcpStream::connect_timeoutfrom the test process on the VM host, in no namespace, before / during / after a job's cleanup — opening with a negative control on a port nobody listens on, so an instrument that cannot say "no" fails instead of passing three times. That is the 20th matrix leg.DNS through the
tc-capable sidecarThe round-2 ask was functional DNS through the integrated v0.5.9 path, not rule shape:
sha256:43d80f022dce…and carries/sbin/tc(iproute2-v7.0.0);sandbox_netns.rsreads no env var for the sidecar image, andseller_exec.rspasses the hardcodedDEFAULT_NETFILTER_IMAGEintoestablish;payload runtime=runscand resolvers that answer (172.29.7.53,fd00:d57::53,2001:db8:bb::53) — not TEST-NET-1;v0.5.9tag removed, the functional DNS test fails at containment establishment with…maxplayer-netfilter:v0.5.9: not found. The green run could not have gone through any other image. Tag restored, same test green, then the local tag removed again — it was never pushed.Round-3 review fixes (
df93a558→13745fd8)One commit, three source fixes, each with a control that was reverted, run, and observed RED before being restored — because the defect that killed rounds 1 and 2 was a test that passes without reproducing the failure it names.
wait_until_settledcall fromDropcleanup_does_not_remove_ahead_of_a_create_that_is_still_in_flightErrcustodya_reaped_client_whose_container_is_still_there_keeps_custodymalformed1. A delayed create was never fenced against cleanup. Adoption supplies a name to remove; it says nothing about when the container appears. The create runs on a blocking thread and cancelling the future above it does not stop that thread, so cleanup ran immediately, asked docker to remove a container that did not exist yet, was told
No such container— which cleanup correctly treats as benign — and returned satisfied. The create then landed and left a container nobody held. No ordering of removes fixes that, so aCreationFencemoves the removes to the far side of the create settling; the ticket lives inside thespawn_blockingclosure and never in the future, so a cancellation cannot release it early. The test asserts thatDrophad not returned before the create settled, not that the helper works.2. Custody ended on a reaped client rather than a gone container.
--rmis a request to the daemon, not a receipt from it: a nonzero exit, a deadline kill, and a stdin write that failed before the wait all return from the same call, and none of them means the container is gone. Custody now ends only when the client was reaped and absence is confirmed; a confirmer that cannot answer keeps the name a cleanup target, at the cost of onedocker rmthat saysNo such container, which cleanup already treats as success. This also removed a daemon dependency from an offline test — the previous custody test reached a livedocker inspectand passed only because a daemon happened to answer.3. The ND readback tolerated tokens it could not consume. A canonical ND ACCEPT carrying a leading unflagged token after
OUTPUT, or a dangling final!, verified with the same accepted predicates — the unconsumed tokens changed nothing the checker went on to read. The prior tests asserted this at the parser; the new one asserts it at the decision site, runningverify_readbackover the real live-measured v6 readback mutated only in those two ways, with the unmutated readback as a positive control. Malformed lines are kept rather than dropped so the rule count stays honest.Live gates re-run at this head against a real gVisor daemon: containment matrix 20/0 (87.65s), 995's DNS suite 9/0 (44.42s, functional — three resolvers answered,
payload runtime=runsc), egress 1/0. Both test binaries were rebuilt and their content hashes differ from the previous head's (matrixa4349e53…→470c733f…, lib930cd590…→73705c4b…), which is what rules out a stale artifact.Worth writing down: running the matrix binary without
--ignoredexits 0 and prints1 passed; 20 ignored— a green that measures nothing, since every live case is#[ignore]d. That false green was produced during this round and discarded; the recorded run is the one reporting20 passed … 1 filtered out.cargo fmt --checkis not applied here: it reports diffs in 80 files, nearly all untouched by this branch, which is toolchain drift rather than this branch's doing. Reformatting the crate would be exactly the out-of-scope diff earlier rounds were marked down for.Still not claimed, unchanged: attribution of
known_open_preexisting_defect_other_private_and_non53_must_be_deniedto this branch (needs a baseline build, out of scope this round); that a saved row's assertions fit its case id (that mapping is hand-made); andintegrated.denied.neighbour-port/integrated.exception.proxy-pinholeare asserted by a canary, not a production launch.Renewed revise, round 2 of 3 — head
e725c38fThree production-path defects the round-2 verdict named, all closed.
1. Production could be pointed at another docker executable.
MX_SANDBOX_DOCKER_BINanddocker_program()are removed repo-wide — a grep for either name returns nothing.DockerCli::system()holds the constant"docker", and the client is now threaded explicitly throughrun_docker,run_docker_fenced,run_bounded*,container_is_absent,force_remove,reapable_holders_live,reap_orphansandNetnsHolder::adopt{,_bounded}.sandbox_netns.rsperforms zeroenv::varreads; the stand-in constructor is#[cfg(test)], so it does not exist in a production build.2. The test seam was a process-global.
DOCKER_INJECT_LOCK,INJECTED_DOCKER,injected_docker_program()and the RAII guard are deleted. Tests callestablish_with(client, bounds, ..);establish()is a thin wrapper passingDockerCli::system()andFenceBounds::production(). An argument rather than a global was the point: under the old seam, cleanup issued fromDropcould read another test's client. The three establish gates now run together in parallel with no serialising lock (2.19s).3. Cleanup raced the create it had to outlast.
Dropno longer removes anyway when the fence expires — it hands custody toHolderCleanup, which sweeps only after the create has settled and then polls until the daemon confirms absence (20ms→500ms backoff), on threadmx-holder-cleanupwith an inline fallback.FenceBoundsmakes the delayed path measurable in under a second while production keeps itsDOCKER_DEADLINE-anchored waits. The gate asserts ordering —rmaftercreate-endin the stand-in's event log — not elapsed time, so a build that merely slept longer cannot pass it.Gates at this head. Offline suite 1681 passed / 0 failed / 36 ignored, exit 0. Live containment matrix in the approved VM under
runscagainst a real daemon: 22 passed / 0 failed (90.84s), up from 20 for the two gates added here — a contained job actually connecting to a host proxy through the pinhole and being refused on a port outside it, and a cancelled establish leaving no holder behind.Both live gates were observed RED first, and the controls are production-side. With the pinhole rendered for a different address, the job could not reach the real proxy at
172.17.0.1:49221. With the fence ticket held by the future, a cancelled establish left a real container running on the real daemon — the exact orphan the fix exists to prevent, not a simulated one. Source was restored byte-identical after each control (sha256 79113d63…), and the leaked debris was removed and the VM re-verified clean.A defect found on the way, reported rather than fixed. A single-port proxy range is a supported configuration —
PortRange::newacceptsstart == endand the parser documents"49200"— but it rendersdst_port N-N, which the tc flower classifier rejects outright (max value should be greater than min value), leaving the namespace partially filtered and the job refused rather than contained. Every existing tc-path test uses a multi-port range, so nothing had exercised it. It is outside the three blockers this round was sent to close, so the new gate uses the multi-port shape production ships (49220-49229); pinning it to one port would have measured that bug instead of the proxy leg.Unchanged from earlier rounds:
cargo fmt --checkis still not applied, and the attribution ofknown_open_preexisting_defect_other_private_and_non53_must_be_deniedis still not claimed.Renewed revise, round 3 of 3 (final) — head
f1a655c5The three residuals the renewed round-2 verdict left open, each closed with a gate observed RED before it was allowed to pass. All credited repairs from earlier rounds are preserved.
1. Outer cleanup released too early. It could release on a timeout, or on a holder that was momentarily absent while creation had not settled.
confirm_absentbecomesconfirm_all_absent: every owned joiner is confirmed before the holder, and the names it could not confirm come back instead of a bool, so a partial sweep can no longer read as a clean one. A deadline that expires while a create is still in flight now reports a KNOWN leak rather than releasing.run_bounded_blockingstarts its clock before the spawn and writes stdin on its own thread, joining after reap — so the whole create flow is bounded, stdin and output and queue included. RED first: a blocked stdin write ran 30.48s against a 300ms deadline.2. The cancellation oracle proved nothing. It could pass on an empty output or a first absent list. It now credits cancellation only when cancellation actually hit a create that was still running:
cancelled_in_flight > 0is asserted, so a run where every establish finished first fails instead of quietly scoring cleanup-after-success. Absence is credited only from adocker psthat really answered, covers the owned holder and its sidecars by name prefix, and must hold for 5 consecutive answers within 30s, with a failed query resetting the streak.Reachanddocker_exitseparate a refused connection from a tool that could not run, so a DENIED is credited only when the probe actually ran — the tool-failure control the verdict asked for. The real host TCP route credit from round 2 is retained unchanged.3. A supported single-port pinhole could not be installed at all. This is the defect round 2 reported rather than fixed. iptables renders
49221:49221; the tc translation turned that intodst_port 49221-49221; the flower classifier rejects it (Illegal "dst_port" / max value should be greater than min value), so the filters never installed and establishment refused the launch. Fail-closed, but a supported configuration that cannot start is still broken. Equal endpoints now collapse to the bare port, with positive and negative offline coverage and a live gate that a realtchas to accept. No scope expansion: multi-port rendering is unchanged and separately gated.Gates at this head. Offline suite 1687 passed / 0 failed / 36 ignored, exit 0. Live containment matrix in the approved VM under
runscagainst a real daemon: 23 passed / 0 failed (101.39s).cargo check --all-targetsexit 0. The acceptance script run against the saved record for this head: ACCEPTED, exit 0 — leg 0 head containment, leg 1 offline suite, leg 2 saved matrix.The evidence contract that failed last round is satisfied. Round 2's record exited on a missing
source_head; this one carries all four required headers, and its matrix log is the complete run — 23 named... oklines in a 29-line file. The round-2 log had been truncated bytail -20, which cuta_job_prepared_and_launched_by_production_is_contained_on_its_veth, the test three rows depend on; nothing here is tailed or filtered. The validator was not weakened —REQUIRED_CASESandREQUIRED_HEADERSare untouched — and no earlier DNS run was relabelled.RED controls, both production-side. Reverting the equal-endpoint collapse made the live singleton gate fail on a real kernel with
Illegal "dst_port"and a refused establish. Delaying cancellation so every establish won made the oracle refuse to pass. Both sources were restored byte-identical (diffplus sha256) before the final binary was built.Still not claimed, named rather than left to be discovered.
cargo fmt --checkis still not applied across the branch (~80 files, toolchain drift). Attribution ofknown_open_preexisting_defect_other_private_and_non53_must_be_deniedstill needs a baseline build ofb9ee77f1….seller_node::run::tests::a_settled_offer_won_by_another_seat_is_not_claimedis a pre-existingAddrInUseflake under parallel load (3/3 isolated passes, and it passed in the clean final run); it is unrelated to the three changed files.