seller delivery: bound the lifetime of the push WORK, not the patience of its caller - #1006
Merged
jbojcic1 merged 63 commits intoSep 17, 2026
Merged
Conversation
added 3 commits
September 14, 2026 05:49
…s caller F3: the delivery turn was tied to the future that started the push. A revoked push still occupied the turn while it waited for a blocking slot; queued work, the local config rewrite, pack generation and buffering honoured no cancellation and no deadline; and the async supervisor holding the guard could be cancelled while its blocking thread survived. - new delivery_turn module: the turn carries the exclusion token and an ABSOLUTE deadline. begin/end race on one atomic, so the turn is handed back only when the work actually stopped, or when it provably never started. Never because the caller timed out. - the turn is moved onto the blocking thread that does the operation and dropped there, so a dead supervisor or a shut-down runtime cannot free it early. - deadline/cancellation checks at every phase the turn is held across: queue admission, config rewrite, pack negotiation, every buffered pack chunk, before the signer queue is joined, and before each wire request. - DELIVERY_DRAIN_BOUND = 270s (DELIVERY_PUSH_TIMEOUT 150s + one 120s HTTP leg), const-asserted, documented phase by phase. Preserved unchanged: post-mint/pre-submission authority check, absolute deadline check, exact OID/ref binding, fresh per-leg authentication, no hidden replay, redirects denied, relay policy and TTL.
…ith it Two separate defects, one rule. Releasing on the supervisor alone was the original bug: the caller stops waiting and the next delivery opens a receive-pack against the same remote while the abandoned push is still on the wire. Releasing on the work alone is its mirror, and the first shape of this fix had it: the blocking operation returned and the turn was gone while the delivery arm that supervises it was still inside the section the turn exists to exclude. `the_delivery_that_waited_for_the_lock_signs_after_the_wait` caught it - delivery 2 entered before delivery 1 left. So ownership is handed back only when the work has actually stopped (or provably never started) AND the supervising side is done. Each side publishes its half with SeqCst and asks; whichever is second releases. Also: the phase gate the transport asks at every boundary now asks the delivery's own authority first and the turn's lifetime second, composed once in seller_git instead of two gates plumbed everywhere - so a push refused before it mints is refused by the same name, and recorded in the same place, as one refused after the mint.
Three tests the verdict asked for, none of them simulated: - revoked WHILE its token is being signed: the supervisor dies with the mint parked inside the signer round trip, the delivery's own authority left alive on purpose so only the turn can stop it. Nothing reaches the relay and the signer queue is never joined again. This one found a real gap - asking the lifetime only BEFORE the mint answers a question the queue wait makes stale, so send() now asks again after the mint, next to the authority re-check. - revoked before dispatch: no blocking slot ever came free, so the workdir's push config is byte-identical afterwards, no token is minted, nothing dials. - a caller timeout across a LIVE upload: a real GET and a real POST, the POST held open at the relay while the arm that started it times out and returns. A second real delivery is launched into that window, proved PENDING on acquisition (try_lock refused, no Enter in the journal, no third request), and completes for real only after the abandoned upload stops. Peak concurrency at the remote stays 1. Fixture: hold_request_number, the existing appointment applied to a chosen leg rather than the first one - a held POST is the only instant where the bytes are genuinely unrecallable.
|
Someone is attempting to deploy a commit to the MakePrisms Team on Vercel. A member of the Team first needs to authorize it. |
… cannot be The drain bound asserted 150+120 over phases it did not reach. libgit2 walks the object graph, inserts into the packbuilder and searches for deltas BEFORE the first pack byte, and an HTTP timeout cannot bound work that happens before HTTP. Traversal and insert are now interruptible: libgit2 checks what the pack progress callback returns on that path (pack-objects.c:256-270), at half-millisecond granularity, so the delivery's work gate is asked there too. git2 0.19 discards a Rust closure's answer in that trampoline and hard-codes 0, so the refusal travels as a typed panic raised inside git2's own catch (no unwind crosses a C frame) and is converted straight back into a TransportError at the push call. The delta search is NOT interruptible: pack-objects.c:979 and :1356 throw the callback's answer away, and there is no other hook in that loop. That span is now named in the bound's own documentation with its evidence, measured from its true start, and reported with the number when it outlasts its budget -- never asserted away. A hard stop there needs an executor that can be killed, which is an architectural change and is not smuggled in behind a constant. Proof enters the dangerous state on purpose: a 16MB delivery whose local pack phase is HELD past its work deadline, with the remote provably at one request (advertisement sent, nothing uploaded) at the moment it is held. The seat stays taken while it is held, the boundary ends it, nothing is ever uploaded, and the next delivery gets the seat only after the held thread actually returns.
Contributor
Author
|
Correction pushed — new head Reading the pinned dependency source (
A second blocker sat on the first: The bound, stated honestly. A hard bound on that span needs a killable executor — local phase in a child process, deadline enforced by a signal, turn released when the child is reaped (bound = deadline + one leg + reap). That is an architectural change; it is written down rather than smuggled in behind a constant. The new test enters the dangerous state on purpose. Gate at Negative controls: 6/6 detected, each compiled, each failing on an assertion, files restored and verified by SHA-256 — now including pack-phase hook deleted → the held-phase test goes red, and delta overrun reported silently → the reporting test goes red. One footnote worth stating: the first run of the extended control driver reported the four older controls as "not detected". They had not regressed — the driver's per-control |
added 24 commits
September 14, 2026 08:17
Reporting a breach is not stopping the work. libgit2's delta search discards the only cancellation answer it is offered (pack-objects.c:979), so in-process the span can be measured and never ended -- and a delivery that entered it owned this seat's turn until libgit2 chose to give it back. The one thing on a POSIX host that ends work its author refuses to end is the kernel. This adds the executor that uses it: the blocking local phases run in a child process, the deadline is enforced by SIGKILL to that child's process GROUP, and the turn is released only after the kernel has confirmed the exit and cleanup has run. A kill that is merely issued releases nothing. Custody is unchanged, which is what makes the child safe. The push path already took a MINTER rather than a token, so the child gets no key and no token up front: it asks over the pipe, the parent runs the same closure -- same destination binding, same authority check, same deadline -- and returns one scoped token whose life is its round trip. The parent's deadline therefore binds the child before any kill lands: past it, no header, so no authenticated leg can even begin. The kill ends the work; the refusal ends the authority. Feasibility first, and settled by gates rather than by assertion. The shipped binary hosts the child entrypoint (F1) in the internal-subcommand namespace it already reserves. The pipe protocol carries a request and returns an outcome over real pipes to that real binary (F2), with nothing on argv and an environment cleared to a named allowlist -- proven from inside the child, which reports what it actually received. A child that ignores SIGTERM and never yields is ended anyway and its exit confirmed (F3), the kill reaches a descendant that would otherwise outlive the delivery, and dropping the handle leaves nothing behind. The bound is stated as the conditional thing it is: 270s + a 5s reap bound, under SIGKILL, process-group and parent-scheduled assumptions that are written down with the cases that break them -- uninterruptible kernel sleep above all. Where they break it fails CLOSED: an unreapable child keeps the seat rather than handing it on, because an unreapable child is not evidence that it stopped.
…iling The round-1 verdict names the trap this executor could still have fallen into: issuing a kill is not proof of an exit, and a design that assumes a silent child is a dead child is the same defect wearing new clothes. Two places in the executor still made that assumption. Drop swallowed a failed reap. Every reporting path already returned Unreaped to a caller that must keep its seat closed, but a drop on a panic or an early return has nowhere to return anything to, so the one path that most needs to fail closed was the one that failed silent. A failed reap now increments a process-wide UNCONFIRMED_CHILDREN that is never decremented -- an unconfirmed child is a permanent fact about this process, not a transient one -- and a seat consults it before treating its delivery lane as free. The release rule is now a named function rather than a shape inferred from whichever paths happen to call it. exclusion_after_reap returns Release ONLY on a reap the kernel completed; a stalled reap, an unreadable status, any error at all returns Retain. Unknown exit is treated as still running. That costs liveness on the affected seat, deliberately, and it is named as lost liveness rather than described as bounded recovery. read_frame had no ceiling. A peer writing without bound is memory this process never bounded, and an unbounded parent-side buffer is a phase outside the drain bound -- moving the pack work into a child while leaving the pipe unbounded would not have closed anything. Frames are capped at 1 MiB, which is three orders of magnitude above the longest field this protocol carries, and a frame past it is a protocol violation rather than an allocation. Both rules are gated: the release rule against all three outcomes, the cap against an oversized frame.
The executor existed; nothing called it. This is the wiring, and it is the whole point of the change: the seller node's delivery push no longer runs the local phase on a thread that cannot be interrupted. - seller_git::neutralize_then_push_in_child_off_runtime drives one delivery push in a child process while holding this delivery's turn, and returns only when that child has exited and been reaped. - The turn is released on a CONFIRMED EXIT and on nothing else. An unreaped child retains it for the life of the process rather than hand the seat to a second delivery while the first may still be packing. - The per-request minter stays in the parent. The authority is asked again AFTER the mint and BEFORE the header crosses the pipe, so a token for a leg this delivery no longer owns never reaches the child. - run.rs resolves the child by current_exe, never PATH, and FAILS the delivery if it cannot be resolved rather than fall back to a push that cannot be stopped.
Seven gates against `neutralize_then_push_in_child_off_runtime` — the call the delivery arm actually makes — not against the executor in isolation. - A local phase that ignores SIGTERM and never speaks again is ended AT ITS DEADLINE, its exit is CONFIRMED (the fixture records its own pid; the test asserts the pid is gone), and the turn comes back only then. The hold is measured: at least the budget, and less than budget + REAP_BOUND. - A push that finishes returns its oid and hands the turn back, so the gate above cannot be satisfied by an executor that kills everything. - A delivery revoked before dispatch spawns NO child at all. - The child receives the header the PARENT minted, for the destination it named; authority that ends during the mint keeps that token on the parent's side of the pipe; an unauthenticated remote cannot obtain one by asking, and the child that asks is killed and confirmed gone. - The release rule itself: Unreaped RETAINS the turn, every confirmed-exit outcome releases it. The test says in the open why an unreapable child cannot be manufactured in user space, and that the rule is therefore gated at the single decision the production path consults.
…ivery wait Round 1 was failed for claiming a stop bound nobody had watched a second delivery wait out, and for calling a parked test minter a signer interleaving. - THE MINT IS BOUNDED, by the delivery's own deadline, and runs OFF the drive thread to make that true. A signer whose queue is full or whose reply is held used to block the parent inside `mint()`, and a parent blocked in the signer is a parent that never issues the kill. The abandoned thread holds no lock of ours. Gated by a minter that never returns at all: the deadline still lands, the child is killed, its exit confirmed, the turn comes back. - THE CLEANUP IS BOUNDED. Joining the pump thread is unbounded: it sits in a read that ends at EOF, and EOF needs the LAST holder of the write end to close it — a process that escaped the group we killed still holds it, and then the parent never returns at all. We wait for the channel to disconnect instead, under the same REAP_BOUND as the reap, and losing that race FAILS CLOSED: `CleanupUnbounded` retains the turn, because a reaped child whose pipe outlived it does not prove the local phase is over. - A SECOND DELIVERY IS OBSERVED PENDING, through the seat's own serializer and the seat's own lock, while the first one's local phase is wedged: sampled repeatedly, not probed once with try_lock, and the handover instant is compared against the first delivery's return rather than assumed to follow it. The matching control does the same behind a first delivery that SUCCEEDS, so the gate cannot be satisfied by a seat that only ever hands over after a kill. Tokens do cross this pipe — the parent mints a scoped header and writes it to the child, which is the point of the round trip. The signing key does not: it stays in the actor the parent calls.
The bound is a claim about two processes and a clock, so the gate that makes it prints what it measured — budget, actual hold, overrun, handover to the delivery that was waiting, and how many times that delivery was observed pending — under `-- --nocapture`. A reviewer reproduces the measurement, not the sentence.
…we saw The parent's own writes were outside the deadline. `write_frame` serialized and then blocked in `write_all` on the drive thread — the only thread that can issue the kill — so a child that simply did not read its stdin parked the supervisor behind ordinary pipe backpressure, with the kill unreachable. Writes now happen on their own thread and the drive waits for the acknowledgement under the same absolute deadline as every other phase; a write that does not complete in time is the same overrun, stopped the same way. The initial push request is written inside that loop rather than before its first check. The frame cap capped one direction. A cap on what this protocol will READ is not a cap on the protocol, so frames are encoded and refused at the writer. The parent's inbound queue was unbounded: a child writing small frames while the supervisor was minting cost this process unbounded memory. It is a synchronous queue now, so the party that stalls is the one that can be killed. The child's stderr was piped and never drained, which turned the transport's own overrun diagnostic into a stalled child and printed nothing; it is relayed, capped. `waitpid` can fail, and that was reported as a protocol fault — which the release rule releases on. An unknown exit wearing a releasable name is a seat handed to a second delivery while the first may still be packing. It has its own outcome now, it retains, and the rule is re-checked once at the single point every outcome leaves the supervisor: if this process has not seen the child exit, nothing releasable leaves there. A panic unwound straight through `RunningWork` and freed the seat; custody is a guard whose default is retention and whose release is explicit. The count of children this process could not confirm dead now has a consumer on the production dispatch — it refuses the next delivery — rather than a comment saying a seat consults it. The child supplied `authority: None` and `lifetime: None` and never read the budget it was sent, so the transport's pre-wire gates did nothing in the one process that actually transmits. The child derives its own deadline from that budget, and asks the parent across the pipe, immediately before it transmits, whether this delivery still owns its turn. No answer is not permission.
…the peer The production-child gates called the real parent wrapper and handed it `/bin/sh` fixtures, fake minters and a recording token. They are honest about the parent's deadline and reap arithmetic and they say nothing about the artifact that actually delivers, so the claim "a delivery runs in a child" rested on a shell script wearing the child's name. This drives `CARGO_BIN_EXE_maxplayer` — the artifact release-platforms.json builds — through the real wrapper, against the crate's smart-HTTP fixture over TLS, and reads the pushed object back out of the bare repo afterwards. The credential is minted in the parent and crosses the pipe when the child asks for it; the private key stays on the parent's side, which is the reason the re-exec exists. Verification is ON for that push, which took a fixture change. Every other fixture test in the workspace reaches the server with `GIT_SSL_NO_VERIFY=1`, and a child cannot be told that: the variable is deliberately absent from CHILD_ENV_ALLOWLIST. `SSL_CERT_FILE` is on that allowlist, for a host whose trust store is not the default, so the fixture now hands out the certificate it actually serves and the child is given it the way a musl image would be given one. Its own control is a second test binary that differs in one line — no trust anchor — and requires the push to fail with the remote ref untouched and no request ever reaching the handler; a separate binary because the transport's client is a process-wide OnceLock whose roots are fixed the first time it is built. Two more controls, because a gate that cannot go red is decoration. The wrapper logs `path=inprocess` on success — the push IS in-process, inside the child, and that line reaches the parent's console only because the child's stderr is relayed now — so a parent quietly pushing by itself would look identical from outside. Replacing the child with an executable that runs and exits mute must therefore deliver nothing, and does. And the pack upload is parked ON THE WIRE by the fixture at `POST /git-receive-pack`, released only after the outcome is in hand, so the finite stop cannot be attributed to the remote letting go: the delivery ends after its budget, reports killed-and-confirmed, and the remote ref never moves. Measured on darwin-arm64 only. Nothing here is evidence about the two Linux targets.
…re they share Found by running the workspace suite rather than the file: the held-wire delivery ended in 30ms instead of waiting out its 4s budget. `SSL_CERT_FILE` is the child's only trust input and there is one per PROCESS, so three tests each minting their own fixture certificate raced, and the losing child spent its life failing a handshake against a neighbour's CA — never reaching the held pack upload the test was about. The gate was green under `--test-threads 1` and red in the suite, which is the worst shape a gate can have. Serialized on an explicit lock, with the reason written down: the constraint is not a test artifact. A seller node has one environment too.
…trand a childless turn The remaining round-3 findings, in the order they bite: * A delivery refused between begin() and spawn kept the seat forever: the custody guard forgot its turn unconditionally. It is now armed immediately before the child is spawned, and an unarmed drop RELEASES. * Nothing acted on a revocation until the deadline. The parent now waits in CANCELLATION_POLL slices and re-asks its authority on each tick, killing and reaping the child and returning Revoked - a confirmed exit, so the seat moves on. The check-to-send window is bounded by the poll, not closed. * A malformed frame ended the reader, and the cleanup then read the resulting channel disconnect as 'the pipe closed'. The reader now forwards the error and keeps reading; only a zero-byte read is end of file; the cleanup bound is checked at the TOP of every loop iteration, and an unobserved end returns CleanupUnobserved, which retains. * Protocol: a second hello, a result before hello, a reported oid that is not this job's, an unbounded number of mint requests, and an authorization that both grants and refuses were all accepted. None are now. * The child's write budget is computed at write time from the time left. * A child-program override must be an absolute path to an existing file. Module docs narrowed to what is actually proved. Gates: 12 in-file unit tests (cleanup bound under a refilling queue, malformed frame is not EOF, override policy, ambiguous authorization), 3 childless-turn tests proving a REAL next acquisition takes the seat, 6 protocol/revocation tests against real spawned children measuring the revocation reaction against the poll interval rather than the deadline.
The parent measured what was left of the delivery's deadline and handed it across as a duration. The child started that clock when it READ the frame, so everything between the parent's write and that read - the scheduler, a loaded host, a slow decode - was time the parent had already spent and the child was given anyway. Stamp the same deadline twice from one moment: the remaining duration, which is a ceiling the child can never exceed, and the same deadline as an absolute wall-clock instant. The child subtracts its own now from the second and takes whichever is smaller. A clock stepped backward between the two reads cannot lift the child past the ceiling; a forward step only shortens it. Neither replaces the parent's kill, and the module header no longer says the transit is unaccounted for, because it no longer is. The gate is a real TCP listener counting accepted connections, not an error string: a child that read its request after the deadline opens none, and the positive control with its whole budget in hand opens one.
…ker the test wrote Both gates stored a PENDING byte from the second delivery's own task immediately before awaiting the serializer, then sampled that byte. It proved the test had reached a line. It did not prove the serializer had answered anything, so the ordered second-acquisition Pending proof was not in the file. Hold the second delivery's future in the test, pin it, and poll it. Every assertion now reads the Poll that serialized_bounded_push itself returned, with the runtime's own waker, while the first delivery's wedged child is demonstrably alive. The first poll is also the ask, so the spin that waited for the second delivery to announce itself is gone with the marker it watched. The PENDING/ACQUIRED bytes survive only as corroboration that the push body did not run; the Poll is the oracle.
…oth ways out One cell of this grid was credited and the rest were named missing: the pack upload held until the deadline. The two axes are which leg is held - the info/refs advertisement, before libgit2 has built anything, or the git-receive-pack POST that carries the pack - and what ends the delivery, its deadline or a revocation while it hangs. Those are different code paths: a deadline is the parent's timer firing, a revocation is the parent's cancellation poll re-asking authority and killing early. Three new cells through the SHIPPED binary, sharing one runner with the existing one: advertisement x deadline, advertisement x revocation, pack x revocation. The revocation cells carry a 60s budget so that reaching the deadline would be a failure rather than a pass, and they end in seconds. Whether the seat came back is read from work_ended, not from a message: the custody guard hands the turn on by dropping RunningWork and retains by mem::forget, so a retained turn cannot report ended.
…r actor Every gate in this file handed the parent a closure that returned a literal. That proves a token crosses the pipe on demand and nothing about what production calls, which is an actor: a task that owns the seller key, reached through a bounded queue, answered on a channel, bounded at both legs by the push deadline. Build production's minter instead - same destination binding, same authority re-ask before signing, same deadline refusal - over SignerHandle::http_auth_header_blocking, with the key loaded from a real home and consumed into the actor's task by spawn. The shipped child then delivers over verified TLS and the bare repo's own ref is read back to say so, every authorized leg carries a NIP-98 header, and the child's environment allowlist is checked to have nothing key-shaped on it.
…tring Two integration assertions told a revocation from a deadline breach by looking for the word revoked in a sentence. That is a string standing in for a type, and it holds only while the mapping keeps the two apart - exactly the thing nothing was checking. Pin the mapping itself, on the typed ExecutorError variants, at the one place they become prose: both are Cancelled and not Io (an Io would retain the seat), the two sentences differ, a revocation never claims a deadline, and the reap measurement survives both. Negative control run: folding the Revoked arm into the Killed text fails this test (unit-run2-negative.log, exit 101).
…suite The custody guard was changed so that an unwind BEFORE the spawn releases the seat and an unwind after it retains - the fix for a refusal between begin() and spawn taking the seat forever. The test guarding the old blanket-retain rule was left asserting the old rule, so cargo test -p maxplayer-core --lib was RED: a_panic_through_the_child_push_custody_keeps_the_turn, one failure in 1620 (full-run2.log). It was a stale test, not a broken guard, and the useful repair is to gate the rule that actually shipped, on both sides: * a_panic_after_the_custody_is_armed_keeps_the_turn - arm() is called at the point this process stops being able to say no child exists, so from there an unwind is an unknown and the seat stays held. * a_panic_before_any_child_could_exist_hands_the_turn_back - the new half, and the reason arm exists: 'we cannot say whether a child exists' and 'we know none does' are different facts, and answering the second with the first is what turned an ordinary refusal into a dead seat. cargo test -p maxplayer-core --lib seller_git::tests:: - 16 passed (unit-run3.log).
This gate queued a second delivery behind a first whose child it assumed was already running after a 300ms sleep. That assumption is about how fast this machine forks, not about the code: alone it passed, and in the full workspace run - where these binaries run in parallel - the pidfile was not there yet and the test failed reading it (full-run3.log). Poll the real condition instead, bounded at 20s: the file names a pid AND that process is alive. The precondition is unchanged, so 'pending' still means 'queued behind a held turn' rather than 'raced and won'; a child that never starts is still a failure, not a hang. cargo test -p maxplayer-core --test delivery_push_observed_pending - 2 passed (pending-run3.log).
…authority calls Two stale fixtures, both left behind by the enforcement added in e41c00c, both of which made cargo test --workspace red: * a_push_that_finishes... had its child report the oid abc123 while the delivery was gated on a 40-hex object. The parent now compares the two, so the fixture was exercising the refusal path by accident and the test failed (full-run4.log). The fixture reports the gated oid, and the refusal it used to hit gets a gate of its own - a_child_that_reports_an_oid_nobody_gated_is_a_ protocol_fault - which names both oids in the message and checks the seat still comes back, because a reaped protocol fault is a stop, not an unknown. * authority_that_ends_during_the_mint... triggered its revocation on a COUNT of authority calls: Ok twice, then Err. The parent now re-asks authority on a cancellation poll every CANCELLATION_POLL, so a tick or two spent both Oks before the child asked for a mint and the delivery was killed before reaching the moment under test - reproducibly, twice out of two (prodchild-repeat1/2). The condition was never a call count; it was the mint returning. The minter now sets a flag and authority ends the first time it is asked after that. cargo test -p maxplayer-core --test delivery_push_production_child - 9 passed, twice (prodchild-run3-1/2.log).
… on a clock Finding A. The absolute stamp handed to the child was a fresh wall-clock now plus a duration measured BEFORE the request was cloned, so copy time was given back to the child as extra life. The acknowledgement wait was sized by a duration measured before the frame was encoded and handed over, so encoding was charged to nobody. Both now recompute from the absolute deadline at the moment the wait actually starts, and the stamp comes from one pair of readings with only arithmetic between them. The module header claimed the seat is free within deadline + one reap bound. That was wider than the code: the kill/confirmed exit and the end-of-file observation are two consecutive windows, so the seat bound is deadline + 2 * REAP_BOUND, and no claim at all is made about a process that inherited the pipe and escaped the group. Said that way instead. Finding B. Revocation was acted on in one place only - the arm that runs when no frame arrived in time - which made the poll interval conditional on the child being QUIET. A child that keeps the parent busy was never a timeout, so the owner was never re-asked. It is now asked on ELAPSED TIME at the top of the loop, which is what the claim always said. The two long waits that never re-asked at all are now sliced the same way: a write the child has not acknowledged, and a mint whose reply the signer is holding. A revocation during the mint abandons the minting thread and the token is dropped on this side of the pipe, which is what makes the post-mint property hold while the mint is still outstanding. An authority check that comes back refused now ENDS the delivery after the refusal is written, instead of answering no and letting the child run on. Existing suites green: lib delivery_executor 13, protocol_and_revocation 6, production_child 9, custody 4 (r2-*.log).
…quiet Two gates for finding B, each with its negative control run at a recorded source hash. A flood of authority checks. The child asks whether it still holds its turn in a tight loop and reads every answer, so the frame wait never expires. Revoked mid-flood, the delivery must end for revocation within the poll interval, and the gate also asserts the traffic was real (the child was answered >20 times) and that the owner was asked MORE often than the child asked - the poll has to be the parent's clock, not a by-product of the child's silence. Negative control: with the clock-based ask and the end-after-refusal removed, this gate ran the full 30s deadline and failed, exit 101 (neg-B1.log, source neg-B1-source.sha256 4d6f4ed65f7fad00b85b5bcb7ef7596202e20a14a19b863ee3618555fe35b551). An answer stuck in the pipe. The child asks without limit and reads nothing, so the kernel's pipe buffer fills and the parent is left inside a write holding a frame the child will not take. Revoked there, it must still end within the poll. Negative control: with the acknowledgement wait put back to one un-sliced block of the remaining deadline, this gate ran 30.18s and failed, exit 101 (neg-B2.log, source neg-B2-source.sha256 76396932910eaffa1ea55ad5e9396cab053464de8598c84e968bee4351559b0d). Source restored to 098679a500420336f36a772249f63ad3c89b6849c4fd73254336d3b7e26a0019 after each control. Suite green at 8 passed (r2-bgates1.log).
…ll queue Finding C, first part. The existing actor gate mints for a delivery that is going well. Both bounded legs of http_auth_header_blocking - try_send against the full bounded queue, and recv_timeout on the answer - were ungated, and they are the two ways a leg fails on a loaded seat. The actor is the real one, holding a real key from a real home, spawned on its own current-thread runtime whose only worker is then occupied by a blocking sleep. Nothing is mocked: the task simply cannot be polled, which is what a stalled actor is. Held reply must come back 'did not answer before this push's deadline'; with 80 abandoned commands left in a queue of 64, the next ask must come back 'signer queue stayed full past this push's deadline'. Both must RETURN inside their own deadline - a leg that cannot be authorized is failed unauthorized, never parked on the seat's lock. Control: releasing the stall and minting again returns a real Nostr header, so both refusals were about the hold and not a dead actor; and the header is checked byte-wise against the home's secret, which it never contains. r2-c1.log: 1 passed.
…econd delivery wait Finding C, second part. The contention gates in maxplayer-core hold the seat with a shell that ignores SIGTERM. The verdict credited that and named what it is not: the first held phase was never libgit2 packing inside the shipped binary. That is the state a seat is actually stuck in - a repository open, a delta search behind it, TLS to a real smart-HTTP server, a pack half-written onto the wire - and no gate had a second delivery waiting behind it. Delivery one is now the shipped binary pushing a real pack over verified TLS, parked by the fixture at POST /git-receive-pack. Delivery two runs the seat's own serialized_bounded_push and is polled while that pack is held: >=20 observed Poll::Pending returns, its push body never entered, then the seat handed over only after delivery one returned from its deadline kill with a confirmed exit. The hold is still parked when the assertions run, and the remote ref is read back untouched. r2-c3.log: 1 passed in 4.20s.
… pending Finding C, third part, and NOT a revocation - the distinction is the point. Revocation is the owner withdrawing and the executor acting on it. Here the delivery's own tokio task is destroyed mid-flight, the shape of a cancelled request or a shutting-down supervisor, while its child runs and holds the seat. The seat's exclusion is an OwnedMutexGuard moved into the turn and then into the blocking call, so an abort takes the awaiting task and leaves the work: the guard is not the aborter's to drop. Gated: the second delivery returns Poll::Pending from the real serializer while the aborted delivery's child is still alive, the abort really was a cancellation (JoinError::is_cancelled), and the seat moves only after that child is confirmed gone. Measured while writing it, and asserted rather than assumed: the abort drops the turn control, the executor sees it at its next cancellation poll, and the handover lands inside CANCELLATION_POLL + REAP_BOUND - well before the deadline. A first draft required 20 Pending samples at 40ms and failed at 5, because the window is genuinely short; sampling is now 5ms and the SHORTNESS is asserted, so an abort that parked the seat until its deadline would fail this gate. r2-c2b.log: 3 passed.
…side the isolated one The renewed grading named this a material narrowing, not a gaming attempt, and it was right: the one-shot refusal isolates 'a token minted for a revoked delivery does not cross the pipe' cleanly, and in doing so stopped proving what happens next. An owner that goes away stays away, and the delivery has to END - telling the child no about one leg and letting it run to its deadline is not the seat's safety property. Both are kept because neither implies the other. The new case holds authority revoked from the mint onward and requires all three: the SENTINEL header is withheld, the delivery comes back as a revocation rather than a deadline breach or a success, and it ends in well under its budget. The child traps TERM and loops, so 'the delivery ended' is checked against the process being gone. Also corrected a comment in the drive loop that still claimed the timeout arm was the only place a revocation is acted on; that sentence was a description of the defect finding B names. r2-persistent.log: 10 passed.
A custody test that cannot fail is a comment. scripts/custody-mutation-control.sh
applies the two mutations that matter to CustodyBailiff::attempt_handoff, one at
a time, and REQUIRES a red suite against each before restoring the file and
requiring a green one:
M-PREMATURE delete the confirmed-exit condition — the seat moves on 'the work
ended', i.e. on a signal nobody looked at. CAUGHT (3 assertions)
M-DEADLINE replace the work-stopped condition with the clock — cleanup by
calendar rather than by observation. CAUGHT (5)
The deadline control now also pins the case that makes M-DEADLINE dangerous
rather than merely wrong: the child has been reaped and the work has NOT
returned, which is the executor between its reap and its bounded cleanup drain.
Every term a clock-driven fence reads is satisfied there and the seat must still
not move.
The watchdog could stop a child exactly on time and the seat would still wait forever. Child::try_wait is the only call that turns a kill into a CONFIRMED exit, and the child handle lived on the supervisor's stack, so confirmation was reachable from the synchronous executor and nowhere else. An executor parked in the owner's authority check (E:1382) or in its own request clone (E:1473) never reached it: the kill was independent of that stall, the confirmation was not, and the bailiff refuses without one. The child handle moves into ExitGuard - the mutex that already made a late signal impossible - so whichever thread reaches it first may reap. The watchdog now reaps what it killed and publishes the exit itself, charging the SAME REAP_BOUND budget so confirming from there buys no window the seat was never promised. Both halves are published, because the bailiff requires ENDED as well as a confirmation and both sat behind the same stalled return. It is sound at exactly one instant: after the kernel reported the child's exit. A signal, a passed deadline, a spent budget and a caller that gave up all leave it unpublished, and an unknown exit still retains the seat. T-S6 parks the executor in the authority check and asserts it never returns while the seat comes back within WATCHDOG_TICK + REAP_BOUND + CUSTODY_TICK.
… is a kill Every write to a child this process just killed fails with EPIPE, so the broken pipe says nothing about whose fault it was. Both write-failure sites returned Protocol unconditionally, so the deadline watchdog doing its job was reported as a malformed delivery - the one cause an operator would act on differently. Attribution is on the watchdog's own flag, not on 'the deadline has passed': a clock reading would claim every late failure as a stop, including a real protocol fault that landed after the deadline. Custody is untouched. Both causes release, and the reap that precedes the call already decided the question - an unreaped or unwaitable child returns through ? as Unreaped or WaitFailed and retains before this is reached. The end-to-end path is a real race between the drive loop's slice expiring and the kill landing, which is why gate-final-2 caught this intermittently; the rule is now pinned by two unit tests that cannot race.
A's abort budget is 60s so the deadline cannot steal the stop under test. That length was also the hole: a cancellation ignored entirely, with A left to die at its natural deadline kill, satisfied every assertion here. The gate proved the seat came back, not that the abort brought it back. Three oracles added or corrected: - the stop is now measured FROM THE ABORT, against the product's own terms (one cancellation poll + two reap windows + slack), far below the 60s budget - that gap IS the discriminator; - B's acquisition must precede A's own deadline, so a run where the deadline did the work cannot pass; - the join checks is_cancelled rather than is_err, because is_err is equally satisfied by a panic inside the delivery task, which frees the seat by failing rather than by being cancelled. Whole file green in 6.77s, with both 60s-budget abort cases inside it.
The module claimed the production wiring was rebuilt 'same order, same calls' from run.rs:7722-7757, naming the authority check before signing as one of them. The helper did not make that call: destination binding, then deadline, then signer. The one omitted call is the one the chain is about - a minter parked inside the signer is exactly when an authority can end underneath it. production_minter now takes the real PushAuthority::check closure and asks it between binding the destination and signing, with production's own error wrapping. All four deliveries build a live PushAuthority and pass it to the transport as well, so each request leaves through the same check production uses. The expired-turn gate keeps its authority LIVE on purpose: it is about the turn doing the stopping, so a dead authority must not be what stops it. Claim kept rather than dropped, because the wiring now matches it. Four tests green in 7.55s.
…eipts
expect_red used to accept any failure of a named test. It now requires the
NAMED ASSERTION: a compile failure is rejected as "NOT EVIDENCE", each named
test must be reported FAILED, the failure count must match exactly, and the
required assertion substring must appear in the output. A mutant that goes red
for the wrong reason is reported as WRONG REASON, not as a catch.
Every phase is bound to hashes: source sha256 and root-tree before mutating,
while mutated, and after restoring, with restoration required to hash-match
pre. Restoration is from pristine copies taken before the run, never git
checkout, so uncommitted work is not reverted. Receipts carry the head, the
anchor count, the red log's own sha256, and the green controls.
oracle-red-before-green.sh records red-before-green provenance for T1/T2/T3.
Two findings are recorded in the script rather than buried:
- T2's deadline is defended in four independent places. Removing the
watchdog alone survives; watchdog plus the drive loop's expiry check
survives; adding the child's own clock survives. The redundancy is real,
so the mutant used is single-site and matches T2's own wording: the
parent's wait for the mint is made unbounded, and both gates refuse the
60.04s stop against a 2.5s budget.
- T3's cancellation propagation could not be killed by mutating the product.
Deleting the drive loop's periodic authority ask survives (7.91s); so does
deleting the parent's authority ANSWER in all four places it is produced.
On this branch an aborted wire delivery is not stopped by authority
propagation at all, and which mechanism does stop it is a real question
about the branch, reported rather than answered by widening this fold.
What round 2 asked for is settled by a labelled FIXTURE control: removing
the abort reproduces "a cancellation ignored until the natural deadline",
and the new abort-relative bound is what refuses it.
Receipts: logs/rbg/receipts.txt — three mutants caught for their named
assertion, three controls green unmutated, restoration hash-matched.
…nd does not do
THE CAUSE. The 101 was one test, on its LAST assertion:
a_local_phase_that_refuses_to_stop_is_ended_at_the_deadline_and_its_exit_is_confirmed
panicked at delivery_push_production_child.rs:138:
the fixture child must have recorded its pid: Os { code: 2, kind: NotFound }
Everything before that line passed: the delivery returned Cancelled, the refusal said "was
killed" AND "confirmed the exit", and the return landed inside [budget, budget + REAP_BOUND).
The parent behaved. What was missing was the FIXTURE's pidfile, written by the child's first
line — so the test failed on a PREMISE, not on a bound, and nothing about the product's custody
behaviour is implicated.
The deadline these tests arm has to cover the neutralize, the spawn, and a /bin/sh reaching that
first line. Lose that race and the parent correctly kills a child that has written nothing.
FOUR EXPERIMENTS, in scripts/pidfile-101-cause.sh, receipts in logs/pidfile101/:
E1 Prepending `sleep 2` to the fixture body reproduces the gate's failure exactly — same test,
same panic, same 9-passed-1-failed shape. Mechanism confirmed; nothing about the parent,
the deadline or the kill is touched. CAUGHT.
E2 12 idle runs and 12 under 2x-ncpu spin load: 0 failures either way.
E3 E2's null result explained, as a number. Shrinking the budget shows the test still passes
at 250ms, so the headroom at 1500ms is about 1350ms — a busy CPU alone never spends it.
E4 The gate's actual condition, which is different in kind: the workspace's own 40 test
binaries run in parallel, continuously, while the named test runs against them.
15 of 15 failed, all 15 on the missing pidfile. The window is real and reachable.
THE CHANGE. Startup now has its own named allowance (CHILD_STARTUP), and every bound in the two
affected tests is stated relative to the DEADLINE rather than to the call — "not before it, and
within REAP_BOUND after it". No bound is weakened; what changes is that process startup is no
longer charged against the budget being measured. Both tests, and the file, pass idle: 10/10.
WHAT IT DOES NOT DO, measured rather than assumed. Re-running E4 against the fixed tests still
fails 13 of 15, on the same missing pidfile (logs/pidfile101-fix/). The allowance raises the
headroom about eightfold, from ~1.35s to ~11.35s; it does not make the race impossible. That
load — forty binaries looping without pause — is deliberately far harsher than a gate, which
runs each suite once. So this is reported as what it is: exposure reduced by a known factor, not
eliminated, with the residual risk named. Removing it entirely means letting the child's own
startup gate the deadline, which is a change to the executor's API and not this round's scope.
No retry, no tolerance, no ignored failure.
The file I added in d116d2b carries #![cfg(unix)] but no feature gate, while every module it imports is gated: delivery_executor, git_transport and seller_git are #[cfg(feature = "git-delivery")] in lib.rs. Under default features the test target therefore tried to import three modules that are not compiled, and `cargo check -p maxplayer-core --locked --all-targets` failed with three errors that were mine. The workspace check exits 0 because feature unification enables git-delivery for the whole graph, and an --all-features test gate compiles the modules too, so neither instrument could see this. The per-package default-features check is the one that can. Adding #![cfg(feature = "git-delivery")] takes that command from 18 errors to 14 and removes this file from the failures entirely. The remaining 14 are pre-existing in delivery_push_custody.rs, delivery_push_observed_pending.rs and delivery_push_unconfirmed_lane.rs from 32be84b and c6c4c3f, and are deliberately untouched here. No product code changes. Under --all-features the test compiles and runs exactly as before.
…arge the failed reap
Review round 3, item 1 — both halves of the source FAIL.
1a. ENDED publication could release the seat BEFORE the delivery was cleaned up.
Reaping says the child is gone; it does not say that nothing from this
delivery still holds the pipe. A child that leaves a descendant behind is
reaped at once while that descendant keeps the write end and keeps running,
so publishing at the reap handed the seat on while a process from this
delivery was still alive — and CleanupUnbounded/CleanupUnobserved are raised
far too late to take a released seat back.
The confirmation is now owed TWO facts, an observed exit and an observed
EOF, and is published by whichever arrives SECOND. The EOF fact is reported
by the PUMP thread through the new CleanupSink, not by the supervisor, so
the property round 2 established is preserved: a stalled executor still
cannot hold the seat. Only EOF establishes cleanup — a failed read and a
parent that stopped listening say nothing about who holds the descriptor,
so both leave the seat retained, fail-closed.
1b. The watchdog's own reap charged REAP_BOUND only when it observed an exit.
The timeout and error paths returned WITHOUT charging, so the supervisor's
later reap re-read a budget this thread had already spent and could spend
the same window again: the sum the seat's bound is stated in no longer
bounded the time actually spent, on exactly the path where the wait is
longest. Every exit from that loop is now charged, once, on the way out.
(Charging per iteration would shrink the window being measured against and
end the wait early; the loop is extracted as reap_after_watchdog_kill so a
test can drive the accounting directly.)
Tests, red before green, receipts in logs/rbg-r3/receipts.txt:
an_exit_confirmation_is_withheld_until_cleanup_is_established
MUTANT R3-1a/M-PUBLISH-AT-REAP -> CAUGHT, 1 named failure
a_watchdog_reap_that_times_out_is_charged_against_the_bound
MUTANT R3-1b/M-UNCHARGED-TIMEOUT -> CAUGHT, 1 named failure
All five mutants caught; four unmutated controls green; every mutant restored
the tree to the same hash ca20853.
… wire delivery
Review round 3, item 2. Round 2 recorded two product mutants that survived T3
and left the stopping mechanism unidentified. It is identified now, and the
reason both survived is not gate weakness: NEITHER MUTANT WAS ON THIS PATH.
This gate calls neutralize_then_push_in_child_off_runtime with authority: None,
so the executor's authority ask and answer are not wired into an aborted wire
delivery at all, and deleting them could not change a run that never used them.
End to end, an aborted wire delivery stops like this:
1. first.abort() drops the delivery future at its await. The blocking push is
NOT cancelled — it runs under spawn_blocking, and dropping that JoinHandle
leaves the closure running.
2. Dropping the future drops the supervisor's TurnControl, whose Drop calls
end(). That is the only thing the abort itself does.
3. The still-running closure holds a WorkLifetime over the same turn, wired
into the transport as its per-leg/per-chunk gate. The next check sees
WorkEnded, the leg fails, and the child is killed and reaped.
So: TURN REVOCATION OBSERVED BY THE TRANSPORT GATE — not authority propagation,
not task cancellation.
Proven at product level, not by a fixture: R3-2/M-NO-REVOKE-ON-DROP makes
TurnControl::drop a no-op and both abort gates go red at the abort-relative
bound (B took the seat 58.741692709s and 60.052326334s after the abort, past
the 13.05s this stop is allowed — exactly the "cancellation ignored until the
delivery's own 60s deadline" shape the oracle exists to refuse). Both TIMEOUT
gates in the same file stay GREEN, so the mutant discriminates the abort path
rather than breaking the file's premise.
The fixture control T3/C-NO-ABORT is kept and stays labelled as a fixture
control; it is no longer the only thing standing behind this gate.
…uld close it
Review round 3, item 4. Round 2 raised the startup allowance and left the
residual measured but not accounted for. Accounting for it here.
THE RESIDUAL: the allowance reduces the window, it does not close it. Under the
same harsh condition used to find it (the workspace's forty test binaries
looping in parallel), this test still loses the race 13 of 15 on the missing
pidfile. Headroom rose ~1.35s -> ~11.35s, about eightfold, and a gate-shaped
load does not spend it — the gate-final.log 101 has not recurred in the round-2
or round-3 runs. A lost race presents as ENOENT on the pidfile: a failed PREMISE
that looks like a failed bound, and it is never to be waived as a flake.
WHY NO TEST-SIDE FIX CLOSES IT: the budget is handed over before the child
exists. delivery_turn takes an absolute deadline, and the executor arms the
watchdog at the earliest instant a pid exists so that no interval exists in
which a child could go wrong unwatched. Both are load-bearing, so the clock
necessarily starts before the child does, and every in-test remedy is a guess at
startup cost. This constant is a bigger guess.
THE CHANGE IT NEEDS, stated: the work deadline must start at OBSERVED READINESS.
* arm_deadline_watchdog(deadline) becomes two-phase: a startup bound armed at
spawn, and the work deadline armed on an observed readiness signal.
* the executor's entry points take a budget plus that observation rather than
one absolute Instant, so "deadline before the child is up" is unspeakable.
* the child protocol gains the readiness signal — a wire change; today the
child's first act is writing its pid and there is no frame for "up".
WHY DEFERRED: arming later reopens the window the design closes — a child that
hangs before readiness would be bounded only by the new startup bound, making
that bound a second safety property needing its own custody and gates. That is a
product change to the mechanism this PR exists to make trustworthy and belongs
to its own review, not to a corner of this one.
No retry, no tolerance, no waiver. Comment only; no behaviour changes.
…annot
The :482 assertion in the wire-abort gates claimed an overlap: "the seat was
unobserved for long enough that an overlap could have hidden there". It could
not have. That assertion was built entirely from observer-sampled stamps, and
the overlap it named is decided by stamps the participants record themselves --
A's own Token drop (released_at) and B's own entry into its push body
(acquired_at). No polling cadence can move either number. So the check failed
when the POLLER was descheduled, never when two deliveries overlapped, which is
why it reds intermittently under real gate load and passes on an idle host.
The limit is NOT widened. Widening enlarges the interval in which the seat goes
unwatched while still proving nothing: it converts a failing oracle into a
silent one, which is worse than the red.
Instead the file now states its claim so that a descheduled observer cannot
decide it:
- The child check is stated on PRESENCE rather than absence. It used to
demand that A's child be OBSERVED ABSENT before B acquired; absence is
observed late under load, so the stamp drifts past the acquisition and the
gate reds on a run where nothing overlapped. A sample that finds the child
ALIVE proves it was alive at that instant, and starving the observer takes
such samples away rather than inventing later ones. The assertion is now
that A's child was never seen alive at or after the instant B took the seat.
- The cadence number is reported, not asserted. It says how often the observer
got to look, which is a fact about the observer; it is printed because it
tells a reader how strongly a run CORROBORATES the checks above.
Controls, same host, same injected stall of 200ms in the polling loop and no
product change whatsoever:
- the previous oracle RED at 203.635959ms (limit 50ms), exit 101
- this one GREEN at a widest gap of 203.261042ms, exit 0
so the red was manufactured by descheduling the observer, and it is gone.
Discriminating power is unchanged where it matters. With the abort suppressed
(T3/C-NO-ABORT) the two abort gates still fail, at the abort-relative bound --
"B took the seat 30.05508425s after A was aborted, past the 13.05s this stop is
allowed" -- while both timeout gates pass. The failing count is now 2, which is
what the receipt expects; the run that reported 3 was counting this cadence
assert firing on a TIMEOUT row, and that drift is what stopped the combined
receipts run.
RESIDUAL, stated rather than claimed: I could not demonstrate the new
alive-after-acquire assertion going red. Two mutants were tried and neither
produced the overlap it looks for. Confirming the exit on signal-sent survives,
because release also requires supervisor_done and the supervisor's completion
implies its own reap, so the early confirmation is masked. Corrupting both gates
at once breaks the turn machinery instead: the seat is never handed back and the
release clock is left poisoned, so the run dies at "A's turn was never handed
back" rather than on an overlap. The assertion is therefore corroboration whose
falsifiability is UNPROVEN here, and the exclusion this file rests on is the
participant-stamped ordering, which is unchanged and still asserted.
This is the GET-TIMEOUT observation row only. It is not the abort-mutant
stopping causality, and no common cause with E4 is claimed.
The replacement for the cadence assertion was not yet an oracle. It had never
been observed to fail, which is the same silence the no-widening rule forbids
wearing a different shape: widening a limit, demoting a check to a diagnostic
and replacing it with an assertion that cannot fail all land in one place. It
fails now, on a named mutant, and the sampling defect that made it unfalsifiable
is fixed.
THE SAMPLING DEFECT. The process table was sampled once per iteration, BEFORE
the poll. B stamps its acquisition DURING the poll, so on the last iteration --
the one where B takes the seat and may finish -- the alive sample preceded the
acquisition by construction, and the one interval this check exists to witness
was invisible to it. That is why it stayed green against an overlap a plain
process-table check could see. The child is now sampled on both sides of the
poll.
THE MUTANT: R3-3/M-REAP-WITHOUT-WAITING, one site, in `kill_and_reap`. It calls
the SIGNAL a reap: the exit is claimed and the confirmation published without
ever waiting for it, so the signalled child stays in the process table, unreaped,
while the seat is handed on to the next delivery. That is exactly what the
fail-closed rule in this module forbids -- releasing a seat on an exit nobody
observed -- and it produces a real overlap rather than a broken harness.
RED on all four rows, at the named assertion:
A's child was observed ALIVE 3.875us AFTER B entered its push body: two
deliveries were live against the same workdir at once
with 2.416us, 3.041us and 3.375us on the other three, exit 101; green again on
the same head with the mutant reverted.
COVERAGE REPORTING, NOT AN EXCLUSION CLAIM. The demoted cadence number now says
so in the test in those words, so that a later reader cannot cite it as proof
that two deliveries did not overlap. It is not one and never was.
Two earlier mutants did NOT produce an overlap, recorded because they are
evidence about the product rather than gaps in the gate. Confirming the exit on
signal-sent is masked: release also requires `supervisor_done`, and the
supervisor's completion implies its own reap. Skipping the fail-closed backstop
at the end of the confirming run changes nothing either, because every arm of
`drive` has already reaped by the time control reaches it -- that backstop is a
second line of defence, not the reap.
Still the GET-TIMEOUT observation row only: not the abort-mutant stopping
causality, and no common cause with E4 is claimed.
The overlap assertion was guarded by `if let Some(alive_at) = child_last_alive_at`,
so it was SKIPPED ENTIRELY when no alive sample had been taken -- and starving the
observer is precisely what removes those samples. Under the conditions this check
exists to survive it degraded to silence rather than to failure, and a reader of a
green run could not tell "nothing overlapped" from "nobody looked". That is
fail-open, and it was a second reason the check's teeth were unshown, separate
from the mutant question settled in the previous commit.
The run must now show it looked at the interval it judges. Every lookup records
`last_sample_at` whatever it sees, and some sample must fall at or after the
instant B entered its push body. Absent evidence is not evidence of absence; here
it is a failure.
CONTROL C-STARVED-OBSERVER, the observer stops looking after its first three
iterations, which is what starvation does -- it removes samples rather than moving
them later. Same host, same fixtures, no product change:
before this commit: 4 passed, exit 0 -- GREEN having observed nothing
after this commit: 4 failed, exit 101, at the coverage assertion
no sample of A's child was taken at or after B entered its push body -- the
last look was 1.696457333s BEFORE it -- so this run observed nothing about
the interval it exists to judge and cannot corroborate exclusion
with 55.241791ms on another row. The green half is the point: the old form passed
a run in which the observer never once looked at the window it was judging.
The overlap mutant from the previous commit, R3-3/M-REAP-WITHOUT-WAITING, is still
caught at the named assertion -- child observed alive 8.208us, 9.541us, 11.375us
and 14us after B entered its push body -- so closing the fail-open path did not
blunt the check it guards. Clean tree green on the same head.
Still the GET-TIMEOUT observation row only: not the abort-mutant stopping
causality, and no common cause with E4 is claimed.
…after The pump established `PumpEnd::Eof` from a read that returned zero bytes and then sent the final marker BEFORE telling anyone. That send is a blocking send into a bounded queue (`MAX_QUEUED_FRAMES`), so with the queue full at the instant the pipe ended the pump parked in `send` and the cleanup half of the seat's confirmation was not published until the PARENT came back and drained. Custody handoff waited on the receiver's appetite for frames, and the cleanup bound reported that delay afterwards as a delivery that failed to clean up. Cleanup is now established on the observation itself, ahead of the send, and an observed EOF is no longer downgraded to `ParentStopped` by a send that fails after it: the parent going away says nothing about who holds the write end. The post-loop establish is kept as a backstop; `establish_cleanup` takes the confirmation once, so publishing twice cannot confirm twice. The test fills the queue to capacity and never drains it, so a publication observed there can only mean publication no longer waits for a receive.
…kill cannot miss a fork Every wedge in these suites was `while :; do sleep 0.05; done`: a shell forking a `sleep` twenty times a second. The executor kills the child's process GROUP, and on macOS that kill is not atomic against a fork in progress. `killpg` collects the group's members into a snapshot and signals the snapshot (`pgrp_iterate`, xnu bsd/kern/kern_proc.c), so a `sleep` the shell is forking at that instant is inserted after the snapshot and never signalled. It inherits the child's stdout, and the executor — correctly — holds the seat until that pipe reaches end of file: its stated second REAP_BOUND window, which is neither the abort path nor the deadline path a test was measuring. Linux re-checks pending signals inside copy_process and restarts the fork; the window is a macOS one, and the gate runs on macOS. Measured, idle M2 Pro, Darwin 25.6, kill issued the way the executor issues it: 6 of 1500 group kills of the old wedge left such a survivor, each holding the pipe for 53–63 ms (an escaped `sleep 0.05` living out its life); a `sleep 0.005` wedge, 6 of 1500 at 8–10 ms. Under load the survivor lives as long as fork/exec takes on that host, which this branch has already measured past ten seconds under the gate's parallel test binaries (delivery_push_production_child, CHILD_STARTUP). That is the shape of the two `CleanupUnbounded` failures at 5.1 s recorded against that file, and of the abort handover measured at 1.788 s against a 50 ms poll in delivery_push_observed_pending: a prompt kill, then a seat waiting for a process the kill never reached. The `sleep 5` tails written right after a child's last frame are the same race with the kill landing microseconds after the fork; not reproduced in 400 probe kills, closed on the same principle. The production child forks nothing, so its stand-in must not either. tests/wedge/mod.rs blocks the shell ITSELF in a `read` on a FIFO it holds open for writing: `exec` and `read` are builtins, no data ever arrives, no writer ever closes, and the child is one process from its first line to its kill. 0 survivors in 400 probed group kills. Changed: tests/wedge/mod.rs (new); delivery_push_observed_pending.rs (2 wedges); delivery_push_production_child.rs (5); delivery_push_protocol_and_revocation.rs (4 wedges and 3 `sleep 5` tails); delivery_push_stalled_supervisor.rs (the deaf child, which forked once a second). Left as they are, on purpose: custody's descendant test and executor_platform's group-kill test, which fork in order to test the fork, and the cadence test's own `sleep 0.04` cycle, which is the behaviour under test.
…t from the handover `an_aborted_delivery_task_does_not_hand_the_seat_on_while_its_child_still_runs` failed on a clean tree in two opposite directions, and both were the test measuring itself. `samples >= 5` at a 5 ms cadence required the executor to leave the child alive for 25 ms after the abort. The executor acts at its next authority ask, which is 0–50 ms away, so on a host that hands the seat over in 35–65 ms this failed 22–46 % of runs: 350 runs at 5e61933, idle 22/50, light CPU load 31/100, heavy CPU load 28/100, fork/exec storm 46/100, every failure this one assertion with 1–4 samples. The sampler now polls at 1 ms and asserts on every sample; the count is printed, not required. The first poll is taken microseconds after the child is asserted alive, and an exit is confirmed only by a reap that polls at 2 ms, so "observed pending while the child was alive" no longer depends on the executor being slow. `handover * 2 < remaining_at_abort` compared the HANDOVER to the deadline. The handover contains the executor's end-of-file window — up to REAP_BOUND on its own — which a stdout holder the group kill missed can fill. The recorded failure (handover 1.788 s, 19 samples, 2.78 s remaining) has that shape: the child was gone within the sampler's first ~100 ms, and the seat then waited for a pipe. See the wedge commit for the race and the numbers; this host did not reproduce the 1.788 s run itself (350 runs), so its attribution is by mechanism. The instant that separates abort cleanup from deadline cleanup is the KILL: deadline cleanup cannot issue one before the deadline. The test now passes its own authority closure — always yes, as production's PushAuthority is during an abort — and reads the kill from the last recorded ask, which is the ask whose turn check refused; the kill follows it on the same thread. Asserted, each against a bound the executor states: the refusing ask within CANCELLATION_POLL (+1 s for a blocking thread to be scheduled) of the abort AND within half of what the delivery still had; the child gone before the deadline and within REAP_BOUND of that ask; the handover within poll + 2 × REAP_BOUND (+2 s) and, because the fixture is one process whose stdout closes with it, before the deadline. Fail-closed facts kept as they were: the seat moves only after the child is gone, and after it was last seen alive. Measured at this head on the same host: idle 150/150, whole binary 40/40, fork/exec storm 100/100; refusing ask 0.3–59 ms after the abort, handover 5–64 ms. Negative control: with `Turn::check` ignoring `cancelled`, the test fails on "child still alive at the original deadline" in 3 of 3 runs, and passes again on restore.
… group kill The header listed one way a descendant escapes the group kill: it left the group first. There is a second that needs no setsid. XNU signals a SNAPSHOT of the group's members (`pgrp_iterate`, bsd/kern/kern_proc.c), so a descendant the child is forking at that instant is inserted after the snapshot and never signalled; Linux re-checks pending signals inside copy_process and restarts the fork. Such a survivor inherits the child's stdout, and step 12 waits for it — up to that step's whole window — before the seat moves. Measured against a forking fixture: 6 of 1500 group kills left a survivor (tests/wedge/mod.rs has the numbers and the single-process fixture that closes it). The shipped child forks nothing, so today this is reachable only from a test fixture that does. Documentation only; no code path changes.
`delivery_push_observed_pending.rs` was the one delivery-push suite with no `#![cfg]` gate, so `cargo check -p maxplayer-core --all-targets` under default features compiled it against a library that has no `delivery_executor`, no `seller_node` and no `libc`: 8 errors at 5e61933, 9 plus 2 in the new `wedge` module after the wedge commits. With the gate the suite is empty under default features, exactly as `delivery_push_stalled_supervisor.rs` was made to be in fafb4e7, and the wedge module is compiled only by suites that have the feature. Plain `cargo check -p maxplayer-core` (default features) exits 0 at 5e61933 and at this head; the 14-error state the handoff describes did not reproduce on this host with cargo 1.94.1 in either configuration, and its origin is unknown. What remains under `--all-targets` is the pre-existing set: delivery_push_custody.rs (7), delivery_push_unconfirmed_lane.rs (2), src/relay_info.rs (1). Deliberately a separate commit, so the deferred state is changed in the open and only where this PR's own new suite was missing the gate its siblings have.
…e in front of the budget Gate run 7 at 53d1322 failed `a_second_delivery_is_observed_pending_until_the_held_local_phase_is_killed_and_reaped` with "the second delivery's future returned Ready while the first one's local phase was still running", the binary finishing in 2.04 s — the first delivery's own 2 s deadline. The test's deadline is fixed at task start, before the child exists (the executor's arming contract), and its watch runs 1.2 s from the moment the child is seen running. Once the shell took more than 0.8 s to reach its first line, the watch outlived the deadline it was watching, the deadline's kill handed the seat on, and the sampler read that as an early handover. Why that shell was slow on an otherwise idle host (load 1.9) is not established; that it CAN be is: this branch measured fixture startups past 1.35 s under load and gave `delivery_push_production_child` its `CHILD_STARTUP` allowance for exactly this failure shape. The same fix, same constant. Both children in this file now get the allowance: `deadline = now + CHILD_STARTUP + budget`. The killed-and-reaped test states its bound against the DEADLINE — returned not before it, and within the executor's reap plus end-of-file windows after it — instead of against the budget, which silently included startup. The abort test keeps its discrimination against what was still left on the deadline; the allowance only means a slow shell cannot spend that remainder before the abort. The measured line reports the allowance and the time past the deadline. The killed-and-reaped test now runs for the allowance plus the budget, about 12 s. Measured at this head: whole binary 30 of 30 idle, 25 of 25 under a fork/exec storm.
…rit the child's stdout The `CleanupUnbounded` failures in delivery_push_production_child have a cause now, and the fixture's fork race was not it: gate run 8 at 0914881 failed `a_push_that_finishes_returns_its_oid_and_hands_the_turn_back` with "child was reaped but its output pipe was still held 5005ms later" — a child that printed two lines and exited, forked nothing, and was reaped; and a 37-run loop of the same binary failed `an_unauthenticated_remote_cannot_obtain_a_token_by_asking` the same way at 5010 ms. Something outside the child's process group held the write end of its stdout for more than REAP_BOUND. That something is a sibling child of the same test process. Rust's std creates a child's stdio pipes with `pipe()` and then a separate `fcntl(FD_CLOEXEC)` on platforms without `pipe2` — macOS — and spawns with `posix_spawn` without `POSIX_SPAWN_CLOEXEC_DEFAULT` (library/std/src/sys/ process/unix/unix.rs, 1.94.1). Between the two calls the new pipe ends are inheritable, and a `posix_spawn` on another thread at that instant copies them into ITS child, where they survive the exec. Probed on this host with std alone: eight children spawned at the same instant hand one of them another's stdout in 23 of 200 rounds, the write end held for the sibling's whole life (0 of 200 without siblings). A test binary whose suites all spawn at startup is that burst; the wedged siblings live 10 s, so step 12's wait for end of file runs out at 5 s and the seat is retained for a child this process watched exit. Caught in the act, with `lsof` run at the moment the wait timed out (a temporary, env-gated diagnostic, not committed): in 2 of 42 runs the write end of the finishing child's stdout pipe was open as descriptor 64, then 67, of a SIBLING test's wedged child — `/bin/sh .../maxplayer-push-child-refuses-<pid>/child.sh __delivery-push`, the `refuses` fixture, spawned by another test at the same instant. Its own stdio is 0–2; a descriptor in the sixties is an inherited copy. The fix at the layer that owns the pipe: `KillableChild::spawn` holds a process-wide lock across `Command::spawn`, so no two of this executor's children are ever created concurrently and neither can inherit the other's pipe. That closes the case the gate hits — every child in these suites, and every delivery child in production, is spawned here. THE RESIDUAL, NAMED: any other spawn site in the process — a job's agent, `git`, `docker` — that runs concurrently with a delivery child's spawn can still inherit that child's stdout, and step 12 then waits for it and, past its bound, retains the seat for the life of the process. On the shipped darwin target that is a liveness hazard the executor's design did not account for: end of file on the pipe is evidence about the pipe's holders, and on macOS those can include processes unrelated to the delivery. Closing it needs either every spawn site to take this lock, or step 12 to stop treating an unrelated holder as evidence about the delivery (for example a process-group census after the reap). That is a design decision for the PR, recorded on [`SPAWN_LOCK`] rather than made here. Measured with the lock: delivery_push_production_child 80 of 80 whole-binary runs green, 0 `CleanupUnbounded`; without it, 1 in 37 and 1 in 8 gate runs.
…ay what to do when it fires 12d29d0 named a residual and offered two ways to close it: every spawn site in the seller takes SPAWN_LOCK, or step 12 stops treating an unrelated holder of the child's stdout as evidence about the delivery. Decision, 2026-09-17: neither, for now. Both change what this module promises about when the seat may move, the two spawns have to overlap within microseconds, and a delivery is one spawn per job. So the residual is stated where the module states the rest of its assumptions — the "where it can fail" list in the header — with its consequence (the fail-closed retention, reported as CleanupUnbounded) and its remedy (restart the seller). The lock's own documentation records the decision instead of the open question. Linux is unaffected: pipe2(O_CLOEXEC) has no window. Documentation only; no code path changes.
…eature they need CI's default-features and `acp` jobs run `cargo test -p maxplayer-core`, which compiles every test target. `delivery_push_custody.rs` and `delivery_push_unconfirmed_lane.rs` had only `#![cfg(unix)]` and import the executor, the transport, `libc` and `seller_git`, all of which exist only with `git-delivery`: 7 and 2 errors, the deferred state the handoff described, red in both jobs. The same gate their sibling suites carry; under `--all-features` the suites are unchanged. Deliberately its own commit, as the handoff asked for any change to that state.
Two Linux CI jobs (all-features release; money-path) failed `a_push_that_finishes_returns_its_oid_and_hands_the_turn_back` and `a_child_that_reports_an_oid_nobody_gated_is_a_protocol_fault` with "writing the push request: Broken pipe". Both fixtures printed Hello and Done and exited at once, without reading the request. The parent writes the request after Hello; on a fast runner the child was already gone, the write hit EPIPE, and the drive reported the protocol fault before it read the Done frame the test was about. A real child cannot answer before it has its job, so both fixtures now `read` the request first, as every other answering fixture in the file already does. No executor change; the race was between a fixture's exit and the parent's write.
…check, not by racing the reap Review of edac10f, two findings on the abort test, both correct. First: the test observed the child alive, then constructed and polled the second delivery and demanded Pending. Between the observation and the poll, correct cleanup can kill, reap and release the seat, and the poll sees Ready although exclusion was right. The comment claimed a 2 ms reap poll guaranteed a live-child interval; `kill_and_reap` tries `wait` at once and can return without sleeping, and a scheduler gap suffices anyway. The sampling loop had the same gap between its `alive` check and its poll. The repair is synchronization, not a longer assumption. `seller_git` composes the gate as the delivery's authority first and the turn's lifetime second, so an authority ask that has not returned is an executor that cannot yet consult the turn or kill — its documented slow-owner case. The test's own authority closure (always yes, as PushAuthority is during an abort) is armed before the abort and HOLDS the executor's next ask open. While it is held the child is alive and cannot die, and every Pending the second delivery returns in that window is a fact about a live child: twenty polls, each preceded by a live check, none racing anything. Then the ask is released and the executor's own cleanup follows. The hold carries a safety bound so a failed assertion cannot park the executor's blocking thread for the runtime's life; a hold that ran out fails the test. Second: the ask timestamp was labelled THE KILL. It is the authority check; the signal is issued after the check returns and the turn refuses, under the child guard. Every interval is now named for what it observes: `armed_to_ask` is the poll cadence (< CANCELLATION_POLL + 1 s), `released_to_child_gone` is the child disappearing after the released check (< REAP_BOUND + 1 s), `released_to_handover` is the seat moving (< 2 × REAP_BOUND + 2 s, and under half of the remainder at release, the abort-versus-deadline discriminator). After the release no Pending is asserted; the ordering facts — the push body ran after the child was last seen alive, and the child is gone when it did — hold whatever the sampler's cadence. Measured at this head: armed_to_ask 4–14 ms, released_to_child_gone and released_to_handover about 5 ms, remaining at release about 12.6 s. Idle 100 of 100, fork storm 60 of 60, whole binary 15 of 15; one full workspace gate 1975 passed, 0 failed, 23 ignored. Negative control, the turn ignoring a dropped control: 2 of 2 runs fail on "child still alive at the ORIGINAL DEADLINE", and pass again on restore.
…'s absence at the seat's entry Review of f66528e, two findings on the abort test, both correct. First, the arm/abort order. `AskHold::enter` returns at once while unarmed, and the composed gate checks the authority before the turn. So an ask that had passed the unarmed branch and paused before the lifetime check could resume after arm-and-abort, see the cancellation, kill and reap — and no later ask would ever be held, so `wait_until_held` would time out although exclusion was correct. The order is now: arm, wait for the executor to be acknowledged inside the held ask while the first task is still live, THEN abort and join. Until the abort the turn is live, so an ask that slipped past the arming returns yes on both halves and changes nothing; after it, the executor is held where it cannot consult the turn until the release. The mandatory twenty Pending polls, the expiry check and the acknowledgement remain required. Second, the ordering claim. `acquired >= last_alive_at` followed from the sampler's own sequence — it read the child alive, then polled, and `acquired` was recorded inside that poll — so an early release that admitted the contender and reaped afterwards would have passed it. The witness now sits at the boundary that matters: the second delivery's push body checks for the aborted child at the instant it is given the seat, and records what it found. A seat released before the exit was confirmed finds the child there, alive or as a zombie nobody waited for, which `kill(pid, 0)` also reports. The sampler-based assertion is gone; the sampler still provides the timings, labelled as before. Negative controls at this head, each RED on the intended assertion and GREEN on restore: - the turn ignores a dropped control → "child still alive at the ORIGINAL DEADLINE": 2 of 2 runs red; - `kill_and_reap` issues the kill and returns without confirming the exit → the boundary witness: "entered its push body while the aborted delivery's child still existed": 2 of 2 runs red. Measured: idle 100 of 100, fork storm 60 of 60, whole binary 15 of 15; armed_to_ask 30–45 ms, released_to_child_gone and released_to_handover about 5 ms. One full workspace gate: 43 suites, 1975 passed, 0 failed, 23 ignored.
Merged
maxy-player
pushed a commit
to maxy-player/maxplayerai
that referenced
this pull request
Sep 17, 2026
web/app/test/muse-buyer-skill.test.mjs asserts every version literal in web/app/.well-known/skills/muse-buyer/ equals this tree's Cargo.toml version unless the line labels itself a field report, so the bundle is a CI-enforced version surface that RELEASE.md step 1 does not name. Precedent: 0485f99. Version literals only; no other prose changed. The Tier 2 field-report line (verification.md:44, maxplayer 0.5.7) is exempt by the test and left alone. The source-checked claims still hold at 0.5.10: MakePrisms#1006 is the only PR in this release and it touches the seller delivery path plus one internal, unadvertised __delivery-push CLI arm -- nothing in the buyer, wallet or settlement path.
jbojcic1
pushed a commit
that referenced
this pull request
Sep 17, 2026
Bump 0.5.9 -> 0.5.10 across the version surfaces RELEASE.md step 1 names (Cargo.toml [workspace.package].version, Cargo.lock workspace crate entries, npm/*/package.json version, npm/maxplayer/package.json optionalDependencies) and add the v0.5.10 RELEASE_NOTES.md section covering #1006. Gates: verify-release-version.sh 0.5.10 = 0 verify-release-surface.sh --no-artifacts 0.5.10 = 0 verify-release-workflow.sh .github/workflows/release.yml = 0
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to the round-3 verdict on #994 (F3/F5), against
156901027755f0717e80b94fde72d20e62fe787f. Not a revision of #994 — that PR's review record is untouched.The defect. The delivery lock's lifetime was the lifetime of the future that started the push, not of the work. So: a push revoked while it waited for a blocking thread still occupied the seat's turn and still did its local work when the slot came free; queued and pre-HTTP work honoured no cancellation and no deadline; and an async supervisor cancelled at an await took the lock guard with it while its blocking thread was still on the wire — letting the next delivery open a second
git-receive-packagainst the same remote.The rule. The turn is handed back when both sides are finished with it: the work has actually stopped (or provably never started) and the supervising arm is done with the section the turn excludes. Neither alone is sufficient, and each alone has been a bug — supervisor-alone is the original defect; work-alone is its mirror, which the first shape of this fix had and which
the_delivery_that_waited_for_the_lock_signs_after_the_waitcaught (delivery 2 entered before delivery 1 left). A caller's timeout never frees the seat for live work: it revokes, declares its own side finished, and still returnsTimedOut.New
crates/maxplayer-core/src/delivery_turn.rs: the turn carries the exclusion token itself (the lock's owned guard, moved in, so a dying supervisor cannot take exclusion with it) plus an absolute deadline fixed at creation.begin()is queue admission on the blocking thread;RunningWorkis dropped on the thread that did the work.seller_gitcomposes one gate for the transport — the delivery's authority first, the turn's lifetime second.The gate is asked at: queue admission · before the push-config rewrite · before pack generation · pack negotiation · before the mint (a dead delivery never joins the signer queue) · after the mint (the queue wait is exactly where a turn dies unnoticed) · every buffered pack chunk · before every wire request.
Drain bound.
run.rs:1739, with a build-timeconst _: () = assert!(...)on the sum and a test pinning the literal. It is not an HTTP timeout: it is the work's absolute deadline, checked at every boundary above, plus the single in-flight leg whose bytes cannot be recalled.Stated precisely, and see the correction comment below for the evidence: 270s covers every phase this process can interrupt, plus libgit2's delta search, which discards its cancellation answer (
pack-objects.c:979) and is therefore bounded by the delivery's object list rather than by a clock. That span is measured from its true start and an overrun is reported with the number;UNINTERRUPTIBLE_DELTA_BUDGET = 5sis what it is expected to fit in, held finite and strictly inside the work deadline by a second compile-time assertion. A hard bound there needs a killable executor (local phase in a child process, deadline enforced by a signal) — an architectural change, written down rather than smuggled in.Tests (real signer actor, real transport, real HTTPS git fixture; no sleep as scheduling proof — the fixture parks a request and announces it, and ordering is read off one journal):
.git/configbyte-identical, no mint, no dial;try_lockrefused, noEnter, no third request), then completes and lands its ref only after the abandoned upload stops. Peak concurrency stays 1.Fixture gains
FixtureOptions::hold_request_number— the existing appointment, applied to a chosen leg.Gate (from the worktree at
0575a695, all exit 0):cargo test -p maxplayer-core --features wallet --lib→ 1537 passed / 0 failed / 2 ignored;--test delivery_push_contention→ 6;--test h2_no_hidden_replay→ 2;--test relay_push_fresh_auth→ 6;--test push_destination_binding→ 2;--test relay_git_http_auth→ 2;--test git_config_isolation→ 1.--features walletis required (seller_nodeis behind it; it enablesgit-delivery).cargo fmt --checkis red at the pinned base — 2013 diffs across files this branch never touches — so it is not claimed as a gate; reformatting the crate here would be a scope violation.Negative controls. Four, each removing one protection, each compiled and each failing on an assertion (not setup), with files restored and verified by SHA-256: caller-timeout-frees-the-seat → the live-upload test goes red; post-mint lifetime gate deleted → the signer-queue test goes red; admission gate removed → the pre-dispatch test goes red;
supervisor_doneignored → the serialization test goes red.Preserved: post-mint authority check, absolute deadline check, exact OID/ref binding, fresh per-leg auth, no hidden replay, redirects denied, relay policy and TTL.
No merge, no tag, no protected-branch push.