Skip to content

Reproduce the restore-time 1224 panic as two live mappings on one scratch - #523

Merged
thebhef merged 1 commit into
mainfrom
task156-restore-crash-investigation
Oct 14, 2026
Merged

thebhef merged 1 commit into
mainfrom
task156-restore-crash-investigation

Conversation

@thebhef

@thebhef thebhef commented Sep 23, 2026

Copy link
Copy Markdown
Owner

The field crash — a tokio worker panicking at seg_chain.rs:76 with
Windows OS error 1224 half a second after restored 57529481 frames from prior capture — was not a stale in-process store. It was the
previous cannet process, still alive and still holding every segment
file in the same project cache directory while the relaunched one
restored that directory and grew a chain in it.

A process-wide registry of live mappings, run over the whole cannet-gui
and cannet-spill suites, found no production path that keeps two stores
or two caches over one directory: there is exactly one
TraceStore::new_disk and one SignalCacheStore::new, and every swap path
drops the old value before the new one lands. The owner's own log
settles it instead — the crashing process started at 17:47:01.954 and
panicked at 17:47:09.095, while the process that had just imported that
BLF was still emitting its health tick at 17:47:15.902 with its webview
already gone. Two processes, fourteen seconds of overlap, one scratch.

The collision needs the two chains to be at different lengths, and the
manifest guarantees that: the older chain grew into segment i after its
last flush, so that file exists and is mapped while the manifest the
newer one reopens from still describes a chain one segment short. The
newer chain's next push therefore calls create_segment on a live
mapping, which opens with truncate(true) and set_len.

Both tests here reproduce that in about ten milliseconds against the
public API — one over SampleSeq (the pyramid levels), one over
DiskRawStore (the by-id postings, the family the field scratch shows).
Both fail on every CI OS rather than only on Windows: the Windows arm
is the panic, taken with catch_unwind so it reads as an assertion, and
the POSIX arm is the assertion that the held chain's samples survive,
because there set_len on a mapped file succeeds and silently zeroes the
other chain's segment underneath it. Both are #[ignore]d, since making
them pass is the fix phase's job; cargo test --workspace does not pass
--include-ignored, so they stay inert until the attribute comes off.

Reviewers should look at what the tests assert rather than how they are
shaped: that reopening a segment chain is legal twice over, and that
growing one is what the fix has to make safe.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com

thebhef commented Sep 23, 2026 •

Copy link
Copy Markdown
Owner Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

@thebhef
thebhef force-pushed the task155-logger-clamp branch from 229f9ca to 4f16288 Compare September 24, 2026 01:39
@thebhef
thebhef force-pushed the task156-restore-crash-investigation branch from 8313560 to 33117e1 Compare September 24, 2026 01:39
This was referenced Sep 24, 2026

thebhef commented Oct 14, 2026 •

Copy link
Copy Markdown
Owner Author

Merge activity

  • Oct 14, 9:10 PM UTC: A user started a stack merge that includes this pull request via Graphite.
  • Oct 14, 10:29 PM UTC: Graphite rebased this pull request as part of a merge.
  • Oct 14, 10:32 PM UTC: @thebhef merged this pull request with Graphite.

@thebhef
thebhef changed the base branch from task155-logger-clamp to graphite-base/523 October 14, 2026 22:23
@thebhef
thebhef changed the base branch from graphite-base/523 to main October 14, 2026 22:27
…atch

The field crash — a tokio worker panicking at seg_chain.rs:76 with
Windows OS error 1224 half a second after `restored 57529481 frames
from prior capture` — was not a stale in-process store. It was the
*previous* cannet process, still alive and still holding every segment
file in the same project cache directory while the relaunched one
restored that directory and grew a chain in it.

A process-wide registry of live mappings, run over the whole cannet-gui
and cannet-spill suites, found no production path that keeps two stores
or two caches over one directory: there is exactly one
TraceStore::new_disk and one SignalCacheStore::new, and every swap path
drops the old value before the new one lands. The owner's own log
settles it instead — the crashing process started at 17:47:01.954 and
panicked at 17:47:09.095, while the process that had just imported that
BLF was still emitting its health tick at 17:47:15.902 with its webview
already gone. Two processes, fourteen seconds of overlap, one scratch.

The collision needs the two chains to be at different lengths, and the
manifest guarantees that: the older chain grew into segment i after its
last flush, so that file exists and is mapped while the manifest the
newer one reopens from still describes a chain one segment short. The
newer chain's next push therefore calls create_segment on a live
mapping, which opens with truncate(true) and set_len.

Both tests here reproduce that in about ten milliseconds against the
public API — one over SampleSeq (the pyramid levels), one over
DiskRawStore (the by-id postings, the family the field scratch shows).
Both fail on every CI OS rather than only on Windows: the Windows arm
is the panic, taken with catch_unwind so it reads as an assertion, and
the POSIX arm is the assertion that the held chain's samples survive,
because there set_len on a mapped file succeeds and silently zeroes the
other chain's segment underneath it. Both are #[ignore]d, since making
them pass is the fix phase's job; `cargo test --workspace` does not pass
--include-ignored, so they stay inert until the attribute comes off.

Reviewers should look at what the tests assert rather than how they are
shaped: that reopening a segment chain is legal twice over, and that
*growing* one is what the fix has to make safe.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@thebhef
thebhef force-pushed the task156-restore-crash-investigation branch from 32bfa20 to 930a202 Compare October 14, 2026 22:28
@thebhef
thebhef merged commit 4cddf5c into main Oct 14, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant