Repository navigation
SEP 49: The TDSE response on native (implementation) - #771
Merged
Merged
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HAYtF5P9QqEd3HQx3WJkA3
The synthesis to the nodes, the assembly of P, g S and g' S and their projection move out of numpy_ref into a module that imports neither backend; numpy_ref calls it. New tests state the taper scaling, node synthesis, edge-array scaling, off-band zero and the diagnostics' floors independently of the module. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HAYtF5P9QqEd3HQx3WJkA3
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HAYtF5P9QqEd3HQx3WJkA3
_interrupts.raise_held, for a host source's wait inside native's advance (SEP 49 section 6.7); a no-op outside a commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HAYtF5P9QqEd3HQx3WJkA3
Start with subprocess (never multiprocessing) in a session of its own, protocol on private descriptors with stdout pointed at stderr, SIGINT and SIGTERM ignored, a watchdog that exits on parent-pipe EOF; length-prefixed frames, JSON control, arrays as raw bytes, no pickle, bounded frames; handshake with protocol version and provider identity; a reader thread and a 0.2 s wait loop that polls signals (held ones included) and the stop flag; kill at once on cancellation, a short grace only for close; reap and close on every path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HAYtF5P9QqEd3HQx3WJkA3
The binding, basis_from_preset, hacc_preset, hacc_basis and DiagnosticSession talk to Pyroraptor's adapter in a worker; solprop's process never imports the provider. Budgets, the archive, validation, normalisation and warnings stay in the binding, on plain records. Transport copies join the memory ledger and the estimate; the run's stop flag reaches the binding; a stop during a call kills the worker, keeps the charge and records the evaluation as interrupted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HAYtF5P9QqEd3HQx3WJkA3
The host Adams march releases a trial's reservation on every exit and attaches a failed release to the original error as a note; tests of an archive failure after an accepted commit and of the note. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HAYtF5P9QqEd3HQx3WJkA3
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HAYtF5P9QqEd3HQx3WJkA3
The maintainer's decision: a worker is never used again after a failure, so a provider exception or a refused result ends it and the binding's next call prepares a fresh one. Resolve the three cross-reference warnings Read the Docs raised; drop a dead flag, derive the preset record names, and record the decision in Part II. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HAYtF5P9QqEd3HQx3WJkA3
Queue a failure for any reader exception; leave the worker through os._exit on every path; map a write to a dead worker to its exit; kill in close's finally; static reader threads and a finaliser; no blocking close of a stream still read; a stop owns an exit seen after it; -P and BLAS idle defaults for the worker; raise_held only inside an interruptible() scope on the main thread. Bounded manufactured hangs, signal guards and a zombie-aware orphan check in the tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HAYtF5P9QqEd3HQx3WJkA3
…into feature/sep-0049-tdse-on-native
The resident Adams march calls a generic host source -- begin(cost) after its budget check, evaluate(field, z, purpose, out) at each source evaluation, and end(outcome) after the decision and the commit -- from the advancing thread, outside every parallel region; the binding layer takes the interpreter lock and keeps a Python exception for the session to raise. The engine's own response pass is skipped and its finish applies the collar, absorber and propagator. Device builds refuse a TDSE problem by name. The native adapter binds the driver's binding through the shared source assembly, records the TDSE metadata and memory rows, restores atomic ledgers and opens an interruptible scope around advance; auto now runs TDSE on native. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HAYtF5P9QqEd3HQx3WJkA3
…into feature/sep-0049-tdse-on-native
The host source now reaches the engine: the binding hands `evaluate` its two rows with a no-op capsule owner (nanobind copied the ownerless views), the field read-only, and the stage's entry kernel writes the unscaled lab field, noise included, into the hook's field row instead of the hook reading the fine spectrum (the field divided by n_to). The rows are two, as the SEP says, under the generic `host_source`. A failed `end` after an accepted commit keeps the step's proposal, count and replay cursor. `NativeBackend.close` no longer unbinds under a leased state. An abandoned attempt's failed release rides on the run's error as a note, and the kept error is cleared per attempt. The native TDSE tests go through the C++ gather and the binding (field, F and the history's first row against the reference), use a remainder strong enough that F moves every compared field far beyond its budget and check that it does, restore the eight-step 128 eps budget, and exercise a real held SIGTERM, the floor's note, the terminal proposal and close under a session. Each defect, put back, fails them. Part II records the review and closes Q12. A stale import in test_tdse_edge_free is fixed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HAYtF5P9QqEd3HQx3WJkA3
The review's follow-ups to phase 3: NativeBackend.close drops the host source once the state is closed, since the state's reference to it is invisible to the collector and a cycle through a kept exception was never freed (the held-signal test now checks the binding is collected); and item 4's run is eight binary-exact steps, where 8e-8 m in 10 nm steps split its last step into two. ADR-078 records the host source hook, its scope and the device refusal, the provider worker and the failure boundary, amending ADR-077's exception for the TDSE response. testing.md states the oracle boundary for shared code and the worker-contract tests. Part II records phase 4, the size against section 6.8, and the maintainer's manual check with a real Pyroraptor as a pending, copy-pasteable procedure (install, a two-mode script, pass criteria), dry-run with the manufactured provider. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HAYtF5P9QqEd3HQx3WJkA3
Maintainer decisions of 2026-11-12. ADR-078 is accepted. Until Pyroraptor has a release, solprop uses whatever provider is installed (by default the latest main): `_PROVIDER_REVISION` and `pinned_revision` are removed, and the handshake refuses only another protocol version or an execution space other than `TDSEExecution.device`. The worker's reported revision, build and execution space remain the run's build record; presets and bases name that revision in their source, the consistency cache keys on it (`tdse.provider_revision`), and the archive contexts drop `provider_pin`. Tests check that another revision is recorded, not refused. The SEP's §6.5, §7.2, §7.4, §9.3, Q10 and Part II, dependencies.md and ADR-078 say so, with a release note. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HAYtF5P9QqEd3HQx3WJkA3
The manual check of section 7.4, run with Pyroraptor main (7658b61) on a Slurm cluster: on 32 EPYC 7763 cores, compare gave numpy_ref and native equal ledgers and final fields 4.1e-13 apart (about 100 s a call after 8.4 min of model preparation), and cancel ended a run 0.13 s after SIGINT mid-call with the worker gone, the call charged and nothing reserved; on an A40 with Pyroraptor's CUDA build beside native's CPU wheel, a device mismatch was refused by name and cancel ended the run 0.47 s after SIGINT mid-call with the device released. No GPU compare: an A40's FP64 makes a call hours long, and that comparison stays manual (Q10). The script gains a refusal mode and starts its cancellation 5 s into the first call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HAYtF5P9QqEd3HQx3WJkA3
Brings in #770, #768 and #774 (main 33172a5). The TDSE changes of this branch move with #768's split of simulation.py: the numpy_ref-only rule of _admit_tdse_execution is gone from driver/settings.py, and driver/tdse_run.py takes the stop flag to the binding, the installed provider's revision for the consistency gate, no provider_pin in the archive context, and auto resolving to native. ADR-077 keeps main's text with ADR-078's amendment note; layers.svg is redrawn with docs/dev/figures/build.py. A docstring line in test_tdse_hacc_installed.py is rewrapped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HAYtF5P9QqEd3HQx3WJkA3
CI's Linux wheel failed check_strict_fma.py with 36 fused multiply-adds "in" march::reached. In that wheel reached is 0x5d bytes with none; the 36 lie in 9 KB of unlabelled code LTO placed after it, the dispersive response's Kokkos kernels (calls to GeneralDispersive::respond, sin, tanh, pow, exp). The extension is stripped, so objdump labels only exported functions and the check counted that code as reached's. It now bounds each strict function by its symbol's size from `objdump -T` and `-t`, with a test; the CI wheel passes (333 strict functions, none fused). No native source changes; the non-TDSE runs are unchanged bit for bit. ruff formats the Python block embedded in the SEP's Part II; the block is formatted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HAYtF5P9QqEd3HQx3WJkA3
- test_snapshots: the TDSE snapshot test read the in-process haCC model and plans, which live in the provider's worker since phase 2; it now checks that the snapshots keep the same worker, model view and plans. The failure was the test's own hook, logged by the snapshot transaction; the snapshot path does not touch the model. - test_tdse_worker: the import-isolation claim sets aside provider modules another test of the process loaded, then checks that the run through the worker imports neither pyroraptor nor the adapter. - test_source_worker, test_tdse_worker: each test carries its own tier, so the slow ones are not also medium. - check_skips: the POSIX-only worker tests' reason is expected on Windows. - test_tdse_native: the interaction formulation's transported startup row is each engine's own complex product, which MSVC fuses on Arm64; it is compared within 4 eps of each value, the exponential row still bit for bit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HAYtF5P9QqEd3HQx3WJkA3
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
The coverage lane missed the source worker's child side, which runs in a process of its own, and several new failure paths of the TDSE binding. Tests: - The worker's main, its private streams and the module's entry block run in this process on pipes standing in for descriptors 0 and 1, with os._exit recorded: requests in order until close, an adapter exception and a reply the codec refuses reported as errors, provider printing kept out of the frames, SIGINT and SIGTERM ignored, a failed start reported in place of the handshake, and the watchdog's exit at end of file or on garbage. - Real workers: an answer before the handshake, a message out of turn, a close that overruns its grace, a grandchild holding standard error across a kill, an unstarted worker's close and kill. - Frames: truncated, oversized and mistyped messages; the reply reader's endings; numpy scalars, callables and re-sent records in the codec. - Binding: a provider that cannot start, a worker that dies in a call (a charged "worker" failure with its stderr attached), a stop during the consistency diagnostic's call, a missing result record, a refused shadow component, a fresh worker reporting another build, and the wrapping of worker and local preparation errors. - Adapter: requests outside its protocol, and a result info its model comparison cannot read. Code: - A result that is not a record is refused by name, as a contract failure, instead of failing on a missing attribute as "unexpected". - A worker's failure waits for its standard-error reader once it has exited, so the last lines it wrote are on the error (they could be lost to a race before). - The diagnostic's close is always sent, suppressed when the worker is already ended; a killed worker refuses it at once, so the "alive" check only added a branch no run could take. Removed as unreachable: - tdse._json_default's dataclass branch: provider records now cross the worker as Records, and nothing else the binding encodes is a dataclass. - tdse._component's "order must be one of" refusal: the worker evaluates every quantity at each order the shadow reports, and the binding asks only for orders it has checked against those labels. - tdse_source._cubic_polarisation's checks: PropagationProblem.validate, which both backends run first, admits optmat_cubic only with its real Kerr coefficient and no plasma, Raman or dispersive Kerr beside it. - Worker._exited's and _end's None checks: only a started worker, which always has its process and all three pipes, reaches them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HAYtF5P9QqEd3HQx3WJkA3
The A/B on non-TDSE cases shows no regression (geometric mean 0.986 against the limit 1.015), and the bitwise check is unchanged on all seven cases. CI's coverage lane now covers the worker's own side. Archive the SEP, move its index and toctree entries, drop its roadmap entry, link it from the TDSE manual page, and record the completion and the issues the open questions moved to (#785, #777). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HAYtF5P9QqEd3HQx3WJkA3
…or inactivity) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HAYtF5P9QqEd3HQx3WJkA3
…-on-native # Conflicts: # docs/dev/architecture.md # docs/dev/figures/layers.svg
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
This was referenced Nov 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implementation of SEP 49 (
docs/dev/seps/sep-0049-tdse-on-native.md, tracking issue #764): phases 0–4 of 4, including the manual check with a real Pyroraptor (§7.4). Draft; phase reports are in the SEP's Part II on this branch.This branch merges the native restructuring (#770), which is not on
mainyet. Until #770 merges, this diff also shows #770's commits. Once #770 lands,mainwill be merged in again.Done
poe sizeat the base commit, and the host TDSE source assembly's time per evaluation with a manufactured binding: 0.13–0.26 ms on the test window, 1.2–1.9 ms on a production-sized lattice (17 nodes,n_to = 9216). That is below the SEP's 10–100 ms estimate.solprop.backends.tdse_sourceis a pure move ofnumpy_ref/tdse.py, plusnode_fields. It imports neither backend, andnumpy_refcalls it. Newtests/backends/test_tdse_source.pystates each shared term independently: taper, node synthesis, edge scaling, the off-band zero and the floors. A deliberate defect in each named term fails a test; the table is in Part II.solprop.driver._source_workeris provider-agnostic. It covers subprocess start, a session of its own, stdout redirected to stderr, ignored SIGINT and SIGTERM, a watchdog on parent EOF, length-prefixed frames with no pickle and bounded sizes, a handshake, and a wait loop that polls signals (held ones included) and the stop flag. It kills the worker on cancellation and reaps it on every path.solprop.driver._pyroraptoris the adapter. The binding, presets, bases andDiagnosticSessionall go through the worker, so solprop's process never imports the provider.tdse_transportmemory rows.numpy_ref._interrupts.raise_heldis added for the native hook to use later.include/solprop_native/host_source.hppdefines a genericHostSource. The resident Adams march calls it from the thread that calledadvance, outside any parallel region:begin(cost)runs after the budget check;evaluate(field, z, purpose, out)runs at each source evaluation;end(outcome)runs after the decision, and after the commit for an accepted attempt.abandoned, which only releases its reservation.Fin place into its source row. Both rows are allocated when the source is bound, and appear ashost_sourcein the ledger.nb::python_errorunder the interpreter lock. The march reports a host failure with its stage; before the commit the attempt is abandoned, after the commit it is returned._host_source.TDSEHostSourceuses the shared assembly with the reference's own scalars and prefactor. The native backend:atomic_calls,tdse_execution,tdse_last_evaluationandtdse_source;tdse_*rows;advancein_interrupts.interruptible();backend="auto"now runs TDSE on native.kernel-contract.md§6,backends.md,AGENTS.md,native/AGENTS.md, the TDSE manual and Choosing a backend, and added a release note.SOLPROP_FFTW_PLAN=estimate SOLPROP_CALIBRATION=offon one thread:gnlse_soliton,rdw_one_mode(both Adams formulations),argon_blueshift,radial_filament_256anddense_output.poe tidy-nativereports no finding on the changed C++.478b3808; details in Part II):Ftherefore never reached the engine. The views now have a no-op capsule owner, and the field view is read-only.n_to(read from the fine spectrum). It now reads the unscaled stage input, which the entry kernel writes.Fbit for bit, and history row zero bit for bit without the absorber, withinfft_bound(n_to)with it. This covers both formulations, both descriptors, and runs with and without input noise.Fmoves the field a millionfold beyond its budget.close()under a session lease no longer unbinds.endkeeps the proposal and the replay cursor.0625619d):close()now drops the host source after the state is closed. The state's reference to it was invisible to the GC, so a cycle through a kept exception was never freed. The held-signal test now asserts with a weakref that the binding is collected.auto. It amends ADR-077.testing.mdgains a section on shared code and the worker contract.backends.mdandkernel-contract.mdcite the ADR. The rest of §8 was already done.compare(native againstnumpy_ref, at most three atomic calls) andcancel(SIGINT during a call). Both modes were dry-run with the manufactured provider.main) and records the revision the provider reports. The handshake still refuses another protocol version or a device mismatch.main7658b61 and solprop 7a5ca1f, run on a Slurm cluster:compare(32 cores):numpy_refand native gave identical ledgers and status, and final fields 4.1e-13 apart (relative). Each atomic call took about 100 s, after about 8.4 min of model preparation.cancel:KeyboardInterruptcame 0.13 s after SIGINT, sent 5 s into a call. The worker was gone, the call was charged and nothing was left reserved.device="cpu"was refused by name.cancel cudaraisedKeyboardInterrupt0.47 s after SIGINT, the worker was gone and the GPU had no process and 0 MiB in use afterwards.compare, because the A40 is too slow at FP64. That comparison is left manual for the provider's next production GPU use (Q10).Phase 3 line counts against the SEP's estimates for the hook:
src/solpropSize
Against
main(97a902bd),tools/size_report.py(src/solprop, lines):mainBy file (code, docstrings, comments, total;
main→ this PR):driver/tdse.pydriver/_source_worker.pydriver/_pyroraptor.pybackends/numpy_ref/tdse.pybackends/tdse_source.pybackends/numpy_ref/backend.pybackends/host/adams.py_interrupts.pydriver/simulation.pyThe
growthgate's excess is the worker (_source_worker.py) plus the adapter's request handlers and the parent's view of the haCC model (_pyroraptor.py,tdse.py). The binding's validation and normalisation stayed in solprop's process, as §6.5 asks. I looked for code that could go without losing behaviour and removed only a dead flag, a duplicated preset-field list and a dead dataclass branch. Nothing larger can go without moving validation into the worker or merging the parent and worker views of the haCC model; both are design changes. The private cross-package import ratchet is unchanged at 89; this PR's one new private-module import was replaced by public names.Tests added
tests/backends/test_tdse_source.py(10)tests/driver/test_source_worker.py(21, real workers; 3slow)tests/driver/test_tdse_worker.py(8, real workers with a manufactured haCC provider; 1slow)tests/backends/test_tdse_native.py(23, native againstnumpy_ref, SEP §7.3)test_tdse_march.py, and one test intest_interrupts.pytests/source_adapters.pyandtests/source_channel.pyThe existing binding tests now serve their fakes through the same adapter and codec in-process; their claims are unchanged.
Run locally: the affected modules, 958 passed before the follow-up commit and 698 of the driver set after it. The 4 that failed are native-only: the editable native build here is stale after merging
main, and these phases change no native code.Maintainer decision (2026-11-12)
A worker is never used again after a failure of any kind. That covers a provider exception, a result the binding refuses, a worker exit, a protocol error and a cancellation. The binding's next evaluation or diagnostic starts and prepares a fresh worker. This is recorded in Part II and tested (
test_after_a_provider_exception_the_retry_gets_a_fresh_worker).Deviations (details in Part II)
raisedwith classificationinterrupted, so the archive format is unchanged.endtakes an outcome (rejected, accepted or abandoned) rather than a boolean.errorreason with a newhostcategory, not a reason of its own.Fbit for bit. The stage output is compared through whole runs instead; open question Q12 explains why.Left
🤖 Generated with Claude Code
https://claude.ai/code/session_01HAYtF5P9QqEd3HQx3WJkA3