Skip to content

WIP: SEP 52 milestone 4 — native Workspace seam, field-table records, generated stubs, engine parity, ADR-079 - #859

Open
jtravs wants to merge 33 commits into
mainfrom
feature/sep-0052-backend-shape-p4
Open

jtravs wants to merge 33 commits into
mainfrom
feature/sep-0052-backend-shape-p4

Conversation

@jtravs

@jtravs jtravs commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Milestone 4, the last, of SEP 52, tracked in #849. Stacked on #858 (← #854 ← #853): merge those first. The details are in the SEP's Part II report for milestone 4. The SEP is not yet Final: the report's handoff lists what the Final commit must do once the maintainer has reviewed this.

Phase 4: the native seam

  • Field-table records: each C++ record declares its fields once (dtype, rank, extent relations). One converter (native/include/solprop_native/fields.hpp, native/src/records.{hpp,cpp}) reads a dataclass or mapping in a single pass. It refuses missing or unknown fields, wrong dtype or rank, strided arrays, broken extent relations, overflow and integer enums, and names the field path. _core.schemas() exports every table, and a test holds each Python record to its C++ table.
  • _core.Workspace(problem) with .start(run), .probe(start), .diagnostics(), .counters() and .close(). Problem.setup carries the set-once data. The lease, the coupled-flow guard, method selection and the marches' entry points moved into the C++ library (native::Workspace, gtest-covered).
  • Deleted: the State class (55 bound names), every configure, upload and bind call, the old probe path (_trial_step, _ResidentLowStorage, _ResidentAdams, _ZTables evaluation) and most dict converters. Runtime setup is one configure_runtime record.
  • Stubs generated whole by nanobind stubgen, with nb::sig naming the Python dataclasses. No hand-written stub remains.
  • Split sums travel in native's AcceptRecord, measured where they always were.
  • Engine-level parity (tests/backends/test_engine_parity.py): each case is written once against the protocols and run on both backends' engines (RK, low-storage, Adams interaction/exponential/carrier).

Phase 5: documentation and ADR

  • Developer pages (backends.md, architecture.md, kernel-contract.md, benchmarking.md, testing.md, stepper.md) and native/README.md's binding rules.
  • ADR-079 (proposed until Final).

Performance (SEP §7; one exclusive 32-core EPYC 7763 node, 3×3):

  • Whole-run time from a built backend: rk +1.9 %, low-storage +2.1 %, Adams +1.9 %.
  • ab --length 0.05 with both builds timing the same work: within 3 % on rdw, radial_filament_256 and gnlse_soliton × default/interaction/rk; kerr_beam_192 1.007.
  • Scan preparation +0.5–1 %. Probe trial unchanged; −16 % for low-storage.
  • Device targets are qualified by compilation only (D5).

Size versus the §6.7 estimate: solprop_native 14,154 → 11,655 (est. 10–11k). backends missed its estimate: the workspace's operations are now public and documented, and the records were added. bindings.cpp 4,559 → 3,861, about 1,700 of it the private _testing. Hand-written stub: none.

For the maintainer: see "For the maintainer" in the milestone 4 report (basis_rows/trial_tables; an intermittent z-table tolerance test that predates this; ADR-079 to accept at Final). The issues to update at merge are #110, #618, #171, #828, #344, #107 and #356.

🤖 Generated with Claude Code

jtravs and others added 27 commits November 16, 2026 02:21
…52 phase 4)

The new native seam beside the old one. Each record that crosses declares
its fields once, next to its definition (fields.hpp), and one converter
(src/records.hpp) reads a dataclass, mapping or tuple into it by its table
and writes it back, refusing a malformed field by its path before copying
anything; _core.schemas() exports every table. Problem's groups are bound
from their tables, and Problem absorbs the set-once data (stage, z and flux
tables, flow guard, controllers, low-storage settings, monitor and edge).

native::Workspace (workspace.hpp) holds the lease, the coupled-flow guard,
the marches' construction and entry points, the session (Run) and the timing
probe, moved out of the binding unit; the old State binds over it. _core
gains Workspace(problem).start(run)/probe(start)/diagnostics(), Probe,
configure_runtime and the Session's ticketed export, snapshot, noise_phase
and records; _testing gains engines, inject_fault, evaluate and round_trip.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
march_policy_from, watch_policy_from, watch_state_from, watch_config_from and
stage_table_from now read through the field-table converter, so the old seam
and the new one refuse a malformed record alike; the workspace's native test
builds a complete envelope problem.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t); the State seam deleted (SEP 52)

NativeBackend builds one Problem with its set-once data and one Workspace for
its life; a session starts it from one run record (RunStart: the run records,
field, noise, march state, history rows, statistics plans, monitor weights and
edge tables), and the timing probe is Workspace.probe. The State class, its
configure, upload, bind and metadata names, start_session, the per-step probe
paths (_trial_step, _ResidentLowStorage, _ZTables' evaluation), _ResidentAdams,
_records and the old dict converters go; the _testing entry points take a
workspace. Tests not yet ported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…AcceptRecord (SEP 52)

Tests that reached the native State now reach the backend's Workspace through
_testing (load, field, finish, rk_trial, bind_monitor, edge_reading,
table_values, table_load, prepare_raman, adams_rows_from_host,
aperture_statistics, low_storage_norms, quadrature_peak, configure_march) and
diagnostics(). _testing.engines' accept returns the split collar's sums its
finish measured (GeneralState::split_sums, read without draining); the
low-storage engine keeps its last squared norm for the forced-thread folds.
The adapter projects a TDSE host source's rows in the memory ledger before a
run binds them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…(SEP 52)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…en (SEP 52)

problem_stub.py now writes _core.pyi and _core_testing.pyi whole from
nanobind.stubgen; the bindings name each record argument and result with
nb::sig (the run start and probe start, the export recipe, the monitor
continuation, the run records, the protocol records, the batch arrays), and
--check holds both stubs. The broken adams_host benchmark, which reached the
deleted _ResidentAdams, goes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pace seam (SEP 52)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…art (SEP 52)

The converter reads a std::span field as a view of the caller's array, valid for
the call; RunStart's field and noise are such views, which Workspace::start loads
straight into the state, so a large field is no longer copied into the record
first (kerr_beam_192's low-storage window was 4.6 % slower on an EPYC 7763 with
the copy). The probe keeps its own copy, which it reloads after a read-back.
Also the tests the cluster sweep found: the carrier frame's host exponent
(NativeBackend._delta_phi, host path), the session-failure test's lowering with
its setup, and the plane plan's buffer constants, now the run records'.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nostics

Workspace.counters() returns the on-axis scans, the Runge-Kutta stage
evaluations and the z-table positions alone; a session's start and close read
the axis scans from it instead of the whole diagnostics record (92 us each on an
EPYC 7763 at 32 threads).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… types

A run's records cross at every start, so the converter's cost is a run's: it
read each record twice (a check pass, then a copy), asked every number's type
for its module and name by string to rule out NumPy's bool, looked each field up
by a fresh name string, rebuilt a dataclass's field names, and formed every
array refusal's text before checking. It now reads once into a record a refusal
discards, compares the type with NumPy's bool type looked up once, gets fields
by interned names, holds each dataclass type's names and its checked table, and
words a refusal only when it raises. RunRecords' round trip went 16.9 -> 6.3 us
and a workspace's start and close 26.4 -> 16.5 us on an Apple M5 Max.

The benchmark worker's window can take the whole run, the load included
(SOLPROP_BENCHMARK_WINDOW=run, for ab's per-arm environment): a build that loads
the state in the session's start then times the same work as one that loads it
before. The recorded history's window is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e (SEP 52)

A run records its execution metadata at every plane and reads the split
collar's cap at every batch. The metadata read the whole diagnostics record each
time (58 us on an EPYC 7763: the memory ledger, the CPU record, the planning);
it now reads the workspace's facts once it is built and only its live counters
(Workspace.counters, which gains the Raman transforms and the recycled stage)
after. Session.records takes the names it is asked for, so the cap alone no
longer drains the quadrature peak with a fence or converts the monitor's record,
and a run without a monitor reads no cap at all; the monitor's record leaves the
peak to the alarm's check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Milestone 3 (the reference restructured) with phase 4's native seam. The
conflicts were tests both sides ported: the reference is driven through its
session, workspace and engines, native through its workspace's _testing
entry points. test_transverse_monitor.py read the deleted host driver.

[skip ci]

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e (SEP 52)

- tests/backends/test_engine_parity.py: each case written once against
  RungeKuttaEngine or AdamsEngine and played on both backends' engines
  (tests/engine_pairs.py: the reference's own engines over its workspace,
  native's through _testing.engines): rejected and accepted Runge-Kutta
  trials, the first step's stage registers, the split collar's sums in
  AcceptRecord and the boundary reading; a dense extension; the low-storage
  in-flight interval; and the reference march's own Adams calls replayed on
  the interaction formulation with transported history, the exponential one
  and the carrier frame (step_carrier, chord_phase). SEP 25 budgets from
  march_bounds, each term held clear of its bound.
- The tests found native's protocol records two-dimensional (occupancy, the
  Adams norms, the carrier rotations): one-dimensional now. _testing.engines
  gives a dense configuration's Runge-Kutta engine its extension holder
  (Workspace::trial_engine(dense), which the session's method now shares).
- The estimate's native trial and the whole-trial norm test go through the
  engines.
- The converter reads an enumeration by its name only; the Adams march's
  history crosses with its identities named, and the watch state is written
  by its table.

[skip ci]

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…leted driver; drop an unused _testing binding (SEP 52)

The native comments and docstrings still cited HostDriver's members, which
milestone 3 deleted: each now names the member that does the same work
(HostSession, RungeKuttaMarch, NumpyWorkspace, RungeKuttaDense,
HostLowStorage). _testing.set_absorber_quiet had no user.

[skip ci]

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
backends.md (the binding, the z tables' slots, the Adams march's start
record), architecture.md section 9, kernel-contract.md (the occupancy and
the Adams march's start record, the engine's parity), benchmarking.md (the
whole-run window), testing.md (engine-level parity), stepper.md (the
low-storage probe), native/README.md, a ledger entry for the seam's A/B in
benchmarks.md, and ADR-079 (proposed until the SEP is Final). The native
headers no longer cite the deleted adapter's _ZTables, _ResidentLowStorage
and _score_trial.

[skip ci]

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
[skip ci]

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…2-backend-shape-p4

Milestone 3's CI fixes. [skip ci]

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…p ci]

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@read-the-docs-community

read-the-docs-community Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

@codecov

codecov Bot commented Nov 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

jtravs and others added 3 commits November 16, 2026 09:00
…n _testing

The compiled Adams march forms its Lagrange basis and coefficient tables in
C++, so no production path reached their bindings (maintainer's decision).
The native helper record names only the Chebyshev rows; Helpers.basis is
optional and refuses by name on a record without one, and the Helpers.tables
slot, which only the native record filled, goes with numpy_ref's one-call
branch. tests/native_lagrange.py reaches the C++ through _testing for the
plain-form and exact-table comparisons; the stubs are regenerated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tests

The aperture packer moves to the session that calls it, a dead device helper
and an unneeded cast go, the session status indexes its tables by int, and
the workspace tests type their spoiled records, comparisons and counts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jtravs
jtravs marked this pull request as ready for review November 16, 2026 10:05
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T10:58:52.278980Z 2872c35 New commits
🔒 Security Review ✅ Completed 2026-11-16T10:58:20.975335Z 2872c35 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Base automatically changed from feature/sep-0052-backend-shape-p3 to main November 16, 2026 10:16
…and workspace

GCC (as the CUDA host compiler) read the member-pointer declarations as
parenthesised: they are aliases now. The marches' entry points read their
optionals checked, and the probe takes its start by reference.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… test checks its optional

nvcc's front end does not discard an if-constexpr requires branch inside the
setter lambda, so a group's shape is kept through two overloads instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant