Skip to content

A scan worker builds a backend per point, and whether keeping one SimulationRunner would pay is unmeasured where it could #298

Closed

Description

@jtravs

What is missing

Outside a scan, a SimulationRunner keeps one backend for many inputs. Scan workers construct a backend per point, so a large spatial scan pays allocation and transform planning per point. SEP 22 left this out.

Why it may not matter

On a CPU, the measurement says it hardly does. SEP 22 reports, at 384 by 384 by 128 over ten replayed steps, that the runner's per-input saving is inside the spread: -4.6 % and +1.9 %, with construction at 0.04 to 0.35 s against 11 to 18 s of stepping. It also says that device builds, where construction and planning cost more, were not measured. A run shorter than those ten steps would also shift the balance. So measure before building. Nothing has been run for this issue.

To do

  • Time construction against stepping per point on a GPU build, and on a CPU scan whose points take few steps. benchmarks/history/ records construction_s, in 13 rows on each of three GPUs, which is a start.
  • If it pays, make the worker keep one runner while each point's problem matches its runner's, using the check the runner already makes, and fall back to a new backend when it does not.

Must not change: a point's output, to rounding, and what the scan records of a point.

Done when

The measurement is in the benchmark ledger with machine, width and date. Then either a change that saves at least 10 % of a scan's wall time on a named case, or this issue closes with the numbers.


Checked against commit 8a9f764. Takes over the first half of the roadmap's entry "Runner reuse and field-free members in scans."

Activity

  1. added
    performanceSlower or larger than it should be, with no wrong result
    area: scansScans, batches and their queues
    on Nov 1, 2026
  2. jtravs commented on Nov 5, 2026

    @jtravs
    ContributorAuthor

    Measured, and it pays on a device, so #520 now does it.

    A new backend in a warm process against keeping one, on an NVIDIA H200 (CUDA 13.1) at 8cd170e2, default check, the median of 3:

    Build the simulation Translate Point, new backend Point, kept Saved
    768 x 768 x 600, 8 steps, 4 planes 4.80 s 3.68 s 4.69 s 2.59 s 2.10 s
    384 x 384 x 600, 51 steps, 17 planes 1.39 s 0.93 s 4.24 s 3.26 s 0.98 s

    So a kept backend saves 45 % and 23 % of a point's own time where the builder shares one translation, and 16 % and 15 % even of a point that also builds and translates from scratch, though that kind can't keep one. The issue's bar was 10 %.

    What #520 does: a Local scan on a CUDA or HIP build keeps the backend while each point passes the check a runner makes, which is what Simulation.with_input gives. Any other point closes it first, so two are never held; a point that raises takes it down; the scan closes it. Pool and Batch don't keep one. A CPU doesn't either: SEP 22 found no gain there, and its memory goes back to the next point's build.

    A kept point takes the same path as one alone, Simulation._run with a backend handed in. Its record differs in created and planning_seconds, its hash doesn't, and its fields match within the march's bound, with each point's own noise on the backend (tests/driver/test_scan_kept_backend.py, on numpy_ref and native, with the device predicate patched). Replacing that noise with the backend's is caught.

    A whole scan's time on a card isn't measured yet: the MI325X's pass includes it. #520 will close this.

  3. jtravs commented on Nov 5, 2026

    @jtravs
    ContributorAuthor

    A whole scan now, on an NVIDIA A40 with a CUDA build at ec7b5bab: 5 points of a Local scan through Simulation.with_input, field-free, default check, 8 steps and 4 planes. Two rounds of each, in one process, alternating:

    A backend per point Kept Ratio
    384 x 384 x 600 37.8 s, 36.7 s 33.0 s, 32.4 s 0.88
    768 x 768 x 600 153.9 s, 154.0 s 141.6 s, 141.7 s 0.92

    That saves about 0.9 s and 2.5 s a point, and those are small shares here only because this card's binary64 step is slow: 2.6 s at 768, against 0.22 s on the H200. The H200's 2.1 s a point above is about 45 % of its point. The kept-backend tests pass on this CUDA build, 149 with the scan and batch modules, including a kept point held to the same point run alone. No whole scan has been timed on a data-centre card yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: scansScans, batches and their queuesperformanceSlower or larger than it should be, with no wrong resultpriority: lowWorth doing, no hurry

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions