Skip to content

feat(models): run the merged 3-model graph in one NPU session - #415

Draft
youtalk wants to merge 1 commit into
autowarefoundation:mainfrom
youtalk:feat/merged-npu-inference
Draft

youtalk wants to merge 1 commit into
autowarefoundation:mainfrom
youtalk:feat/merged-npu-inference

Conversation

@youtalk

@youtalk youtalk commented Sep 10, 2026 •

Copy link
Copy Markdown
Member

Why

The Renesas execution provider for the R-Car X5H NPU permits one NPU session per process. VisionPilot runs three ONNX models — AutoDrive, AutoSteer, AutoSpeed — each owning its own Ort::Session, so under that constraint only one of the three can ride the NPU and the other two fall back to the CPU silently, costing roughly an order of magnitude in latency.

Work in openadkit resolves this outside VisionPilot by composing the three models into one graph and rewriting it to be fully NPU-legal, emitting a sidecar contract.json that describes the postprocessing it moved to the host. This PR adds the consuming side: VisionPilot can load a merged model, target the Renesas EP, and perform the host-side postprocessing the rewritten graph requires.

What this adds

  • A ModelBackend interface replacing the hard-coded three-session dispatch in InferencePipeline. SplitBackend is today's behaviour lifted verbatim; MergedBackend runs one session with three inputs.
  • Contract-driven host postprocessing covering both the v6 and v7 rewrite variants without a code branch: the fused drive head (per-row alpha division plus activations), the lane soft-argmax decoded in double precision on the host (which left the NPU graph because int8 softmax quantisation caused visible trajectory wobble), and the per-level speed DFL assembly with the stride multiply moved host-side.
  • A renesas provider in OnnxEngine that resolves a compiled artifacts directory into a model path plus provider options.
  • A startup fidelity gate that runs one warm-up frame with profiling enabled, parses the emitted profile JSON for a per-execution-provider node histogram, and refuses to start unless the NPU actually executed nodes. engine.require_npu_nodes optionally pins the expected count so a recompile that sheds subgraphs to the CPU fails at startup rather than quietly costing latency.
  • AutoSpeed::post_process extracted into a free decode_detections() with an explicit cls_is_probability flag, because the rewritten graph emits the class tensor already sigmoided and applying sigmoid twice would compress every score toward 0.5.

Existing behaviour is unchanged

model.merged defaults to false, so provider = cpu | cuda | tensorrt continues to construct three sessions and behave exactly as it does today. This was verified by building a standalone probe against the built libraries and running the literal pre-branch vision_pilot.conf through it: three separate sessions with their own startup banners, and a byte-identical [OnnxEngine] provider=cuda device=0 line. The SplitBackend lift was also compared statement by statement against the removed block — same std::launch::async policy, same future creation and get() order, same clock placement, same arena-shrink configuration.

One deliberate removal on the split path: curr_01_asp, a redundant per-frame copy of curr_01_as, is gone. Both networks already shared that buffer's contents and ONNX Runtime does not write into caller-owned input tensors, so this removes roughly a 6 MB memcpy per frame for every provider.

Testing

130 tests, all passing, zero build warnings, verified on this branch with docker build --target tests -f docker/Dockerfile.cpu .. The contract parsing, all three postprocessing rules, the detection decoder, the artifact resolution, the profile parser, the offload decision logic, the startup validation refusals, and the backend selection are all unit-tested against plain data — no NPU, no vendor wheel, and no model file required. That coverage exists because every validation decision was deliberately extracted out of the session-holding classes into free functions.

The repo had no unit-test target for modules/models, so this adds one: tests/models/models_tests (gtest), wired in behind the existing BUILD_TESTING option, which still defaults to OFF; docker/Dockerfile.cpu gains a tests stage that builds and runs it.

The split-versus-merged equivalence test is included but CMake-gated and skipped when the asset is absent: the merged ONNX is roughly 147 MB and is not in this repository, matching how the existing model weights are handled.

What is not verified here

The merged path's own numerics need the board and the vendor EP; they stay verified by openadkit's consistency check and final-check runs. Nothing in CI exercises the Renesas provider end to end.

The merged path degrades the signal that steers the vehicle. Frozen attention costs lane agreement against the CPU reference, measured at 0.653 / 0.905 / 0.954. That is why the path is opt-in and why the startup gate logs the active attn_mode and warns explicitly whenever it is anything other than keep — including when it is unset, which is treated as unknown rather than safe. Whoever enables model.merged on a vehicle should read that figure first.

Configuration

All new keys are additive with safe defaults and are documented in config/vision_pilot.conf. model.merged = true is required when provider = renesas and refused loudly at startup otherwise, because a split backend there would place two of three models on exactly the silent CPU fallback this work exists to remove.

Relation to the other two PRs

Last of three independent changes, and the only one that is X5H-specific — the other two are provider-agnostic CPU-side wins, which is why they are suggested first. This branch builds and passes on main as it stands. It touches the same InferencePipeline::process() block as the input-tensor-fusion PR and adds the same tests/models + Dockerfile.cpu test harness, so whichever of the two lands second needs a mechanical rebase; no logic overlaps.

Note on spell-check-differential

The check is red, and none of it comes from this branch. spell-check-differential scans every changed file in full, so it reports words that already live in those files on main — Ipopt, libnice, CPACK, cipo, latc, Matx, dets, and the forbidden ROS2 among them. The same check has been failing on main in spell-check-daily every night for at least a week, and the last two merged PRs (#408, #411) merged red on it too. Every word this branch actually introduces is either already in the shared dictionary or added to .cspell.json here, verified locally with cspell against the repo config.

The Renesas execution provider for the R-Car X5H NPU permits one NPU
session per process. VisionPilot runs three ONNX models -- AutoDrive,
AutoSteer, AutoSpeed -- each owning its own Ort::Session, so under that
constraint only one of the three can ride the NPU and the other two fall
back to the CPU silently, costing roughly an order of magnitude in
latency.

Add the consuming side of the merged-graph work: VisionPilot can now
load a single model that contains all three networks, target the Renesas
provider, and perform the host-side postprocessing the rewritten graph
requires.

  * A ModelBackend interface replaces the hard-coded three-session
    dispatch in InferencePipeline. SplitBackend is today's behaviour
    lifted verbatim; MergedBackend runs one session with three inputs.
  * Contract-driven host postprocessing covers both rewrite variants
    without a code branch: the fused drive head, the lane soft-argmax
    decoded in double precision on the host, and the per-level speed DFL
    assembly with the stride multiply moved host-side.
  * A renesas provider in OnnxEngine resolves a compiled artifacts
    directory into a model path plus provider options.
  * A startup fidelity gate runs one warm-up frame with profiling
    enabled, parses the profile JSON for a per-provider node histogram,
    and refuses to start unless the NPU actually executed nodes.
    engine.require_npu_nodes optionally pins the expected count so a
    recompile that sheds subgraphs to the CPU fails at startup rather
    than quietly costing latency.
  * AutoSpeed::post_process becomes a free decode_detections() with an
    explicit cls_is_probability flag, because the rewritten graph emits
    the class tensor already sigmoided.

model.merged defaults to false, so provider = cpu | cuda | tensorrt
keeps constructing three sessions and behaves exactly as before.
model.merged = true is required when provider = renesas and refused
loudly otherwise, because a split backend there would place two of three
models on exactly the silent CPU fallback this exists to remove.

One deliberate removal on the split path: curr_01_asp, a redundant
per-frame copy of curr_01_as, is gone. Both networks already shared that
buffer's contents and ONNX Runtime does not write into caller-owned
input tensors, so this removes roughly a 6 MB memcpy per frame.

The merged path degrades the signal that steers the vehicle: frozen
attention costs lane agreement against the CPU reference, measured at
0.653 / 0.905 / 0.954. That is why the path is opt-in and why the
startup gate logs the active attn_mode and warns whenever it is anything
other than keep -- including when it is unset, which is treated as
unknown rather than safe.

Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant