Conversation
The Renesas execution provider for the R-Car X5H NPU permits one NPU
session per process. VisionPilot runs three ONNX models -- AutoDrive,
AutoSteer, AutoSpeed -- each owning its own Ort::Session, so under that
constraint only one of the three can ride the NPU and the other two fall
back to the CPU silently, costing roughly an order of magnitude in
latency.
Add the consuming side of the merged-graph work: VisionPilot can now
load a single model that contains all three networks, target the Renesas
provider, and perform the host-side postprocessing the rewritten graph
requires.
* A ModelBackend interface replaces the hard-coded three-session
dispatch in InferencePipeline. SplitBackend is today's behaviour
lifted verbatim; MergedBackend runs one session with three inputs.
* Contract-driven host postprocessing covers both rewrite variants
without a code branch: the fused drive head, the lane soft-argmax
decoded in double precision on the host, and the per-level speed DFL
assembly with the stride multiply moved host-side.
* A renesas provider in OnnxEngine resolves a compiled artifacts
directory into a model path plus provider options.
* A startup fidelity gate runs one warm-up frame with profiling
enabled, parses the profile JSON for a per-provider node histogram,
and refuses to start unless the NPU actually executed nodes.
engine.require_npu_nodes optionally pins the expected count so a
recompile that sheds subgraphs to the CPU fails at startup rather
than quietly costing latency.
* AutoSpeed::post_process becomes a free decode_detections() with an
explicit cls_is_probability flag, because the rewritten graph emits
the class tensor already sigmoided.
model.merged defaults to false, so provider = cpu | cuda | tensorrt
keeps constructing three sessions and behaves exactly as before.
model.merged = true is required when provider = renesas and refused
loudly otherwise, because a split backend there would place two of three
models on exactly the silent CPU fallback this exists to remove.
One deliberate removal on the split path: curr_01_asp, a redundant
per-frame copy of curr_01_as, is gone. Both networks already shared that
buffer's contents and ONNX Runtime does not write into caller-owned
input tensors, so this removes roughly a 6 MB memcpy per frame.
The merged path degrades the signal that steers the vehicle: frozen
attention costs lane agreement against the CPU reference, measured at
0.653 / 0.905 / 0.954. That is why the path is opt-in and why the
startup gate logs the active attn_mode and warns whenever it is anything
other than keep -- including when it is unset, which is treated as
unknown rather than safe.
Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The Renesas execution provider for the R-Car X5H NPU permits one NPU session per process. VisionPilot runs three ONNX models — AutoDrive, AutoSteer, AutoSpeed — each owning its own
Ort::Session, so under that constraint only one of the three can ride the NPU and the other two fall back to the CPU silently, costing roughly an order of magnitude in latency.Work in openadkit resolves this outside VisionPilot by composing the three models into one graph and rewriting it to be fully NPU-legal, emitting a sidecar
contract.jsonthat describes the postprocessing it moved to the host. This PR adds the consuming side: VisionPilot can load a merged model, target the Renesas EP, and perform the host-side postprocessing the rewritten graph requires.What this adds
ModelBackendinterface replacing the hard-coded three-session dispatch inInferencePipeline.SplitBackendis today's behaviour lifted verbatim;MergedBackendruns one session with three inputs.renesasprovider inOnnxEnginethat resolves a compiled artifacts directory into a model path plus provider options.engine.require_npu_nodesoptionally pins the expected count so a recompile that sheds subgraphs to the CPU fails at startup rather than quietly costing latency.AutoSpeed::post_processextracted into a freedecode_detections()with an explicitcls_is_probabilityflag, because the rewritten graph emits the class tensor already sigmoided and applying sigmoid twice would compress every score toward 0.5.Existing behaviour is unchanged
model.mergeddefaults to false, soprovider = cpu | cuda | tensorrtcontinues to construct three sessions and behave exactly as it does today. This was verified by building a standalone probe against the built libraries and running the literal pre-branchvision_pilot.confthrough it: three separate sessions with their own startup banners, and a byte-identical[OnnxEngine] provider=cuda device=0line. TheSplitBackendlift was also compared statement by statement against the removed block — samestd::launch::asyncpolicy, same future creation andget()order, same clock placement, same arena-shrink configuration.One deliberate removal on the split path:
curr_01_asp, a redundant per-frame copy ofcurr_01_as, is gone. Both networks already shared that buffer's contents and ONNX Runtime does not write into caller-owned input tensors, so this removes roughly a 6 MBmemcpyper frame for every provider.Testing
130 tests, all passing, zero build warnings, verified on this branch with
docker build --target tests -f docker/Dockerfile.cpu .. The contract parsing, all three postprocessing rules, the detection decoder, the artifact resolution, the profile parser, the offload decision logic, the startup validation refusals, and the backend selection are all unit-tested against plain data — no NPU, no vendor wheel, and no model file required. That coverage exists because every validation decision was deliberately extracted out of the session-holding classes into free functions.The repo had no unit-test target for
modules/models, so this adds one:tests/models/models_tests(gtest), wired in behind the existingBUILD_TESTINGoption, which still defaults toOFF;docker/Dockerfile.cpugains atestsstage that builds and runs it.The split-versus-merged equivalence test is included but CMake-gated and skipped when the asset is absent: the merged ONNX is roughly 147 MB and is not in this repository, matching how the existing model weights are handled.
What is not verified here
The merged path's own numerics need the board and the vendor EP; they stay verified by openadkit's consistency check and final-check runs. Nothing in CI exercises the Renesas provider end to end.
The merged path degrades the signal that steers the vehicle. Frozen attention costs lane agreement against the CPU reference, measured at 0.653 / 0.905 / 0.954. That is why the path is opt-in and why the startup gate logs the active
attn_modeand warns explicitly whenever it is anything other thankeep— including when it is unset, which is treated as unknown rather than safe. Whoever enablesmodel.mergedon a vehicle should read that figure first.Configuration
All new keys are additive with safe defaults and are documented in
config/vision_pilot.conf.model.merged = trueis required whenprovider = renesasand refused loudly at startup otherwise, because a split backend there would place two of three models on exactly the silent CPU fallback this work exists to remove.Relation to the other two PRs
Last of three independent changes, and the only one that is X5H-specific — the other two are provider-agnostic CPU-side wins, which is why they are suggested first. This branch builds and passes on
mainas it stands. It touches the sameInferencePipeline::process()block as the input-tensor-fusion PR and adds the sametests/models+Dockerfile.cputest harness, so whichever of the two lands second needs a mechanical rebase; no logic overlaps.Note on
spell-check-differentialThe check is red, and none of it comes from this branch.
spell-check-differentialscans every changed file in full, so it reports words that already live in those files onmain—Ipopt,libnice,CPACK,cipo,latc,Matx,dets, and the forbiddenROS2among them. The same check has been failing onmaininspell-check-dailyevery night for at least a week, and the last two merged PRs (#408, #411) merged red on it too. Every word this branch actually introduces is either already in the shared dictionary or added to.cspell.jsonhere, verified locally with cspell against the repo config.