trtc: declarative TensorRT compilation with a remote builder - #7
Open
wokalski wants to merge 8 commits into
Open
trtc: declarative TensorRT compilation with a remote builder#7wokalski wants to merge 8 commits into
wokalski wants to merge 8 commits into
Conversation
wokalski
force-pushed
the
tensorrt-util
branch
2 times, most recently
from
July 2, 2026 18:22
9e96350 to
6937429
Compare
One Bundle declaration per model drives all three stages: export (project env, exact locked torch) -> ONNX + plan.json; build (target GPU, TensorRT matching the plan's uv.lock-derived pin; no torch, no model code) -> engines + manifest.json; serve (manifest validated against TRT version and compute capability before any engine loads). The builder (trtc serve) is a deliberately dumb HTTP job broker: one job = one ONNX + query params = one engine back; multi-component models are composed client-side. It runs `trtc build` in its own fixed environment — no per-job dependency resolution, a wrong TensorRT fails the job loudly. Bare .onnx targets work without a bundle or plan file. `trtc launch` rents a vast.ai GPU running the builder image and prints TRTC_BUILDER=<url>. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…a.com trtc-builder workspace member pins the builder image's venv (trtc + the locked TensorRT) like a derivation. pypi.org only carries a wheel_stub sdist for tensorrt-cu12-libs that downloads the real wheel at build time — anti-hermetic and broken inside image derivations — so the nvidia index supplies real hashed wheels via the lock. tinfer[inference] gains trtc for the engine runtime. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ript trt_bundle.py declares what compiles (components, coupled dynamic axes, export-mode graph rewrites, model.pth metadata finalize); the export path loads with runtime_engine="pytorch" so re-compiles never validate a prior build's engines against the wrong GPU. tensorrt_export.py keeps only the model-owned adapters (ONNX iSTFT, unrolled diffusion sampler); tensorrt_runtime.py keeps engine naming + shape buckets over trtc's generic manifest-validating runner. compile_converted_model.py is superseded by `trtc compile`. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
trtc-builder is a flake image target (x2container, like tinfer-server): the fixed builder venv plus the gpuContainer contract shared by every GPU image — host driver provides libcuda; both container-toolkit injection paths on the search path after nix libs (the nix loader never reads the container's ld.so.cache); NVIDIA_VISIBLE_DEVICES/DRIVER_CAPABILITIES markers. Only libstdc++ from nix on the builder's path so host-injected FHS binaries (nvidia-smi) never resolve a foreign libc. Unlike the serving image, the builder keeps tensorrt's engine-builder payload — it is the thing that builds. CI pushes it alongside tinfer, tagged trt<major.minor> from the lock so a shared registry holds one image per TensorRT version and clients select by their plan's pin. launch-builder wraps `trtc launch` with a prebuilt env (trtc resolved at build time, vastai from the vast-cli input). x2container bump: member-subset images now carry every workspace member's pyproject so `uv export --locked` validates. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The base-less image cannot exec host-injected FHS binaries, so nvidia-smi returned nothing and manifests carried null GPU facts (silently skipping the runtime's compute-capability check). query_gpu now ctypes-binds libcuda.so.1 — the same host-injected library the engine build itself binds, found via the same search path — for device name and compute capability, and reads the kernel module version from /proc/driver/nvidia/version. No subprocess; degrades to None off-GPU. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
trtc is generic infrastructure, now shared across projects from https://github.com/dialohq/trtc (same workspace layout, imported at 6937429). tinfer[inference] pulls it via a uv git source pinned in the lock. Builder images ship from that repo's CI as ghcr.io/dialohq/trtc-builder:trt<major.minor> (one per supported TensorRT version via a workflow matrix); rent one with `nix run github:dialohq/trtc#launch-builder`. The tinfer flake keeps only the shared gpuContainer contract for the serving image, and docker.yml returns to pushing tinfer alone. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The tinfer image build failed fetching the trtc git dependency — uv shells out to git, absent from x2container's env derivations. Fixed upstream (dialohq/x2container.nix@8fc9a63, gitMinimal in nativeBuildInputs). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
New workspace member
trtc: oneBundledeclaration per model drives export (torch→ONNX+plan, locally in the project env), build (ONNX→engine on deployment-class hardware) and serve (manifest-validated engine loading). Seetrtc/README.md. Four commits, linear on main.uv.lock(tomllib parse; currently 11.1.0.106) into the plan, the builder image tag (trt<major.minor>), and the manifest the runtime validates against. TensorRT wheels resolve hermetically from pypi.nvidia.com (real hashed wheels, no wheel_stub phone-home at build time).trtc buildin its own fixed venv (trtc-builder member = trtc + locked TensorRT) — no per-job resolution, wrong TRT fails loudly. One image per TRT version in a shared registry.gpuContainerin the flake): host driver provides libcuda; both container-toolkit injection paths on the search path after nix libs; NVIDIA_* markers — consumed by tinfer-server and trtc-builder alike.trt_bundle.pydeclares components/axes/rewrites/finalize;tensorrt_export.pyshrinks to adapters;tensorrt_runtime.pyto engine naming + buckets over trtc's generic runner (slots under main'sStyleTTS2TensorRTAcceleratorunchanged). Replacescompile_converted_model.py.nix run .#launch-builder(wraps the Pythontrtc launch) rents a vast.ai GPU on the version-matched image and printsTRTC_BUILDER=<url>.Verification
trtc/tests): spec→profile derivation matches the deleted profile functions exactly, lock-pin resolution, path-traversal rejection, full HTTP round trip against a live server.trtc-builder:trt11.1from this PR's CI): HF LJSpeech checkpoint → convert → local CPU export → per-component REST jobs → decoder engine built in 72.6s, diffusion in 34.8s on TRT 11.1 → engines + assembled manifest downloaded →model.pthruntime_config stamped. An earlier round also validated loading and running both engines through the tinfer runtime.Known follow-up: the no-base image can't exec host-injected
nvidia-smi(no FHS loader), so manifest GPU facts (compute_capability) are currently null and the runtime's arch check is skipped — GPU facts should come from the CUDA API instead.🤖 Generated with Claude Code
https://claude.ai/code/session_01YUXNbBVdgPJaraZEz8y6nr