diff --git a/optimized/tensorRT/README.md b/optimized/tensorRT/README.md index 8d552261..304e18a2 100644 --- a/optimized/tensorRT/README.md +++ b/optimized/tensorRT/README.md @@ -119,30 +119,32 @@ Omit `--dit` / `--decoder` for an interactive arrow-key picker. Relative ### DiT precision (`--precision`) -`fp16mixed` is the default for every model — no flag needed for the recommended setup: +`fp16` is the default for every model — no flag needed for the recommended setup: | model | default | why | |------------------|-------------|------------------------------------------------------------| -| `medium` | `fp16mixed` | FMHA-fused (96 fused attention nodes) **and** fp32-accurate at every length | -| `sm-music`/`sm-sfx` | `fp16mixed` | standard attention — already fuses in fp16-mixed | - -`--precision` also takes `fp8` (all DiTs), `bf16` (medium only) and `fp32` explicitly: - -- **`fp16mixed`** — canonical: FP16 trunk, FP32 islands around RMSNorm and RoPE - generation, and an FP16 attention core (QK^T → Softmax → P·V) so TRT's FMHA fuser - fires. Teacher-forced velocity cos **1.0000** vs the FP32 engine at every length. - Bit-reproducible run to run. -
Medium used to default to `bf16` because this engine's attention core was - stuck in FP32 and therefore unfused — 0 fused MHA nodes. Bounding the RoPE island - before QK^T fixed that (**4.3× faster at L=4096**), which retired the reason to - prefer `bf16`. Engines built before 2026-07 are the slow variant; rebuild with +| `medium` | `fp16` | FMHA-fused (96 fused attention nodes) **and** fp32-accurate at every length | +| `sm-music`/`sm-sfx` | `fp16` | standard attention — already fuses in fp16 | + +`--precision` also takes `fp8` (all DiTs) and `fp32` explicitly: + +- **`fp16`** — canonical **and default for every DiT** (formerly `fp16mixed`; every + tier is mixed-precision, so the qualifier was dropped). FP16 trunk, FP32 islands + around RMSNorm and RoPE generation, and an FP16 attention core (QK^T → Softmax → + P·V) so TRT's FMHA fuser fires. Teacher-forced velocity cos **1.0000** vs the FP32 + engine at every length. Bit-reproducible run to run. +
Medium used to default to a uniform-`bf16` engine because this engine's + attention core was stuck in FP32 and therefore unfused — 0 fused MHA nodes. + Bounding the RoPE island before QK^T fixed that (**4.3× faster at L=4096**), which + retired the `bf16` tier entirely (it also drifted at long sequence — see below). + Engines built before 2026-07 are the slow variant; rebuild with `build_from_onnx.py sa3-m`. - **`fp8`** — *medium: the max-speed clean tier, calibrated; sm-music/sm-sfx: a clean weight-halving tier (see the end of this bullet).* On **medium**: fp8 E4M3 on the 176 linear GEMMs + bf16 fused FMHA (96 nodes) + a **baked fp32 RoPE constant table** (position cos/sin computed host-side at build and frozen as a graph constant — no in-graph trig, so the island is precision-policy-robust and cross-runtime-stable). **~1.3× faster than - `fp16mixed` at every length** (H200, same-run round-robin: 1.40× @L129 / 1.32× @L1292 / + `fp16` at every length** (H200, same-run round-robin: 1.40× @L129 / 1.32× @L1292 / 1.34× @L4092 — also ahead of `bf16`) and **clean at long sequence** (latent std 0.86 vs eager 0.95, **0.000% clip** at 2-min and 6-min), so it stays clean exactly where `bf16` clips. The fp8 scales are **calibrated on real conditioning** (per-tensor activation amax + @@ -151,41 +153,37 @@ Omit `--dit` / `--decoder` for an interactive arrow-key picker. Relative velocity-cos vs fp32 on adversarial seeds **0.52/0.57/0.64 → 0.92/0.94/0.92** (steps 1–7 match the fully-calibrated reference within ~0.001). It remains a **speed tier over an already-good default, not a fidelity upgrade over it**: single-step velocity cos ~0.92–0.94 - (below fp16mixed's ~1.0) but the 8-step render stays coherent — it just no longer collapses + (below fp16's ~1.0) but the 8-step render stays coherent — it just no longer collapses at the highest-noise first step the way the uncalibrated engine did. **Capped at L≤4096** — the baked table is sized to the profile max, which is *also* the SAME-L decoder's own cap, so this is not a new end-to-end limit; longer renders are rejected (a re-bake would be - needed). **Not seed-reproducible vs fp16-mixed.** Built weakly-typed `EXPLICIT_BATCH` + + needed). **Not seed-reproducible vs fp16.** Built weakly-typed `EXPLICIT_BATCH` + `BF16` + `FP8` + `OBEY_PRECISION_CONSTRAINTS` (rebuild: `build_from_onnx.py sa3-m-fp8`; producer: `build/build_dit_bf16.py` RoPE-baker + `build/transplant_scales.py` calibrated-scale transplant; identity check: `scripts/verify_fp8_rope.py`; calibration by @ryanontheinside, [#47](https://github.com/Stability-AI/stable-audio-3/pull/47)).
On **sm-music / sm-sfx** fp8 is a **different, simpler recipe** — fp8 E4M3 grafted onto - the linear GEMMs of the fp16mixed graph (attention stays fp16-fused, fp32 islands untouched; + the linear GEMMs of the fp16 graph (attention stays fp16-fused, fp32 islands untouched; no baked RoPE — these DiTs never had bf16's long-angle problem). It's a **clean weight-halving - tier** (engine 479 vs 936 MB, velocity-cos **~0.99** vs eager, clip% at/below fp16mixed) that + tier** (engine 479 vs 936 MB, velocity-cos **~0.99** vs eager, clip% at/below fp16) that is only **marginally faster (~1.1×)**: a small DiT's ~5 ms forward at batch 1 is overhead-bound, - so fp8's GEMM savings barely show. Default stays fp16mixed. Rebuild: `build_from_onnx.py + so fp8's GEMM savings barely show. Default stays fp16. Rebuild: `build_from_onnx.py sa3-sm-music-fp8` / `sa3-sm-sfx-fp8`; producer: `build/make_dit_fp8_smalldit.py`. Not - seed-reproducible vs fp16mixed. -- **`bf16`** — *medium only.* Same `dit.onnx` as fp32, built with `BuilderFlag.BF16`; - a uniform bf16 trunk also lets the FMHA fuser fire, and it is ~3% faster than - `fp16mixed`. **But it drifts at long sequence**: weakly-typed BF16 lets TRT - evaluate RoPE's rotation angle in bf16, and that angle reaches ~4155 rad at - L=4092 where bf16's spacing is 32 rad (bigger than a full 2π rotation), so - position information for the fast-rotating dims is destroyed. The latent inflates - ~2.5× over the 8 steps and a 6-minute render clips 2–3% of samples. Clean at short - lengths; use `fp16mixed` for anything long. Also **not seed-reproducible vs - fp16-mixed** — a different-but-equal take per seed. Same varlen profile - (L 1..4096, opt 1292) and full mode/feature set as the other precisions. + seed-reproducible vs fp16. - **`fp32`** — pure FP32, bit-equivalent to PyTorch eager (~2× slower, ~2× VRAM). +> **Retired: `bf16` (was medium-only).** A uniform-bf16 trunk was ~3% faster than +> `fp16` but **drifted at long sequence** — weakly-typed BF16 evaluated RoPE's +> rotation angle in bf16, which reaches ~4155 rad at L=4092 where bf16's spacing is +> 32 rad (> 2π), destroying position info for the fast-rotating dims (latent inflates +> ~2.5×, a 6-min render clips 2–3% of samples). `fp16` matches its FMHA fusion +> without the drift, so `bf16` was removed. `--precision bf16` (and the old +> `--precision fp16mixed`) now silently alias to `fp16`. + ```bash -# medium defaults to fp16mixed. fp8 is the max-speed clean tier (~1.3x faster at -# every length, clean at long L where bf16 clips): +# medium defaults to fp16. fp8 is the max-speed clean tier (~1.3x faster at +# every length, clean at long sequence): ./sa3 --prompt "..." --dit medium --decoder same-l --precision fp8 -# bf16 trades long-sequence accuracy for the last ~3% of speed: -./sa3 --prompt "..." --dit medium --decoder same-l --precision bf16 ``` The TRT DiT engines are static batch=1 (the ONNX bakes batch=1), so CFG runs as a @@ -225,7 +223,7 @@ variance once the graph is built). ```bash .venv/bin/python scripts/bench_dit_profile.py \ - --engines "canonical=models/sm_90/sa3-sm-music/dit_fp16mixed.trt" \ + --engines "canonical=models/sm_90/sa3-sm-music/dit_fp16.trt" \ --lvals 1,32,128,256,512,1024,1292,2048,4096 --warmup 3 --runs 7 ``` @@ -275,10 +273,10 @@ optimized/tensorRT/ │ └── build_dit_profile.py ← DiT with custom (min, opt, max) profile shapes └── models/ ← .trt engines (auto-downloaded per arch; ~8 GB) └── sm_/ ← arch dir matches `nvidia-smi --query-gpu=compute_cap` - ├── t5gemma/t5gemma_fp16mixed.trt - ├── sa3-sm-music/dit_fp16mixed.trt - ├── sa3-sm-sfx/dit_fp16mixed.trt - ├── sa3-m/dit_fp16mixed.trt ← + dit_fp8.trt / dit_bf16.trt / dit_fp32.trt if selected + ├── t5gemma/t5gemma_fp16.trt + ├── sa3-sm-music/dit_fp16.trt + ├── sa3-sm-sfx/dit_fp16.trt + ├── sa3-m/dit_fp16.trt ← + dit_fp8.trt / dit_fp32.trt if selected ├── same-s/{enc,dec}_dynamic_bf16.trt └── same-l/{enc,dec}_dynamic_triton_swa.trt ``` diff --git a/optimized/tensorRT/build/README.md b/optimized/tensorRT/build/README.md index de5b1af5..38670ea3 100644 --- a/optimized/tensorRT/build/README.md +++ b/optimized/tensorRT/build/README.md @@ -13,18 +13,18 @@ Run the build on the target GPU; TensorRT bakes the arch into the engine, so the ``` consumer flow producer flow ───────────── ───────────── -HuggingFace onnx// ←─────── publish (incl. dit_fp16mixed.onnx) +HuggingFace onnx// ←─────── publish (incl. dit_fp16.onnx) tensorRT// ←─── compile + commit source ckpts │ │ ↓ ↓ build.py build_*.py build_from_onnx.py (build_t5gemma.py, (just compile, build_dit.py, - STRONGLY_TYPED; build_dit_fp16mixed.py, + STRONGLY_TYPED; build_dit_fp16.py, no graphsurgeon) build_same_*.py) ``` -The SA3 DiT ships both an FP32 canonical `dit.onnx` (regenerable from PyTorch source) and a pre-processed `dit_fp16mixed.onnx` (canonical + FP32 islands around RMSNorm and RoPE *generation*, FP16 attention core, rest converted to FP16). Consumers use the pre-processed one; producers refresh both when the model retrains. +The SA3 DiT ships both an FP32 canonical `dit.onnx` (regenerable from PyTorch source) and a pre-processed `dit_fp16.onnx` (canonical + FP32 islands around RMSNorm and RoPE *generation*, FP16 attention core, rest converted to FP16). Consumers use the pre-processed one; producers refresh both when the model retrains. ## Consumer flow (default) @@ -46,7 +46,7 @@ python build.py # interactive menu Output dir: models/sm_100/ [1] ✓ t5gemma (text encoder + tokenizer) - ✓ t5gemma/t5gemma_fp16mixed.trt 538.1 MB + ✓ t5gemma/t5gemma_fp16.trt 538.1 MB ✓ t5gemma/tokenizer.json 32.8 MB [2] ✗ same-s encoder ✗ same-s/enc_dynamic_bf16.trt (missing) @@ -223,17 +223,17 @@ python build_dit.py sa3-sm-sfx python build_dit.py sa3-m ``` -After the DiT ONNXes are exported, run the FP16-mixed precision-island surgery on each one (see `build_dit_fp16mixed.py`): +After the DiT ONNXes are exported, run the FP16-mixed precision-island surgery on each one (see `build_dit_fp16.py`): ```bash -python build_dit_fp16mixed.py \ +python build_dit_fp16.py \ --input /onnx/sa3-sm-music/dit.onnx \ - --onnx /onnx/sa3-sm-music/dit_fp16mixed.onnx \ - --engine ../models//sa3-sm-music/dit_fp16mixed.trt + --onnx /onnx/sa3-sm-music/dit_fp16.onnx \ + --engine ../models//sa3-sm-music/dit_fp16.trt # repeat for sa3-sm-sfx and sa3-m ``` -This wraps every RMSNorm chain, attention `Softmax`, and the RoPE region in `Cast(FP32) → op → Cast(FP16)` islands, converts the rest of the weights to FP16, then **bounds the RoPE island before QK^T** (`bound_attention_core()`) so the attention core — QK^T → Softmax → P·V — runs FP16, and finally compiles a `STRONGLY_TYPED` TRT engine. It writes BOTH the modified `dit_fp16mixed.onnx` (~half the size of the original) AND the TRT engine. Publishing the modified ONNX is what lets consumers compile their own engines with plain `build_from_onnx.py` (no `onnx-graphsurgeon` dependency on the consumer side). +This wraps every RMSNorm chain, attention `Softmax`, and the RoPE region in `Cast(FP32) → op → Cast(FP16)` islands, converts the rest of the weights to FP16, then **bounds the RoPE island before QK^T** (`bound_attention_core()`) so the attention core — QK^T → Softmax → P·V — runs FP16, and finally compiles a `STRONGLY_TYPED` TRT engine. It writes BOTH the modified `dit_fp16.onnx` (~half the size of the original) AND the TRT engine. Publishing the modified ONNX is what lets consumers compile their own engines with plain `build_from_onnx.py` (no `onnx-graphsurgeon` dependency on the consumer side). The attention-core step is not cosmetic. Without it the RoPE island emits q/k in FP32 and nothing casts them back, so the whole O(L²) core stays FP32 and TRT's FMHA fuser cannot fire: **0 fused MHA nodes and 4.3× slower at L=4096** on the medium DiT, for no accuracy gain (teacher-forced velocity cos vs the FP32 engine is 1.0000 with the step, 0.9998 without). PyTorch does the same thing the step does — `apply_rotary_pos_emb` ends with `t.to(out_dtype)` and attention is then a fused `scaled_dot_product_attention` over FP16 inputs — and TRT's FMHA kernels accumulate softmax in FP32 internally anyway. `--no-bound-attn` skips it, only useful for reproducing the pre-2026-07 engines. @@ -253,12 +253,13 @@ git push ## Medium `fp8` — the max-speed RoPE-baked engine, calibrated -On top of the `fp16mixed` default and the selectable `bf16`, the medium DiT ships an **`fp8`** +On top of the `fp16` default (the medium-only `bf16` tier was retired — its uniform-bf16 +build survives only internally as the fp8 RoPE-baker), the medium DiT ships an **`fp8`** engine (`dit_fp8.trt`, `--precision fp8`): fp8 E4M3 on the 176 linear GEMMs + 96 bf16 fused -FMHA + the **same baked fp32 RoPE constant table** as bf16. Measured on H200 it is ~1.3× -faster than `fp16mixed` at every length (1.40× / 1.32× / 1.34× @L129/1292/4092, also ahead of -bf16) and clean at long sequence (latent std 0.86, 0.000% clip at 6-min), so it stays clean -exactly where bf16 clips. +FMHA + the **same baked fp32 RoPE constant table**. Measured on H200 it is ~1.3× +faster than `fp16` at every length (1.40× / 1.32× / 1.34× @L129/1292/4092) and clean at +long sequence (latent std 0.86, 0.000% clip at 6-min), so it stays clean exactly where the +old bf16 tier clipped. The fp8 scales are **calibrated on real conditioning** (updated 2026-07-31). The scale VALUES — per-tensor activation amax + per-channel weight scales — come from @ryanontheinside's fp8 work @@ -267,8 +268,8 @@ and grafted onto this baked / bf16-fused structure by `transplant_scales.py`. Ca **speed-free** (31.2 vs the earlier uncalibrated 30.8 ms/fwd, inside run noise) and lifts worst-case per-step fidelity — worst-step velocity-cos vs fp32 on adversarial seeds goes **0.52/0.57/0.64 → 0.92/0.94/0.92**, with sampling steps 1–7 tracking the fully-calibrated #47 -reference within ~0.001. It is still a **speed tier over the `fp16mixed` default, not a fidelity -upgrade over it** (single-step velocity cos ~0.92–0.94 < fp16mixed's ~1.0; the 8-step render +reference within ~0.001. It is still a **speed tier over the `fp16` default, not a fidelity +upgrade over it** (single-step velocity cos ~0.92–0.94 < fp16's ~1.0; the 8-step render stays coherent), but it no longer collapses at the highest-noise first step the way the uncalibrated engine did. See the runtime README's precision section for positioning. @@ -329,21 +330,21 @@ merged here). Everyday consumers never recalibrate — they pull the published c `sm-music` and `sm-sfx` also ship a selectable **`fp8`** engine (`--precision fp8`), but it is **not** the medium's baked-RoPE recipe — those DiTs use standard (non-differential) attention and never had the bf16 long-angle RoPE problem, so there is nothing to bake. Their fp8 is a straight -**graft of fp8 E4M3 Q/DQ onto the linear GEMMs of the known-good `dit_fp16mixed.onnx`** — attention -stays fp16-fused and the fp32 RMSNorm/RoPE islands are left exactly as the fp16mixed producer made +**graft of fp8 E4M3 Q/DQ onto the linear GEMMs of the known-good `dit_fp16.onnx`** — attention +stays fp16-fused and the fp32 RMSNorm/RoPE islands are left exactly as the fp16 producer made them. Built `STRONGLY_TYPED` (`build_from_onnx.py sa3-sm-music-fp8` / `sa3-sm-sfx-fp8`); the QDQ -carry the precision. Identity: 186 fp8 GEMMs + fp16 fused attention + the fp16mixed fp32 islands. +carry the precision. Identity: 186 fp8 GEMMs + fp16 fused attention + the fp16 fp32 islands. Positioning is honest: this is a **clean weight-halving tier** (engine 479 vs 936 MB, velocity-cos -~0.99 vs eager, clip% at or below fp16mixed), only **marginally faster** (~1.10–1.17×) — a small +~0.99 vs eager, clip% at or below fp16), only **marginally faster** (~1.10–1.17×) — a small DiT's ~5 ms forward at batch 1 is overhead-bound, so fp8's GEMM-math savings barely show. Default -stays `fp16mixed`; fp8 is for when the smaller engine / weight footprint helps. Not seed-reproducible -vs fp16mixed. +stays `fp16`; fp8 is for when the smaller engine / weight footprint helps. Not seed-reproducible +vs fp16. > ⚠ Do **not** produce these with `build_dit_fp8.py` (#47's ModelOpt path): on the small graphs its > island-flatten + reapply does not restore the fp32 islands correctly and the engine collapses to > velocity-cos ~0.69 with clipping (the GEMMs are fine — it's the islands). Grafting onto the -> fp16mixed ONNX keeps the islands correct by construction. +> fp16 ONNX keeps the islands correct by construction. **Producer (refresh the ONNX).** `make_dit_fp8_smalldit.py` calibrates per-linear activation scales from the eager model (own-domain few-shot prompts + one full render) and grafts the fp8 Q/DQ. Two @@ -353,7 +354,7 @@ fp16) and floored at 1e-4 (fp16 underflows tiny scales to 0, which TRT rejects): ```bash python make_dit_fp8_smalldit.py \ --model-config /model_config.json --checkpoint /model.safetensors \ - --fp16mixed-onnx onnx/sa3-sm-music/dit_fp16mixed.onnx \ + --fp16-onnx onnx/sa3-sm-music/dit_fp16.onnx \ --domain Music --out onnx/sa3-sm-music/dit_fp8.onnx # --domain SFX for sm-sfx ``` @@ -362,9 +363,9 @@ python make_dit_fp8_smalldit.py \ | File | Role | Flow | |---|---|---| | `build.py` | Interactive menu (default entry point) | consumer | -| `build_from_onnx.py` | One target → download ONNX from HF + compile to TRT. **For the SA3 DiTs, pulls `dit_fp16mixed.onnx` (the pre-processed island-wrapped graph)** so the consumer just needs to invoke `STRONGLY_TYPED` compilation — no `onnx-graphsurgeon` required | consumer | +| `build_from_onnx.py` | One target → download ONNX from HF + compile to TRT. **For the SA3 DiTs, pulls `dit_fp16.onnx` (the pre-processed island-wrapped graph)** so the consumer just needs to invoke `STRONGLY_TYPED` compilation — no `onnx-graphsurgeon` required | consumer | | `build_dit_profile.py` | Build a DiT with custom `(min, opt, max)` profile shapes (experimental — short-form / fixed-shape variants). Operates on either ONNX flavor. | consumer | -| `build_dit_fp16mixed.py` | **Producer-side** ONNX surgery: takes the canonical FP32 `dit.onnx`, finds RMSNorm chains + attention `Softmax` + RoPE region, wraps each in `Cast(FP32) ↔ Cast(FP16)` islands, converts non-island weights to FP16, then bounds the RoPE island before QK^T (`bound_attention_core()`, `--no-bound-attn` to skip) so the attention core runs FP16 and TRT's FMHA fuser fires — 96/96 attentions on the medium DiT, 4.3× at L=4096. Writes both the modified `dit_fp16mixed.onnx` AND the TRT engine, which **must** be `STRONGLY_TYPED` (weakly-typed + `BuilderFlag.FP16` re-casts the FP32 islands and silently degrades to naive FP16). Only re-run when the model retrains or the island recipe changes. Requires `onnx` + `onnx-graphsurgeon`. | producer | +| `build_dit_fp16.py` | **Producer-side** ONNX surgery: takes the canonical FP32 `dit.onnx`, finds RMSNorm chains + attention `Softmax` + RoPE region, wraps each in `Cast(FP32) ↔ Cast(FP16)` islands, converts non-island weights to FP16, then bounds the RoPE island before QK^T (`bound_attention_core()`, `--no-bound-attn` to skip) so the attention core runs FP16 and TRT's FMHA fuser fires — 96/96 attentions on the medium DiT, 4.3× at L=4096. Writes both the modified `dit_fp16.onnx` AND the TRT engine, which **must** be `STRONGLY_TYPED` (weakly-typed + `BuilderFlag.FP16` re-casts the FP32 islands and silently degrades to naive FP16). Only re-run when the model retrains or the island recipe changes. Requires `onnx` + `onnx-graphsurgeon`. | producer | | `build_dit_bf16.py` | **Producer-side** shared RoPE-baker for the medium `bf16` AND `fp8` engines: precomputes RoPE's cos/sin in fp64 on the host, freezes them as fp32 constant tables (`--max-t`), rewires the 96 trig sites and lets DCE delete the runtime angle chain — so the trunk runs bf16/fp8 without the long-angle drift. Weights are never loaded (keeps the input's `.data` sidecar). Handles both external `inv_freq` (fp32 `dit.onnx`) and inline (fp8-linear ONNX). Consumer compile: `build_from_onnx.py sa3-m-bf16` / `sa3-m-fp8`. Requires `onnx`. | producer | | `make_calib.py` | **Producer-side** FP8 calibration capture: drives the model's own pingpong `generate()` and records the six DiT engine inputs at every sampling step into a `.npz` (real-conditioning, deployment-matched prompts). Feeds `build_dit_fp8.py` (#47). Only re-run for a from-scratch recalibration on a model retrain. Calibration tooling by @ryanontheinside (#47). Requires `torch` + `stable_audio_3`. | producer | | `transplant_scales.py` | **Producer-side** fp8 scale transplant: grafts #47's calibrated activation + per-channel weight scales onto the RoPE-baked bakedmin ONNX (matches Linears by weight-initializer name, swaps only the scale VALUES; weights untouched). This is what makes the shipped `dit_fp8.onnx` calibrated while keeping bakedmin's bf16 fused-MHA speed. Scale values / calibration by @ryanontheinside (#47). Requires `onnx`. | producer | @@ -374,7 +375,7 @@ python make_dit_fp8_smalldit.py \ | `build_same_s_encoder.py` | Trace + export SAME-S encoder ONNX + build TRT | producer | | `build_same_l_decoder.py` | Trace + export SAME-L decoder ONNX (Triton SWA) + build TRT | producer | | `build_same_l_encoder.py` | Trace + export SAME-L encoder ONNX (Triton SWA) + build TRT | producer | -| `build_dit.py ` | Trace + export DiT FP32 ONNX (cond baked in) + build TRT BF16 engine (legacy; the BF16 output isn't suitable for inference — chain it with `build_dit_fp16mixed.py` afterwards) | producer | +| `build_dit.py ` | Trace + export DiT FP32 ONNX (cond baked in) + build TRT BF16 engine (legacy; the BF16 output isn't suitable for inference — chain it with `build_dit_fp16.py` afterwards) | producer | | `_arch.py` | Shared: GPU arch detection + path helpers | both | | `samel_loader.py` | Helper: load SAME-L from .ckpt | producer | | `samel_{encoder,decoder}_onnx.py` | Helper: clean ONNX rewrites of SAME-L blocks | producer | diff --git a/optimized/tensorRT/build/build.py b/optimized/tensorRT/build/build.py index d27a495e..6c5ee525 100755 --- a/optimized/tensorRT/build/build.py +++ b/optimized/tensorRT/build/build.py @@ -52,7 +52,7 @@ def _from_onnx(name): TARGETS = [ {"label": "t5gemma (text encoder + tokenizer)", "command": _from_onnx("t5gemma"), - "outputs": ["t5gemma/t5gemma_fp16mixed.trt", "t5gemma/tokenizer.json"]}, + "outputs": ["t5gemma/t5gemma_fp16.trt", "t5gemma/tokenizer.json"]}, {"label": "same-s encoder", "command": _from_onnx("same-s-encoder"), "outputs": ["same-s/enc_dynamic_bf16.trt"]}, @@ -84,19 +84,19 @@ def _from_onnx(name): # spacing is 32 rad — more than a full 2*pi rotation). {"label": "DiT medium (SA3-M, FP16-mixed — medium DEFAULT, attention-fused)", "command": _from_onnx("sa3-m"), - "outputs": ["sa3-m/dit_fp16mixed.trt"]}, + "outputs": ["sa3-m/dit_fp16.trt"]}, {"label": "DiT medium (SA3-M, bf16 — selectable; drifts at long sequence)", "command": _from_onnx("sa3-m-bf16"), "outputs": ["sa3-m/dit_bf16.trt"]}, - {"label": "DiT medium (SA3-M, fp8 — selectable; MAX-SPEED clean tier, ~1.3x over fp16mixed)", + {"label": "DiT medium (SA3-M, fp8 — selectable; MAX-SPEED clean tier, ~1.3x over fp16)", "command": _from_onnx("sa3-m-fp8"), "outputs": ["sa3-m/dit_fp8.trt"]}, {"label": "DiT sm-music (FP16-mixed)", "command": _from_onnx("sa3-sm-music"), - "outputs": ["sa3-sm-music/dit_fp16mixed.trt"]}, + "outputs": ["sa3-sm-music/dit_fp16.trt"]}, {"label": "DiT sm-sfx (FP16-mixed)", "command": _from_onnx("sa3-sm-sfx"), - "outputs": ["sa3-sm-sfx/dit_fp16mixed.trt"]}, + "outputs": ["sa3-sm-sfx/dit_fp16.trt"]}, # FP32 variants — opt-in. ~2× engine size, ~2× slower, but bit-equivalent # to PyTorch eager. Useful for precision-debug or reference comparisons. {"label": "[opt-in] same-l decoder FP32", diff --git a/optimized/tensorRT/build/build_dit_fp16mixed.py b/optimized/tensorRT/build/build_dit_fp16.py similarity index 99% rename from optimized/tensorRT/build/build_dit_fp16mixed.py rename to optimized/tensorRT/build/build_dit_fp16.py index 2af6e4a0..af670aac 100644 --- a/optimized/tensorRT/build/build_dit_fp16mixed.py +++ b/optimized/tensorRT/build/build_dit_fp16.py @@ -61,11 +61,11 @@ - Engine size: 926 MB (BF16: 935 MB, FP32: 1842 MB) Usage: - python build_dit_fp16mixed.py + python build_dit_fp16.py [--mode {minimal,rope,full}] # default: rope [--input /tmp/dit_sm-music_fixed_v2.onnx] - [--onnx /tmp/dit_sm-music_fp16mixed.onnx] - [--engine .../models/sm_90/sa3-sm-music/dit_fp16mixed.trt] + [--onnx /tmp/dit_sm-music_fp16.onnx] + [--engine .../models/sm_90/sa3-sm-music/dit_fp16.trt] """ import argparse import os @@ -1001,7 +1001,7 @@ def constant_feeds_island_only(consumers): return model -def convert_to_fp16mixed(input_onnx, output_onnx, mode="minimal", bound_attn=True): +def convert_to_fp16(input_onnx, output_onnx, mode="minimal", bound_attn=True): """Load FP32 ONNX, identify FP32 islands, convert everything else to FP16, and save.""" import onnx @@ -1142,11 +1142,11 @@ def main(): ap = argparse.ArgumentParser(description=__doc__) ap.add_argument("--input", default="/tmp/dit_sm-music_fixed_v2.onnx", help="Input FP32 ONNX") - ap.add_argument("--onnx", default="/tmp/dit_sm-music_fp16mixed.onnx", + ap.add_argument("--onnx", default="/tmp/dit_sm-music_fp16.onnx", help="Output FP16-mixed ONNX (intermediate)") ap.add_argument("--engine", default="/weka2/cj/clod/sa3s/stable-audio-3/" "optimized/tensorRT/models/sm_90/" - "sa3-sm-music/dit_fp16mixed.trt", + "sa3-sm-music/dit_fp16.trt", help="Output TRT engine path") ap.add_argument("--workspace-gb", type=int, default=16) ap.add_argument("--mode", choices=("minimal", "rope", "full"), @@ -1168,7 +1168,7 @@ def main(): if not args.skip_convert: print(f"━━━ Convert FP32 ONNX -> FP16-mixed ONNX (mode={args.mode}) ━━━") - convert_to_fp16mixed(args.input, args.onnx, mode=args.mode, + convert_to_fp16(args.input, args.onnx, mode=args.mode, bound_attn=not args.no_bound_attn) if not args.skip_build: diff --git a/optimized/tensorRT/build/build_from_onnx.py b/optimized/tensorRT/build/build_from_onnx.py index 2c5d337e..0fd2e1b6 100755 --- a/optimized/tensorRT/build/build_from_onnx.py +++ b/optimized/tensorRT/build/build_from_onnx.py @@ -26,7 +26,7 @@ python build_from_onnx.py same-l-encoder python build_from_onnx.py same-l-decoder # SA3 DiT engines use a different builder (FP16-mixed recipe): - python build_dit_fp16mixed.py --input --engine + python build_dit_fp16.py --input --engine python build_from_onnx.py all # build everything for this arch """ import os @@ -73,7 +73,7 @@ "onnx_hf": ["t5gemma/encoder.onnx"], # tokenizer.json ships bundled with the repo at scripts/tokenizer.json # (arch-agnostic), so we don't fetch it here anymore. - "trt_local": "t5gemma/t5gemma_fp16mixed.trt", + "trt_local": "t5gemma/t5gemma_fp16.trt", "flags": set(), # STRONGLY_TYPED carries the FP16/FP32 dtype hints "network": "STRONGLY_TYPED", "workspace_gb": 8, @@ -121,12 +121,12 @@ "plugin": True, }, # SA3 DiT engines: build from the pre-processed FP16-mixed ONNX hosted on - # HF. The producer (build_dit_fp16mixed.py) does the FP32-island surgery + # HF. The producer (build_dit_fp16.py) does the FP32-island surgery # once and uploads the result; consumers just compile with STRONGLY_TYPED # (no onnx-graphsurgeon dependency). "sa3-sm-music": { - "onnx_hf": ["sa3-sm-music/dit_fp16mixed.onnx"], - "trt_local": "sa3-sm-music/dit_fp16mixed.trt", + "onnx_hf": ["sa3-sm-music/dit_fp16.onnx"], + "trt_local": "sa3-sm-music/dit_fp16.trt", "flags": set(), # STRONGLY_TYPED + ONNX dtypes carry precision "network": "STRONGLY_TYPED", "workspace_gb": 16, @@ -134,25 +134,25 @@ "plugin": False, }, "sa3-sm-sfx": { - "onnx_hf": ["sa3-sm-sfx/dit_fp16mixed.onnx"], - "trt_local": "sa3-sm-sfx/dit_fp16mixed.trt", + "onnx_hf": ["sa3-sm-sfx/dit_fp16.onnx"], + "trt_local": "sa3-sm-sfx/dit_fp16.trt", "flags": set(), "network": "STRONGLY_TYPED", "workspace_gb": 16, "profile": _DIT_PROFILE, "plugin": False, }, - # SA3 small DiTs in fp8 — SELECTABLE (default stays fp16mixed). fp8 E4M3 on the + # SA3 small DiTs in fp8 — SELECTABLE (default stays fp16). fp8 E4M3 on the # 186 linear GEMMs, attention left fp16-fused and the fp32 RMSNorm/RoPE islands - # intact — an fp8-QDQ graft onto the fp16mixed graph (build/make_dit_fp8_smalldit.py), + # intact — an fp8-QDQ graft onto the fp16 graph (build/make_dit_fp8_smalldit.py), # NOT the medium's baked-RoPE recipe (these DiTs never had the bf16 long-angle # problem). Built STRONGLY_TYPED: the QDQ nodes carry fp8; TRT fires fp8 tensor-core # GEMMs on the linears while the fp16 FMHA fuser still runs the attention. Same # _DIT_PROFILE (batch=1, dynamic L∈[1,4096]) → identical CLI/feature surface. # This is a CLEAN WEIGHT-HALVING tier (479 vs 936 MB, velocity-cos ~0.99 vs eager, - # clip% at/below fp16mixed), only marginally faster (~1.1×): the small DiTs' ~5 ms + # clip% at/below fp16), only marginally faster (~1.1×): the small DiTs' ~5 ms # forward is overhead-bound at batch 1, so fp8's GEMM savings barely show. sm-* fp8 - # is NOT seed-reproducible vs fp16mixed. + # is NOT seed-reproducible vs fp16. "sa3-sm-music-fp8": { "onnx_hf": ["sa3-sm-music/dit_fp8.onnx", "sa3-sm-music/dit_fp8.onnx.data"], "trt_local": "sa3-sm-music/dit_fp8.trt", @@ -173,8 +173,8 @@ }, "sa3-m": { # 2.9 GB external-data sidecar travels alongside. - "onnx_hf": ["sa3-m/dit_fp16mixed.onnx", "sa3-m/dit_fp16mixed.onnx.data"], - "trt_local": "sa3-m/dit_fp16mixed.trt", + "onnx_hf": ["sa3-m/dit_fp16.onnx", "sa3-m/dit_fp16.onnx.data"], + "trt_local": "sa3-m/dit_fp16.trt", "flags": set(), "network": "STRONGLY_TYPED", "workspace_gb": 16, @@ -190,7 +190,7 @@ # # The ~1.76×@L=256 / 4.70×@L=4096 speedup this engine was shipped for was # measured against an fp16-mixed engine whose attention core was stuck in - # FP32; with that fixed (build_dit_fp16mixed.py's bound_attention_core) bf16 + # FP32; with that fixed (build_dit_fp16.py's bound_attention_core) bf16 # is only ~3% ahead, and it loses on accuracy: weakly-typed BF16 also lets # TRT evaluate RoPE's rotation angle in bf16, and that angle reaches ~4155 # rad at L=4092 where bf16's spacing is 32 rad (> 2π), so position info for @@ -258,13 +258,13 @@ # 0.52/0.57/0.64 to 0.92/0.94/0.92 on adversarial seeds; sampling steps 1-7 match # the fully-calibrated #47 reference within ~0.001. It is still a SPEED tier over the # fp16-mixed default rather than a fidelity upgrade over it (single-step velocity cos - # ~0.92-0.94 < fp16mixed's ~1.0), but it no longer collapses at the highest-noise + # ~0.92-0.94 < fp16's ~1.0), but it no longer collapses at the highest-noise # first step the way the uncalibrated engine did. #47's fp16-attention variant buys # the last ~0.03 of step-0 fidelity for +2 ms/fwd; this tier keeps bf16 attention for # the speed. Identity of the shipped engine: 176 fp8 GEMMs + 96 bf16 fused MHA + # baked fp32 RoPE constant — check with scripts/verify_fp8_rope.py. Same _DIT_PROFILE # (batch=1, L in [1,4096]) → identical CLI/feature surface. medium-only, not - # seed-reproducible vs fp16mixed. + # seed-reproducible vs fp16. # # Producer: the fp8-linear ONNX (fp8 QDQ inserted; inv_freq inline) is RoPE-baked by # build_dit_bf16.py, the SAME baker as the bf16 tier (it handles inline OR external diff --git a/optimized/tensorRT/build/build_same_s_dec_fp16mixed.py b/optimized/tensorRT/build/build_same_s_dec_fp16.py similarity index 98% rename from optimized/tensorRT/build/build_same_s_dec_fp16mixed.py rename to optimized/tensorRT/build/build_same_s_dec_fp16.py index e66653c0..70ee0f3b 100644 --- a/optimized/tensorRT/build/build_same_s_dec_fp16mixed.py +++ b/optimized/tensorRT/build/build_same_s_dec_fp16.py @@ -33,11 +33,11 @@ Cast(to=INT32) is unaffected by trunk dtype since it's an explicit dtype change. Usage: - python build_same_s_dec_fp16mixed.py + python build_same_s_dec_fp16.py [--mode {minimal,rope,full}] # default: rope [--input /weka2/cj/clod/sa3s/stable-audio-3-optimized/onnx/same-s/dec_dynamic_bf16.onnx] - [--onnx /tmp/same_s_dec_fp16mixed.onnx] - [--engine .../models/sm_90/same-s/dec_dynamic_fp16mixed.trt] + [--onnx /tmp/same_s_dec_fp16.onnx] + [--engine .../models/sm_90/same-s/dec_dynamic_fp16.trt] """ import argparse import os @@ -51,7 +51,7 @@ # Reuse the helpers from the DiT FP16-mixed build. Most of them work as-is # — only the FP32-island finder needs SAME-S-specific patterns, which we # override below. -from build_dit_fp16mixed import ( +from build_dit_fp16 import ( strip_noop_fp32_casts, wrap_islands_with_casts, fix_dtype_mismatches, @@ -476,7 +476,7 @@ def cast_to(node): return model -def convert_to_fp16mixed(input_onnx, output_onnx, mode="minimal"): +def convert_to_fp16(input_onnx, output_onnx, mode="minimal"): """Load FP32 ONNX, identify FP32 islands, convert everything else to FP16, and save.""" import onnx @@ -612,12 +612,12 @@ def main(): help="Input FP32 ONNX (the `_bf16` suffix refers to the " "eventual engine flavor; the ONNX itself is FP32)") ap.add_argument("--onnx", - default="/tmp/same_s_dec_fp16mixed.onnx", + default="/tmp/same_s_dec_fp16.onnx", help="Output FP16-mixed ONNX (intermediate)") ap.add_argument("--engine", default="/weka2/cj/clod/sa3s/stable-audio-3/optimized/" "tensorRT/models/sm_90/same-s/" - "dec_dynamic_fp16mixed.trt", + "dec_dynamic_fp16.trt", help="Output TRT engine path") ap.add_argument("--workspace-gb", type=int, default=16) ap.add_argument("--mode", choices=("minimal", "rope", "attention", "full"), @@ -635,7 +635,7 @@ def main(): if not args.skip_convert: print(f"━━━ Convert FP32 ONNX -> FP16-mixed ONNX (mode={args.mode}) ━━━") - convert_to_fp16mixed(args.input, args.onnx, mode=args.mode) + convert_to_fp16(args.input, args.onnx, mode=args.mode) if not args.skip_build: print("\n━━━ Build TRT engine ━━━") diff --git a/optimized/tensorRT/build/make_calib.py b/optimized/tensorRT/build/make_calib.py index a5a84b20..4419fbe6 100644 --- a/optimized/tensorRT/build/make_calib.py +++ b/optimized/tensorRT/build/make_calib.py @@ -7,7 +7,7 @@ recording the six DiT engine inputs at every sampling step. FP8 is the only quantization path in this repo that needs calibration data -(``fp16mixed`` / ``fp32`` need none, MLX is weight-only), so this also +(``fp16`` / ``fp32`` need none, MLX is weight-only), so this also establishes the convention: a ``.calib.npz`` whose keys are the six ONNX input names, each a leading-axis batch of samples. The npz is a reproducible producer artifact: gitignored, never committed, regenerated on demand. diff --git a/optimized/tensorRT/build/make_dit_fp8_smalldit.py b/optimized/tensorRT/build/make_dit_fp8_smalldit.py index a56219ab..548d0e7c 100644 --- a/optimized/tensorRT/build/make_dit_fp8_smalldit.py +++ b/optimized/tensorRT/build/make_dit_fp8_smalldit.py @@ -1,14 +1,14 @@ #!/usr/bin/env python3 """Produce the fp8 ONNX for the SMALL DiTs (sm-music / sm-sfx) by grafting fp8 E4M3 Q/DQ onto -the LINEAR GEMMs of the model's fp16mixed ONNX — leaving attention fp16-fused and the fp32 +the LINEAR GEMMs of the model's fp16 ONNX — leaving attention fp16-fused and the fp32 RMSNorm/RoPE islands intact. This is deliberately NOT the medium's fp8 recipe (baked RoPE + bf16 attention, build_dit_bf16.py + build_dit_fp8.py): the small DiTs never had the bf16 -long-angle RoPE problem, so their fp8 is a straight graft on the known-good fp16mixed graph. +long-angle RoPE problem, so their fp8 is a straight graft on the known-good fp16 graph. Why a dedicated script (vs ModelOpt/build_dit_fp8.py): running ModelOpt PTQ on these models flattens the fp32 islands and its island-reapply (tuned for the medium graph) does NOT restore them correctly here — the resulting engine drops to velocity-cos ~0.69 with clipping. Grafting -onto the fp16mixed ONNX keeps the islands correct by construction (velocity-cos ~0.99 vs eager). +onto the fp16 ONNX keeps the islands correct by construction (velocity-cos ~0.99 vs eager). Two fp16-trunk specifics vs dit_fp8_max/make_fp8_onnx.py (which targets the fp32-trunk medium): * Q/DQ scales are FLOAT16 (the trunk is fp16 → DequantizeLinear must output fp16, else TRT @@ -22,7 +22,7 @@ python make_dit_fp8_smalldit.py \ --model-config /model_config.json --checkpoint /model.safetensors \ - --fp16mixed-onnx onnx/sa3-sm-music/dit_fp16mixed.onnx \ + --fp16-onnx onnx/sa3-sm-music/dit_fp16.onnx \ --domain Music --out onnx/sa3-sm-music/dit_fp8.onnx Then compile with build_from_onnx.py sa3-sm-music-fp8 (STRONGLY_TYPED; the QDQ carry fp8). @@ -39,7 +39,7 @@ SCALE_FLOOR = 1e-4 -def calibrate_act_scales(model_config, checkpoint, fp16mixed_onnx, domain, margin, device): +def calibrate_act_scales(model_config, checkpoint, fp16_onnx, domain, margin, device): """Per-linear activation max|x| from the eager model → {onnx_node_name: scale}. Maps ONNX linear nodes to torch modules by weight name (ONNX 'dit..weight' vs torch 'model.' → match on the suffix after the first dotted component).""" @@ -72,7 +72,7 @@ def __call__(s, x): s.amax = max(s.amax, float(x.abs().amax())); return F.linear sampler_type="pingpong", seed=6000, duration_padding_sec=0.0, return_latents=True) print(f" calibrated {len(mods)} linears on {len(prompts)} {domain} prompts + one full render", flush=True) - m = onnx.load(fp16mixed_onnx, load_external_data=False); g = m.graph + m = onnx.load(fp16_onnx, load_external_data=False); g = m.graph inits = {i.name for i in g.initializer}; prod = {o: n for n in g.node for o in n.output} def wsrc(n): w = n.input[1] @@ -106,8 +106,8 @@ def topo_sort(g): del g.node[:]; g.node.extend(result) -def graft_fp8(fp16mixed_onnx, node_scale, gscale, out): - model = onnx.load(fp16mixed_onnx, load_external_data=True); g = model.graph +def graft_fp8(fp16_onnx, node_scale, gscale, out): + model = onnx.load(fp16_onnx, load_external_data=True); g = model.graph have = False for op in model.opset_import: if op.domain in ("", "ai.onnx"): @@ -161,7 +161,7 @@ def main(): ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) ap.add_argument("--model-config", required=True) ap.add_argument("--checkpoint", required=True) - ap.add_argument("--fp16mixed-onnx", required=True, help="the model's canonical dit_fp16mixed.onnx") + ap.add_argument("--fp16-onnx", required=True, help="the model's canonical dit_fp16.onnx") ap.add_argument("--out", required=True, help="output dit_fp8.onnx (a .data sidecar is written alongside)") ap.add_argument("--domain", default="Music", choices=["Music", "SFX", "Instrument", "One-shot"], help="reprompt few-shot domain for activation calibration (Music for sm-music, SFX for sm-sfx)") @@ -170,9 +170,9 @@ def main(): a = ap.parse_args() t0 = time.time() print(f"[make_dit_fp8_smalldit] calibrating ({a.domain}) ...", flush=True) - node_scale, gscale = calibrate_act_scales(a.model_config, a.checkpoint, a.fp16mixed_onnx, a.domain, a.margin, a.device) + node_scale, gscale = calibrate_act_scales(a.model_config, a.checkpoint, a.fp16_onnx, a.domain, a.margin, a.device) print(f" {len(node_scale)} node scales, global={gscale:.5f}; grafting fp8 ...", flush=True) - graft_fp8(a.fp16mixed_onnx, node_scale, gscale, a.out) + graft_fp8(a.fp16_onnx, node_scale, gscale, a.out) print(f"DONE -> {a.out} ({time.time()-t0:.0f}s)", flush=True) diff --git a/optimized/tensorRT/build/transplant_scales.py b/optimized/tensorRT/build/transplant_scales.py index 7d9e7891..323562a7 100644 --- a/optimized/tensorRT/build/transplant_scales.py +++ b/optimized/tensorRT/build/transplant_scales.py @@ -23,7 +23,7 @@ 5. build_from_onnx.py sa3-m-fp8 compile the calibrated ONNX to dit_fp8.trt Why transplant instead of shipping #47's engine directly: #47's calibrated ONNX -is fp16mixed-derived and its engine uses fp16 attention + a runtime fp32 RoPE +is fp16-derived and its engine uses fp16 attention + a runtime fp32 RoPE island (+~2 ms/fwd). Our bakedmin structure keeps the faster bf16 fused MHA and a baked fp32 RoPE constant. Calibration is a property of the SCALE VALUES, not of the structure, so grafting #47's scales onto the bakedmin graph gives #47's diff --git a/optimized/tensorRT/install.sh b/optimized/tensorRT/install.sh index 19a56dec..b844c1d8 100755 --- a/optimized/tensorRT/install.sh +++ b/optimized/tensorRT/install.sh @@ -68,6 +68,9 @@ $(dim "After install, run with:") $(cyan ./.venv/bin/python) sa3_trt.py --prompt "..." --dit sm-music --decoder same-s $(dim "or (with uv):") $(cyan uv run) sa3_trt.py --prompt "..." --dit sm-music --decoder same-s +$(dim "Quantized decoders/encoders (auto-download from HF):") $(yellow "--dec-precision") $(magenta "fp8 | fp8_fast") + $(dim "fp8 ~1.14x near-transparent · fp8_fast SAME-S only ~1.22x lossier · --precision fp32 for max fidelity") + $(dim "build tier engines locally with quantize/build_tiers.py; tier ONNX + sm_90/sm_120 engines live on HF") USAGE exit 0;; *) errmsg "unknown arg: $1"; exit 2;; @@ -324,21 +327,21 @@ fi SHARED=( # T5Gemma engine — arch-specific. (tokenizer.json is arch-agnostic and # ships bundled with the repo at scripts/tokenizer.json — no download.) - "${HF_SUBDIR}/t5gemma/t5gemma_fp16mixed.trt" + "${HF_SUBDIR}/t5gemma/t5gemma_fp16.trt" ) # Only TRT engines are downloaded from HF. MEDIUM=( - "${HF_SUBDIR}/sa3-m/dit_fp16mixed.trt" + "${HF_SUBDIR}/sa3-m/dit_fp16.trt" "${HF_SUBDIR}/same-l/enc_dynamic_triton_swa.trt" "${HF_SUBDIR}/same-l/dec_dynamic_triton_swa.trt" ) SM_MUSIC=( - "${HF_SUBDIR}/sa3-sm-music/dit_fp16mixed.trt" + "${HF_SUBDIR}/sa3-sm-music/dit_fp16.trt" "${HF_SUBDIR}/same-s/enc_dynamic_bf16.trt" "${HF_SUBDIR}/same-s/dec_dynamic_bf16.trt" ) SM_SFX=( - "${HF_SUBDIR}/sa3-sm-sfx/dit_fp16mixed.trt" + "${HF_SUBDIR}/sa3-sm-sfx/dit_fp16.trt" "${HF_SUBDIR}/same-s/enc_dynamic_bf16.trt" "${HF_SUBDIR}/same-s/dec_dynamic_bf16.trt" ) @@ -377,7 +380,7 @@ else SKIPPED=0 for hf_path in "${DEDUP[@]}"; do # Strip only "tensorRT/" so the arch stays in the local path: - # tensorRT/sm_90/sa3-m/dit_fp16mixed.trt → models/sm_90/sa3-m/dit_fp16mixed.trt. + # tensorRT/sm_90/sa3-m/dit_fp16.trt → models/sm_90/sa3-m/dit_fp16.trt. local_rel="${hf_path#tensorRT/}" local_path="${MODELS_DIR}/${local_rel}" if [ -f "${local_path}" ] && [ -s "${local_path}" ]; then diff --git a/optimized/tensorRT/output/ab_gradio/1780480567813_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780480567813_pt_Death_Metal_s1_spec.png new file mode 100644 index 00000000..7b15a858 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780480567813_pt_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780480567813_trt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780480567813_trt_Death_Metal_s1_spec.png new file mode 100644 index 00000000..adb8310b Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780480567813_trt_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780480576475_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780480576475_pt_Death_Metal_s1_spec.png new file mode 100644 index 00000000..7b15a858 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780480576475_pt_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780480576475_trt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780480576475_trt_Death_Metal_s1_spec.png new file mode 100644 index 00000000..adb8310b Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780480576475_trt_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780480587601_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780480587601_pt_Death_Metal_s1_spec.png new file mode 100644 index 00000000..7b15a858 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780480587601_pt_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780480587601_trt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780480587601_trt_Death_Metal_s1_spec.png new file mode 100644 index 00000000..adb8310b Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780480587601_trt_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780480778299_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780480778299_pt_Death_Metal_s1_spec.png new file mode 100644 index 00000000..7b15a858 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780480778299_pt_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780480778299_trt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780480778299_trt_Death_Metal_s1_spec.png new file mode 100644 index 00000000..adb8310b Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780480778299_trt_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780481520748_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780481520748_pt_Death_Metal_s1_spec.png new file mode 100644 index 00000000..7b15a858 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780481520748_pt_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780481520748_trt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780481520748_trt_Death_Metal_s1_spec.png new file mode 100644 index 00000000..b4d69681 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780481520748_trt_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780481695716_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780481695716_pt_Death_Metal_s1_spec.png new file mode 100644 index 00000000..7b15a858 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780481695716_pt_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780481695716_trt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780481695716_trt_Death_Metal_s1_spec.png new file mode 100644 index 00000000..b4d69681 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780481695716_trt_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780481699952_pt_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780481699952_pt_Death_Metal_s2_spec.png new file mode 100644 index 00000000..531bca9f Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780481699952_pt_Death_Metal_s2_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780481699952_trt_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780481699952_trt_Death_Metal_s2_spec.png new file mode 100644 index 00000000..441152b3 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780481699952_trt_Death_Metal_s2_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780482623536_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482623536_pt_Death_Metal_s1_spec.png new file mode 100644 index 00000000..7b15a858 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482623536_pt_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780482623536_trtcanon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482623536_trtcanon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..26d30539 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482623536_trtcanon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780482623536_trtfp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482623536_trtfp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..d03ccf56 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482623536_trtfp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780482784560_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482784560_pt_Death_Metal_s1_spec.png new file mode 100644 index 00000000..7b15a858 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482784560_pt_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780482784560_trtcanon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482784560_trtcanon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..26d30539 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482784560_trtcanon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780482784560_trtfp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482784560_trtfp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..d03ccf56 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482784560_trtfp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780482789056_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482789056_pt_Death_Metal_s1_spec.png new file mode 100644 index 00000000..7b15a858 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482789056_pt_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780482789056_trtcanon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482789056_trtcanon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..26d30539 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482789056_trtcanon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780482789056_trtfp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482789056_trtfp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..d03ccf56 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482789056_trtfp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780482829728_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482829728_pt_Death_Metal_s1_spec.png new file mode 100644 index 00000000..55793924 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482829728_pt_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780482829728_trtcanon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482829728_trtcanon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..13b9ab27 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482829728_trtcanon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780482829728_trtfp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482829728_trtfp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..e7d03383 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482829728_trtfp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780482836657_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482836657_pt_Death_Metal_s1_spec.png new file mode 100644 index 00000000..55793924 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482836657_pt_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780482836657_trtcanon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482836657_trtcanon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..13b9ab27 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482836657_trtcanon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780482836657_trtfp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482836657_trtfp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..e7d03383 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482836657_trtfp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780482855978_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482855978_pt_Death_Metal_s1_spec.png new file mode 100644 index 00000000..8a0f2fc7 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482855978_pt_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780482855978_trtcanon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482855978_trtcanon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..4cf90724 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482855978_trtcanon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780482855978_trtfp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482855978_trtfp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..6bad3afc Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482855978_trtfp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780482858838_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482858838_pt_Death_Metal_s1_spec.png new file mode 100644 index 00000000..8a0f2fc7 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482858838_pt_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780482858838_trtcanon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482858838_trtcanon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..4cf90724 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482858838_trtcanon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780482858838_trtfp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482858838_trtfp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..6bad3afc Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482858838_trtfp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780482862115_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482862115_pt_Death_Metal_s1_spec.png new file mode 100644 index 00000000..8a0f2fc7 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482862115_pt_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780482862115_trtcanon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482862115_trtcanon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..4cf90724 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482862115_trtcanon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780482862115_trtfp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482862115_trtfp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..6bad3afc Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482862115_trtfp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780482865013_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482865013_pt_Death_Metal_s1_spec.png new file mode 100644 index 00000000..8a0f2fc7 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482865013_pt_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780482865013_trtcanon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482865013_trtcanon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..4cf90724 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482865013_trtcanon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780482865013_trtfp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482865013_trtfp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..6bad3afc Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482865013_trtfp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780482944632_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482944632_pt_Death_Metal_s1_spec.png new file mode 100644 index 00000000..11ecd98a Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482944632_pt_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780482944632_trtcanon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482944632_trtcanon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..5e86e2c2 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482944632_trtcanon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780482944632_trtfp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482944632_trtfp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..0a19729b Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482944632_trtfp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780482947783_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482947783_pt_Death_Metal_s1_spec.png new file mode 100644 index 00000000..11ecd98a Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482947783_pt_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780482947783_trtcanon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482947783_trtcanon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..5e86e2c2 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482947783_trtcanon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780482947783_trtfp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482947783_trtfp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..0a19729b Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482947783_trtfp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780482958211_pt_Death_Meta_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482958211_pt_Death_Meta_s1_spec.png new file mode 100644 index 00000000..89a7bd91 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482958211_pt_Death_Meta_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780482958211_trtcanon_Death_Meta_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482958211_trtcanon_Death_Meta_s1_spec.png new file mode 100644 index 00000000..eab01437 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482958211_trtcanon_Death_Meta_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780482958211_trtfp32_Death_Meta_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482958211_trtfp32_Death_Meta_s1_spec.png new file mode 100644 index 00000000..4b62a4e0 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482958211_trtfp32_Death_Meta_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483154115_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483154115_pt_Death_Metal_s1_spec.png new file mode 100644 index 00000000..c1bc94f8 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483154115_pt_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483154115_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483154115_samel_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..2fba4f5f Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483154115_samel_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483154115_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483154115_samel_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..83503d0d Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483154115_samel_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483154115_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483154115_sames_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..e4604463 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483154115_sames_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483154115_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483154115_sames_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..c86bfba5 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483154115_sames_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483219388_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483219388_pt_Death_Metal_s1_spec.png new file mode 100644 index 00000000..c1bc94f8 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483219388_pt_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483219388_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483219388_samel_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..2fba4f5f Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483219388_samel_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483219388_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483219388_samel_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..83503d0d Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483219388_samel_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483219388_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483219388_sames_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..59b92ff3 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483219388_sames_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483219388_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483219388_sames_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..a79354bf Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483219388_sames_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483387713_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483387713_pt_Death_Metal_s1_spec.png new file mode 100644 index 00000000..c1bc94f8 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483387713_pt_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483387713_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483387713_samel_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..2fba4f5f Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483387713_samel_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483387713_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483387713_samel_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..83503d0d Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483387713_samel_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483387713_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483387713_sames_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..9e7c33ad Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483387713_sames_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483387713_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483387713_sames_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..641bf750 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483387713_sames_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483391579_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483391579_pt_Death_Metal_s1_spec.png new file mode 100644 index 00000000..c1bc94f8 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483391579_pt_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483391579_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483391579_samel_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..2fba4f5f Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483391579_samel_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483391579_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483391579_samel_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..83503d0d Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483391579_samel_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483391579_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483391579_sames_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..95fa185b Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483391579_sames_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483391579_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483391579_sames_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..1edeb6e1 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483391579_sames_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483395100_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483395100_pt_Death_Metal_s1_spec.png new file mode 100644 index 00000000..c1bc94f8 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483395100_pt_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483395100_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483395100_samel_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..2fba4f5f Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483395100_samel_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483395100_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483395100_samel_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..83503d0d Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483395100_samel_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483395100_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483395100_sames_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..5cd883aa Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483395100_sames_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483395100_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483395100_sames_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..43e9cd7d Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483395100_sames_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483432118_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483432118_pt_Death_Metal_s1_spec.png new file mode 100644 index 00000000..976d1ffd Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483432118_pt_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483432118_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483432118_samel_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..8cdca875 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483432118_samel_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483432118_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483432118_samel_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..358fad38 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483432118_samel_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483432118_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483432118_sames_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..5cf69fa6 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483432118_sames_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483432118_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483432118_sames_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..a56353d1 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483432118_sames_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483437783_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483437783_pt_Death_Metal_s1_spec.png new file mode 100644 index 00000000..976d1ffd Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483437783_pt_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483437783_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483437783_samel_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..8cdca875 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483437783_samel_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483437783_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483437783_samel_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..358fad38 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483437783_samel_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483437783_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483437783_sames_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..0a6d51b2 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483437783_sames_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483437783_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483437783_sames_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..f7a4c625 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483437783_sames_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483441862_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483441862_pt_Death_Metal_s1_spec.png new file mode 100644 index 00000000..976d1ffd Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483441862_pt_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483441862_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483441862_samel_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..8cdca875 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483441862_samel_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483441862_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483441862_samel_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..358fad38 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483441862_samel_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483441862_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483441862_sames_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..a52d750a Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483441862_sames_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780483441862_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483441862_sames_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..fc037ca2 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483441862_sames_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780484010428_pt_samel_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484010428_pt_samel_Death_Metal_s1_spec.png new file mode 100644 index 00000000..c6579601 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484010428_pt_samel_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780484010428_pt_sames_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484010428_pt_sames_Death_Metal_s1_spec.png new file mode 100644 index 00000000..51ab5c3c Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484010428_pt_sames_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780484010428_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484010428_samel_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..1e178a3a Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484010428_samel_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780484010428_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484010428_samel_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..1cfd3a8b Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484010428_samel_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780484010428_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484010428_sames_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..e4604463 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484010428_sames_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780484010428_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484010428_sames_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..c86bfba5 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484010428_sames_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780484169479_pt_samel_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484169479_pt_samel_Death_Metal_s1_spec.png new file mode 100644 index 00000000..c6579601 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484169479_pt_samel_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780484169479_pt_sames_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484169479_pt_sames_Death_Metal_s1_spec.png new file mode 100644 index 00000000..51ab5c3c Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484169479_pt_sames_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780484169479_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484169479_samel_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..1e178a3a Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484169479_samel_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780484169479_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484169479_samel_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..1cfd3a8b Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484169479_samel_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780484169479_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484169479_sames_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..59b92ff3 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484169479_sames_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780484169479_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484169479_sames_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..a79354bf Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484169479_sames_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780484174384_pt_samel_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484174384_pt_samel_Death_Metal_s1_spec.png new file mode 100644 index 00000000..c6579601 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484174384_pt_samel_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780484174384_pt_sames_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484174384_pt_sames_Death_Metal_s1_spec.png new file mode 100644 index 00000000..51ab5c3c Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484174384_pt_sames_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780484174384_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484174384_samel_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..1e178a3a Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484174384_samel_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780484174384_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484174384_samel_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..1cfd3a8b Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484174384_samel_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780484174384_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484174384_sames_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..9e7c33ad Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484174384_sames_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780484174384_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484174384_sames_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..641bf750 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484174384_sames_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780484750885_pt_samel_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484750885_pt_samel_Death_Metal_s1_spec.png new file mode 100644 index 00000000..c1bc94f8 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484750885_pt_samel_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780484750885_pt_sames_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484750885_pt_sames_Death_Metal_s1_spec.png new file mode 100644 index 00000000..331eaa4e Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484750885_pt_sames_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780484750885_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484750885_samel_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..2fba4f5f Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484750885_samel_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780484750885_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484750885_samel_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..83503d0d Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484750885_samel_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780484750885_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484750885_sames_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..8f058a5a Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484750885_sames_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780484750885_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484750885_sames_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..59663eff Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484750885_sames_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780484770421_pt_samel_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484770421_pt_samel_Death_Metal_s1_spec.png new file mode 100644 index 00000000..c1bc94f8 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484770421_pt_samel_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780484770421_pt_sames_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484770421_pt_sames_Death_Metal_s1_spec.png new file mode 100644 index 00000000..331eaa4e Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484770421_pt_sames_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780484770421_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484770421_samel_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..2fba4f5f Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484770421_samel_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780484770421_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484770421_samel_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..83503d0d Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484770421_samel_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780484770421_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484770421_sames_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..d0d3546d Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484770421_sames_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780484770421_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484770421_sames_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..488a4043 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484770421_sames_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780485629288_pt_samel_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780485629288_pt_samel_Death_Metal_s1_spec.png new file mode 100644 index 00000000..c1bc94f8 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485629288_pt_samel_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780485629288_pt_sames_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780485629288_pt_sames_Death_Metal_s1_spec.png new file mode 100644 index 00000000..331eaa4e Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485629288_pt_sames_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780485629288_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780485629288_samel_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..2fba4f5f Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485629288_samel_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780485629288_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780485629288_samel_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..83503d0d Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485629288_samel_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780485629288_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780485629288_sames_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..22bd2ec2 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485629288_sames_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780485629288_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780485629288_sames_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..904b0608 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485629288_sames_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780485714220_pt_samel_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780485714220_pt_samel_Death_Metal_s1_spec.png new file mode 100644 index 00000000..c1bc94f8 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485714220_pt_samel_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780485714220_pt_sames_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780485714220_pt_sames_Death_Metal_s1_spec.png new file mode 100644 index 00000000..331eaa4e Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485714220_pt_sames_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780485714220_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780485714220_samel_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..2fba4f5f Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485714220_samel_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780485714220_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780485714220_samel_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..83503d0d Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485714220_samel_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780485714220_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780485714220_sames_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..49fd93a4 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485714220_sames_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780485714220_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780485714220_sames_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..0b0d1691 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485714220_sames_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780485720493_pt_samel_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780485720493_pt_samel_Death_Metal_s1_spec.png new file mode 100644 index 00000000..c1bc94f8 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485720493_pt_samel_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780485720493_pt_sames_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780485720493_pt_sames_Death_Metal_s1_spec.png new file mode 100644 index 00000000..331eaa4e Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485720493_pt_sames_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780485720493_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780485720493_samel_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..2fba4f5f Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485720493_samel_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780485720493_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780485720493_samel_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..83503d0d Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485720493_samel_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780485720493_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780485720493_sames_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..1a4fb98a Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485720493_sames_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780485720493_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780485720493_sames_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..42c49db8 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485720493_sames_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780485772267_pt_samel_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780485772267_pt_samel_Death_Metal_s2_spec.png new file mode 100644 index 00000000..c0ce21a8 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485772267_pt_samel_Death_Metal_s2_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780485772267_pt_sames_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780485772267_pt_sames_Death_Metal_s2_spec.png new file mode 100644 index 00000000..a465cf4f Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485772267_pt_sames_Death_Metal_s2_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780485772267_samel_canon_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780485772267_samel_canon_Death_Metal_s2_spec.png new file mode 100644 index 00000000..79d4183a Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485772267_samel_canon_Death_Metal_s2_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780485772267_samel_fp32_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780485772267_samel_fp32_Death_Metal_s2_spec.png new file mode 100644 index 00000000..ed69fabd Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485772267_samel_fp32_Death_Metal_s2_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780485772267_sames_canon_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780485772267_sames_canon_Death_Metal_s2_spec.png new file mode 100644 index 00000000..546ad02d Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485772267_sames_canon_Death_Metal_s2_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780485772267_sames_fp32_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780485772267_sames_fp32_Death_Metal_s2_spec.png new file mode 100644 index 00000000..d924da05 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485772267_sames_fp32_Death_Metal_s2_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780485778528_pt_samel_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780485778528_pt_samel_Death_Metal_s2_spec.png new file mode 100644 index 00000000..c0ce21a8 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485778528_pt_samel_Death_Metal_s2_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780485778528_pt_sames_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780485778528_pt_sames_Death_Metal_s2_spec.png new file mode 100644 index 00000000..a465cf4f Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485778528_pt_sames_Death_Metal_s2_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780485778528_samel_canon_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780485778528_samel_canon_Death_Metal_s2_spec.png new file mode 100644 index 00000000..79d4183a Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485778528_samel_canon_Death_Metal_s2_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780485778528_samel_fp32_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780485778528_samel_fp32_Death_Metal_s2_spec.png new file mode 100644 index 00000000..ed69fabd Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485778528_samel_fp32_Death_Metal_s2_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780485778528_sames_canon_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780485778528_sames_canon_Death_Metal_s2_spec.png new file mode 100644 index 00000000..a188e08c Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485778528_sames_canon_Death_Metal_s2_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780485778528_sames_fp32_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780485778528_sames_fp32_Death_Metal_s2_spec.png new file mode 100644 index 00000000..b9485032 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485778528_sames_fp32_Death_Metal_s2_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486038662_pt_samel_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780486038662_pt_samel_Death_Metal_s2_spec.png new file mode 100644 index 00000000..d1584165 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486038662_pt_samel_Death_Metal_s2_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486038662_pt_sames_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780486038662_pt_sames_Death_Metal_s2_spec.png new file mode 100644 index 00000000..7802aa7c Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486038662_pt_sames_Death_Metal_s2_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486038662_samel_canon_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780486038662_samel_canon_Death_Metal_s2_spec.png new file mode 100644 index 00000000..10876409 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486038662_samel_canon_Death_Metal_s2_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486038662_samel_fp32_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780486038662_samel_fp32_Death_Metal_s2_spec.png new file mode 100644 index 00000000..778bcc5b Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486038662_samel_fp32_Death_Metal_s2_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486038662_sames_canon_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780486038662_sames_canon_Death_Metal_s2_spec.png new file mode 100644 index 00000000..2d905d40 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486038662_sames_canon_Death_Metal_s2_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486038662_sames_fp32_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780486038662_sames_fp32_Death_Metal_s2_spec.png new file mode 100644 index 00000000..eadd2a18 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486038662_sames_fp32_Death_Metal_s2_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486225881_pt_samel_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780486225881_pt_samel_Death_Metal_s2_spec.png new file mode 100644 index 00000000..673885d6 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486225881_pt_samel_Death_Metal_s2_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486225881_pt_sames_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780486225881_pt_sames_Death_Metal_s2_spec.png new file mode 100644 index 00000000..057ad46a Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486225881_pt_sames_Death_Metal_s2_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486225881_samel_canon_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780486225881_samel_canon_Death_Metal_s2_spec.png new file mode 100644 index 00000000..b8d2e10c Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486225881_samel_canon_Death_Metal_s2_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486225881_samel_fp32_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780486225881_samel_fp32_Death_Metal_s2_spec.png new file mode 100644 index 00000000..8cf678aa Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486225881_samel_fp32_Death_Metal_s2_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486225881_sames_canon_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780486225881_sames_canon_Death_Metal_s2_spec.png new file mode 100644 index 00000000..e3043ab7 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486225881_sames_canon_Death_Metal_s2_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486225881_sames_fp32_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780486225881_sames_fp32_Death_Metal_s2_spec.png new file mode 100644 index 00000000..457c9f67 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486225881_sames_fp32_Death_Metal_s2_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486238402_pt_samel_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780486238402_pt_samel_Death_Metal_s2_spec.png new file mode 100644 index 00000000..673885d6 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486238402_pt_samel_Death_Metal_s2_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486238402_pt_sames_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780486238402_pt_sames_Death_Metal_s2_spec.png new file mode 100644 index 00000000..057ad46a Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486238402_pt_sames_Death_Metal_s2_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486238402_samel_canon_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780486238402_samel_canon_Death_Metal_s2_spec.png new file mode 100644 index 00000000..b8d2e10c Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486238402_samel_canon_Death_Metal_s2_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486238402_samel_fp32_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780486238402_samel_fp32_Death_Metal_s2_spec.png new file mode 100644 index 00000000..8cf678aa Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486238402_samel_fp32_Death_Metal_s2_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486238402_sames_canon_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780486238402_sames_canon_Death_Metal_s2_spec.png new file mode 100644 index 00000000..774e621f Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486238402_sames_canon_Death_Metal_s2_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486238402_sames_fp32_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780486238402_sames_fp32_Death_Metal_s2_spec.png new file mode 100644 index 00000000..6d840a16 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486238402_sames_fp32_Death_Metal_s2_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486700361_pt_samel_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486700361_pt_samel_Death_Metal_s1_spec.png new file mode 100644 index 00000000..c1bc94f8 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486700361_pt_samel_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486700361_pt_sames_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486700361_pt_sames_Death_Metal_s1_spec.png new file mode 100644 index 00000000..331eaa4e Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486700361_pt_sames_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486700361_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486700361_samel_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..2fba4f5f Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486700361_samel_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486700361_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486700361_samel_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..83503d0d Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486700361_samel_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486700361_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486700361_sames_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..22bd2ec2 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486700361_sames_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486700361_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486700361_sames_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..904b0608 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486700361_sames_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486716579_pt_samel_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486716579_pt_samel_Death_Metal_s1_spec.png new file mode 100644 index 00000000..292b3476 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486716579_pt_samel_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486716579_pt_sames_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486716579_pt_sames_Death_Metal_s1_spec.png new file mode 100644 index 00000000..ab6f2483 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486716579_pt_sames_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486716579_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486716579_samel_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..a10d99df Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486716579_samel_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486716579_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486716579_samel_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..71feeeea Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486716579_samel_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486716579_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486716579_sames_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..574f211b Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486716579_sames_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486716579_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486716579_sames_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..3603e14c Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486716579_sames_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486723920_pt_samel_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486723920_pt_samel_Death_Metal_s1_spec.png new file mode 100644 index 00000000..292b3476 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486723920_pt_samel_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486723920_pt_sames_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486723920_pt_sames_Death_Metal_s1_spec.png new file mode 100644 index 00000000..ab6f2483 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486723920_pt_sames_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486723920_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486723920_samel_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..a10d99df Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486723920_samel_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486723920_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486723920_samel_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..71feeeea Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486723920_samel_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486723920_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486723920_sames_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..db99739b Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486723920_sames_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486723920_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486723920_sames_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..89aac57a Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486723920_sames_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486731811_pt_samel_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486731811_pt_samel_Death_Metal_s1_spec.png new file mode 100644 index 00000000..c1bc94f8 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486731811_pt_samel_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486731811_pt_sames_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486731811_pt_sames_Death_Metal_s1_spec.png new file mode 100644 index 00000000..331eaa4e Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486731811_pt_sames_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486731811_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486731811_samel_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..2fba4f5f Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486731811_samel_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486731811_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486731811_samel_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..83503d0d Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486731811_samel_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486731811_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486731811_sames_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..6c0a096b Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486731811_sames_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486731811_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486731811_sames_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..504157dd Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486731811_sames_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486740562_pt_samel_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486740562_pt_samel_Death_Metal_s1_spec.png new file mode 100644 index 00000000..c1bc94f8 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486740562_pt_samel_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486740562_pt_sames_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486740562_pt_sames_Death_Metal_s1_spec.png new file mode 100644 index 00000000..331eaa4e Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486740562_pt_sames_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486740562_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486740562_samel_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..2fba4f5f Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486740562_samel_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486740562_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486740562_samel_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..83503d0d Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486740562_samel_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486740562_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486740562_sames_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..68b1d986 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486740562_sames_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486740562_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486740562_sames_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..f8426956 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486740562_sames_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486749056_pt_samel_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486749056_pt_samel_Death_Metal_s1_spec.png new file mode 100644 index 00000000..c1bc94f8 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486749056_pt_samel_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486749056_pt_sames_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486749056_pt_sames_Death_Metal_s1_spec.png new file mode 100644 index 00000000..331eaa4e Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486749056_pt_sames_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486749056_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486749056_samel_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..2fba4f5f Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486749056_samel_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486749056_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486749056_samel_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..83503d0d Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486749056_samel_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486749056_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486749056_sames_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..a7463fe5 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486749056_sames_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486749056_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486749056_sames_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..d71afc80 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486749056_sames_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486757188_pt_samel_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486757188_pt_samel_Death_Metal_s1_spec.png new file mode 100644 index 00000000..c1bc94f8 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486757188_pt_samel_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486757188_pt_sames_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486757188_pt_sames_Death_Metal_s1_spec.png new file mode 100644 index 00000000..331eaa4e Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486757188_pt_sames_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486757188_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486757188_samel_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..2fba4f5f Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486757188_samel_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486757188_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486757188_samel_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..83503d0d Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486757188_samel_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486757188_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486757188_sames_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..0d31bc64 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486757188_sames_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486757188_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486757188_sames_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..8208246b Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486757188_sames_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486770233_pt_samel_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486770233_pt_samel_Death_Metal_s1_spec.png new file mode 100644 index 00000000..aaecf14d Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486770233_pt_samel_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486770233_pt_sames_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486770233_pt_sames_Death_Metal_s1_spec.png new file mode 100644 index 00000000..8fba811c Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486770233_pt_sames_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486770233_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486770233_samel_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..ac55ada0 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486770233_samel_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486770233_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486770233_samel_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..2de5a7a0 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486770233_samel_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486770233_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486770233_sames_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..0b79cc8f Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486770233_sames_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486770233_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486770233_sames_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..fb12b1a3 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486770233_sames_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486778839_pt_samel_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486778839_pt_samel_Death_Metal_s1_spec.png new file mode 100644 index 00000000..ab77d8a4 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486778839_pt_samel_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486778839_pt_sames_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486778839_pt_sames_Death_Metal_s1_spec.png new file mode 100644 index 00000000..fff2a513 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486778839_pt_sames_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486778839_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486778839_samel_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..dfe3c931 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486778839_samel_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486778839_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486778839_samel_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..a7faa5f8 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486778839_samel_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486778839_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486778839_sames_canon_Death_Metal_s1_spec.png new file mode 100644 index 00000000..20b763d7 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486778839_sames_canon_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/output/ab_gradio/1780486778839_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486778839_sames_fp32_Death_Metal_s1_spec.png new file mode 100644 index 00000000..c432a3b2 Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486778839_sames_fp32_Death_Metal_s1_spec.png differ diff --git a/optimized/tensorRT/quantize/README.md b/optimized/tensorRT/quantize/README.md new file mode 100644 index 00000000..7aac3f8a --- /dev/null +++ b/optimized/tensorRT/quantize/README.md @@ -0,0 +1,182 @@ +# SA3 Encoder + Decoder — Quantized Tiers (SAME-S / SAME-L) + +Train-free quantized variants of the Stable Audio 3 **encoders and decoders**, each a single +**self-contained ONNX** derived from the published bf16 export. TensorRT builds the engine; quality is +preserved by GPTQ weight-error compensation and per-op precision placement (details below). + +Two tiers, both derived from the published bf16 export: + +- **`fp8`** — fp8 **compute** on the FFN GEMMs (fp8 tensor cores) → real speedup **and** a smaller engine, near-transparent. +- **`fp8_fast`** — fp8 pushed into the attention projections too (SAME-S only) — faster still, lower quality. + +For max fidelity, use the bf16 baseline (or `--precision fp32`). + +> **Retired — `w8_bf16` (int8 weight-only).** Its `DequantizeLinear` constant-folds to bf16 at build, so +> the *engine* came out byte-for-byte the size and speed of the bf16 baseline (only the *ONNX* was +> smaller) with slightly lossier weights — strictly dominated by bf16. The lesson worth keeping: **ONNX +> size ≠ engine size.** fp8 stays 1-byte all the way through (small ONNX *and* small engine *and* fp8 +> math); an int8-weight-only tier only shrinks the download, then expands back to bf16 in the engine. + +--- + +## Tiers at a glance + +The **bf16 baseline is the first row** of each table. Decoder dB is PSNR vs that bf16 baseline; encoder +dB/cos is vs **eager** encode (the latent). dB is a heuristic — confirm near-transparent tiers by ear; +`_fast` has audible artifacts. Speed is whole-model median at seq 1292 (~2 min audio). + +### SAME-S decoder + +| file | quantized | compute | onnx | speed | dB vs bf16 (music / sfx) | +|---|---|---|---|---|---| +| `dec_dynamic_bf16.onnx` | — *(bf16 baseline)* | bf16 | 219 MB | 1.00× | ref (7.25 ms @1292 · 23.2 @4096) | +| `dec_fp8.onnx` | fp8 weights + activations (FFN) | fp8 FFN GEMM | 58 MB | **1.14×** | 30.9 / 41.0 · near-transparent | +| `dec_fp8_fast.onnx` | fp8 weights + activations (all linears) | fp8 all GEMMs | 58 MB | **1.22×** | 26.1 / 36.5 · lossy | + +### SAME-L decoder + +| file | quantized | compute | onnx | speed | dB vs bf16 | +|---|---|---|---|---|---| +| `dec_dynamic_triton_swa.onnx` | — *(bf16 baseline)* | bf16 | 1193 MB | 1.00× | ref (54.0 ms @1292 · 174.4 @4096) | +| `dec_fp8.onnx` | fp8 weights + activations (FFN) | fp8 FFN GEMM | 937 MB | **1.15×** | 43.4 · near-transparent | + +SAME-L has **no `_fast` tier**: its attention projections are fp32 islands (fp8 there collapses quality — see islands below). + +### SAME-S encoder + +The **encoder is cheap** (~1 ms SAME-S, ~7 ms SAME-L), so quant mostly buys **size** — `fp8` shrinks both the onnx and the engine. + +| file | quantized | compute | onnx | speed | dB / cos vs eager | +|---|---|---|---|---|---| +| `enc_dynamic_bf16.onnx` | — *(bf16 baseline)* | bf16 | 216 MB | 1.00× | 36.4 / 0.997 (~1 ms) | +| `enc_fp8.onnx` | fp8 (FFN) | fp8 FFN GEMM | 54 MB | 1.08× | 29.9 / 0.986 · near-transparent | +| `enc_fp8_fast.onnx` | fp8 (all linears) | fp8 all GEMMs | 54 MB | 1.14× | 22.3 / 0.924 · lossy | + +### SAME-L encoder + +| file | quantized | compute | onnx | speed | dB / cos vs eager | +|---|---|---|---|---|---| +| `enc_dynamic_triton_swa.onnx` | — *(bf16 baseline)* | bf16 | 1192 MB | 1.00× | 46.0 / 0.9997 (~7 ms) | +| `enc_fp8.onnx` | fp8 (FFN) | fp8 FFN GEMM | 895 MB | 1.03× | 30.8 / 0.989 · near-transparent | + +Encoder input is audio `(1,2,N)`; output latent `(1,256,N/4096)`. **Every encoder onnx carries a +silence-pad node** (see below), so any `N` is accepted. + +### `fp8` vs `fp8_fast` + +Both run the **FFN** GEMMs (`ff.0` / `ff.2`) in fp8. The difference is the **attention projections**: +- **`fp8`** leaves `to_qkv` / `to_out` in bf16 compute (weight-only fp8 for storage) — attention inputs + stay full-precision → near-transparent. +- **`fp8_fast`** runs `to_qkv` / `to_out` in fp8 too — the extra **+0.08×** (1.14 → 1.22×), but it + fp8-rounds Q/K/V *before* the still-bf16 attention core, and the softmax path is fp8-sensitive → the + quality drop (31 → 26 dB). SAME-S only. + +--- + +## Sequence length, padding & building + +The onnx graphs are **fully dynamic** — every model is correct at **any** length (verified L=1→8192 +vs eager, odd/even/prime, no size-dependent error; quant tiers track bf16 identically at all lengths). +The only bound is the **TensorRT optimization profile** you build with. Two rules: + +- **Build wide.** Use `min=1` and a `max` covering your longest track. Reference build uses + **latent `[1, 1292, 8192]`** (decoders, ≈12:40) and **audio `[1, 2097152, 33554432]`** (encoders). + `L=1–31` then work and match eager (these can't be chunked); long files run **natively, no chunking** + (a 12:40 decode / 6:20 encode is a single call). The old `[32,4096]` was just a conservative default. +- **Encoders auto-pad.** The SAME-L encoder needs the audio length to be a multiple of 4096 (the + downsample ratio); the built-in pad node silence-pads the tail up to the next multiple, matching what + eager does internally. Callers feed any `N`. (SAME-S never needed it; padded anyway for a uniform API.) + +The `encode_chunked` path in the runtime is a **stale workaround** — it predated wide profiles and was +chasing what turned out to be out-of-profile garbage, not a real divergence. Single-shot is accurate at +any length (SAME-L enc cos 0.9995 on a real 285 s track). Chunking is unnecessary. + +--- + +## Where the quantization goes, per tier + +Every decoder block is `latent_proj → [ attn: to_qkv → (QKᵀ·softmax·V) → to_out ] → [ ff: ff.0 → GELU → ff.2 ]`, +with RMSNorm + RoPE around attention. Each tier touches a different subset: + +### `dec_fp8` — fp8 on the FFN +- **FFN** (`ff.0` up-proj, `ff.2` down-proj) `+ latent_proj`: **weight fp8 + activation fp8** (per-tensor clipping scale) → **fp8 GEMM**. Weights GPTQ-compensated. +- **Attention projections** (`to_qkv`, `to_out`): SAME-S → **weight-only fp8** (bf16 compute — kept out of fp8 math because they feed softmax); SAME-L → **fp32 islands** (untouched). +- **Attention core** (QKᵀ·softmax·V): bf16 flash (unchanged). +- **RMSNorm / RoPE:** fp32. + +### `dec_fp8_fast` — fp8 everywhere it pays (SAME-S only) +- **All linears** (FFN + `to_qkv` + `to_out` + `latent_proj`): **weight fp8 + activation fp8** → fp8 GEMM. +- **Attention core** still bf16 — but now it's fed **fp8-rounded Q/K/V** from the projections. *That rounding is the quality cost* (the softmax path is fp8-sensitive), buying +0.08× speed over `dec_fp8`. +- **RMSNorm / RoPE:** fp32. + +--- + +## The levers (reference) + +### What you quantize — the two halves of every GEMM + +| lever | format | compute or storage | speed | quality | notes | +|---|---|---|---|---|---| +| **Weights** | fp8 (per-tensor*) or int8 (per-channel) | **storage** (dequant→bf16 at build) | none | near-lossless; **GPTQ** compensates rounding | 1 byte either format — fp8 ≠ smaller than int8 | +| **Activations** | fp8 only (dynamic, per-tensor **clip** scale) | turns the GEMM into fp8 tensor-core compute | **unlocks the speedup** | the **lossy half** (~31 dB floor; GPTQ can't fix it) | calibration-critical — amax scale → collapse; use a clipping scale | + +\* fp8 weight scale **must be per-tensor in a strongly-typed build** (per-channel silently collapses the GEMM). int8 is happily per-channel. int8 *activations* are a non-starter — softmax breaks. + +### Where you apply it — which op + +| lever | touches | fp8? | speed | quality | verdict | +|---|---|---|---|---|---| +| **FFN GEMM** | `ff.0`, `ff.2` | ✅ weight+act | 1.14× (both) / 1.07× (ff.0 only) | ff.0 transparent; **ff.2 sensitive** (±outliers) | the main speed win → `dec_fp8` | +| **Attn projections** | `to_qkv`, `to_out` | ✅ weight+act | +0.08× on top | lossy — fp8-rounds Q/K/V *before* attention | `dec_fp8_fast` (SAME-S) | +| **Attn core** | QKᵀ·softmax·V | ❌ no fp8 MHA kernel @ hd=64 | ~none | — | **dead at hd=64** (Sage 0.46–0.97×, fully-fp8 softmax slower); the real lever is algorithmic **windowing** (~1.6× on SAME-L long-seq), not quant | + +### Leave-alone islands (constraints, not levers) + +| op | rule | why | +|---|---|---| +| **RMSNorm** | keep fp32/bf16 | cheap; quantizing buys nothing | +| **RoPE** | **must stay fp32** | bf16 RoPE angle → long-sequence clip bug (≥2 min renders) | +| **Attn projections (SAME-L)** | fp32 islands | fp8 compute there collapses quality in the strongly-typed graph | + +The mental model: **activations = speed (and the quality floor); weights = size (near-free with GPTQ); attn core = neither (use windowing).** + +--- + +## Which to pick + +- **Max fidelity** → the bf16 baseline (or `--precision fp32` for bit-exact eager) +- **Faster + smaller, near-transparent (default quant)** → `dec_fp8` +- **Fastest, quality-tolerant** → `dec_fp8_fast` (SAME-S) + +## Reducing SAME-L size (optional) +Both SAME-L tiers are 937 MB because the attention projections stay fp32 (680 MB). Weight-only-quantizing +those islands (int8/fp8 weights, bf16 compute — no speed or quality cost) pulls both to ~500 MB. + +## Building the engines + +The tier **ONNX files ship on HuggingFace** (`stabilityai/stable-audio-3-optimized`, alongside the +existing `dec_dynamic_bf16.onnx` / `enc_dynamic_*`). To build an engine with the wide profile: + +```bash +python build_tiers.py --arch {same-s|same-l} --kind {enc|dec} [--fp8] +# SAME-S fp8 / fp8_fast tiers need --fp8; SAME-L carries fp8 in-graph (strongly-typed) and needs no flag. +``` + +`build_tiers.py` bakes the **wide** profile (decoder `latent [1,1292,8192]`, encoder `audio [1,·,33.5M]`), +so L=1..31 work and long files run natively. SAME-L uses the `diff_attn_swa` plugin (PREFER_JIT). + +## Reproducing the tier ONNX (grafters) + +The quantized ONNX are produced by grafting QDQ / int8 / fp8-stored weights onto the published bf16 export: + +| grafter | produces | +|---|---| +| `fp8_gptq.py` | SAME-S `dec_fp8` / `dec_fp8_fast` (fp8-stored + GPTQ, ORT Hessian capture) | +| `fp8_gptq_samel.py` | SAME-L `dec_fp8` (fp8-stored + GPTQ, eager Hessian capture — SWA blocks ORT) | +| `gptq_w8.py` / `gptq_samel.py` | int8 weight-only ONNX — the *retired* `w8_bf16` tier (kept for reference) | +| `enc_quant_sames.py` / `enc_quant_samel.py` | the encoder tiers | +| `pad_encoder.py` | adds the silence-pad node so any audio length is accepted | + +> **Note:** the grafters reference campaign-local calibration data (real-audio latents) and model +> checkpoints via absolute paths at the top of each file — adjust those to reproduce. Calibration + +> Hessians are computed from real-audio latents. The ready-made ONNX on HF need none of this. diff --git a/optimized/tensorRT/quantize/build_tiers.py b/optimized/tensorRT/quantize/build_tiers.py new file mode 100644 index 00000000..36f18e68 --- /dev/null +++ b/optimized/tensorRT/quantize/build_tiers.py @@ -0,0 +1,68 @@ +#!/usr/bin/env python3 +"""Build a SA3 encoder/decoder tier engine with a WIDE dynamic profile. + + python build_tiers.py --arch {same-s,same-l} --kind {enc,dec} [--fp8] + +Why "wide": the historical [32,4096]-frame profile was a conservative default, not a model limit. +With min=1 the models are correct at L=1..31 (verified vs eager — these can't be chunked), and long +files run natively (a ~12:40 decode / 6:20 encode is a single call, no chunking). See README. + +Profiles built here: + decoder : latent (1,256,1) .. (1,256,1292) .. (1,256,8192) # 1 frame .. ~12:40 + encoder : audio (1,2,1) .. (1,2,2097152) .. (1,2,33554432) # 1 samp .. ~12:40 + +SAME-S -> weakly-typed EXPLICIT_BATCH + BF16 (+ FP8 builder flag for fp8/fp8_fast tiers). +SAME-L -> strongly-typed + diff_attn_swa plugin + NO FP8 flag (the in-graph fp8 QDQ nodes + carry the precision). Plugin impl is chosen by GPU arch: AOT on sm_120 (JIT isn't + stream-capturable there), JIT on sm_90 (override with SA3_SWA_PLUGIN=aot|jit). +`--fp8` is required for fp8 / fp8_fast SAME-S tiers; harmless/ignored for SAME-L and w8/bf16. +""" +import argparse, os, sys +sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "scripts")) +import tensorrt as trt + +DEC_PROFILE = ((1, 256, 1), (1, 256, 1292), (1, 256, 8192)) +ENC_PROFILE = ((1, 2, 1), (1, 2, 2097152), (1, 2, 33554432)) + +def build(onnx_path, out_path, arch, kind, fp8): + lg = trt.Logger(trt.Logger.ERROR) + trt.init_libnvinfer_plugins(lg, "") + strong = arch == "same-l" + flags = 0 + if strong: + # SWA plugin: sm_120 (Blackwell) needs AOT — the JIT impl isn't stream-capturable there + # and silently drops the decode inside the runtime's mega-graph. sm_90 uses JIT. Choose by + # the GPU arch (override with SA3_SWA_PLUGIN=aot|jit); set it before importing the plugin. + import torch + gpu = "sm_%d%d" % torch.cuda.get_device_capability() + want_aot = gpu == "sm_120" or os.environ.get("SA3_SWA_PLUGIN") == "aot" + os.environ.setdefault("SA3_SWA_PLUGIN", "aot" if want_aot else "jit") + import diff_attn_nocast_plugin # registers samel::diff_attn_swa (AOT + JIT impls) # noqa: F401 + flags |= 1 << int(trt.NetworkDefinitionCreationFlag.STRONGLY_TYPED) + flags |= 1 << int(trt.NetworkDefinitionCreationFlag.PREFER_AOT_PYTHON_PLUGINS if want_aot + else trt.NetworkDefinitionCreationFlag.PREFER_JIT_PYTHON_PLUGINS) + else: + flags |= 1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH) + b = trt.Builder(lg); net = b.create_network(flags); p = trt.OnnxParser(net, lg) + if not p.parse_from_file(onnx_path): + for i in range(p.num_errors): print(p.get_error(i)) + raise SystemExit("parse failed") + cfg = b.create_builder_config(); cfg.set_memory_pool_limit(trt.MemoryPoolType.WORKSPACE, 20 << 30) + if not strong: + cfg.set_flag(trt.BuilderFlag.BF16) + if fp8: cfg.set_flag(trt.BuilderFlag.FP8) # SAME-L carries fp8 in-graph; only SAME-S needs the flag + name, prof = ("audio", ENC_PROFILE) if kind == "enc" else ("latent", DEC_PROFILE) + pr = b.create_optimization_profile(); pr.set_shape(name, *prof); cfg.add_optimization_profile(pr) + ser = b.build_serialized_network(net, cfg) + if ser is None: raise SystemExit("build failed") + with open(out_path, "wb") as f: f.write(ser) + print(f"built {out_path} ({ser.nbytes/1e6:.0f} MB) — {arch} {kind} {'fp8 ' if fp8 else ''}wide", flush=True) + +if __name__ == "__main__": + ap = argparse.ArgumentParser() + ap.add_argument("onnx"); ap.add_argument("out") + ap.add_argument("--arch", required=True, choices=["same-s", "same-l"]) + ap.add_argument("--kind", required=True, choices=["enc", "dec"]) + ap.add_argument("--fp8", action="store_true", help="required for SAME-S fp8/fp8_fast tiers") + a = ap.parse_args() + build(a.onnx, a.out, a.arch, a.kind, a.fp8) diff --git a/optimized/tensorRT/quantize/enc_quant_samel.py b/optimized/tensorRT/quantize/enc_quant_samel.py new file mode 100644 index 00000000..8e021bb6 --- /dev/null +++ b/optimized/tensorRT/quantize/enc_quant_samel.py @@ -0,0 +1,124 @@ +#!/usr/bin/env python3 +"""SAME-L ENCODER quant tiers (w8 + fp8). Eager-capture the encoder FFN activations (Hessian + p99.9 +clip scale) via hooks (SWA plugin blocks ORT), map eager<->onnx FFN linears by order+weight-value, +graft int8 weight-only (w8) and fp8 FFN GEMM (fp8, per-tensor weight, per-linear clip scale). Attention +proj stay fp32 islands. Self-contained.""" +import sys, os, re, json, glob +import numpy as np, torch, onnx +from onnx import TensorProto, helper, numpy_helper +sys.path.insert(0, "/weka2/cj/clod/sa3s/stable-audio-3/optimized/tensorRT/scripts"); sys.path.insert(0, "/weka2/cj/clod/fp8_calib/build") +from stable_audio_3.factory import create_autoencoder_from_config +from stable_audio_3.loading_utils import copy_state_dict +torch.set_grad_enabled(False) +WD = "/weka2/cj/clod/sames_fp8"; E4M3 = 448.0 +cfg = json.load(open("/weka2/cj/clod/sa3s/models/SAME-L/SAME-L.json")) +ae = create_autoencoder_from_config(cfg["model"], cfg["sample_rate"]) +ck = torch.load("/weka2/cj/clod/sa3s/models/SAME-L/SAME-L.ckpt", map_location="cpu", weights_only=False) +copy_state_dict(ae, ck.get("state_dict", ck) if isinstance(ck, dict) else ck); ae = ae.cuda().eval() +# eager encoder FFN linears, in module order +eproj = [m for nm, m in ae.named_modules() if re.search(r"encoder\..*\.ff\.ff\.0\.proj$", nm)] +eout = [m for nm, m in ae.named_modules() if re.search(r"encoder\..*\.ff\.ff\.2$", nm)] +print(f"eager encoder FFN: {len(eproj)} proj + {len(eout)} out", flush=True) +HP = {}; PP = {} # keyed by (role, idx) +def mk(role, i): + def hook(m, inp): + A = inp[0].reshape(-1, inp[0].shape[-1]).double(); h = (A.T @ A).cpu().numpy() + HP[(role, i)] = h if (role, i) not in HP else HP[(role, i)] + h + p = float(np.percentile(np.abs(inp[0].detach().float().cpu().numpy()), 99.9)); PP.setdefault((role, i), []).append(p) + return hook +H = [m.register_forward_pre_hook(mk("proj", i)) for i, m in enumerate(eproj)] + [m.register_forward_pre_hook(mk("out", i)) for i, m in enumerate(eout)] +AUD = np.load(f"{WD}/calib_audio_samel.npz")["audio"].astype(np.float32) +for ci in range(AUD.shape[0]): + ae.encode(torch.tensor(AUD[ci:ci + 1], device="cuda")); print(f" captured clip {ci}", flush=True) +for h in H: h.remove() +ASC = {k: max(float(np.percentile(v, 90)) / E4M3, 1e-4) for k, v in PP.items()} # per-linear clip scale (robust over clips) +def fp8_gptq(Wnp, Hnp, sc, bs=128, pd=0.05): + dev = "cuda"; W = torch.tensor(np.asarray(Wnp, np.float32), device=dev); Hh = torch.tensor(np.asarray(Hnp, np.float32), device=dev) + cin = W.shape[1]; idx = torch.arange(cin, device=dev); Hh[~torch.isfinite(Hh)] = 0 + d = torch.diag(Hh).clone(); dead = d <= 0; d[dead] = 1; Hh[idx, idx] = d; W[:, dead] = 0; Hh = (Hh + Hh.T) / 2; damp = pd * d.mean(); Hinv = None + for k in range(6): + try: L = torch.linalg.cholesky(Hh + torch.eye(cin, device=dev) * (damp * 4.0 ** k)); Hinv = torch.linalg.cholesky(torch.cholesky_inverse(L), upper=True); break + except Exception: continue + if Hinv is None: Hinv = torch.diag(1.0 / torch.sqrt(d + damp)) + Q = torch.zeros_like(W) + for i1 in range(0, cin, bs): + i2 = min(i1 + bs, cin); W1 = W[:, i1:i2].clone(); Q1 = torch.zeros_like(W1); Err = torch.zeros_like(W1); Hi = Hinv[i1:i2, i1:i2] + for j in range(i2 - i1): + w = W1[:, j]; dd = Hi[j, j].clamp_min(1e-12); qf = (w / sc).to(torch.float8_e4m3fn).to(torch.float32) + q = qf * sc; Q1[:, j] = qf; err = (w - q) / dd; Err[:, j] = err; W1[:, j:] -= err[:, None] * Hi[j, j:][None, :] + Q[:, i1:i2] = Q1 + if i2 < cin: W[:, i2:] -= Err @ Hinv[i1:i2, i2:] + return Q +def warr(inits, prod, n): + w = n.input[1] + if w in inits: return w, numpy_helper.to_array(inits[w]), False, None + p = prod.get(w) + if p and p.op_type == "Transpose" and p.input and p.input[0] in inits: return p.input[0], numpy_helper.to_array(inits[p.input[0]]), True, p + return None, None, None, None +SRC = os.environ["ENC_ONNX"] +def onnx_ffn(g, inits, prod): + pr, ou = {}, {} + for n in g.node: + if n.op_type != "MatMul": continue + mp = re.search(r"blocks\.(\d+)/ff/proj", n.name); mo = re.search(r"blocks\.(\d+)/ff/out", n.name) + if mp: pr[int(mp.group(1))] = n + elif mo: ou[int(mo.group(1))] = n + return [pr[i] for i in sorted(pr)], [ou[i] for i in sorted(ou)] +def build_tier(mode, out): + m = onnx.load(SRC, load_external_data=True); g = m.graph + inits = {i.name: i for i in g.initializer}; prod = {o: n for n in g.node for o in n.output} + oproj, oout = onnx_ffn(g, inits, prod) + assert len(oproj) == len(eproj) and len(oout) == len(eout), f"{len(oproj)}/{len(eproj)} {len(oout)}/{len(eout)}" + fp8 = mode == "fp8" + if fp8: g.initializer.append(helper.make_tensor("fp8_zero", TensorProto.FLOAT8E4M3FN, [], [0.0])) + new_nodes, new_inits, drop = [], [], set() + for role, elist, olist in [("proj", eproj, oproj), ("out", eout, oout)]: + for i, (em, n) in enumerate(zip(elist, olist)): + ws, W, vt, tn = warr(inits, prod, n) + Wg = W if vt else W.T # [out,in] + Weag = em.weight.detach().double().cpu().numpy() + assert np.allclose(Wg, Weag, atol=1e-2), f"map mismatch {role}{i} (L2 {np.abs(Wg-Weag).mean():.4f})" + pfx = n.name.strip("/").replace("/", "_"); qax = 0 if vt else 1; wdq = f"{pfx}_wdq" + if fp8: + scale = np.float32(max(np.abs(Wg).max() / E4M3, 1e-8)) # per-tensor + Qi = fp8_gptq(Wg, HP[(role, i)], scale); Qs = Qi if vt else Qi.T.contiguous() + raw = Qs.to(torch.float8_e4m3fn).view(torch.uint8).cpu().numpy().tobytes() + new_inits.append(helper.make_tensor(f"{pfx}_w8", TensorProto.FLOAT8E4M3FN, list(Qs.shape), raw, raw=True)) + new_inits.append(numpy_helper.from_array(np.array(scale, np.float16), f"{pfx}_wsc")) + new_nodes.append(helper.make_node("DequantizeLinear", [f"{pfx}_w8", f"{pfx}_wsc", "fp8_zero"], [wdq], name=f"{pfx}_DQw")) + a = ASC[(role, i)]; new_inits.append(numpy_helper.from_array(np.array(a, np.float16), f"{pfx}_asc")) + aq, adq = f"{pfx}_aq", f"{pfx}_adq" + new_nodes.append(helper.make_node("QuantizeLinear", [n.input[0], f"{pfx}_asc", "fp8_zero"], [aq], name=f"{pfx}_Qa")) + new_nodes.append(helper.make_node("DequantizeLinear", [aq, f"{pfx}_asc", "fp8_zero"], [adq], name=f"{pfx}_DQa")) + n.input[0] = adq + else: + scale = np.maximum(np.abs(Wg).max(axis=1) / 127.0, 1e-4).astype(np.float32) # per-channel int8 + sc_b = scale.reshape(-1, 1) if not vt else scale.reshape(-1, 1) + Wq = np.round((Wg / scale.reshape(-1, 1))).clip(-127, 127).astype(np.int8); Qs = Wq if vt else Wq.T + new_inits.append(numpy_helper.from_array(Qs, f"{pfx}_w8")) + new_inits.append(numpy_helper.from_array(scale.astype(np.float16), f"{pfx}_wsc")) + new_inits.append(helper.make_tensor(f"{pfx}_wzp", TensorProto.INT8, [scale.size], [0] * scale.size)) + new_nodes.append(helper.make_node("DequantizeLinear", [f"{pfx}_w8", f"{pfx}_wsc", f"{pfx}_wzp"], [wdq], name=f"{pfx}_DQw", axis=qax)) + if vt: + for k, inp in enumerate(tn.input): + if inp == ws: tn.input[k] = wdq + else: n.input[1] = wdq + drop.add(ws) + g.initializer.extend(new_inits); g.node.extend(new_nodes) + keep = [i for i in g.initializer if i.name not in drop]; del g.initializer[:]; g.initializer.extend(keep) + avail = {i.name for i in g.initializer} | {i.name for i in g.input} | {""}; order, rem = [], list(g.node) + while rem: + nx, prog = [], False + for nd in rem: + if all(i in avail for i in nd.input): order.append(nd); [avail.add(o) for o in nd.output]; prog = True + else: nx.append(nd) + rem = nx + if not prog: raise RuntimeError("topo stuck") + del g.node[:]; g.node.extend(order) + for f in (out, out + ".data"): + if os.path.exists(f): os.remove(f) + onnx.save(m, out, save_as_external_data=True, all_tensors_to_one_file=True, location=os.path.basename(out) + ".data", size_threshold=1024) + print(f" {mode} -> {out}", flush=True) +build_tier("w8", f"{WD}/enc_samel_w8.onnx") +build_tier("fp8", f"{WD}/enc_samel_fp8.onnx") +print("done", flush=True) diff --git a/optimized/tensorRT/quantize/enc_quant_sames.py b/optimized/tensorRT/quantize/enc_quant_sames.py new file mode 100644 index 00000000..562a5500 --- /dev/null +++ b/optimized/tensorRT/quantize/enc_quant_sames.py @@ -0,0 +1,122 @@ +#!/usr/bin/env python3 +"""SAME-S ENCODER quant tiers. ORT-capture FFN/attn-proj activations from calibration AUDIO (Hessian + +p99.9 clip scale), then emit 3 self-contained onnx: w8 (int8 weight-only), fp8 (FFN fp8-compute + +attn weight-only fp8), fast (all-linears fp8-compute). Per-channel weight scales (weakly-typed OK).""" +import os, re +import numpy as np, onnx, torch +from onnx import TensorProto, helper, numpy_helper +import onnxruntime as ort +torch.set_grad_enabled(False) +WD = "/weka2/cj/clod/sames_fp8"; E4M3 = 448.0 +SRC = os.environ.get("ENC_ONNX") +if not SRC: + from huggingface_hub import hf_hub_download + SRC = hf_hub_download("stabilityai/stable-audio-3-optimized", "onnx/same-s/enc_dynamic_bf16.onnx") +AUD = np.load(f"{WD}/calib_audio_sames.npz")["audio"].astype(np.float32) # [6,2,N] +def warr(inits, prod, n): + w = n.input[1] + if w in inits: return w, numpy_helper.to_array(inits[w]), False, None + p = prod.get(w) + if p and p.op_type == "Transpose" and p.input and p.input[0] in inits: return p.input[0], numpy_helper.to_array(inits[p.input[0]]), True, p + return None, None, None, None +# ---- capture (ORT): Hessian H=AtA + p99.9 abs per linear ---- +m = onnx.load(SRC, load_external_data=True); g = m.graph +inits = {i.name: i for i in g.initializer}; prod = {o: n for n in g.node for o in n.output} +tin = {} +for n in g.node: + if n.op_type == "MatMul": + ws, W, vt, tn = warr(inits, prod, n) + if ws is not None and W.ndim == 2: tin[n.name] = n.input[0] +n0 = len(g.output); ex = {o.name for o in g.output} +for t in set(tin.values()): + if t not in ex: g.output.append(helper.make_tensor_value_info(t, TensorProto.FLOAT, None)) +for f in (f"{WD}/_enc.onnx", f"{WD}/_enc.onnx.data"): + if os.path.exists(f): os.remove(f) +onnx.save(m, f"{WD}/_enc.onnx", save_as_external_data=True, all_tensors_to_one_file=True, location="_enc.onnx.data", size_threshold=1024) +so = ort.SessionOptions(); so.graph_optimization_level = ort.GraphOptimizationLevel.ORT_DISABLE_ALL +sess = ort.InferenceSession(f"{WD}/_enc.onnx", so, providers=["CUDAExecutionProvider", "CPUExecutionProvider"]) +iname = sess.get_inputs()[0].name; ks = list(tin.keys()); ts = [tin[k] for k in ks] +Hs = {k: None for k in ks}; P99 = {k: [] for k in ks} +for ci in range(AUD.shape[0]): + acts = sess.run(ts, {iname: AUD[ci:ci + 1]}) + for k, a in zip(ks, acts): + A = a.reshape(-1, a.shape[-1]).astype(np.float64); Hs[k] = A.T @ A if Hs[k] is None else Hs[k] + A.T @ A + av = np.abs(a).ravel(); P99[k].append(av if av.size < 400000 else np.random.default_rng(ci).choice(av, 400000, replace=False)) + print(f" captured clip {ci}", flush=True) +del sess +ASCALE = {k: max(float(np.percentile(np.concatenate(v), 99.9)) / E4M3, 1e-4) for k, v in P99.items()} # per-linear clip scale +print(f" clip scales: min {min(ASCALE.values()):.4f} max {max(ASCALE.values()):.4f}", flush=True) +# ---- GPTQ (int8 or fp8), blocked, per-output-channel ---- +def gptq(Wnp, Hnp, sc_np, fmt, bs=128, pd=0.05): + dev = "cuda"; W = torch.tensor(np.asarray(Wnp, np.float32), device=dev); H = torch.tensor(np.asarray(Hnp, np.float32), device=dev) + sc = torch.tensor(sc_np.astype(np.float32), device=dev); cin = W.shape[1]; idx = torch.arange(cin, device=dev) + H[~torch.isfinite(H)] = 0; d = torch.diag(H).clone(); dead = d <= 0; d[dead] = 1; H[idx, idx] = d; W[:, dead] = 0 + H = (H + H.T) / 2; damp = pd * d.mean(); Hinv = None + for k in range(6): + try: L = torch.linalg.cholesky(H + torch.eye(cin, device=dev) * (damp * 4.0 ** k)); Hinv = torch.linalg.cholesky(torch.cholesky_inverse(L), upper=True); break + except Exception: continue + if Hinv is None: Hinv = torch.diag(1.0 / torch.sqrt(d + damp)) + Q = torch.zeros_like(W) + for i1 in range(0, cin, bs): + i2 = min(i1 + bs, cin); W1 = W[:, i1:i2].clone(); Q1 = torch.zeros_like(W1); Err = torch.zeros_like(W1); Hi = Hinv[i1:i2, i1:i2] + for j in range(i2 - i1): + w = W1[:, j]; dd = Hi[j, j].clamp_min(1e-12) + qf = torch.clamp(torch.round(w / sc), -127, 127) if fmt == "int8" else (w / sc).to(torch.float8_e4m3fn).to(torch.float32) + q = qf * sc; Q1[:, j] = qf; err = (w - q) / dd; Err[:, j] = err; W1[:, j:] -= err[:, None] * Hi[j, j:][None, :] + Q[:, i1:i2] = Q1 + if i2 < cin: W[:, i2:] -= Err @ Hinv[i1:i2, i2:] + return Q +# ---- graft one tier (fresh load -> self-contained) ---- +def build_tier(mode, out): + m = onnx.load(SRC, load_external_data=True); g = m.graph + inits = {i.name: i for i in g.initializer}; prod = {o: n for n in g.node for o in n.output} + fp8 = mode in ("fp8", "fast"); noact = re.compile(r"to_qkv|to_out") if mode == "fp8" else re.compile(r"$^") + if fp8: g.initializer.append(helper.make_tensor("fp8_zero", TensorProto.FLOAT8E4M3FN, [], [0.0])) + new_nodes, new_inits, drop = [], [], set() + for n in [x for x in g.node if x.op_type == "MatMul" and x.name in Hs]: + ws, W, vt, tn = warr(inits, prod, n) + if ws is None: continue + Wg = W if vt else W.T; div = E4M3 if fp8 else 127.0 + scale = np.maximum(np.abs(Wg).max(axis=1) / div, 1e-8).astype(np.float32) + Qi = gptq(Wg, Hs[n.name], scale, "fp8" if fp8 else "int8"); Qs = Qi if vt else Qi.T.contiguous() + pfx = n.name.strip("/").replace("/", "_"); qax = 0 if vt else 1; wdq = f"{pfx}_wdq" + do_act = fp8 and not noact.search(n.name) + if fp8: + raw = Qs.to(torch.float8_e4m3fn).view(torch.uint8).cpu().numpy().tobytes() + new_inits.append(helper.make_tensor(f"{pfx}_w8", TensorProto.FLOAT8E4M3FN, list(Qs.shape), raw, raw=True)) + new_inits.append(numpy_helper.from_array(scale, f"{pfx}_wsc")) + new_nodes.append(helper.make_node("DequantizeLinear", [f"{pfx}_w8", f"{pfx}_wsc", "fp8_zero"], [wdq], name=f"{pfx}_DQw", axis=qax)) + if do_act: + a = ASCALE[n.name]; new_inits.append(helper.make_tensor(f"{pfx}_asc", TensorProto.FLOAT, [], [a])) + aq, adq = f"{pfx}_aq", f"{pfx}_adq" + new_nodes.append(helper.make_node("QuantizeLinear", [n.input[0], f"{pfx}_asc", "fp8_zero"], [aq], name=f"{pfx}_Qa")) + new_nodes.append(helper.make_node("DequantizeLinear", [aq, f"{pfx}_asc", "fp8_zero"], [adq], name=f"{pfx}_DQa")) + n.input[0] = adq + else: + new_inits.append(numpy_helper.from_array(Qs.to(torch.int8).cpu().numpy(), f"{pfx}_w8")) + new_inits.append(numpy_helper.from_array(scale, f"{pfx}_wsc")) + new_inits.append(helper.make_tensor(f"{pfx}_wzp", TensorProto.INT8, [scale.size], [0] * scale.size)) + new_nodes.append(helper.make_node("DequantizeLinear", [f"{pfx}_w8", f"{pfx}_wsc", f"{pfx}_wzp"], [wdq], name=f"{pfx}_DQw", axis=qax)) + if vt: + for i, inp in enumerate(tn.input): + if inp == ws: tn.input[i] = wdq + else: n.input[1] = wdq + drop.add(ws) + g.initializer.extend(new_inits); g.node.extend(new_nodes) + keep = [i for i in g.initializer if i.name not in drop]; del g.initializer[:]; g.initializer.extend(keep) + avail = {i.name for i in g.initializer} | {i.name for i in g.input} | {""}; order, rem = [], list(g.node) + while rem: + nx, prog = [], False + for nd in rem: + if all(i in avail for i in nd.input): order.append(nd); [avail.add(o) for o in nd.output]; prog = True + else: nx.append(nd) + rem = nx + if not prog: raise RuntimeError("topo stuck") + del g.node[:]; g.node.extend(order) + for f in (out, out + ".data"): + if os.path.exists(f): os.remove(f) + onnx.save(m, out, save_as_external_data=True, all_tensors_to_one_file=True, location=os.path.basename(out) + ".data", size_threshold=1024) + print(f" {mode:5s} -> {out}", flush=True) +for mode, out in [("w8", f"{WD}/enc_sames_w8.onnx"), ("fp8", f"{WD}/enc_sames_fp8.onnx"), ("fast", f"{WD}/enc_sames_fast.onnx")]: + build_tier(mode, out) +print("done", flush=True) diff --git a/optimized/tensorRT/quantize/fp8_gptq.py b/optimized/tensorRT/quantize/fp8_gptq.py new file mode 100644 index 00000000..08854c44 --- /dev/null +++ b/optimized/tensorRT/quantize/fp8_gptq.py @@ -0,0 +1,104 @@ +#!/usr/bin/env python3 +"""fp8-STORED + GPTQ decoder tier — self-contained, every linear 8-bit. +ALL linears: per-output-channel fp8 weight, GPTQ-compensated, STORED as fp8 (small onnx). + - FFN / non-attention: ALSO per-tensor fp8 activation QDQ -> fp8 GEMM (speed, 1.14x) + - attention proj (to_qkv/to_out): weight-only fp8, DQ to bf16 -> bf16 compute (quality-safe) +Fresh reload before grafting so the output .data is self-contained. argv: .""" +import sys, os, json, re +import numpy as np, onnx, torch +from onnx import TensorProto, helper, numpy_helper +import onnxruntime as ort +torch.set_grad_enabled(False) +DEC, OUT = sys.argv[1], sys.argv[2]; WD = "/weka2/cj/clod/sames_fp8" +AS = json.load(open(f"{WD}/dec_fp8_act_scales_real.json")); nsc = AS["node_scale"]; gs = AS["global_scale"] +LATS = np.load(f"{WD}/calib_latents.npz")["latents"] +NOACT = re.compile(os.environ.get("NOACT_RE", r"to_qkv|to_out")) # weight-only fp8 (bf16); set "$^" for all-linears fp8 +E4M3 = 448.0 +def warr(inits, prod, n): + w = n.input[1] + if w in inits: return w, numpy_helper.to_array(inits[w]), False, None + p = prod.get(w) + if p and p.op_type == "Transpose" and p.input and p.input[0] in inits: return p.input[0], numpy_helper.to_array(inits[p.input[0]]), True, p + return None, None, None, None +# ---- capture Hessians for ALL linears (ORT), on a temp copy ---- +m = onnx.load(DEC, load_external_data=True); g = m.graph +inits = {i.name: i for i in g.initializer}; prod = {o: n for n in g.node for o in n.output} +tin = {n.name: n.input[0] for n in g.node if n.op_type == "MatMul" and warr(inits, prod, n)[0] is not None and warr(inits, prod, n)[1].ndim == 2} +n0 = len(g.output); ex = {o.name for o in g.output} +for t in set(tin.values()): + if t not in ex: g.output.append(helper.make_tensor_value_info(t, TensorProto.FLOAT, None)) +for _f in (f"{WD}/_fpg.onnx", f"{WD}/_fpg.onnx.data"): + if os.path.exists(_f): os.remove(_f) +onnx.save(m, f"{WD}/_fpg.onnx", save_as_external_data=True, all_tensors_to_one_file=True, location="_fpg.onnx.data", size_threshold=1024) +so = ort.SessionOptions(); so.graph_optimization_level = ort.GraphOptimizationLevel.ORT_DISABLE_ALL +sess = ort.InferenceSession(f"{WD}/_fpg.onnx", so, providers=["CUDAExecutionProvider", "CPUExecutionProvider"]) +iname = sess.get_inputs()[0].name; ks = list(tin.keys()); ts = [tin[k] for k in ks]; Hs = {k: None for k in ks} +for li in range(LATS.shape[0]): + acts = sess.run(ts, {iname: LATS[li:li + 1, :, :768].astype(np.float32)}) + for k, a in zip(ks, acts): + A = a.reshape(-1, a.shape[-1]).astype(np.float64); Hs[k] = A.T @ A if Hs[k] is None else Hs[k] + A.T @ A + print(f" captured {li}", flush=True) +del sess +# ---- GPTQ (torch, blocked, per-output-channel fp8 weight) ---- +def fp8_gptq(Wnp, Hnp, sc_np, bs=128, pd=0.05): + dev = "cuda"; W = torch.tensor(np.asarray(Wnp, np.float32), device=dev); H = torch.tensor(np.asarray(Hnp, np.float32), device=dev) + sc = torch.tensor(sc_np.astype(np.float32), device=dev); cin = W.shape[1]; idx = torch.arange(cin, device=dev) + H[~torch.isfinite(H)] = 0; d = torch.diag(H).clone(); dead = d <= 0; d[dead] = 1; H[idx, idx] = d; W[:, dead] = 0 + H = (H + H.T) / 2; damp = pd * d.mean(); Hinv = None + for k in range(6): + try: L = torch.linalg.cholesky(H + torch.eye(cin, device=dev) * (damp * 4.0 ** k)); Hinv = torch.linalg.cholesky(torch.cholesky_inverse(L), upper=True); break + except Exception: continue + if Hinv is None: Hinv = torch.diag(1.0 / torch.sqrt(d + damp)) + Q = torch.zeros_like(W) + for i1 in range(0, cin, bs): + i2 = min(i1 + bs, cin); W1 = W[:, i1:i2].clone(); Q1 = torch.zeros_like(W1); Err = torch.zeros_like(W1); Hi = Hinv[i1:i2, i1:i2] + for j in range(i2 - i1): + w = W1[:, j]; dd = Hi[j, j].clamp_min(1e-12) + qf = (w / sc).to(torch.float8_e4m3fn).to(torch.float32); q = qf * sc + Q1[:, j] = qf; err = (w - q) / dd; Err[:, j] = err; W1[:, j:] -= err[:, None] * Hi[j, j:][None, :] + Q[:, i1:i2] = Q1 + if i2 < cin: W[:, i2:] -= Err @ Hinv[i1:i2, i2:] + return Q +# ---- fresh reload (all raw_data present -> self-contained output), then graft ---- +m = onnx.load(DEC, load_external_data=True); g = m.graph +inits = {i.name: i for i in g.initializer}; prod = {o: n for n in g.node for o in n.output} +g.initializer.append(helper.make_tensor("fp8_zero", TensorProto.FLOAT8E4M3FN, [], [0.0])) +new_nodes, new_inits, drop, nact = [], [], set(), 0 +for n in [x for x in g.node if x.op_type == "MatMul" and x.name in Hs]: + ws, W, vt, tn = warr(inits, prod, n) + if ws is None: continue + Wg = W if vt else W.T + scale = np.maximum(np.abs(Wg).max(axis=1) / E4M3, 1e-8).astype(np.float32) + Qs = fp8_gptq(Wg, Hs[n.name], scale); Qs = Qs if vt else Qs.T.contiguous() + raw = Qs.to(torch.float8_e4m3fn).view(torch.uint8).cpu().numpy().tobytes() + pfx = n.name.strip("/").replace("/", "_"); qax = 0 if vt else 1 + new_inits.append(helper.make_tensor(f"{pfx}_w8", TensorProto.FLOAT8E4M3FN, list(Qs.shape), raw, raw=True)) + new_inits.append(numpy_helper.from_array(scale, f"{pfx}_wsc")) + wdq = f"{pfx}_wdq" + new_nodes.append(helper.make_node("DequantizeLinear", [f"{pfx}_w8", f"{pfx}_wsc", "fp8_zero"], [wdq], name=f"{pfx}_DQw", axis=qax)) + if not NOACT.search(n.name): # FFN -> fp8 activation too (fp8 GEMM) + a = float(max(nsc.get(n.name, gs), 1e-4)); new_inits.append(helper.make_tensor(f"{pfx}_asc", TensorProto.FLOAT, [], [a])) + aq, adq = f"{pfx}_aq", f"{pfx}_adq" + new_nodes.append(helper.make_node("QuantizeLinear", [n.input[0], f"{pfx}_asc", "fp8_zero"], [aq], name=f"{pfx}_Qa")) + new_nodes.append(helper.make_node("DequantizeLinear", [aq, f"{pfx}_asc", "fp8_zero"], [adq], name=f"{pfx}_DQa")) + n.input[0] = adq; nact += 1 + if vt: + for i, inp in enumerate(tn.input): + if inp == ws: tn.input[i] = wdq + else: n.input[1] = wdq + drop.add(ws) +g.initializer.extend(new_inits); g.node.extend(new_nodes) +_keep = [i for i in g.initializer if i.name not in drop]; del g.initializer[:]; g.initializer.extend(_keep) +avail = {i.name for i in g.initializer} | {i.name for i in g.input} | {""}; order, rem = [], list(g.node) +while rem: + nx, prog = [], False + for nd in rem: + if all(i in avail for i in nd.input): order.append(nd); [avail.add(o) for o in nd.output]; prog = True + else: nx.append(nd) + rem = nx + if not prog: raise RuntimeError("topo stuck") +del g.node[:]; g.node.extend(order) +for f in (OUT, OUT + ".data"): + if os.path.exists(f): os.remove(f) +onnx.save(m, OUT, save_as_external_data=True, all_tensors_to_one_file=True, location=os.path.basename(OUT) + ".data", size_threshold=1024) +print(f"fp8-stored: {len(drop)} linears 8-bit ({nact} with fp8 activation/GEMM, {len(drop)-nact} weight-only) -> {OUT}", flush=True) diff --git a/optimized/tensorRT/quantize/fp8_gptq_samel.py b/optimized/tensorRT/quantize/fp8_gptq_samel.py new file mode 100644 index 00000000..0b1937e7 --- /dev/null +++ b/optimized/tensorRT/quantize/fp8_gptq_samel.py @@ -0,0 +1,119 @@ +#!/usr/bin/env python3 +"""SAME-L fp8-STORED + GPTQ. Eager Hessian+amax capture (ORT can't run the SWA plugin), fp8-GPTQ the +FFN weights, graft fp8-stored weight + per-tensor fp8 activation QDQ onto the SAME-L onnx. Attention +proj stay fp32 islands (fp8 compute breaks them). Self-contained. Writes samel_fp8_gptq.onnx.""" +import sys, os, re, json +from pathlib import Path +import numpy as np, torch, onnx +from onnx import TensorProto, helper, numpy_helper +sys.path.insert(0, "/weka2/cj/clod/sa3s/stable-audio-3/optimized/tensorRT/scripts") +sys.path.insert(0, "/weka2/cj/clod/fp8_calib/build") +from stable_audio_3.factory import create_autoencoder_from_config +from stable_audio_3.loading_utils import copy_state_dict +torch.set_grad_enabled(False) +ML = "/weka2/cj/clod/sa3s/models/SAME-L"; AB = "/weka2/cj/clod/sames_fp8/decoder_ab"; E4M3 = 448.0 +cfg = json.load(open(f"{ML}/SAME-L.json")) +ae = create_autoencoder_from_config(cfg["model"], cfg["sample_rate"]) +ck = torch.load(f"{ML}/SAME-L.ckpt", map_location="cpu", weights_only=False) +copy_state_dict(ae, ck.get("state_dict", ck) if isinstance(ck, dict) else ck); ae = ae.to("cuda").eval() +def ekey(nm): + m = re.search(r"transformers\.(\d+)\.ff\.ff\.0\.proj$", nm); x = f"b{m.group(1)}.proj" if m else None + if x: return x + m = re.search(r"transformers\.(\d+)\.ff\.ff\.2$", nm) + if m: return f"b{m.group(1)}.out" + return "latent_proj" if re.search(r"latent_proj$", nm) else None +def okey(nm): + m = re.search(r"blocks\.(\d+)/ff/proj", nm); x = f"b{m.group(1)}.proj" if m else None + if x: return x + m = re.search(r"blocks\.(\d+)/ff/out", nm) + if m: return f"b{m.group(1)}.out" + return "latent_proj" if re.search(r"latent_proj", nm) else None +mods = {} +for nm, mod in ae.named_modules(): + if hasattr(mod, "weight") and getattr(mod.weight, "ndim", 0) == 2: + k = ekey(nm) + if k: mods[k] = mod +print(f"hooked {len(mods)} FFN linears", flush=True) +H = {k: None for k in mods}; AMAX = {k: 0.0 for k in mods} +def mk_hook(k): + def hook(m, inp): + A = inp[0].reshape(-1, inp[0].shape[-1]).double(); h = (A.T @ A).cpu().numpy() + H[k] = h if H[k] is None else H[k] + h; AMAX[k] = max(AMAX[k], float(inp[0].abs().max())) + return hook +handles = [m.register_forward_pre_hook(mk_hook(k)) for k, m in mods.items()] +lats = [np.load("/weka2/cj/clod/fp8_listening/latents_2min.npz")["bf16"]] +lats += [np.load(f) for f in sorted(Path(AB).glob("samel_lat_*.npy"))] +for i, L in enumerate(lats): + ae.decode(torch.tensor(L, device="cuda", dtype=torch.float32)); print(f" captured {i} ({L.shape[-1]})", flush=True) +for h in handles: h.remove() +def fp8_gptq(Wnp, Hnp, sc_np, bs=128, pd=0.05): + dev = "cuda"; W = torch.tensor(np.asarray(Wnp, np.float32), device=dev); Hh = torch.tensor(np.asarray(Hnp, np.float32), device=dev) + sc = torch.tensor(sc_np.astype(np.float32), device=dev); cin = W.shape[1]; idx = torch.arange(cin, device=dev) + Hh[~torch.isfinite(Hh)] = 0; d = torch.diag(Hh).clone(); dead = d <= 0; d[dead] = 1; Hh[idx, idx] = d; W[:, dead] = 0 + Hh = (Hh + Hh.T) / 2; damp = pd * d.mean(); Hinv = None + for k in range(6): + try: L = torch.linalg.cholesky(Hh + torch.eye(cin, device=dev) * (damp * 4.0 ** k)); Hinv = torch.linalg.cholesky(torch.cholesky_inverse(L), upper=True); break + except Exception: continue + if Hinv is None: Hinv = torch.diag(1.0 / torch.sqrt(d + damp)) + Q = torch.zeros_like(W) + for i1 in range(0, cin, bs): + i2 = min(i1 + bs, cin); W1 = W[:, i1:i2].clone(); Q1 = torch.zeros_like(W1); Err = torch.zeros_like(W1); Hi = Hinv[i1:i2, i1:i2] + for j in range(i2 - i1): + w = W1[:, j]; dd = Hi[j, j].clamp_min(1e-12) + qf = (w / sc).to(torch.float8_e4m3fn).to(torch.float32); q = qf * sc + Q1[:, j] = qf; err = (w - q) / dd; Err[:, j] = err; W1[:, j:] -= err[:, None] * Hi[j, j:][None, :] + Q[:, i1:i2] = Q1 + if i2 < cin: W[:, i2:] -= Err @ Hinv[i1:i2, i2:] + return Q +# ---- graft onto the SAME-L onnx (fresh load -> self-contained) ---- +from huggingface_hub import hf_hub_download +DEC = hf_hub_download("stabilityai/stable-audio-3-optimized", "onnx/same-l/dec_dynamic_triton_swa.onnx") +m = onnx.load(DEC, load_external_data=True); g = m.graph +inits = {i.name: i for i in g.initializer}; prod = {o: n for n in g.node for o in n.output} +def warr(n): + w = n.input[1] + if w in inits: return w, numpy_helper.to_array(inits[w]), False, None + p = prod.get(w) + if p and p.op_type == "Transpose" and p.input and p.input[0] in inits: return p.input[0], numpy_helper.to_array(inits[p.input[0]]), True, p + return None, None, None, None +g.initializer.append(helper.make_tensor("fp8_zero", TensorProto.FLOAT8E4M3FN, [], [0.0])) +new_nodes, new_inits, drop, done = [], [], set(), 0 +for n in [x for x in g.node if x.op_type == "MatMul"]: + key = okey(n.name) + if key not in mods: continue # skip attn proj (fp32 islands) / non-FFN + ws, W, vt, tn = warr(n) + if ws is None: continue + Wg = mods[key].weight.detach().double().cpu().numpy() # eager [out,in] + scale = np.float32(max(np.abs(Wg).max() / E4M3, 1e-8)) # per-TENSOR scalar (strongly-typed fp8 GEMM needs per-tensor weight scale) + Qi = fp8_gptq(Wg, H[key], scale); Qstore = Qi if vt else Qi.T.contiguous() # onnx orientation + raw = Qstore.to(torch.float8_e4m3fn).view(torch.uint8).cpu().numpy().tobytes() + pfx = n.name.strip("/").replace("/", "_") + new_inits.append(helper.make_tensor(f"{pfx}_w8", TensorProto.FLOAT8E4M3FN, list(Qstore.shape), raw, raw=True)) + new_inits.append(numpy_helper.from_array(np.array(scale, np.float16), f"{pfx}_wsc")) # scalar fp16 (fp16 trunk) + a = 0.05; new_inits.append(numpy_helper.from_array(np.array(a, np.float16), f"{pfx}_asc")) # repo-calibrated FFN activation CLIPPING scale + wdq, aq, adq = f"{pfx}_wdq", f"{pfx}_aq", f"{pfx}_adq" + new_nodes.append(helper.make_node("DequantizeLinear", [f"{pfx}_w8", f"{pfx}_wsc", "fp8_zero"], [wdq], name=f"{pfx}_DQw")) + new_nodes.append(helper.make_node("QuantizeLinear", [n.input[0], f"{pfx}_asc", "fp8_zero"], [aq], name=f"{pfx}_Qa")) + new_nodes.append(helper.make_node("DequantizeLinear", [aq, f"{pfx}_asc", "fp8_zero"], [adq], name=f"{pfx}_DQa")) + n.input[0] = adq + if vt: + for i, inp in enumerate(tn.input): + if inp == ws: tn.input[i] = wdq + else: n.input[1] = wdq + drop.add(ws); done += 1 +g.initializer.extend(new_inits); g.node.extend(new_nodes) +_keep = [i for i in g.initializer if i.name not in drop]; del g.initializer[:]; g.initializer.extend(_keep) +avail = {i.name for i in g.initializer} | {i.name for i in g.input} | {""}; order, rem = [], list(g.node) +while rem: + nx, prog = [], False + for nd in rem: + if all(i in avail for i in nd.input): order.append(nd); [avail.add(o) for o in nd.output]; prog = True + else: nx.append(nd) + rem = nx + if not prog: raise RuntimeError("topo stuck") +del g.node[:]; g.node.extend(order) +OUT = "/weka2/cj/clod/sames_fp8/samel_fp8_gptq.onnx" +for f in (OUT, OUT + ".data"): + if os.path.exists(f): os.remove(f) +onnx.save(m, OUT, save_as_external_data=True, all_tensors_to_one_file=True, location=os.path.basename(OUT) + ".data", size_threshold=1024) +print(f"SAME-L fp8-stored+GPTQ on {done} FFN linears -> {OUT}", flush=True) diff --git a/optimized/tensorRT/quantize/gptq_samel.py b/optimized/tensorRT/quantize/gptq_samel.py new file mode 100644 index 00000000..9baedba5 --- /dev/null +++ b/optimized/tensorRT/quantize/gptq_samel.py @@ -0,0 +1,130 @@ +#!/usr/bin/env python3 +"""GPTQ int8 for the SAME-L decoder FFN linears. ORT can't run the SWA plugin, so capture the +per-linear input activations from the EAGER SAME-L autoencoder (hooks), build Hessians, GPTQ, and +graft int8 (fp16 scale) onto the SAME-L onnx (attention proj kept fp32). Writes samel_w8_gptq.onnx.""" +import sys, os, re +from pathlib import Path +import numpy as np, torch, onnx +from onnx import TensorProto, helper, numpy_helper +sys.path.insert(0, "/weka2/cj/clod/sa3s/stable-audio-3/optimized/tensorRT/scripts") +sys.path.insert(0, "/weka2/cj/clod/fp8_calib/build") +from stable_audio_3.factory import create_autoencoder_from_config +from stable_audio_3.loading_utils import copy_state_dict +import json +torch.set_grad_enabled(False) +ML = "/weka2/cj/clod/sa3s/models/SAME-L"; AB = "/weka2/cj/clod/sames_fp8/decoder_ab" +cfg = json.load(open(f"{ML}/SAME-L.json")) +ae = create_autoencoder_from_config(cfg["model"], cfg["sample_rate"]) +ck = torch.load(f"{ML}/SAME-L.ckpt", map_location="cpu", weights_only=False) +copy_state_dict(ae, ck.get("state_dict", ck) if isinstance(ck, dict) else ck) +ae = ae.to("cuda").eval() +# hook the decoder FFN linears (exclude attention proj). Normalize eager+onnx names to (block, role). +def ekey(nm): + m = re.search(r"transformers\.(\d+)\.ff\.ff\.0\.proj$", nm) + if m: return f"b{m.group(1)}.proj" + m = re.search(r"transformers\.(\d+)\.ff\.ff\.2$", nm) + if m: return f"b{m.group(1)}.out" + if re.search(r"latent_proj$", nm): return "latent_proj" + return None +def okey(nm): + m = re.search(r"blocks\.(\d+)/ff/proj", nm) + if m: return f"b{m.group(1)}.proj" + m = re.search(r"blocks\.(\d+)/ff/out", nm) + if m: return f"b{m.group(1)}.out" + if re.search(r"latent_proj", nm): return "latent_proj" + return None +mods = {} +for nm, mod in ae.named_modules(): + if hasattr(mod, "weight") and getattr(mod.weight, "ndim", 0) == 2: + k = ekey(nm) + if k: mods[k] = mod +print(f"hooked {len(mods)} eager FFN linears: {sorted(mods)[:3]} ...", flush=True) +H = {k: None for k in mods} +def mk_hook(k): + def hook(m, inp): + A = inp[0].reshape(-1, inp[0].shape[-1]).double() + h = (A.T @ A).cpu().numpy() + H[k] = h if H[k] is None else H[k] + h + return hook +handles = [m.register_forward_pre_hook(mk_hook(k)) for k, m in mods.items()] +# calibration latents: medium 2-min + 380s + the 10 real-song SAME-L latents +lats = [np.load("/weka2/cj/clod/fp8_listening/latents_2min.npz")["bf16"]] # 1292 samples ... +lats += [np.load(f) for f in sorted(Path(AB).glob("samel_lat_*.npy"))] # + 10×376 = ample for the Hessian +for i, L in enumerate(lats): + ae.decode(torch.tensor(L, device="cuda", dtype=torch.float32)); print(f" captured latent {i} ({L.shape[-1]})", flush=True) +for h in handles: h.remove() +def gptq(Wnp, Hnp, scale_np, pd=0.05, bs=128): + dev = "cuda" + W = torch.tensor(np.asarray(Wnp, np.float32), device=dev) # [out, in] + H = torch.tensor(np.asarray(Hnp, np.float32), device=dev) + sc = torch.tensor(scale_np.astype(np.float32), device=dev) # [out] + cin = W.shape[1]; idx = torch.arange(cin, device=dev) + H[~torch.isfinite(H)] = 0.0 + d = torch.diag(H).clone(); dead = d <= 0; d[dead] = 1.0; H[idx, idx] = d; W[:, dead] = 0.0 + H = (H + H.T) / 2; damp = pd * d.mean(); Hinv = None + for k in range(6): # damped Cholesky of H⁻¹ (upper) + try: + L = torch.linalg.cholesky(H + torch.eye(cin, device=dev) * (damp * 4.0 ** k)) + Hinv = torch.linalg.cholesky(torch.cholesky_inverse(L), upper=True); break + except Exception: + continue + if Hinv is None: Hinv = torch.diag(1.0 / torch.sqrt(d + damp)) + Q = torch.zeros_like(W) + for i1 in range(0, cin, bs): # BLOCKED: small updates in-block, 1 matmul for the rest + i2 = min(i1 + bs, cin); W1 = W[:, i1:i2].clone(); Q1 = torch.zeros_like(W1); Err = torch.zeros_like(W1) + Hi = Hinv[i1:i2, i1:i2] + for j in range(i2 - i1): + w = W1[:, j]; dd = Hi[j, j].clamp_min(1e-12) + qi = torch.clamp(torch.round(w / sc), -127, 127); err = (w - qi * sc) / dd + Q1[:, j] = qi; Err[:, j] = err; W1[:, j:] -= err[:, None] * Hi[j, j:][None, :] + Q[:, i1:i2] = Q1 + if i2 < cin: W[:, i2:] -= Err @ Hinv[i1:i2, i2:] + return torch.clamp(torch.round(Q), -127, 127).to(torch.int8).cpu().numpy() +# graft onto the onnx +from huggingface_hub import hf_hub_download +DEC = hf_hub_download("stabilityai/stable-audio-3-optimized", "onnx/same-l/dec_dynamic_triton_swa.onnx") +m = onnx.load(DEC, load_external_data=True); g = m.graph +inits = {i.name: i for i in g.initializer}; prod = {o: n for n in g.node for o in n.output} +def warr(n): + w = n.input[1] + if w in inits: return w, numpy_helper.to_array(inits[w]), False, None + p = prod.get(w) + if p and p.op_type == "Transpose" and p.input and p.input[0] in inits: return p.input[0], numpy_helper.to_array(inits[p.input[0]]), True, p + return None, None, None, None +new_nodes, new_inits, done = [], [], 0 +for n in [x for x in g.node if x.op_type == "MatMul"]: + key = okey(n.name) + if key not in mods: continue # skip attn proj / non-FFN + ws, W, vt, tn = warr(n) + if ws is None: continue + Wg = mods[key].weight.detach().double().cpu().numpy() # eager [out,in] + scale = np.maximum(np.abs(Wg).max(axis=1) / 127.0, 1e-4).astype(np.float32) + Qi = gptq(Wg, H[key], scale) # [out,in] + qax = 0 if vt else 1 + inits[ws].CopyFrom(numpy_helper.from_array(Qi if vt else Qi.T, ws)) + pfx = n.name.strip("/").replace("/", "_") + new_inits += [numpy_helper.from_array(scale.astype(np.float16), f"{pfx}_wsc"), + helper.make_tensor(f"{pfx}_wzp", TensorProto.INT8, [scale.size], [0] * scale.size)] + dq = f"{pfx}_wdq" + new_nodes.append(helper.make_node("DequantizeLinear", [ws, f"{pfx}_wsc", f"{pfx}_wzp"], [dq], name=f"{pfx}_DQw", axis=qax)) + if vt: + for i, inp in enumerate(tn.input): + if inp == ws: tn.input[i] = dq + else: n.input[1] = dq + done += 1 +g.initializer.extend(new_inits); g.node.extend(new_nodes) +avail = {i.name for i in g.initializer} | {i.name for i in g.input} | {""} +order, rem = [], list(g.node) +while rem: + nx, prog = [], False + for nd in rem: + if all(i in avail for i in nd.input): order.append(nd); [avail.add(o) for o in nd.output]; prog = True + else: nx.append(nd) + rem = nx + if not prog: raise RuntimeError("topo stuck") +del g.node[:]; g.node.extend(order) +OUT = "/weka2/cj/clod/sames_fp8/samel_w8_gptq.onnx" +for f in (OUT, OUT + ".data"): + if os.path.exists(f): os.remove(f) +onnx.save(m, OUT, save_as_external_data=True, all_tensors_to_one_file=True, location=os.path.basename(OUT) + ".data", size_threshold=1024) +print(f"GPTQ int8 grafted on {done} SAME-L FFN linears -> {OUT}", flush=True) diff --git a/optimized/tensorRT/quantize/gptq_w8.py b/optimized/tensorRT/quantize/gptq_w8.py new file mode 100644 index 00000000..5263cc21 --- /dev/null +++ b/optimized/tensorRT/quantize/gptq_w8.py @@ -0,0 +1,87 @@ +#!/usr/bin/env python3 +"""GPTQ int8 per-channel weight-only quantization of a decoder onnx. Captures each linear's input +activations (ORT), builds the Hessian H = AᵀA, and quantizes weights column-by-column with +inverse-Hessian error compensation (the OBQ/GPTQ update). Writes a W8 onnx. argv: +[Lslice].""" +import os, sys +import numpy as np, onnx +from onnx import TensorProto, helper, numpy_helper +import onnxruntime as ort +DEC, OUT = sys.argv[1], sys.argv[2] +LSL = int(sys.argv[3]) if len(sys.argv) > 3 else 768 +LATS = np.load("/weka2/cj/clod/sames_fp8/calib_latents.npz")["latents"] +m = onnx.load(DEC, load_external_data=True); g = m.graph +inits = {i.name: i for i in g.initializer}; prod = {o: n for n in g.node for o in n.output} +def warr(n): + w = n.input[1] + if w in inits: return w, numpy_helper.to_array(inits[w]), False, None + p = prod.get(w) + if p and p.op_type == "Transpose" and p.input and p.input[0] in inits: return p.input[0], numpy_helper.to_array(inits[p.input[0]]), True, p + return None, None, None, None +linears = [] +for n in g.node: + if n.op_type == "MatMul": + ws, W, vt, tn = warr(n) + if ws is not None and W.ndim == 2: linears.append((n, ws, W, vt, tn)) +print(f"{len(linears)} linears; capturing activations (L={LSL}) ...", flush=True) +# --- capture inputs via ORT (add as temp outputs, then restore) --- +in_names = [n.input[0] for n, *_ in linears] +n0 = len(g.output) +existing = {o.name for o in g.output} +for t in in_names: + if t not in existing: g.output.append(helper.make_tensor_value_info(t, TensorProto.FLOAT, None)) +onnx.save(m, "/weka2/cj/clod/sames_fp8/_gptq_cap.onnx", save_as_external_data=True, all_tensors_to_one_file=True, location="_gptq_cap.onnx.data", size_threshold=1024) +del g.output[n0:] +so = ort.SessionOptions(); so.graph_optimization_level = ort.GraphOptimizationLevel.ORT_DISABLE_ALL +sess = ort.InferenceSession("/weka2/cj/clod/sames_fp8/_gptq_cap.onnx", so, providers=["CUDAExecutionProvider", "CPUExecutionProvider"]) +iname = sess.get_inputs()[0].name +Hs = {n.name: None for n, *_ in linears} +for li in range(LATS.shape[0]): + acts = sess.run(in_names, {iname: LATS[li:li + 1, :, :LSL].astype(np.float32)}) + for (n, *_), a in zip(linears, acts): + A = a.reshape(-1, a.shape[-1]).astype(np.float64) + Hs[n.name] = A.T @ A if Hs[n.name] is None else Hs[n.name] + A.T @ A + print(f" latent {li} done", flush=True) +# --- GPTQ --- +def gptq(W, H, scale, pd=0.01): + W = W.astype(np.float64).copy(); cin = W.shape[1]; H = H.copy() + dead = np.diag(H) == 0; H[dead, dead] = 1.0; W[:, dead] = 0.0 + H[np.diag_indices(cin)] += pd * np.mean(np.diag(H)) + Hinv = np.linalg.cholesky(np.linalg.inv(H)).T # upper-tri Cholesky of H⁻¹ + Qi = np.zeros(W.shape, dtype=np.int8); sc = scale + for i in range(cin): + w = W[:, i]; d = Hinv[i, i] + qi = np.clip(np.round(w / sc), -127, 127); q = qi * sc + Qi[:, i] = qi.astype(np.int8) + W[:, i:] -= np.outer((w - q) / d, Hinv[i, i:]) + return Qi +new_nodes, new_inits = [], [] +for (n, ws, W, vt, tn) in linears: + Wg = W if vt else W.T # -> [out, in] + scale = np.maximum(np.abs(Wg).max(axis=1) / 127.0, 1e-9).astype(np.float32) # per-output [out] + Qi = gptq(Wg, Hs[n.name], scale) # [out, in] int8 + inits[ws].CopyFrom(numpy_helper.from_array(Qi if vt else Qi.T, ws)) + pfx = n.name.strip("/").replace("/", "_"); qax = 0 if vt else 1 + new_inits += [helper.make_tensor(f"{pfx}_wsc", TensorProto.FLOAT, [scale.size], scale.tolist()), + helper.make_tensor(f"{pfx}_wzp", TensorProto.INT8, [scale.size], [0] * scale.size)] + dq = f"{pfx}_wdq" + new_nodes.append(helper.make_node("DequantizeLinear", [ws, f"{pfx}_wsc", f"{pfx}_wzp"], [dq], name=f"{pfx}_DQw", axis=qax)) + if vt: + for i, inp in enumerate(tn.input): + if inp == ws: tn.input[i] = dq + else: n.input[1] = dq +g.initializer.extend(new_inits); g.node.extend(new_nodes) +avail = {i.name for i in g.initializer} | {i.name for i in g.input} | {""} +order, rem = [], list(g.node) +while rem: + nx, prog = [], False + for nd in rem: + if all(i in avail for i in nd.input): order.append(nd); [avail.add(o) for o in nd.output]; prog = True + else: nx.append(nd) + rem = nx + if not prog: raise RuntimeError("topo stuck") +del g.node[:]; g.node.extend(order) +for f in (OUT, OUT + ".data"): + if os.path.exists(f): os.remove(f) +onnx.save(m, OUT, save_as_external_data=True, all_tensors_to_one_file=True, location=os.path.basename(OUT) + ".data", size_threshold=1024) +print(f"GPTQ int8 on {len(linears)} linears -> {OUT}", flush=True) diff --git a/optimized/tensorRT/quantize/pad_encoder.py b/optimized/tensorRT/quantize/pad_encoder.py new file mode 100644 index 00000000..ec3faf9c --- /dev/null +++ b/optimized/tensorRT/quantize/pad_encoder.py @@ -0,0 +1,43 @@ +#!/usr/bin/env python3 +"""Add a dynamic silence-pad node to an encoder onnx so it accepts ANY audio length: pads the sample +axis up to the next multiple of 4096 with zeros (matches what eager does internally). argv: .""" +import sys, os +import onnx +from onnx import helper, TensorProto +IN, OUT = sys.argv[1], sys.argv[2] +m = onnx.load(IN, load_external_data=True); g = m.graph +audio = g.input[0].name; adt = g.input[0].type.tensor_type.elem_type +opset = {o.domain: o.version for o in m.opset_import}.get("", 0) +print(f"input '{audio}' dtype={adt} opset={opset}", flush=True) +g.initializer.extend([ + helper.make_tensor("_pc4096", TensorProto.INT64, [1], [4096]), + helper.make_tensor("_pzeros5", TensorProto.INT64, [5], [0, 0, 0, 0, 0]), + helper.make_tensor("_pidx2", TensorProto.INT64, [1], [2]), + helper.make_tensor("_pc0", adt, [], [0]), +]) +for n in g.node: # rewire consumers to the padded tensor + for i, inp in enumerate(n.input): + if inp == audio: n.input[i] = "_audio_pad" +g.node.extend([ # pad = (4096 - N%4096) % 4096, pad end of axis 2 + helper.make_node("Shape", [audio], ["_pshape"], name="_pad_shape"), + helper.make_node("Gather", ["_pshape", "_pidx2"], ["_pN"], name="_pad_gather", axis=0), + helper.make_node("Mod", ["_pN", "_pc4096"], ["_pr"], name="_pad_mod1"), + helper.make_node("Sub", ["_pc4096", "_pr"], ["_pt"], name="_pad_sub"), + helper.make_node("Mod", ["_pt", "_pc4096"], ["_ppad"], name="_pad_mod2"), + helper.make_node("Concat", ["_pzeros5", "_ppad"], ["_ppads"], name="_pad_concat", axis=0), + helper.make_node("Pad", [audio, "_ppads", "_pc0"], ["_audio_pad"], name="_pad_pad", mode="constant"), +]) +avail = {i.name for i in g.initializer} | {i.name for i in g.input} | {""} +order, rem = [], list(g.node) +while rem: + nx, prog = [], False + for nd in rem: + if all(i in avail for i in nd.input): order.append(nd); [avail.add(o) for o in nd.output]; prog = True + else: nx.append(nd) + rem = nx + if not prog: raise RuntimeError("topo stuck") +del g.node[:]; g.node.extend(order) +for f in (OUT, OUT + ".data"): + if os.path.exists(f): os.remove(f) +onnx.save(m, OUT, save_as_external_data=True, all_tensors_to_one_file=True, location=os.path.basename(OUT) + ".data", size_threshold=1024) +print(f"padded encoder -> {OUT}", flush=True) diff --git a/optimized/tensorRT/scripts/_arch.py b/optimized/tensorRT/scripts/_arch.py new file mode 100644 index 00000000..8bcfcdae --- /dev/null +++ b/optimized/tensorRT/scripts/_arch.py @@ -0,0 +1,13 @@ +"""Current-GPU SM arch as a plain 'sm_XX' string. + +Used by swa_mma_aot to retarget the AOT SWA kernel's PTX `.target`. The plain form (no +architecture-specific 'a' suffix, e.g. sm_90a / sm_120a) is required because TRT's PTX +loader rejects the arch-specific variants; the m16n8k8 TF32 MMA this kernel uses is +available from sm_80 up, so the plain target compiles and loads on every arch. +""" + + +def detect_arch() -> str: + import torch + major, minor = torch.cuda.get_device_capability() + return f"sm_{major}{minor}" diff --git a/optimized/tensorRT/scripts/pt_inference.py b/optimized/tensorRT/scripts/pt_inference.py index 3b296268..b14657e7 100644 --- a/optimized/tensorRT/scripts/pt_inference.py +++ b/optimized/tensorRT/scripts/pt_inference.py @@ -35,7 +35,7 @@ # Defaults — matches the user's "medium / SAME-L / fp32" scope. PT_MODEL_DIR = Path("/weka2/cj/clod/sa3s/models/SA3-M-hf") -T5_TRT_PATH = TRT_REPO / "models" / "sm_90" / "t5gemma" / "t5gemma_fp16mixed.trt" +T5_TRT_PATH = TRT_REPO / "models" / "sm_90" / "t5gemma" / "t5gemma_fp16.trt" SAMEL_CKPT_DIR = Path("/weka2/cj/clod/sa3s/models/SAME-L") # local SAME-L = TRT decoder's weight source SAMES_CKPT_DIR = Path("/weka2/cj/clod/sa3s/models/SAME-S") # local SAME-S = TRT SAME-S decoder's weight source diff --git a/optimized/tensorRT/scripts/sa3_gradio.py b/optimized/tensorRT/scripts/sa3_gradio.py index 91654ca2..3e8580e9 100644 --- a/optimized/tensorRT/scripts/sa3_gradio.py +++ b/optimized/tensorRT/scripts/sa3_gradio.py @@ -59,7 +59,7 @@ def discover_variants(dit_name: str) -> list[tuple[str, Path]]: """Return [(label, path)] of available DiT engine files for this model. Scans models///dit_*.trt. The canonical engine - (dit_fp16mixed.trt) is always first if present; other variants follow + (dit_fp16.trt) is always first if present; other variants follow alphabetically. For the medium DiT also appends a pseudo-variant "pytorch fp32 (GT)" that dispatches to the PyTorch-eager backend. """ @@ -70,14 +70,14 @@ def discover_variants(dit_name: str) -> list[tuple[str, Path]]: if not d.exists(): return [] files = sorted(d.glob("dit_*.trt")) - canonical_name = "dit_fp16mixed.trt" + canonical_name = "dit_fp16.trt" canonical = [f for f in files if f.name == canonical_name] others = [f for f in files if f.name != canonical_name] out = [] for f in canonical + others: label = f.name[len("dit_"):-len(".trt")] if f.name.startswith("dit_") and f.name.endswith(".trt") else f.name if f.name == canonical_name: - label = "fp16mixed (canonical)" + label = "fp16 (canonical)" elif "buggy" in f.name: label = label + " ← old, broken" out.append((label, f)) @@ -88,34 +88,25 @@ def discover_variants(dit_name: str) -> list[tuple[str, Path]]: def discover_decoder_variants(decoder_name: str) -> list[tuple[str, Path]]: - """Return [(label, path)] of available decoder engine files. Scans - models///dec_*.trt. - - Canonical (what's referenced by DECODER_PATHS) is always first if present. - Naming convention: dec_dynamic_.trt (e.g. - dec_dynamic_bf16.trt, dec_dynamic_fp32.trt, dec_dynamic_triton_swa.trt). + """Return [(label, path)] of decoder quantization tiers for this decoder — + canonical first. Uses the known tier set (canonical / fp8 / fp8_fast; + see quantize/README.md); engines auto-download from HF on selection if missing. + Any extra local dec_*.trt not in the known set are appended. """ - d = MODELS_ROOT / decoder_name - if not d.exists(): - return [] - files = sorted(d.glob("dec_*.trt")) - canonical = canon.DECODER_PATHS.get(decoder_name) - canonical_name = canonical.name if canonical else "" - canonicals = [f for f in files if f.name == canonical_name] - others = [f for f in files if f.name != canonical_name] - out = [] - for f in canonicals + others: - # Strip 'dec_dynamic_' prefix and '.trt' suffix for the label. - stem = f.name - if stem.startswith("dec_dynamic_"): - label = stem[len("dec_dynamic_"):-len(".trt")] - elif stem.startswith("dec_"): - label = stem[len("dec_"):-len(".trt")] - else: - label = stem - if f.name == canonical_name: - label = f"{label} (canonical)" - out.append((label, f)) + out, known = [], set() + for tier, fname in canon.DECODER_TIER_FILENAME.get(decoder_name, {}).items(): + label = f"{tier} (canonical)" if tier == "canonical" else tier + out.append((label, canon.ARCH_DIR / decoder_name / fname)) + known.add(fname) + d = canon.ARCH_DIR / decoder_name + if d.exists(): + for f in sorted(d.glob("dec_*.trt")): + if f.name in known: + continue + stem = f.name + label = (stem[len("dec_dynamic_"):-len(".trt")] if stem.startswith("dec_dynamic_") + else stem[len("dec_"):-len(".trt")]) + out.append((label, f)) return out @@ -196,10 +187,12 @@ def get_inference(dit: str, decoder: str, dit_variant_path: str, # Override the canonical engine path lookups before constructing. canon.DIT_CHOICES[dit]["engine"] = Path(dit_variant_path) canon.DECODER_PATHS[decoder] = Path(dec_variant_path) + dec_tier = canon.decoder_tier_from_filename(decoder, Path(dec_variant_path).name) print(f"\n → loading SA3Inference({dit!r}, {decoder!r}, " f"dit={Path(dit_variant_path).name}, " - f"dec={Path(dec_variant_path).name})") + f"dec={Path(dec_variant_path).name}, tier={dec_tier})") inf = SA3Inference(dit, decoder, + dec_precision=dec_tier, default_T_lat=default_T_lat, default_steps=default_steps, default_seconds=default_seconds, diff --git a/optimized/tensorRT/scripts/sa3_trt.py b/optimized/tensorRT/scripts/sa3_trt.py index 4a6c5699..39170d4f 100644 --- a/optimized/tensorRT/scripts/sa3_trt.py +++ b/optimized/tensorRT/scripts/sa3_trt.py @@ -445,6 +445,7 @@ class SA3Inference: def __init__(self, dit: str, decoder: str, *, precision: str | None = None, + dec_precision: str = "canonical", default_T_lat: int = 324, default_steps: int = 8, default_seconds: float = 30.0, models_dir: Path | None = None, @@ -455,16 +456,16 @@ def __init__(self, dit: str, decoder: str, *, Args: dit: one of DIT_CHOICES — "sm-music" / "sm-sfx" / "medium" decoder: one of DECODER_PATHS — "same-s" / "same-l" - precision: None (default → "fp16mixed" for every model), or an - explicit "fp16mixed" (canonical: FP16 trunk, FP32 + precision: None (default → "fp16" for every model), or an + explicit "fp16" (canonical: FP16 trunk, FP32 RMSNorm/RoPE islands, FMHA-fused FP16 attention core), "fp8" (medium only; max-speed clean tier — fp8 linears + bf16 fused FMHA + baked fp32 RoPE, ~1.3× faster than - fp16mixed and clean at long sequence, capped L≤4096), - "bf16" (medium only; ~3% faster but its bf16 RoPE - angle drifts at long sequence, and not seed- - reproducible vs fp16mixed), or "fp32" (bit-equiv - PyTorch eager, ~2× slower). Engines auto-download + fp16 and clean at long sequence, capped L≤4096), + or "fp32" (bit-equiv + PyTorch eager, ~2× slower). ("fp16mixed" and the + retired medium-only "bf16" alias to "fp16".) + Engines auto-download from HF if the requested file is missing. default_T_lat: latent length to build the initial graph at default_steps: pingpong steps for the initial graph @@ -479,18 +480,16 @@ def __init__(self, dit: str, decoder: str, *, raise ValueError(f"unknown dit={dit!r}; valid: {list(DIT_CHOICES)}") if decoder not in DECODER_PATHS: raise ValueError(f"unknown decoder={decoder!r}; valid: {list(DECODER_PATHS)}") - # Resolve precision default per model: fp16-mixed everywhere. - # bf16 is medium-only and selectable. + # Resolve precision default per model: fp16 everywhere. Retired tokens + # (fp16mixed, bf16) alias to fp16. if precision is None: precision = canon.default_precision(dit) + precision = canon.normalize_precision(precision) if precision not in canon.PRECISIONS: raise ValueError(f"unknown precision={precision!r}; valid: {canon.PRECISIONS}") - if precision == "bf16" and dit != "medium": - raise ValueError("precision='bf16' is only available for dit='medium' " - "(sm-music/sm-sfx already fuse in fp16mixed)") if precision == "fp8" and dit != "medium": raise ValueError("precision='fp8' is only available for dit='medium' " - "(fp8 linears + bf16 FMHA + baked fp32 RoPE; sm-music/sm-sfx ship fp16mixed only)") + "(fp8 linears + bf16 FMHA + baked fp32 RoPE; sm-music/sm-sfx ship fp16 only)") # Quiet: patch canon's stage/sub/_stage_vram to no-ops so loading # doesn't spam stdout (gradio in particular wants a clean log). @@ -504,7 +503,7 @@ def __init__(self, dit: str, decoder: str, *, if models_dir is not None and str(models_dir) != str(canon.MODELS_DIR): new_root = Path(models_dir).resolve() new_arch_dir = new_root / ARCH - canon.T5GEMMA_PATH = new_arch_dir / "t5gemma" / "t5gemma_fp16mixed.trt" + canon.T5GEMMA_PATH = new_arch_dir / "t5gemma" / "t5gemma_fp16.trt" for kk in DIT_CHOICES: DIT_CHOICES[kk]["engine"] = new_arch_dir / DIT_CHOICES[kk]["engine"].relative_to(canon.ARCH_DIR) for kk in DECODER_PATHS: @@ -521,7 +520,7 @@ def __init__(self, dit: str, decoder: str, *, self.quiet = quiet # 1. Lazy-download any missing engines (precision-aware). - needed = canon.get_engine_files(dit, decoder, precision, with_encoder=with_encoder) + needed = canon.get_engine_files(dit, decoder, precision, with_encoder=with_encoder, dec_tier=dec_precision) _ensure_files(needed) # 2. Heavy imports (torch + tensorrt + plugin). @@ -547,10 +546,10 @@ def __init__(self, dit: str, decoder: str, *, engine_specs = { "t5": canon.T5GEMMA_PATH, "dit": canon.get_dit_engine_path(dit, precision), - "dec": canon.get_decoder_engine_path(decoder, precision), + "dec": canon.resolve_decoder_engine(decoder, precision, dec_precision), } if with_encoder: - engine_specs["enc"] = ENCODER_PATHS[decoder] + engine_specs["enc"] = canon.resolve_encoder_engine(decoder, dec_precision) t0 = time.time() with concurrent.futures.ThreadPoolExecutor(max_workers=len(engine_specs)) as ex: futs = {name: ex.submit(TRTRunner, path) for name, path in engine_specs.items()} @@ -721,17 +720,20 @@ def main(): ap.add_argument("--inpaint-range", default=None) ap.add_argument("--dit", choices=list(DIT_CHOICES.keys()), default=None) ap.add_argument("--decoder", choices=list(DECODER_PATHS.keys()), default=None) - ap.add_argument("--precision", choices=list(canon.PRECISIONS), default=None, - help="DiT engine precision. Default is 'fp16mixed' for every model: " + ap.add_argument("--dec-precision", choices=list(canon.DECODER_TIERS), default="canonical", + help="Decoder/encoder quantization tier (orthogonal to --precision, which is the " + "DiT). 'canonical' (bf16, default); 'fp8' (~1.14x, near-transparent); " + "'fp8_fast' (SAME-S only; fp8 attention projections too, ~1.22x, lossier). " + "Engine auto-downloads from HF. (For max-fidelity decode use --precision fp32.)") + ap.add_argument("--precision", choices=list(canon.PRECISIONS) + list(canon._PRECISION_ALIAS), default=None, + help="DiT engine precision. Default is 'fp16' for every model: " "canonical FP16 trunk + FP32 RMSNorm/RoPE islands + FMHA-fused FP16 " "attention core, fp32-accurate at every length. 'fp8' (medium only) " "is the max-speed clean tier — fp8 linears + bf16 fused FMHA + baked " - "fp32 RoPE, ~1.3x faster than fp16mixed at every length and clean at " - "long sequence (L<=4096). 'bf16' (medium only) is ~3%% faster than " - "fp16mixed but evaluates RoPE's angle in bf16 and drifts at long " - "sequence (clips on a 6-min render); not seed-reproducible vs " - "fp16mixed. 'fp32' = bit-equiv PyTorch eager, slower. " - "Auto-downloads from HF.") + "fp32 RoPE, ~1.3x faster than fp16 at every length and clean at " + "long sequence (L<=4096). 'fp32' = bit-equiv PyTorch eager, slower. " + "('fp16mixed' and the retired medium-only 'bf16' are accepted as " + "deprecated aliases for 'fp16'.) Auto-downloads from HF.") ap.add_argument("--models-dir", default=str(canon.MODELS_DIR)) ap.add_argument("--seconds", type=float, default=30.0) ap.add_argument("--steps", type=int, default=8) @@ -749,6 +751,10 @@ def main(): args = ap.parse_args() if args.steps < 1: ap.error(f"--steps must be ≥ 1 (got {args.steps})") + if args.precision in canon._PRECISION_ALIAS: # retired token → canonical, with a note + _canon_p = canon._PRECISION_ALIAS[args.precision] + print(f" note: --precision {args.precision} is retired → using {_canon_p}", file=sys.stderr) + args.precision = _canon_p # Mute display in quiet mode — match sa3_trt's behavior. if args.quiet: @@ -766,7 +772,7 @@ def _noop_vram(label): return 0 if args.models_dir != str(canon.MODELS_DIR): new_root = Path(args.models_dir).resolve() new_arch_dir = new_root / ARCH - canon.T5GEMMA_PATH = new_arch_dir / "t5gemma" / "t5gemma_fp16mixed.trt" + canon.T5GEMMA_PATH = new_arch_dir / "t5gemma" / "t5gemma_fp16.trt" for kk in DIT_CHOICES: DIT_CHOICES[kk]["engine"] = new_arch_dir / DIT_CHOICES[kk]["engine"].relative_to(canon.ARCH_DIR) for kk in DECODER_PATHS: @@ -837,7 +843,8 @@ def _noop_vram(label): return 0 if args.negative_prompt: suffix = "" if args.cfg != 1.0 else dim(" (ignored: --cfg=1.0)") print(f" {k('neg prompt')} {bold(repr(args.negative_prompt))}{suffix}") - print(f" {k('dit')} {magenta(v(args.dit))} {k('decoder')} {magenta(v(args.decoder))} {k('precision')} {v(args.precision)}") + _dtier = f" {k('dec-tier')} {magenta(v(args.dec_precision))}" if args.dec_precision != "canonical" else "" + print(f" {k('dit')} {magenta(v(args.dit))} {k('decoder')} {magenta(v(args.decoder))}{_dtier} {k('precision')} {v(args.precision)}") print(f" {k('σmax')} {bold(f'{sigma_max:.2f}')}") print(f" {k('seconds')} {v(f'{args.seconds}s')} {k('steps')} {v(args.steps)} {k('seed')} {args.seed}") print(f" {k('cfg')} {v(args.cfg)} {k('mega-graph')} {v('on' if use_mega else 'off (fallback)')}") @@ -853,7 +860,7 @@ def _noop_vram(label): return 0 # Lazy-download needed = canon.get_engine_files(args.dit, args.decoder, args.precision, - with_encoder=bool(args.init_audio)) + with_encoder=bool(args.init_audio), dec_tier=args.dec_precision) _ensure_files(needed) # Heavy imports @@ -879,10 +886,10 @@ def _noop_vram(label): return 0 engine_specs = { "t5": canon.T5GEMMA_PATH, "dit": canon.get_dit_engine_path(args.dit, args.precision), - "dec": canon.get_decoder_engine_path(args.decoder, args.precision), + "dec": canon.resolve_decoder_engine(args.decoder, args.precision, args.dec_precision), } if args.init_audio: - engine_specs["enc"] = ENCODER_PATHS[args.decoder] + engine_specs["enc"] = canon.resolve_encoder_engine(args.decoder, args.dec_precision) t0 = time.time() with concurrent.futures.ThreadPoolExecutor(max_workers=len(engine_specs)) as ex: futs = {name: ex.submit(TRTRunner, path) for name, path in engine_specs.items()} diff --git a/optimized/tensorRT/scripts/sa3_trt_core.py b/optimized/tensorRT/scripts/sa3_trt_core.py index 96df0065..3b0d5dff 100644 --- a/optimized/tensorRT/scripts/sa3_trt_core.py +++ b/optimized/tensorRT/scripts/sa3_trt_core.py @@ -95,15 +95,15 @@ def _detect_gpu_arch() -> str: ARCH = _detect_gpu_arch() ARCH_DIR = MODELS_DIR / ARCH HF_SUBDIR = f"tensorRT/{ARCH}" -T5GEMMA_PATH = ARCH_DIR / "t5gemma" / "t5gemma_fp16mixed.trt" +T5GEMMA_PATH = ARCH_DIR / "t5gemma" / "t5gemma_fp16.trt" # What engine files each DiT choice needs (relative to MODELS_DIR, mirroring HF repo layout). # Each DiT engine bundles its conditioner tensors (padding_embedding + seconds_total # Linear) as graph Constants, so no sidecar weight files are needed. DIT_ENGINE_FILES = { - "sm-music": ["sa3-sm-music/dit_fp16mixed.trt"], - "sm-sfx": ["sa3-sm-sfx/dit_fp16mixed.trt"], - "medium": ["sa3-m/dit_fp16mixed.trt"], # fp16mixed is the medium default + "sm-music": ["sa3-sm-music/dit_fp16.trt"], + "sm-sfx": ["sa3-sm-sfx/dit_fp16.trt"], + "medium": ["sa3-m/dit_fp16.trt"], # fp16 is the medium default } DECODER_FILES = { "same-s": [ @@ -121,15 +121,15 @@ def _detect_gpu_arch() -> str: # T5Gemma engine — downloaded from HF per-arch. (The tokenizer.json is # arch-agnostic and ships bundled with the repo at scripts/tokenizer.json, # so it's NOT in this list.) - "t5gemma/t5gemma_fp16mixed.trt", + "t5gemma/t5gemma_fp16.trt", ] DIT_CHOICES = { - "sm-music": {"engine": ARCH_DIR / "sa3-sm-music" / "dit_fp16mixed.trt", + "sm-music": {"engine": ARCH_DIR / "sa3-sm-music" / "dit_fp16.trt", "default_decoder": "same-s"}, - "sm-sfx": {"engine": ARCH_DIR / "sa3-sm-sfx" / "dit_fp16mixed.trt", + "sm-sfx": {"engine": ARCH_DIR / "sa3-sm-sfx" / "dit_fp16.trt", "default_decoder": "same-s"}, - "medium": {"engine": ARCH_DIR / "sa3-m" / "dit_fp16mixed.trt", # medium default + "medium": {"engine": ARCH_DIR / "sa3-m" / "dit_fp16.trt", # medium default "default_decoder": "same-l"}, } DECODER_PATHS = { @@ -144,78 +144,71 @@ def _detect_gpu_arch() -> str: # ─── Precision-keyed engine maps ───────────────────────────────────────── # -# Three DiT precisions: -# fp16mixed — canonical AND the default for every DiT, medium included since -# 2026-07. FP16 trunk with FP32 islands around RMSNorm and the RoPE -# *generation*; the attention core (QK^T → Softmax → P·V) runs FP16 -# so TRT's FMHA fuser fires (96 fused nodes on medium). Before that -# island was bounded the whole O(L²) core sat in FP32 unfused, which -# is what made bf16 look 4.7× faster. Now: teacher-forced velocity -# cos 1.0000 vs the FP32 engine at every length, and within ~3% of -# bf16's speed. Requires STRONGLY_TYPED (see build_dit_fp16mixed.py). -# bf16 — medium ONLY. Same dit.onnx as fp32, built with BuilderFlag.BF16 -# (EXPLICIT_BATCH), which lets the FMHA fuser fire on a uniform bf16 -# trunk. Marginally the fastest engine, but LOW-FIDELITY AT LONG -# SEQUENCE: it evaluates RoPE's rotation angle in bf16, and that -# angle reaches ~4155 rad at L=4092 where bf16's spacing is 32 rad -# (> 2π), so position information for the fast-rotating dims is -# destroyed. Over 8 sampling steps the latent inflates ~2.5× and the -# decoder clips 2–3% of samples on a 6-min render. Fine at short -# lengths (clean at L=256); prefer fp16mixed for anything long. -# Also not seed-reproducible vs fp16mixed. +# Three DiT precisions (fp16mixed was renamed → fp16 — every tier is mixed, so the +# qualifier was noise; the medium-only bf16 tier was retired. Both old tokens alias +# to fp16, see _PRECISION_ALIAS): +# fp16 — canonical AND the default for every DiT. FP16 trunk with FP32 +# islands around RMSNorm and the RoPE *generation*; the attention +# core (QK^T → Softmax → P·V) runs FP16 so TRT's FMHA fuser fires +# (96 fused nodes on medium). Bounding that island is what lets the +# core fuse — before it, the whole O(L²) core sat in FP32 unfused +# (which is what made the retired bf16 tier look 4.7× faster). Now: +# teacher-forced velocity cos 1.0000 vs the FP32 engine at every +# length, ~3% off the old bf16 speed, and accurate at long sequence +# (bf16 drifted — it evaluated RoPE's angle too coarsely past ~2048 +# rad, ~4155 rad @L4092 vs bf16 spacing 32 rad, clipping 2–3% of a +# 6-min render). Requires STRONGLY_TYPED (see build_dit_fp16.py). # fp8 — all DiTs, fp8 E4M3 on the linear GEMMs (attention + RoPE kept # higher precision). On MEDIUM it's a max-speed clean tier (~1.3× -# over fp16mixed) via the recipe described below. On sm-music / +# over fp16) via the recipe described below. On sm-music / # sm-sfx it is a CLEAN WEIGHT-HALVING tier (engine 479 vs 936 MB), # only marginally faster (~1.10–1.17×): those DiTs' ~5 ms forward is # overhead-bound at batch 1, so fp8's GEMM-math savings barely show. -# Their fp8 is an fp8-QDQ graft onto the fp16mixed graph (fp8 linears -# + fp16 fused attention + the fp16mixed fp32 islands, STRONGLY_TYPED), -# velocity-cos ~0.99 vs eager, clip% at/below fp16mixed. medium recipe: +# Their fp8 is an fp8-QDQ graft onto the fp16 graph (fp8 linears +# + fp16 fused attention + the fp16 fp32 islands, STRONGLY_TYPED), +# velocity-cos ~0.99 vs eager, clip% at/below fp16. medium recipe: # MAX-SPEED clean tier. fp8 E4M3 on the 176 linear # GEMMs + bf16 fused FMHA (96 nodes) + a BAKED fp32 RoPE constant # table: position cos/sin are computed host-side at build time and # frozen as a graph Constant (no in-graph trig), so the island is # precision-policy-robust and cross-runtime-stable — a constant # can't be re-fused to bf16 the way a runtime pin can. ~1.3× faster -# than fp16mixed at every length (measured H200, same-run round- +# than fp16 at every length (measured H200, same-run round- # robin: 1.40× @L129 / 1.32× @L1292 / 1.30× @L4092) and CLEAN at # long sequence (latent std 0.86 vs eager 0.95, 0.000% clip @2min & # full) — the baked RoPE dodges bf16's long-angle drift. It is a # SPEED tier over an already-good default, NOT a fidelity upgrade: -# single-step velocity cos vs fp32 is ~0.92–0.97 (below fp16mixed's +# single-step velocity cos vs fp32 is ~0.92–0.97 (below fp16's # ~1.0) but the 8-step render stays coherent. Built weakly-typed # EXPLICIT_BATCH + BF16 + FP8 + OBEY_PRECISION_CONSTRAINTS (the -# opposite of fp16mixed's STRONGLY_TYPED — see build_from_onnx.py). +# opposite of fp16's STRONGLY_TYPED — see build_from_onnx.py). # The baked table is sized to the profile max L=4096 (= the SAME-L # decoder's own cap), so within the shipped range no re-bake is # needed; L>4096 is rejected (a re-bake would be required). # fp32 — pure-FP32, bit-equivalent to PyTorch eager. ~2× size/latency. # # The lookup tables below resolve the engine filename per (dit/decoder, -# precision). The bf16 DiT recipe is a build-time precision change only (no new -# ONNX): reuse sa3-m/dit.onnx, build with BF16. Decoders/encoders are unchanged -# by bf16 (it's a DiT-trunk fusion recipe), so decoder "bf16" reuses the -# canonical decoder engine. Encoders are FP16-mixed only. +# precision). Decoders/encoders have no DiT-specific recipe, so they reuse their +# canonical fp16 engine for every precision. Encoders are fp16 only. DIT_ENGINE_FILENAME = { - "bf16": "dit_bf16.trt", # medium only; drifts at long sequence - "fp8": "dit_fp8.trt", # all DiTs; fp8 linears (medium: +baked RoPE; small: graft on fp16mixed) - "fp16mixed": "dit_fp16mixed.trt", - "fp32": "dit_fp32.trt", + "fp8": "dit_fp8.trt", # all DiTs; fp8 linears (medium: +baked RoPE; small: graft on fp16) + "fp16": "dit_fp16.trt", # canonical + default for every DiT (fp16 trunk + fp32 RMSNorm/RoPE islands) + "fp32": "dit_fp32.trt", # pure fp32, bit-equivalent to eager } -# DiT precisions actually built per model. bf16 is medium-only; fp8 is available -# for all three (medium via baked-RoPE/bf16-attn; sm-music/sm-sfx via an fp8-QDQ -# graft onto their fp16mixed graph — fp8 linears + fp16 fused attn + fp32 islands). +# DiT precisions actually built per model. fp8 is available for all three (medium +# via baked-RoPE/bf16-attn; sm-music/sm-sfx via an fp8-QDQ graft onto their fp16 +# graph — fp8 linears + fp16 fused attn + fp32 islands). The retired medium-only +# bf16 tier (drifted at long sequence) now aliases to fp16 — see _PRECISION_ALIAS. _DIT_PRECISIONS = { - "sm-music": ("fp16mixed", "fp8", "fp32"), - "sm-sfx": ("fp16mixed", "fp8", "fp32"), - "medium": ("bf16", "fp8", "fp16mixed", "fp32"), + "sm-music": ("fp16", "fp8", "fp32"), + "sm-sfx": ("fp16", "fp8", "fp32"), + "medium": ("fp8", "fp16", "fp32"), } -# Per-DiT default precision — fp16mixed everywhere. Medium moved off bf16 once -# fp16mixed's attention core was fused (4.3x faster than the old fp16mixed engine, +# Per-DiT default precision — fp16 everywhere. Medium moved off bf16 once +# fp16's attention core was fused (4.3x faster than the old fp16 engine, # ~3% off bf16, and fp32-accurate at every sequence length). -DIT_DEFAULT_PRECISION = {"sm-music": "fp16mixed", "sm-sfx": "fp16mixed", - "medium": "fp16mixed"} +DIT_DEFAULT_PRECISION = {"sm-music": "fp16", "sm-sfx": "fp16", + "medium": "fp16"} _DIT_SUBDIR = {"sm-music": "sa3-sm-music", "sm-sfx": "sa3-sm-sfx", "medium": "sa3-m"} # Both SAME decoders' latent profiles start at L=32 (verified against the engines: # min=(1,256,32) for same-l and same-s; the DiT's starts at 1). Below 32, @@ -228,31 +221,97 @@ def _detect_gpu_arch() -> str: DECODER_ENGINE_FILENAME = { "same-l": { - # bf16/fp8 are DiT-only recipes → decoder reuses its canonical fp16-mixed engine. - "bf16": "dec_dynamic_triton_swa.trt", - "fp8": "dec_dynamic_triton_swa.trt", - "fp16mixed": "dec_dynamic_triton_swa.trt", - "fp32": "dec_dynamic_fp32.trt", + # fp8 is a DiT-only recipe → decoder reuses its canonical fp16 engine. + # (dec_dynamic_bf16.trt keeps its name — it's the same-s canonical engine.) + "fp8": "dec_dynamic_triton_swa.trt", + "fp16": "dec_dynamic_triton_swa.trt", + "fp32": "dec_dynamic_fp32.trt", }, "same-s": { - "bf16": "dec_dynamic_bf16.trt", - "fp8": "dec_dynamic_bf16.trt", - "fp16mixed": "dec_dynamic_bf16.trt", - "fp32": "dec_dynamic_fp32.trt", + "fp8": "dec_dynamic_bf16.trt", + "fp16": "dec_dynamic_bf16.trt", + "fp32": "dec_dynamic_fp32.trt", }, } -PRECISIONS = ("bf16", "fp8", "fp16mixed", "fp32") +PRECISIONS = ("fp8", "fp16", "fp32") +# Back-compat: retired precision tokens resolve to their replacement. +# fp16mixed → fp16 : every tier is mixed-precision, so the qualifier was noise. +# bf16 → fp16 : the medium-only bf16 DiT drifted at long sequence; retired. +_PRECISION_ALIAS = {"fp16mixed": "fp16", "bf16": "fp16"} +def normalize_precision(precision): + """Map a (possibly retired) precision token to its canonical name.""" + return _PRECISION_ALIAS.get(precision, precision) if precision else precision + +# ── Decoder / encoder quantization TIERS (orthogonal to the DiT --precision) ── +# Train-free fp8 tiers grafted onto the bf16 export (see quantize/README.md). +# "canonical" = the shipped default decoder engine. The others are downloaded from HF on demand +# (tensorRT///dec_*.trt) — the same wide-profile engines built by quantize/build_tiers.py. +# fp8 fp8 FFN GEMMs (near-transparent, ~1.14×) — the speed pick +# fp8_fast SAME-S only: fp8 on the attention projections too (~1.22×, lossier) +# (int8-weight-only "w8_bf16" was retired: its DequantizeLinear constant-folds to bf16 at build, so +# the engine was byte-for-byte the size/speed of the bf16 baseline with slightly lossier weights — +# strictly dominated. For a max-fidelity decoder use --precision fp32.) +DECODER_TIER_FILENAME = { + "same-s": {"canonical": DECODER_ENGINE_FILENAME["same-s"]["fp16"], + "fp8": "dec_fp8.trt", "fp8_fast": "dec_fp8_fast.trt"}, + "same-l": {"canonical": DECODER_ENGINE_FILENAME["same-l"]["fp16"], + "fp8": "dec_fp8.trt"}, +} +ENCODER_TIER_FILENAME = { + "same-s": {"canonical": "enc_dynamic_bf16.trt", + "fp8": "enc_fp8.trt", "fp8_fast": "enc_fp8_fast.trt"}, + "same-l": {"canonical": "enc_dynamic_triton_swa.trt", + "fp8": "enc_fp8.trt"}, +} +DECODER_TIERS = ("canonical", "fp8", "fp8_fast") + + +def get_decoder_tier_path(decoder_name: str, tier: str = "canonical") -> Path: + tiers = DECODER_TIER_FILENAME.get(decoder_name) + if tiers is None: + raise ValueError(f"unknown decoder={decoder_name!r}; valid: {list(DECODER_TIER_FILENAME)}") + if tier not in tiers: + raise ValueError(f"decoder tier {tier!r} not available for {decoder_name}; valid: {list(tiers)}") + return ARCH_DIR / decoder_name / tiers[tier] + + +def get_encoder_tier_path(decoder_name: str, tier: str = "canonical") -> Path: + tiers = ENCODER_TIER_FILENAME[decoder_name] + return ARCH_DIR / decoder_name / tiers.get(tier, tiers["canonical"]) # enc falls back if a tier is decoder-only + + +def decoder_tier_from_filename(decoder_name: str, filename: str) -> str: + """Reverse-map a decoder engine filename (e.g. 'dec_fp8.trt') to its tier name + ('fp8'); used by the gradio variant picker. Falls back to 'canonical'.""" + for tier, fname in DECODER_TIER_FILENAME.get(decoder_name, {}).items(): + if fname == filename: + return tier + return "canonical" + + +def resolve_decoder_engine(decoder_name: str, precision: str = None, dec_tier: str = "canonical") -> Path: + """dec_tier 'canonical' → the precision-driven canonical/fp32 decoder (back-compat with + --precision, e.g. fp32 still yields dec_dynamic_fp32.trt); any other tier → its engine.""" + if dec_tier == "canonical": + return get_decoder_engine_path(decoder_name, precision) + return get_decoder_tier_path(decoder_name, dec_tier) + + +def resolve_encoder_engine(decoder_name: str, dec_tier: str = "canonical") -> Path: + if dec_tier == "canonical": + return ENCODER_PATHS[decoder_name] + return get_encoder_tier_path(decoder_name, dec_tier) def default_precision(dit_name: str) -> str: - """Default DiT precision: fp16-mixed for every model. + """Default DiT precision: fp16 for every model. - Medium used bf16 until 2026-07, when bounding the fp16-mixed RoPE island let - its attention core fuse — that made fp16-mixed 4.3x faster than before and - within ~3% of bf16, while staying accurate at long sequence (bf16 evaluates - RoPE's angle too coarsely past ~2048 rad and drifts). + (fp16 = FP16 trunk + FP32 RMSNorm/RoPE islands + an FMHA-fused FP16 attention + core. Medium used the now-retired bf16 tier until 2026-07, when bounding the + fp16 RoPE island let its attention core fuse — 4.3x faster than before and + accurate at long sequence, where bf16 drifted.) """ - return DIT_DEFAULT_PRECISION.get(dit_name, "fp16mixed") + return DIT_DEFAULT_PRECISION.get(dit_name, "fp16") def get_dit_engine_path(dit_name: str, precision: str = None) -> Path: @@ -260,15 +319,11 @@ def get_dit_engine_path(dit_name: str, precision: str = None) -> Path: raise ValueError(f"unknown dit={dit_name!r}; valid: {list(_DIT_SUBDIR)}") if precision is None: precision = default_precision(dit_name) + precision = normalize_precision(precision) if precision not in DIT_ENGINE_FILENAME: raise ValueError(f"unknown precision={precision!r}; valid: {PRECISIONS}") - if precision == "bf16" and dit_name != "medium": - raise ValueError( - f"precision='bf16' is only available for --dit medium (FMHA-fused); " - f"{dit_name} uses standard attention and already fuses in fp16mixed. " - f"Valid for {dit_name}: {_DIT_PRECISIONS.get(dit_name)}") # fp8 is available for all three DiTs (medium: baked-RoPE + bf16 attn; sm-music/ - # sm-sfx: fp8-QDQ graft on their fp16mixed graph). Only bf16 stays medium-only. + # sm-sfx: fp8-QDQ graft on their fp16 graph). return ARCH_DIR / _DIT_SUBDIR[dit_name] / DIT_ENGINE_FILENAME[precision] @@ -276,24 +331,33 @@ def get_decoder_engine_path(decoder_name: str, precision: str = None) -> Path: if decoder_name not in DECODER_ENGINE_FILENAME: raise ValueError(f"unknown decoder={decoder_name!r}; valid: {list(DECODER_ENGINE_FILENAME)}") if precision is None: - precision = "fp16mixed" # decoders have no bf16-specific engine; canonical + precision = "fp16" # decoders track the canonical (fp16) engine + precision = normalize_precision(precision) if precision not in DECODER_ENGINE_FILENAME[decoder_name]: raise ValueError(f"unknown precision={precision!r}; valid: {PRECISIONS}") return ARCH_DIR / decoder_name / DECODER_ENGINE_FILENAME[decoder_name][precision] def get_engine_files(dit_name: str, decoder_name: str, precision: str = None, - with_encoder: bool = False) -> list[str]: + with_encoder: bool = False, dec_tier: str = "canonical") -> list[str]: """Relative paths (under ARCH_DIR) needed for the chosen pipeline. Pass this list to _ensure_files() to auto-download anything missing from HF. - precision=None resolves to the per-model default (fp16-mixed).""" + precision=None resolves to the per-model default (fp16); dec_tier picks the + decoder/encoder quantization tier (canonical / fp8 / fp8_fast).""" if precision is None: precision = default_precision(dit_name) + precision = normalize_precision(precision) files = list(SHARED_FILES) files.append(f"{_DIT_SUBDIR[dit_name]}/{DIT_ENGINE_FILENAME[precision]}") - files.append(f"{decoder_name}/{DECODER_ENGINE_FILENAME[decoder_name][precision]}") - if with_encoder: - files.append(f"{decoder_name}/" + ENCODER_PATHS[decoder_name].name) + if dec_tier == "canonical": # precision-driven canonical/fp32 decoder (back-compat) + files.append(f"{decoder_name}/{DECODER_ENGINE_FILENAME[decoder_name][precision]}") + if with_encoder: + files.append(f"{decoder_name}/{ENCODER_PATHS[decoder_name].name}") + else: # a quantization tier + files.append(f"{decoder_name}/{DECODER_TIER_FILENAME[decoder_name][dec_tier]}") + if with_encoder: + enc = ENCODER_TIER_FILENAME[decoder_name] + files.append(f"{decoder_name}/{enc.get(dec_tier, enc['canonical'])}") return files # ─── Display helpers (ANSI color when stdout is a TTY) ─────────────────── @@ -1082,14 +1146,14 @@ def prompt_user_if_missing(args): suggested = DIT_CHOICES[args.dit]["default_decoder"] args.decoder = _arrow_pick("Choose audio decoder:", list(DECODER_PATHS.keys()), default=suggested) print(f" → {args.decoder}") - # Resolve DiT precision default per model: fp16-mixed everywhere. bf16 is - # medium-only and selectable (sm-music / sm-sfx use standard attention and - # already fuse in fp16mixed). + # Resolve DiT precision: default fp16 for every model. Retired tokens + # (fp16mixed, bf16) alias to fp16 with a one-line note. if getattr(args, "precision", None) is None: args.precision = default_precision(args.dit) - if args.precision == "bf16" and args.dit != "medium": - sys.exit("error: --precision bf16 is only available for --dit medium " - "(sm-music / sm-sfx already fuse in fp16mixed).") + elif args.precision in _PRECISION_ALIAS: + _canon = _PRECISION_ALIAS[args.precision] + print(f" note: --precision {args.precision} is retired → using {_canon}", file=sys.stderr) + args.precision = _canon if args.seed is None: args.seed = random.randint(0, 2**31 - 1) return args @@ -1137,20 +1201,22 @@ def main(): ap.add_argument("--decoder", choices=list(DECODER_PATHS.keys()), default=None, help="Audio decoder. 'same-s' pairs with sm-* (110 MB engine). " "'same-l' pairs with medium (1.2 GB engine). Interactive picker if omitted.") - ap.add_argument("--precision", choices=list(PRECISIONS), default=None, - help="DiT engine precision (default is 'fp16mixed' for every model). " - "'fp16mixed' = FP16 trunk + FP32 RMSNorm/RoPE islands with an FMHA-fused " + ap.add_argument("--dec-precision", choices=list(DECODER_TIERS), default="canonical", + help="Decoder/encoder quantization tier (orthogonal to --precision, the DiT): " + "canonical (bf16) | fp8 (~1.14x, near-transparent) | fp8_fast (SAME-S only, " + "~1.22x). Auto-downloads from HF. (For max-fidelity decode use --precision fp32.)") + ap.add_argument("--precision", choices=list(PRECISIONS) + list(_PRECISION_ALIAS), default=None, + help="DiT engine precision (default 'fp16' for every model). " + "'fp16' = FP16 trunk + FP32 RMSNorm/RoPE islands with an FMHA-fused " "FP16 attention core (canonical; fp32-accurate at every length). " "'fp8' = fp8 E4M3 linears (attention + RoPE kept higher precision). On MEDIUM " "it's a max-speed clean tier (bf16 fused FMHA + baked fp32 RoPE, ~1.3× faster, " "clean at long sequence, capped at L<=4096). On sm-music/sm-sfx it's a clean " "weight-halving tier (479 vs 936 MB engine, velocity-cos ~0.99 vs eager) that is " "only marginally faster (~1.1×; their small forward is overhead-bound at batch 1). " - "'bf16' (MEDIUM ONLY) = ~3%% faster still, but it evaluates RoPE's angle " - "in bf16 and drifts at long sequence (clips 2-3%% of samples on a 6-min " - "render); fine for short clips, not seed-reproducible vs fp16mixed. " "'fp32' = pure FP32, matches PyTorch eager bit-for-bit but ~2× slower " - "and ~2× the VRAM. Engines auto-download from HF if missing.") + "and ~2× the VRAM. ('fp16mixed' and the retired medium-only 'bf16' are accepted " + "as deprecated aliases for 'fp16'.) Engines auto-download from HF if missing.") ap.add_argument("--models-dir", default=str(MODELS_DIR), help=f"Directory containing the TRT engines. Default: {MODELS_DIR}") # ── Sampling ── @@ -1207,7 +1273,7 @@ def _stage_vram(label): return 0 if args.models_dir != str(MODELS_DIR): new_root = Path(args.models_dir).resolve() new_arch_dir = new_root / ARCH - T5GEMMA_PATH = new_arch_dir / "t5gemma" / "t5gemma_fp16mixed.trt" + T5GEMMA_PATH = new_arch_dir / "t5gemma" / "t5gemma_fp16.trt" for kk in DIT_CHOICES: DIT_CHOICES[kk]["engine"] = new_arch_dir / DIT_CHOICES[kk]["engine"].relative_to(ARCH_DIR) for kk in DECODER_PATHS: @@ -1309,7 +1375,7 @@ def _stage_vram(label): return 0 # ── Lazy-download any missing engine files for the chosen (dit, decoder) combo ── needed = get_engine_files(args.dit, args.decoder, args.precision, - with_encoder=bool(args.init_audio)) + with_encoder=bool(args.init_audio), dec_tier=args.dec_precision) _ensure_files(needed) # ── Heavy imports (torch + tensorrt + plugin) ── @@ -1337,10 +1403,10 @@ def _stage_vram(label): return 0 engine_specs = { "t5": T5GEMMA_PATH, "dit": get_dit_engine_path(args.dit, args.precision), - "dec": get_decoder_engine_path(args.decoder, args.precision), + "dec": resolve_decoder_engine(args.decoder, args.precision, args.dec_precision), } if args.init_audio: - engine_specs["enc"] = ENCODER_PATHS[args.decoder] + engine_specs["enc"] = resolve_encoder_engine(args.decoder, args.dec_precision) t0 = time.time() with concurrent.futures.ThreadPoolExecutor(max_workers=len(engine_specs)) as ex: futs = {name: ex.submit(TRTRunner, path) for name, path in engine_specs.items()} diff --git a/optimized/tflite/README.md b/optimized/tflite/README.md index 9fd20034..97f8ac98 100644 --- a/optimized/tflite/README.md +++ b/optimized/tflite/README.md @@ -72,7 +72,7 @@ lazy-download from HuggingFace on first use. | `--precision` | legacy name | size (sm DiT / medium DiT / codecs) | quality | CPU speed | |---------------|-------------|-------------------------------------|---------|-----------| | `fp32` (default) | — | 1.8 GB / 5.8 GB / 0.2–1.8 GB | reference | 1× — on CPU this is *also* the fast choice | -| `w16a32` | fp16mixed | 0.9 / 2.9 / 0.1–0.9 GB | ≈lossless (fp16 weights, fp32 activations; 62–75 dB per-forward) | 1.5–3× slower, model-dependent (XNNPACK dequantizes per-matmul) | +| `w16a32` | fp16 | 0.9 / 2.9 / 0.1–0.9 GB | ≈lossless (fp16 weights, fp32 activations; 62–75 dB per-forward) | 1.5–3× slower, model-dependent (XNNPACK dequantizes per-matmul) | | `w8a32` | woint8 | 0.45 / 1.5 / 0.05–0.5 GB | GPTQ int8 weights — codecs transparent (40–46 dB); DiT gives a *different but plausible* sample | ≈fp32 | | `w8a8-dyn` | dynint8 | 0.45 / 1.5 / 0.05–0.5 GB | lowest (int8 weights + activations, per-invoke dynamic scales) | fastest (~1.2–1.3×) | diff --git a/optimized/tflite/scripts/sa3_tflite.py b/optimized/tflite/scripts/sa3_tflite.py index 9c020ae3..f022e1a2 100644 --- a/optimized/tflite/scripts/sa3_tflite.py +++ b/optimized/tflite/scripts/sa3_tflite.py @@ -75,7 +75,7 @@ # fp32 reference models (default — on CPU fp16 is SLOWER than fp32, so unlike # TRT the fastest-and-accurate choice is fp32) # w16a32 fp16 weights / fp32 activations — half size, ≈lossless, 1.5-3× slower -# on an M4 Pro (legacy name: fp16mixed; ≈ TRT's fp16mixed in spirit, +# on an M4 Pro (legacy name: fp16; ≈ TRT's fp16 in spirit, # storage-only here) # w8a32 GPTQ weight-only int8 — ¼ size at fp32 speed (legacy: woint8) # w8a8-dyn GPTQ dynamic int8 — fastest (~1.2-1.3×), lowest quality (legacy: dynint8)