diff --git a/optimized/tensorRT/README.md b/optimized/tensorRT/README.md
index 8d552261..304e18a2 100644
--- a/optimized/tensorRT/README.md
+++ b/optimized/tensorRT/README.md
@@ -119,30 +119,32 @@ Omit `--dit` / `--decoder` for an interactive arrow-key picker. Relative
### DiT precision (`--precision`)
-`fp16mixed` is the default for every model — no flag needed for the recommended setup:
+`fp16` is the default for every model — no flag needed for the recommended setup:
| model | default | why |
|------------------|-------------|------------------------------------------------------------|
-| `medium` | `fp16mixed` | FMHA-fused (96 fused attention nodes) **and** fp32-accurate at every length |
-| `sm-music`/`sm-sfx` | `fp16mixed` | standard attention — already fuses in fp16-mixed |
-
-`--precision` also takes `fp8` (all DiTs), `bf16` (medium only) and `fp32` explicitly:
-
-- **`fp16mixed`** — canonical: FP16 trunk, FP32 islands around RMSNorm and RoPE
- generation, and an FP16 attention core (QK^T → Softmax → P·V) so TRT's FMHA fuser
- fires. Teacher-forced velocity cos **1.0000** vs the FP32 engine at every length.
- Bit-reproducible run to run.
-
Medium used to default to `bf16` because this engine's attention core was
- stuck in FP32 and therefore unfused — 0 fused MHA nodes. Bounding the RoPE island
- before QK^T fixed that (**4.3× faster at L=4096**), which retired the reason to
- prefer `bf16`. Engines built before 2026-07 are the slow variant; rebuild with
+| `medium` | `fp16` | FMHA-fused (96 fused attention nodes) **and** fp32-accurate at every length |
+| `sm-music`/`sm-sfx` | `fp16` | standard attention — already fuses in fp16 |
+
+`--precision` also takes `fp8` (all DiTs) and `fp32` explicitly:
+
+- **`fp16`** — canonical **and default for every DiT** (formerly `fp16mixed`; every
+ tier is mixed-precision, so the qualifier was dropped). FP16 trunk, FP32 islands
+ around RMSNorm and RoPE generation, and an FP16 attention core (QK^T → Softmax →
+ P·V) so TRT's FMHA fuser fires. Teacher-forced velocity cos **1.0000** vs the FP32
+ engine at every length. Bit-reproducible run to run.
+
Medium used to default to a uniform-`bf16` engine because this engine's
+ attention core was stuck in FP32 and therefore unfused — 0 fused MHA nodes.
+ Bounding the RoPE island before QK^T fixed that (**4.3× faster at L=4096**), which
+ retired the `bf16` tier entirely (it also drifted at long sequence — see below).
+ Engines built before 2026-07 are the slow variant; rebuild with
`build_from_onnx.py sa3-m`.
- **`fp8`** — *medium: the max-speed clean tier, calibrated; sm-music/sm-sfx: a clean
weight-halving tier (see the end of this bullet).* On **medium**: fp8 E4M3 on the 176
linear GEMMs + bf16 fused FMHA (96 nodes) + a **baked fp32 RoPE constant table** (position
cos/sin computed host-side at build and frozen as a graph constant — no in-graph trig,
so the island is precision-policy-robust and cross-runtime-stable). **~1.3× faster than
- `fp16mixed` at every length** (H200, same-run round-robin: 1.40× @L129 / 1.32× @L1292 /
+ `fp16` at every length** (H200, same-run round-robin: 1.40× @L129 / 1.32× @L1292 /
1.34× @L4092 — also ahead of `bf16`) and **clean at long sequence** (latent std 0.86 vs
eager 0.95, **0.000% clip** at 2-min and 6-min), so it stays clean exactly where `bf16`
clips. The fp8 scales are **calibrated on real conditioning** (per-tensor activation amax +
@@ -151,41 +153,37 @@ Omit `--dit` / `--decoder` for an interactive arrow-key picker. Relative
velocity-cos vs fp32 on adversarial seeds **0.52/0.57/0.64 → 0.92/0.94/0.92** (steps 1–7
match the fully-calibrated reference within ~0.001). It remains a **speed tier over an
already-good default, not a fidelity upgrade over it**: single-step velocity cos ~0.92–0.94
- (below fp16mixed's ~1.0) but the 8-step render stays coherent — it just no longer collapses
+ (below fp16's ~1.0) but the 8-step render stays coherent — it just no longer collapses
at the highest-noise first step the way the uncalibrated engine did. **Capped at L≤4096** —
the baked table is sized to the profile max, which is *also* the SAME-L decoder's own cap,
so this is not a new end-to-end limit; longer renders are rejected (a re-bake would be
- needed). **Not seed-reproducible vs fp16-mixed.** Built weakly-typed `EXPLICIT_BATCH` +
+ needed). **Not seed-reproducible vs fp16.** Built weakly-typed `EXPLICIT_BATCH` +
`BF16` + `FP8` + `OBEY_PRECISION_CONSTRAINTS` (rebuild: `build_from_onnx.py sa3-m-fp8`;
producer: `build/build_dit_bf16.py` RoPE-baker + `build/transplant_scales.py` calibrated-scale
transplant; identity check: `scripts/verify_fp8_rope.py`; calibration by @ryanontheinside,
[#47](https://github.com/Stability-AI/stable-audio-3/pull/47)).
On **sm-music / sm-sfx** fp8 is a **different, simpler recipe** — fp8 E4M3 grafted onto
- the linear GEMMs of the fp16mixed graph (attention stays fp16-fused, fp32 islands untouched;
+ the linear GEMMs of the fp16 graph (attention stays fp16-fused, fp32 islands untouched;
no baked RoPE — these DiTs never had bf16's long-angle problem). It's a **clean weight-halving
- tier** (engine 479 vs 936 MB, velocity-cos **~0.99** vs eager, clip% at/below fp16mixed) that
+ tier** (engine 479 vs 936 MB, velocity-cos **~0.99** vs eager, clip% at/below fp16) that
is only **marginally faster (~1.1×)**: a small DiT's ~5 ms forward at batch 1 is overhead-bound,
- so fp8's GEMM savings barely show. Default stays fp16mixed. Rebuild: `build_from_onnx.py
+ so fp8's GEMM savings barely show. Default stays fp16. Rebuild: `build_from_onnx.py
sa3-sm-music-fp8` / `sa3-sm-sfx-fp8`; producer: `build/make_dit_fp8_smalldit.py`. Not
- seed-reproducible vs fp16mixed.
-- **`bf16`** — *medium only.* Same `dit.onnx` as fp32, built with `BuilderFlag.BF16`;
- a uniform bf16 trunk also lets the FMHA fuser fire, and it is ~3% faster than
- `fp16mixed`. **But it drifts at long sequence**: weakly-typed BF16 lets TRT
- evaluate RoPE's rotation angle in bf16, and that angle reaches ~4155 rad at
- L=4092 where bf16's spacing is 32 rad (bigger than a full 2π rotation), so
- position information for the fast-rotating dims is destroyed. The latent inflates
- ~2.5× over the 8 steps and a 6-minute render clips 2–3% of samples. Clean at short
- lengths; use `fp16mixed` for anything long. Also **not seed-reproducible vs
- fp16-mixed** — a different-but-equal take per seed. Same varlen profile
- (L 1..4096, opt 1292) and full mode/feature set as the other precisions.
+ seed-reproducible vs fp16.
- **`fp32`** — pure FP32, bit-equivalent to PyTorch eager (~2× slower, ~2× VRAM).
+> **Retired: `bf16` (was medium-only).** A uniform-bf16 trunk was ~3% faster than
+> `fp16` but **drifted at long sequence** — weakly-typed BF16 evaluated RoPE's
+> rotation angle in bf16, which reaches ~4155 rad at L=4092 where bf16's spacing is
+> 32 rad (> 2π), destroying position info for the fast-rotating dims (latent inflates
+> ~2.5×, a 6-min render clips 2–3% of samples). `fp16` matches its FMHA fusion
+> without the drift, so `bf16` was removed. `--precision bf16` (and the old
+> `--precision fp16mixed`) now silently alias to `fp16`.
+
```bash
-# medium defaults to fp16mixed. fp8 is the max-speed clean tier (~1.3x faster at
-# every length, clean at long L where bf16 clips):
+# medium defaults to fp16. fp8 is the max-speed clean tier (~1.3x faster at
+# every length, clean at long sequence):
./sa3 --prompt "..." --dit medium --decoder same-l --precision fp8
-# bf16 trades long-sequence accuracy for the last ~3% of speed:
-./sa3 --prompt "..." --dit medium --decoder same-l --precision bf16
```
The TRT DiT engines are static batch=1 (the ONNX bakes batch=1), so CFG runs as a
@@ -225,7 +223,7 @@ variance once the graph is built).
```bash
.venv/bin/python scripts/bench_dit_profile.py \
- --engines "canonical=models/sm_90/sa3-sm-music/dit_fp16mixed.trt" \
+ --engines "canonical=models/sm_90/sa3-sm-music/dit_fp16.trt" \
--lvals 1,32,128,256,512,1024,1292,2048,4096 --warmup 3 --runs 7
```
@@ -275,10 +273,10 @@ optimized/tensorRT/
│ └── build_dit_profile.py ← DiT with custom (min, opt, max) profile shapes
└── models/ ← .trt engines (auto-downloaded per arch; ~8 GB)
└── sm_/ ← arch dir matches `nvidia-smi --query-gpu=compute_cap`
- ├── t5gemma/t5gemma_fp16mixed.trt
- ├── sa3-sm-music/dit_fp16mixed.trt
- ├── sa3-sm-sfx/dit_fp16mixed.trt
- ├── sa3-m/dit_fp16mixed.trt ← + dit_fp8.trt / dit_bf16.trt / dit_fp32.trt if selected
+ ├── t5gemma/t5gemma_fp16.trt
+ ├── sa3-sm-music/dit_fp16.trt
+ ├── sa3-sm-sfx/dit_fp16.trt
+ ├── sa3-m/dit_fp16.trt ← + dit_fp8.trt / dit_fp32.trt if selected
├── same-s/{enc,dec}_dynamic_bf16.trt
└── same-l/{enc,dec}_dynamic_triton_swa.trt
```
diff --git a/optimized/tensorRT/build/README.md b/optimized/tensorRT/build/README.md
index de5b1af5..38670ea3 100644
--- a/optimized/tensorRT/build/README.md
+++ b/optimized/tensorRT/build/README.md
@@ -13,18 +13,18 @@ Run the build on the target GPU; TensorRT bakes the arch into the engine, so the
```
consumer flow producer flow
───────────── ─────────────
-HuggingFace onnx// ←─────── publish (incl. dit_fp16mixed.onnx)
+HuggingFace onnx// ←─────── publish (incl. dit_fp16.onnx)
tensorRT// ←─── compile + commit source ckpts
│ │
↓ ↓
build.py build_*.py
build_from_onnx.py (build_t5gemma.py,
(just compile, build_dit.py,
- STRONGLY_TYPED; build_dit_fp16mixed.py,
+ STRONGLY_TYPED; build_dit_fp16.py,
no graphsurgeon) build_same_*.py)
```
-The SA3 DiT ships both an FP32 canonical `dit.onnx` (regenerable from PyTorch source) and a pre-processed `dit_fp16mixed.onnx` (canonical + FP32 islands around RMSNorm and RoPE *generation*, FP16 attention core, rest converted to FP16). Consumers use the pre-processed one; producers refresh both when the model retrains.
+The SA3 DiT ships both an FP32 canonical `dit.onnx` (regenerable from PyTorch source) and a pre-processed `dit_fp16.onnx` (canonical + FP32 islands around RMSNorm and RoPE *generation*, FP16 attention core, rest converted to FP16). Consumers use the pre-processed one; producers refresh both when the model retrains.
## Consumer flow (default)
@@ -46,7 +46,7 @@ python build.py # interactive menu
Output dir: models/sm_100/
[1] ✓ t5gemma (text encoder + tokenizer)
- ✓ t5gemma/t5gemma_fp16mixed.trt 538.1 MB
+ ✓ t5gemma/t5gemma_fp16.trt 538.1 MB
✓ t5gemma/tokenizer.json 32.8 MB
[2] ✗ same-s encoder
✗ same-s/enc_dynamic_bf16.trt (missing)
@@ -223,17 +223,17 @@ python build_dit.py sa3-sm-sfx
python build_dit.py sa3-m
```
-After the DiT ONNXes are exported, run the FP16-mixed precision-island surgery on each one (see `build_dit_fp16mixed.py`):
+After the DiT ONNXes are exported, run the FP16-mixed precision-island surgery on each one (see `build_dit_fp16.py`):
```bash
-python build_dit_fp16mixed.py \
+python build_dit_fp16.py \
--input /onnx/sa3-sm-music/dit.onnx \
- --onnx /onnx/sa3-sm-music/dit_fp16mixed.onnx \
- --engine ../models//sa3-sm-music/dit_fp16mixed.trt
+ --onnx /onnx/sa3-sm-music/dit_fp16.onnx \
+ --engine ../models//sa3-sm-music/dit_fp16.trt
# repeat for sa3-sm-sfx and sa3-m
```
-This wraps every RMSNorm chain, attention `Softmax`, and the RoPE region in `Cast(FP32) → op → Cast(FP16)` islands, converts the rest of the weights to FP16, then **bounds the RoPE island before QK^T** (`bound_attention_core()`) so the attention core — QK^T → Softmax → P·V — runs FP16, and finally compiles a `STRONGLY_TYPED` TRT engine. It writes BOTH the modified `dit_fp16mixed.onnx` (~half the size of the original) AND the TRT engine. Publishing the modified ONNX is what lets consumers compile their own engines with plain `build_from_onnx.py` (no `onnx-graphsurgeon` dependency on the consumer side).
+This wraps every RMSNorm chain, attention `Softmax`, and the RoPE region in `Cast(FP32) → op → Cast(FP16)` islands, converts the rest of the weights to FP16, then **bounds the RoPE island before QK^T** (`bound_attention_core()`) so the attention core — QK^T → Softmax → P·V — runs FP16, and finally compiles a `STRONGLY_TYPED` TRT engine. It writes BOTH the modified `dit_fp16.onnx` (~half the size of the original) AND the TRT engine. Publishing the modified ONNX is what lets consumers compile their own engines with plain `build_from_onnx.py` (no `onnx-graphsurgeon` dependency on the consumer side).
The attention-core step is not cosmetic. Without it the RoPE island emits q/k in FP32 and nothing casts them back, so the whole O(L²) core stays FP32 and TRT's FMHA fuser cannot fire: **0 fused MHA nodes and 4.3× slower at L=4096** on the medium DiT, for no accuracy gain (teacher-forced velocity cos vs the FP32 engine is 1.0000 with the step, 0.9998 without). PyTorch does the same thing the step does — `apply_rotary_pos_emb` ends with `t.to(out_dtype)` and attention is then a fused `scaled_dot_product_attention` over FP16 inputs — and TRT's FMHA kernels accumulate softmax in FP32 internally anyway. `--no-bound-attn` skips it, only useful for reproducing the pre-2026-07 engines.
@@ -253,12 +253,13 @@ git push
## Medium `fp8` — the max-speed RoPE-baked engine, calibrated
-On top of the `fp16mixed` default and the selectable `bf16`, the medium DiT ships an **`fp8`**
+On top of the `fp16` default (the medium-only `bf16` tier was retired — its uniform-bf16
+build survives only internally as the fp8 RoPE-baker), the medium DiT ships an **`fp8`**
engine (`dit_fp8.trt`, `--precision fp8`): fp8 E4M3 on the 176 linear GEMMs + 96 bf16 fused
-FMHA + the **same baked fp32 RoPE constant table** as bf16. Measured on H200 it is ~1.3×
-faster than `fp16mixed` at every length (1.40× / 1.32× / 1.34× @L129/1292/4092, also ahead of
-bf16) and clean at long sequence (latent std 0.86, 0.000% clip at 6-min), so it stays clean
-exactly where bf16 clips.
+FMHA + the **same baked fp32 RoPE constant table**. Measured on H200 it is ~1.3×
+faster than `fp16` at every length (1.40× / 1.32× / 1.34× @L129/1292/4092) and clean at
+long sequence (latent std 0.86, 0.000% clip at 6-min), so it stays clean exactly where the
+old bf16 tier clipped.
The fp8 scales are **calibrated on real conditioning** (updated 2026-07-31). The scale VALUES —
per-tensor activation amax + per-channel weight scales — come from @ryanontheinside's fp8 work
@@ -267,8 +268,8 @@ and grafted onto this baked / bf16-fused structure by `transplant_scales.py`. Ca
**speed-free** (31.2 vs the earlier uncalibrated 30.8 ms/fwd, inside run noise) and lifts
worst-case per-step fidelity — worst-step velocity-cos vs fp32 on adversarial seeds goes
**0.52/0.57/0.64 → 0.92/0.94/0.92**, with sampling steps 1–7 tracking the fully-calibrated #47
-reference within ~0.001. It is still a **speed tier over the `fp16mixed` default, not a fidelity
-upgrade over it** (single-step velocity cos ~0.92–0.94 < fp16mixed's ~1.0; the 8-step render
+reference within ~0.001. It is still a **speed tier over the `fp16` default, not a fidelity
+upgrade over it** (single-step velocity cos ~0.92–0.94 < fp16's ~1.0; the 8-step render
stays coherent), but it no longer collapses at the highest-noise first step the way the
uncalibrated engine did. See the runtime README's precision section for positioning.
@@ -329,21 +330,21 @@ merged here). Everyday consumers never recalibrate — they pull the published c
`sm-music` and `sm-sfx` also ship a selectable **`fp8`** engine (`--precision fp8`), but it is
**not** the medium's baked-RoPE recipe — those DiTs use standard (non-differential) attention and
never had the bf16 long-angle RoPE problem, so there is nothing to bake. Their fp8 is a straight
-**graft of fp8 E4M3 Q/DQ onto the linear GEMMs of the known-good `dit_fp16mixed.onnx`** — attention
-stays fp16-fused and the fp32 RMSNorm/RoPE islands are left exactly as the fp16mixed producer made
+**graft of fp8 E4M3 Q/DQ onto the linear GEMMs of the known-good `dit_fp16.onnx`** — attention
+stays fp16-fused and the fp32 RMSNorm/RoPE islands are left exactly as the fp16 producer made
them. Built `STRONGLY_TYPED` (`build_from_onnx.py sa3-sm-music-fp8` / `sa3-sm-sfx-fp8`); the QDQ
-carry the precision. Identity: 186 fp8 GEMMs + fp16 fused attention + the fp16mixed fp32 islands.
+carry the precision. Identity: 186 fp8 GEMMs + fp16 fused attention + the fp16 fp32 islands.
Positioning is honest: this is a **clean weight-halving tier** (engine 479 vs 936 MB, velocity-cos
-~0.99 vs eager, clip% at or below fp16mixed), only **marginally faster** (~1.10–1.17×) — a small
+~0.99 vs eager, clip% at or below fp16), only **marginally faster** (~1.10–1.17×) — a small
DiT's ~5 ms forward at batch 1 is overhead-bound, so fp8's GEMM-math savings barely show. Default
-stays `fp16mixed`; fp8 is for when the smaller engine / weight footprint helps. Not seed-reproducible
-vs fp16mixed.
+stays `fp16`; fp8 is for when the smaller engine / weight footprint helps. Not seed-reproducible
+vs fp16.
> ⚠ Do **not** produce these with `build_dit_fp8.py` (#47's ModelOpt path): on the small graphs its
> island-flatten + reapply does not restore the fp32 islands correctly and the engine collapses to
> velocity-cos ~0.69 with clipping (the GEMMs are fine — it's the islands). Grafting onto the
-> fp16mixed ONNX keeps the islands correct by construction.
+> fp16 ONNX keeps the islands correct by construction.
**Producer (refresh the ONNX).** `make_dit_fp8_smalldit.py` calibrates per-linear activation scales
from the eager model (own-domain few-shot prompts + one full render) and grafts the fp8 Q/DQ. Two
@@ -353,7 +354,7 @@ fp16) and floored at 1e-4 (fp16 underflows tiny scales to 0, which TRT rejects):
```bash
python make_dit_fp8_smalldit.py \
--model-config /model_config.json --checkpoint /model.safetensors \
- --fp16mixed-onnx onnx/sa3-sm-music/dit_fp16mixed.onnx \
+ --fp16-onnx onnx/sa3-sm-music/dit_fp16.onnx \
--domain Music --out onnx/sa3-sm-music/dit_fp8.onnx # --domain SFX for sm-sfx
```
@@ -362,9 +363,9 @@ python make_dit_fp8_smalldit.py \
| File | Role | Flow |
|---|---|---|
| `build.py` | Interactive menu (default entry point) | consumer |
-| `build_from_onnx.py` | One target → download ONNX from HF + compile to TRT. **For the SA3 DiTs, pulls `dit_fp16mixed.onnx` (the pre-processed island-wrapped graph)** so the consumer just needs to invoke `STRONGLY_TYPED` compilation — no `onnx-graphsurgeon` required | consumer |
+| `build_from_onnx.py` | One target → download ONNX from HF + compile to TRT. **For the SA3 DiTs, pulls `dit_fp16.onnx` (the pre-processed island-wrapped graph)** so the consumer just needs to invoke `STRONGLY_TYPED` compilation — no `onnx-graphsurgeon` required | consumer |
| `build_dit_profile.py` | Build a DiT with custom `(min, opt, max)` profile shapes (experimental — short-form / fixed-shape variants). Operates on either ONNX flavor. | consumer |
-| `build_dit_fp16mixed.py` | **Producer-side** ONNX surgery: takes the canonical FP32 `dit.onnx`, finds RMSNorm chains + attention `Softmax` + RoPE region, wraps each in `Cast(FP32) ↔ Cast(FP16)` islands, converts non-island weights to FP16, then bounds the RoPE island before QK^T (`bound_attention_core()`, `--no-bound-attn` to skip) so the attention core runs FP16 and TRT's FMHA fuser fires — 96/96 attentions on the medium DiT, 4.3× at L=4096. Writes both the modified `dit_fp16mixed.onnx` AND the TRT engine, which **must** be `STRONGLY_TYPED` (weakly-typed + `BuilderFlag.FP16` re-casts the FP32 islands and silently degrades to naive FP16). Only re-run when the model retrains or the island recipe changes. Requires `onnx` + `onnx-graphsurgeon`. | producer |
+| `build_dit_fp16.py` | **Producer-side** ONNX surgery: takes the canonical FP32 `dit.onnx`, finds RMSNorm chains + attention `Softmax` + RoPE region, wraps each in `Cast(FP32) ↔ Cast(FP16)` islands, converts non-island weights to FP16, then bounds the RoPE island before QK^T (`bound_attention_core()`, `--no-bound-attn` to skip) so the attention core runs FP16 and TRT's FMHA fuser fires — 96/96 attentions on the medium DiT, 4.3× at L=4096. Writes both the modified `dit_fp16.onnx` AND the TRT engine, which **must** be `STRONGLY_TYPED` (weakly-typed + `BuilderFlag.FP16` re-casts the FP32 islands and silently degrades to naive FP16). Only re-run when the model retrains or the island recipe changes. Requires `onnx` + `onnx-graphsurgeon`. | producer |
| `build_dit_bf16.py` | **Producer-side** shared RoPE-baker for the medium `bf16` AND `fp8` engines: precomputes RoPE's cos/sin in fp64 on the host, freezes them as fp32 constant tables (`--max-t`), rewires the 96 trig sites and lets DCE delete the runtime angle chain — so the trunk runs bf16/fp8 without the long-angle drift. Weights are never loaded (keeps the input's `.data` sidecar). Handles both external `inv_freq` (fp32 `dit.onnx`) and inline (fp8-linear ONNX). Consumer compile: `build_from_onnx.py sa3-m-bf16` / `sa3-m-fp8`. Requires `onnx`. | producer |
| `make_calib.py` | **Producer-side** FP8 calibration capture: drives the model's own pingpong `generate()` and records the six DiT engine inputs at every sampling step into a `.npz` (real-conditioning, deployment-matched prompts). Feeds `build_dit_fp8.py` (#47). Only re-run for a from-scratch recalibration on a model retrain. Calibration tooling by @ryanontheinside (#47). Requires `torch` + `stable_audio_3`. | producer |
| `transplant_scales.py` | **Producer-side** fp8 scale transplant: grafts #47's calibrated activation + per-channel weight scales onto the RoPE-baked bakedmin ONNX (matches Linears by weight-initializer name, swaps only the scale VALUES; weights untouched). This is what makes the shipped `dit_fp8.onnx` calibrated while keeping bakedmin's bf16 fused-MHA speed. Scale values / calibration by @ryanontheinside (#47). Requires `onnx`. | producer |
@@ -374,7 +375,7 @@ python make_dit_fp8_smalldit.py \
| `build_same_s_encoder.py` | Trace + export SAME-S encoder ONNX + build TRT | producer |
| `build_same_l_decoder.py` | Trace + export SAME-L decoder ONNX (Triton SWA) + build TRT | producer |
| `build_same_l_encoder.py` | Trace + export SAME-L encoder ONNX (Triton SWA) + build TRT | producer |
-| `build_dit.py ` | Trace + export DiT FP32 ONNX (cond baked in) + build TRT BF16 engine (legacy; the BF16 output isn't suitable for inference — chain it with `build_dit_fp16mixed.py` afterwards) | producer |
+| `build_dit.py ` | Trace + export DiT FP32 ONNX (cond baked in) + build TRT BF16 engine (legacy; the BF16 output isn't suitable for inference — chain it with `build_dit_fp16.py` afterwards) | producer |
| `_arch.py` | Shared: GPU arch detection + path helpers | both |
| `samel_loader.py` | Helper: load SAME-L from .ckpt | producer |
| `samel_{encoder,decoder}_onnx.py` | Helper: clean ONNX rewrites of SAME-L blocks | producer |
diff --git a/optimized/tensorRT/build/build.py b/optimized/tensorRT/build/build.py
index d27a495e..6c5ee525 100755
--- a/optimized/tensorRT/build/build.py
+++ b/optimized/tensorRT/build/build.py
@@ -52,7 +52,7 @@ def _from_onnx(name):
TARGETS = [
{"label": "t5gemma (text encoder + tokenizer)",
"command": _from_onnx("t5gemma"),
- "outputs": ["t5gemma/t5gemma_fp16mixed.trt", "t5gemma/tokenizer.json"]},
+ "outputs": ["t5gemma/t5gemma_fp16.trt", "t5gemma/tokenizer.json"]},
{"label": "same-s encoder",
"command": _from_onnx("same-s-encoder"),
"outputs": ["same-s/enc_dynamic_bf16.trt"]},
@@ -84,19 +84,19 @@ def _from_onnx(name):
# spacing is 32 rad — more than a full 2*pi rotation).
{"label": "DiT medium (SA3-M, FP16-mixed — medium DEFAULT, attention-fused)",
"command": _from_onnx("sa3-m"),
- "outputs": ["sa3-m/dit_fp16mixed.trt"]},
+ "outputs": ["sa3-m/dit_fp16.trt"]},
{"label": "DiT medium (SA3-M, bf16 — selectable; drifts at long sequence)",
"command": _from_onnx("sa3-m-bf16"),
"outputs": ["sa3-m/dit_bf16.trt"]},
- {"label": "DiT medium (SA3-M, fp8 — selectable; MAX-SPEED clean tier, ~1.3x over fp16mixed)",
+ {"label": "DiT medium (SA3-M, fp8 — selectable; MAX-SPEED clean tier, ~1.3x over fp16)",
"command": _from_onnx("sa3-m-fp8"),
"outputs": ["sa3-m/dit_fp8.trt"]},
{"label": "DiT sm-music (FP16-mixed)",
"command": _from_onnx("sa3-sm-music"),
- "outputs": ["sa3-sm-music/dit_fp16mixed.trt"]},
+ "outputs": ["sa3-sm-music/dit_fp16.trt"]},
{"label": "DiT sm-sfx (FP16-mixed)",
"command": _from_onnx("sa3-sm-sfx"),
- "outputs": ["sa3-sm-sfx/dit_fp16mixed.trt"]},
+ "outputs": ["sa3-sm-sfx/dit_fp16.trt"]},
# FP32 variants — opt-in. ~2× engine size, ~2× slower, but bit-equivalent
# to PyTorch eager. Useful for precision-debug or reference comparisons.
{"label": "[opt-in] same-l decoder FP32",
diff --git a/optimized/tensorRT/build/build_dit_fp16mixed.py b/optimized/tensorRT/build/build_dit_fp16.py
similarity index 99%
rename from optimized/tensorRT/build/build_dit_fp16mixed.py
rename to optimized/tensorRT/build/build_dit_fp16.py
index 2af6e4a0..af670aac 100644
--- a/optimized/tensorRT/build/build_dit_fp16mixed.py
+++ b/optimized/tensorRT/build/build_dit_fp16.py
@@ -61,11 +61,11 @@
- Engine size: 926 MB (BF16: 935 MB, FP32: 1842 MB)
Usage:
- python build_dit_fp16mixed.py
+ python build_dit_fp16.py
[--mode {minimal,rope,full}] # default: rope
[--input /tmp/dit_sm-music_fixed_v2.onnx]
- [--onnx /tmp/dit_sm-music_fp16mixed.onnx]
- [--engine .../models/sm_90/sa3-sm-music/dit_fp16mixed.trt]
+ [--onnx /tmp/dit_sm-music_fp16.onnx]
+ [--engine .../models/sm_90/sa3-sm-music/dit_fp16.trt]
"""
import argparse
import os
@@ -1001,7 +1001,7 @@ def constant_feeds_island_only(consumers):
return model
-def convert_to_fp16mixed(input_onnx, output_onnx, mode="minimal", bound_attn=True):
+def convert_to_fp16(input_onnx, output_onnx, mode="minimal", bound_attn=True):
"""Load FP32 ONNX, identify FP32 islands, convert everything else to
FP16, and save."""
import onnx
@@ -1142,11 +1142,11 @@ def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--input", default="/tmp/dit_sm-music_fixed_v2.onnx",
help="Input FP32 ONNX")
- ap.add_argument("--onnx", default="/tmp/dit_sm-music_fp16mixed.onnx",
+ ap.add_argument("--onnx", default="/tmp/dit_sm-music_fp16.onnx",
help="Output FP16-mixed ONNX (intermediate)")
ap.add_argument("--engine", default="/weka2/cj/clod/sa3s/stable-audio-3/"
"optimized/tensorRT/models/sm_90/"
- "sa3-sm-music/dit_fp16mixed.trt",
+ "sa3-sm-music/dit_fp16.trt",
help="Output TRT engine path")
ap.add_argument("--workspace-gb", type=int, default=16)
ap.add_argument("--mode", choices=("minimal", "rope", "full"),
@@ -1168,7 +1168,7 @@ def main():
if not args.skip_convert:
print(f"━━━ Convert FP32 ONNX -> FP16-mixed ONNX (mode={args.mode}) ━━━")
- convert_to_fp16mixed(args.input, args.onnx, mode=args.mode,
+ convert_to_fp16(args.input, args.onnx, mode=args.mode,
bound_attn=not args.no_bound_attn)
if not args.skip_build:
diff --git a/optimized/tensorRT/build/build_from_onnx.py b/optimized/tensorRT/build/build_from_onnx.py
index 2c5d337e..0fd2e1b6 100755
--- a/optimized/tensorRT/build/build_from_onnx.py
+++ b/optimized/tensorRT/build/build_from_onnx.py
@@ -26,7 +26,7 @@
python build_from_onnx.py same-l-encoder
python build_from_onnx.py same-l-decoder
# SA3 DiT engines use a different builder (FP16-mixed recipe):
- python build_dit_fp16mixed.py --input --engine
+ python build_dit_fp16.py --input --engine
python build_from_onnx.py all # build everything for this arch
"""
import os
@@ -73,7 +73,7 @@
"onnx_hf": ["t5gemma/encoder.onnx"],
# tokenizer.json ships bundled with the repo at scripts/tokenizer.json
# (arch-agnostic), so we don't fetch it here anymore.
- "trt_local": "t5gemma/t5gemma_fp16mixed.trt",
+ "trt_local": "t5gemma/t5gemma_fp16.trt",
"flags": set(), # STRONGLY_TYPED carries the FP16/FP32 dtype hints
"network": "STRONGLY_TYPED",
"workspace_gb": 8,
@@ -121,12 +121,12 @@
"plugin": True,
},
# SA3 DiT engines: build from the pre-processed FP16-mixed ONNX hosted on
- # HF. The producer (build_dit_fp16mixed.py) does the FP32-island surgery
+ # HF. The producer (build_dit_fp16.py) does the FP32-island surgery
# once and uploads the result; consumers just compile with STRONGLY_TYPED
# (no onnx-graphsurgeon dependency).
"sa3-sm-music": {
- "onnx_hf": ["sa3-sm-music/dit_fp16mixed.onnx"],
- "trt_local": "sa3-sm-music/dit_fp16mixed.trt",
+ "onnx_hf": ["sa3-sm-music/dit_fp16.onnx"],
+ "trt_local": "sa3-sm-music/dit_fp16.trt",
"flags": set(), # STRONGLY_TYPED + ONNX dtypes carry precision
"network": "STRONGLY_TYPED",
"workspace_gb": 16,
@@ -134,25 +134,25 @@
"plugin": False,
},
"sa3-sm-sfx": {
- "onnx_hf": ["sa3-sm-sfx/dit_fp16mixed.onnx"],
- "trt_local": "sa3-sm-sfx/dit_fp16mixed.trt",
+ "onnx_hf": ["sa3-sm-sfx/dit_fp16.onnx"],
+ "trt_local": "sa3-sm-sfx/dit_fp16.trt",
"flags": set(),
"network": "STRONGLY_TYPED",
"workspace_gb": 16,
"profile": _DIT_PROFILE,
"plugin": False,
},
- # SA3 small DiTs in fp8 — SELECTABLE (default stays fp16mixed). fp8 E4M3 on the
+ # SA3 small DiTs in fp8 — SELECTABLE (default stays fp16). fp8 E4M3 on the
# 186 linear GEMMs, attention left fp16-fused and the fp32 RMSNorm/RoPE islands
- # intact — an fp8-QDQ graft onto the fp16mixed graph (build/make_dit_fp8_smalldit.py),
+ # intact — an fp8-QDQ graft onto the fp16 graph (build/make_dit_fp8_smalldit.py),
# NOT the medium's baked-RoPE recipe (these DiTs never had the bf16 long-angle
# problem). Built STRONGLY_TYPED: the QDQ nodes carry fp8; TRT fires fp8 tensor-core
# GEMMs on the linears while the fp16 FMHA fuser still runs the attention. Same
# _DIT_PROFILE (batch=1, dynamic L∈[1,4096]) → identical CLI/feature surface.
# This is a CLEAN WEIGHT-HALVING tier (479 vs 936 MB, velocity-cos ~0.99 vs eager,
- # clip% at/below fp16mixed), only marginally faster (~1.1×): the small DiTs' ~5 ms
+ # clip% at/below fp16), only marginally faster (~1.1×): the small DiTs' ~5 ms
# forward is overhead-bound at batch 1, so fp8's GEMM savings barely show. sm-* fp8
- # is NOT seed-reproducible vs fp16mixed.
+ # is NOT seed-reproducible vs fp16.
"sa3-sm-music-fp8": {
"onnx_hf": ["sa3-sm-music/dit_fp8.onnx", "sa3-sm-music/dit_fp8.onnx.data"],
"trt_local": "sa3-sm-music/dit_fp8.trt",
@@ -173,8 +173,8 @@
},
"sa3-m": {
# 2.9 GB external-data sidecar travels alongside.
- "onnx_hf": ["sa3-m/dit_fp16mixed.onnx", "sa3-m/dit_fp16mixed.onnx.data"],
- "trt_local": "sa3-m/dit_fp16mixed.trt",
+ "onnx_hf": ["sa3-m/dit_fp16.onnx", "sa3-m/dit_fp16.onnx.data"],
+ "trt_local": "sa3-m/dit_fp16.trt",
"flags": set(),
"network": "STRONGLY_TYPED",
"workspace_gb": 16,
@@ -190,7 +190,7 @@
#
# The ~1.76×@L=256 / 4.70×@L=4096 speedup this engine was shipped for was
# measured against an fp16-mixed engine whose attention core was stuck in
- # FP32; with that fixed (build_dit_fp16mixed.py's bound_attention_core) bf16
+ # FP32; with that fixed (build_dit_fp16.py's bound_attention_core) bf16
# is only ~3% ahead, and it loses on accuracy: weakly-typed BF16 also lets
# TRT evaluate RoPE's rotation angle in bf16, and that angle reaches ~4155
# rad at L=4092 where bf16's spacing is 32 rad (> 2π), so position info for
@@ -258,13 +258,13 @@
# 0.52/0.57/0.64 to 0.92/0.94/0.92 on adversarial seeds; sampling steps 1-7 match
# the fully-calibrated #47 reference within ~0.001. It is still a SPEED tier over the
# fp16-mixed default rather than a fidelity upgrade over it (single-step velocity cos
- # ~0.92-0.94 < fp16mixed's ~1.0), but it no longer collapses at the highest-noise
+ # ~0.92-0.94 < fp16's ~1.0), but it no longer collapses at the highest-noise
# first step the way the uncalibrated engine did. #47's fp16-attention variant buys
# the last ~0.03 of step-0 fidelity for +2 ms/fwd; this tier keeps bf16 attention for
# the speed. Identity of the shipped engine: 176 fp8 GEMMs + 96 bf16 fused MHA +
# baked fp32 RoPE constant — check with scripts/verify_fp8_rope.py. Same _DIT_PROFILE
# (batch=1, L in [1,4096]) → identical CLI/feature surface. medium-only, not
- # seed-reproducible vs fp16mixed.
+ # seed-reproducible vs fp16.
#
# Producer: the fp8-linear ONNX (fp8 QDQ inserted; inv_freq inline) is RoPE-baked by
# build_dit_bf16.py, the SAME baker as the bf16 tier (it handles inline OR external
diff --git a/optimized/tensorRT/build/build_same_s_dec_fp16mixed.py b/optimized/tensorRT/build/build_same_s_dec_fp16.py
similarity index 98%
rename from optimized/tensorRT/build/build_same_s_dec_fp16mixed.py
rename to optimized/tensorRT/build/build_same_s_dec_fp16.py
index e66653c0..70ee0f3b 100644
--- a/optimized/tensorRT/build/build_same_s_dec_fp16mixed.py
+++ b/optimized/tensorRT/build/build_same_s_dec_fp16.py
@@ -33,11 +33,11 @@
Cast(to=INT32) is unaffected by trunk dtype since it's an explicit dtype change.
Usage:
- python build_same_s_dec_fp16mixed.py
+ python build_same_s_dec_fp16.py
[--mode {minimal,rope,full}] # default: rope
[--input /weka2/cj/clod/sa3s/stable-audio-3-optimized/onnx/same-s/dec_dynamic_bf16.onnx]
- [--onnx /tmp/same_s_dec_fp16mixed.onnx]
- [--engine .../models/sm_90/same-s/dec_dynamic_fp16mixed.trt]
+ [--onnx /tmp/same_s_dec_fp16.onnx]
+ [--engine .../models/sm_90/same-s/dec_dynamic_fp16.trt]
"""
import argparse
import os
@@ -51,7 +51,7 @@
# Reuse the helpers from the DiT FP16-mixed build. Most of them work as-is
# — only the FP32-island finder needs SAME-S-specific patterns, which we
# override below.
-from build_dit_fp16mixed import (
+from build_dit_fp16 import (
strip_noop_fp32_casts,
wrap_islands_with_casts,
fix_dtype_mismatches,
@@ -476,7 +476,7 @@ def cast_to(node):
return model
-def convert_to_fp16mixed(input_onnx, output_onnx, mode="minimal"):
+def convert_to_fp16(input_onnx, output_onnx, mode="minimal"):
"""Load FP32 ONNX, identify FP32 islands, convert everything else to
FP16, and save."""
import onnx
@@ -612,12 +612,12 @@ def main():
help="Input FP32 ONNX (the `_bf16` suffix refers to the "
"eventual engine flavor; the ONNX itself is FP32)")
ap.add_argument("--onnx",
- default="/tmp/same_s_dec_fp16mixed.onnx",
+ default="/tmp/same_s_dec_fp16.onnx",
help="Output FP16-mixed ONNX (intermediate)")
ap.add_argument("--engine",
default="/weka2/cj/clod/sa3s/stable-audio-3/optimized/"
"tensorRT/models/sm_90/same-s/"
- "dec_dynamic_fp16mixed.trt",
+ "dec_dynamic_fp16.trt",
help="Output TRT engine path")
ap.add_argument("--workspace-gb", type=int, default=16)
ap.add_argument("--mode", choices=("minimal", "rope", "attention", "full"),
@@ -635,7 +635,7 @@ def main():
if not args.skip_convert:
print(f"━━━ Convert FP32 ONNX -> FP16-mixed ONNX (mode={args.mode}) ━━━")
- convert_to_fp16mixed(args.input, args.onnx, mode=args.mode)
+ convert_to_fp16(args.input, args.onnx, mode=args.mode)
if not args.skip_build:
print("\n━━━ Build TRT engine ━━━")
diff --git a/optimized/tensorRT/build/make_calib.py b/optimized/tensorRT/build/make_calib.py
index a5a84b20..4419fbe6 100644
--- a/optimized/tensorRT/build/make_calib.py
+++ b/optimized/tensorRT/build/make_calib.py
@@ -7,7 +7,7 @@
recording the six DiT engine inputs at every sampling step.
FP8 is the only quantization path in this repo that needs calibration data
-(``fp16mixed`` / ``fp32`` need none, MLX is weight-only), so this also
+(``fp16`` / ``fp32`` need none, MLX is weight-only), so this also
establishes the convention: a ``.calib.npz`` whose keys are the six ONNX
input names, each a leading-axis batch of samples. The npz is a reproducible
producer artifact: gitignored, never committed, regenerated on demand.
diff --git a/optimized/tensorRT/build/make_dit_fp8_smalldit.py b/optimized/tensorRT/build/make_dit_fp8_smalldit.py
index a56219ab..548d0e7c 100644
--- a/optimized/tensorRT/build/make_dit_fp8_smalldit.py
+++ b/optimized/tensorRT/build/make_dit_fp8_smalldit.py
@@ -1,14 +1,14 @@
#!/usr/bin/env python3
"""Produce the fp8 ONNX for the SMALL DiTs (sm-music / sm-sfx) by grafting fp8 E4M3 Q/DQ onto
-the LINEAR GEMMs of the model's fp16mixed ONNX — leaving attention fp16-fused and the fp32
+the LINEAR GEMMs of the model's fp16 ONNX — leaving attention fp16-fused and the fp32
RMSNorm/RoPE islands intact. This is deliberately NOT the medium's fp8 recipe (baked RoPE +
bf16 attention, build_dit_bf16.py + build_dit_fp8.py): the small DiTs never had the bf16
-long-angle RoPE problem, so their fp8 is a straight graft on the known-good fp16mixed graph.
+long-angle RoPE problem, so their fp8 is a straight graft on the known-good fp16 graph.
Why a dedicated script (vs ModelOpt/build_dit_fp8.py): running ModelOpt PTQ on these models
flattens the fp32 islands and its island-reapply (tuned for the medium graph) does NOT restore
them correctly here — the resulting engine drops to velocity-cos ~0.69 with clipping. Grafting
-onto the fp16mixed ONNX keeps the islands correct by construction (velocity-cos ~0.99 vs eager).
+onto the fp16 ONNX keeps the islands correct by construction (velocity-cos ~0.99 vs eager).
Two fp16-trunk specifics vs dit_fp8_max/make_fp8_onnx.py (which targets the fp32-trunk medium):
* Q/DQ scales are FLOAT16 (the trunk is fp16 → DequantizeLinear must output fp16, else TRT
@@ -22,7 +22,7 @@
python make_dit_fp8_smalldit.py \
--model-config /model_config.json --checkpoint /model.safetensors \
- --fp16mixed-onnx onnx/sa3-sm-music/dit_fp16mixed.onnx \
+ --fp16-onnx onnx/sa3-sm-music/dit_fp16.onnx \
--domain Music --out onnx/sa3-sm-music/dit_fp8.onnx
Then compile with build_from_onnx.py sa3-sm-music-fp8 (STRONGLY_TYPED; the QDQ carry fp8).
@@ -39,7 +39,7 @@
SCALE_FLOOR = 1e-4
-def calibrate_act_scales(model_config, checkpoint, fp16mixed_onnx, domain, margin, device):
+def calibrate_act_scales(model_config, checkpoint, fp16_onnx, domain, margin, device):
"""Per-linear activation max|x| from the eager model → {onnx_node_name: scale}. Maps ONNX
linear nodes to torch modules by weight name (ONNX 'dit..weight' vs torch 'model.'
→ match on the suffix after the first dotted component)."""
@@ -72,7 +72,7 @@ def __call__(s, x): s.amax = max(s.amax, float(x.abs().amax())); return F.linear
sampler_type="pingpong", seed=6000, duration_padding_sec=0.0, return_latents=True)
print(f" calibrated {len(mods)} linears on {len(prompts)} {domain} prompts + one full render", flush=True)
- m = onnx.load(fp16mixed_onnx, load_external_data=False); g = m.graph
+ m = onnx.load(fp16_onnx, load_external_data=False); g = m.graph
inits = {i.name for i in g.initializer}; prod = {o: n for n in g.node for o in n.output}
def wsrc(n):
w = n.input[1]
@@ -106,8 +106,8 @@ def topo_sort(g):
del g.node[:]; g.node.extend(result)
-def graft_fp8(fp16mixed_onnx, node_scale, gscale, out):
- model = onnx.load(fp16mixed_onnx, load_external_data=True); g = model.graph
+def graft_fp8(fp16_onnx, node_scale, gscale, out):
+ model = onnx.load(fp16_onnx, load_external_data=True); g = model.graph
have = False
for op in model.opset_import:
if op.domain in ("", "ai.onnx"):
@@ -161,7 +161,7 @@ def main():
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("--model-config", required=True)
ap.add_argument("--checkpoint", required=True)
- ap.add_argument("--fp16mixed-onnx", required=True, help="the model's canonical dit_fp16mixed.onnx")
+ ap.add_argument("--fp16-onnx", required=True, help="the model's canonical dit_fp16.onnx")
ap.add_argument("--out", required=True, help="output dit_fp8.onnx (a .data sidecar is written alongside)")
ap.add_argument("--domain", default="Music", choices=["Music", "SFX", "Instrument", "One-shot"],
help="reprompt few-shot domain for activation calibration (Music for sm-music, SFX for sm-sfx)")
@@ -170,9 +170,9 @@ def main():
a = ap.parse_args()
t0 = time.time()
print(f"[make_dit_fp8_smalldit] calibrating ({a.domain}) ...", flush=True)
- node_scale, gscale = calibrate_act_scales(a.model_config, a.checkpoint, a.fp16mixed_onnx, a.domain, a.margin, a.device)
+ node_scale, gscale = calibrate_act_scales(a.model_config, a.checkpoint, a.fp16_onnx, a.domain, a.margin, a.device)
print(f" {len(node_scale)} node scales, global={gscale:.5f}; grafting fp8 ...", flush=True)
- graft_fp8(a.fp16mixed_onnx, node_scale, gscale, a.out)
+ graft_fp8(a.fp16_onnx, node_scale, gscale, a.out)
print(f"DONE -> {a.out} ({time.time()-t0:.0f}s)", flush=True)
diff --git a/optimized/tensorRT/build/transplant_scales.py b/optimized/tensorRT/build/transplant_scales.py
index 7d9e7891..323562a7 100644
--- a/optimized/tensorRT/build/transplant_scales.py
+++ b/optimized/tensorRT/build/transplant_scales.py
@@ -23,7 +23,7 @@
5. build_from_onnx.py sa3-m-fp8 compile the calibrated ONNX to dit_fp8.trt
Why transplant instead of shipping #47's engine directly: #47's calibrated ONNX
-is fp16mixed-derived and its engine uses fp16 attention + a runtime fp32 RoPE
+is fp16-derived and its engine uses fp16 attention + a runtime fp32 RoPE
island (+~2 ms/fwd). Our bakedmin structure keeps the faster bf16 fused MHA and
a baked fp32 RoPE constant. Calibration is a property of the SCALE VALUES, not of
the structure, so grafting #47's scales onto the bakedmin graph gives #47's
diff --git a/optimized/tensorRT/install.sh b/optimized/tensorRT/install.sh
index 19a56dec..b844c1d8 100755
--- a/optimized/tensorRT/install.sh
+++ b/optimized/tensorRT/install.sh
@@ -68,6 +68,9 @@ $(dim "After install, run with:")
$(cyan ./.venv/bin/python) sa3_trt.py --prompt "..." --dit sm-music --decoder same-s
$(dim "or (with uv):")
$(cyan uv run) sa3_trt.py --prompt "..." --dit sm-music --decoder same-s
+$(dim "Quantized decoders/encoders (auto-download from HF):") $(yellow "--dec-precision") $(magenta "fp8 | fp8_fast")
+ $(dim "fp8 ~1.14x near-transparent · fp8_fast SAME-S only ~1.22x lossier · --precision fp32 for max fidelity")
+ $(dim "build tier engines locally with quantize/build_tiers.py; tier ONNX + sm_90/sm_120 engines live on HF")
USAGE
exit 0;;
*) errmsg "unknown arg: $1"; exit 2;;
@@ -324,21 +327,21 @@ fi
SHARED=(
# T5Gemma engine — arch-specific. (tokenizer.json is arch-agnostic and
# ships bundled with the repo at scripts/tokenizer.json — no download.)
- "${HF_SUBDIR}/t5gemma/t5gemma_fp16mixed.trt"
+ "${HF_SUBDIR}/t5gemma/t5gemma_fp16.trt"
)
# Only TRT engines are downloaded from HF.
MEDIUM=(
- "${HF_SUBDIR}/sa3-m/dit_fp16mixed.trt"
+ "${HF_SUBDIR}/sa3-m/dit_fp16.trt"
"${HF_SUBDIR}/same-l/enc_dynamic_triton_swa.trt"
"${HF_SUBDIR}/same-l/dec_dynamic_triton_swa.trt"
)
SM_MUSIC=(
- "${HF_SUBDIR}/sa3-sm-music/dit_fp16mixed.trt"
+ "${HF_SUBDIR}/sa3-sm-music/dit_fp16.trt"
"${HF_SUBDIR}/same-s/enc_dynamic_bf16.trt"
"${HF_SUBDIR}/same-s/dec_dynamic_bf16.trt"
)
SM_SFX=(
- "${HF_SUBDIR}/sa3-sm-sfx/dit_fp16mixed.trt"
+ "${HF_SUBDIR}/sa3-sm-sfx/dit_fp16.trt"
"${HF_SUBDIR}/same-s/enc_dynamic_bf16.trt"
"${HF_SUBDIR}/same-s/dec_dynamic_bf16.trt"
)
@@ -377,7 +380,7 @@ else
SKIPPED=0
for hf_path in "${DEDUP[@]}"; do
# Strip only "tensorRT/" so the arch stays in the local path:
- # tensorRT/sm_90/sa3-m/dit_fp16mixed.trt → models/sm_90/sa3-m/dit_fp16mixed.trt.
+ # tensorRT/sm_90/sa3-m/dit_fp16.trt → models/sm_90/sa3-m/dit_fp16.trt.
local_rel="${hf_path#tensorRT/}"
local_path="${MODELS_DIR}/${local_rel}"
if [ -f "${local_path}" ] && [ -s "${local_path}" ]; then
diff --git a/optimized/tensorRT/output/ab_gradio/1780480567813_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780480567813_pt_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..7b15a858
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780480567813_pt_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780480567813_trt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780480567813_trt_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..adb8310b
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780480567813_trt_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780480576475_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780480576475_pt_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..7b15a858
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780480576475_pt_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780480576475_trt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780480576475_trt_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..adb8310b
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780480576475_trt_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780480587601_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780480587601_pt_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..7b15a858
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780480587601_pt_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780480587601_trt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780480587601_trt_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..adb8310b
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780480587601_trt_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780480778299_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780480778299_pt_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..7b15a858
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780480778299_pt_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780480778299_trt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780480778299_trt_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..adb8310b
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780480778299_trt_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780481520748_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780481520748_pt_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..7b15a858
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780481520748_pt_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780481520748_trt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780481520748_trt_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..b4d69681
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780481520748_trt_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780481695716_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780481695716_pt_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..7b15a858
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780481695716_pt_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780481695716_trt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780481695716_trt_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..b4d69681
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780481695716_trt_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780481699952_pt_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780481699952_pt_Death_Metal_s2_spec.png
new file mode 100644
index 00000000..531bca9f
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780481699952_pt_Death_Metal_s2_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780481699952_trt_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780481699952_trt_Death_Metal_s2_spec.png
new file mode 100644
index 00000000..441152b3
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780481699952_trt_Death_Metal_s2_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780482623536_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482623536_pt_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..7b15a858
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482623536_pt_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780482623536_trtcanon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482623536_trtcanon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..26d30539
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482623536_trtcanon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780482623536_trtfp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482623536_trtfp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..d03ccf56
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482623536_trtfp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780482784560_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482784560_pt_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..7b15a858
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482784560_pt_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780482784560_trtcanon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482784560_trtcanon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..26d30539
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482784560_trtcanon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780482784560_trtfp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482784560_trtfp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..d03ccf56
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482784560_trtfp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780482789056_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482789056_pt_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..7b15a858
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482789056_pt_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780482789056_trtcanon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482789056_trtcanon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..26d30539
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482789056_trtcanon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780482789056_trtfp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482789056_trtfp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..d03ccf56
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482789056_trtfp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780482829728_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482829728_pt_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..55793924
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482829728_pt_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780482829728_trtcanon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482829728_trtcanon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..13b9ab27
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482829728_trtcanon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780482829728_trtfp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482829728_trtfp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..e7d03383
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482829728_trtfp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780482836657_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482836657_pt_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..55793924
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482836657_pt_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780482836657_trtcanon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482836657_trtcanon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..13b9ab27
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482836657_trtcanon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780482836657_trtfp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482836657_trtfp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..e7d03383
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482836657_trtfp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780482855978_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482855978_pt_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..8a0f2fc7
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482855978_pt_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780482855978_trtcanon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482855978_trtcanon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..4cf90724
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482855978_trtcanon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780482855978_trtfp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482855978_trtfp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..6bad3afc
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482855978_trtfp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780482858838_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482858838_pt_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..8a0f2fc7
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482858838_pt_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780482858838_trtcanon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482858838_trtcanon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..4cf90724
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482858838_trtcanon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780482858838_trtfp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482858838_trtfp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..6bad3afc
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482858838_trtfp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780482862115_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482862115_pt_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..8a0f2fc7
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482862115_pt_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780482862115_trtcanon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482862115_trtcanon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..4cf90724
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482862115_trtcanon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780482862115_trtfp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482862115_trtfp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..6bad3afc
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482862115_trtfp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780482865013_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482865013_pt_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..8a0f2fc7
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482865013_pt_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780482865013_trtcanon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482865013_trtcanon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..4cf90724
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482865013_trtcanon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780482865013_trtfp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482865013_trtfp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..6bad3afc
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482865013_trtfp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780482944632_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482944632_pt_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..11ecd98a
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482944632_pt_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780482944632_trtcanon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482944632_trtcanon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..5e86e2c2
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482944632_trtcanon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780482944632_trtfp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482944632_trtfp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..0a19729b
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482944632_trtfp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780482947783_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482947783_pt_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..11ecd98a
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482947783_pt_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780482947783_trtcanon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482947783_trtcanon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..5e86e2c2
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482947783_trtcanon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780482947783_trtfp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482947783_trtfp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..0a19729b
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482947783_trtfp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780482958211_pt_Death_Meta_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482958211_pt_Death_Meta_s1_spec.png
new file mode 100644
index 00000000..89a7bd91
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482958211_pt_Death_Meta_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780482958211_trtcanon_Death_Meta_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482958211_trtcanon_Death_Meta_s1_spec.png
new file mode 100644
index 00000000..eab01437
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482958211_trtcanon_Death_Meta_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780482958211_trtfp32_Death_Meta_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780482958211_trtfp32_Death_Meta_s1_spec.png
new file mode 100644
index 00000000..4b62a4e0
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780482958211_trtfp32_Death_Meta_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483154115_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483154115_pt_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..c1bc94f8
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483154115_pt_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483154115_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483154115_samel_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..2fba4f5f
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483154115_samel_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483154115_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483154115_samel_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..83503d0d
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483154115_samel_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483154115_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483154115_sames_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..e4604463
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483154115_sames_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483154115_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483154115_sames_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..c86bfba5
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483154115_sames_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483219388_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483219388_pt_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..c1bc94f8
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483219388_pt_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483219388_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483219388_samel_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..2fba4f5f
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483219388_samel_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483219388_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483219388_samel_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..83503d0d
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483219388_samel_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483219388_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483219388_sames_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..59b92ff3
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483219388_sames_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483219388_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483219388_sames_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..a79354bf
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483219388_sames_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483387713_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483387713_pt_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..c1bc94f8
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483387713_pt_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483387713_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483387713_samel_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..2fba4f5f
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483387713_samel_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483387713_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483387713_samel_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..83503d0d
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483387713_samel_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483387713_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483387713_sames_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..9e7c33ad
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483387713_sames_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483387713_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483387713_sames_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..641bf750
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483387713_sames_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483391579_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483391579_pt_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..c1bc94f8
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483391579_pt_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483391579_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483391579_samel_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..2fba4f5f
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483391579_samel_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483391579_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483391579_samel_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..83503d0d
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483391579_samel_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483391579_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483391579_sames_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..95fa185b
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483391579_sames_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483391579_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483391579_sames_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..1edeb6e1
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483391579_sames_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483395100_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483395100_pt_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..c1bc94f8
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483395100_pt_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483395100_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483395100_samel_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..2fba4f5f
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483395100_samel_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483395100_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483395100_samel_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..83503d0d
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483395100_samel_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483395100_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483395100_sames_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..5cd883aa
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483395100_sames_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483395100_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483395100_sames_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..43e9cd7d
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483395100_sames_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483432118_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483432118_pt_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..976d1ffd
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483432118_pt_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483432118_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483432118_samel_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..8cdca875
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483432118_samel_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483432118_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483432118_samel_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..358fad38
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483432118_samel_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483432118_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483432118_sames_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..5cf69fa6
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483432118_sames_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483432118_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483432118_sames_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..a56353d1
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483432118_sames_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483437783_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483437783_pt_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..976d1ffd
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483437783_pt_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483437783_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483437783_samel_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..8cdca875
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483437783_samel_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483437783_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483437783_samel_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..358fad38
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483437783_samel_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483437783_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483437783_sames_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..0a6d51b2
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483437783_sames_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483437783_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483437783_sames_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..f7a4c625
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483437783_sames_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483441862_pt_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483441862_pt_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..976d1ffd
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483441862_pt_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483441862_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483441862_samel_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..8cdca875
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483441862_samel_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483441862_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483441862_samel_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..358fad38
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483441862_samel_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483441862_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483441862_sames_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..a52d750a
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483441862_sames_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780483441862_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780483441862_sames_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..fc037ca2
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780483441862_sames_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780484010428_pt_samel_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484010428_pt_samel_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..c6579601
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484010428_pt_samel_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780484010428_pt_sames_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484010428_pt_sames_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..51ab5c3c
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484010428_pt_sames_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780484010428_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484010428_samel_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..1e178a3a
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484010428_samel_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780484010428_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484010428_samel_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..1cfd3a8b
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484010428_samel_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780484010428_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484010428_sames_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..e4604463
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484010428_sames_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780484010428_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484010428_sames_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..c86bfba5
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484010428_sames_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780484169479_pt_samel_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484169479_pt_samel_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..c6579601
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484169479_pt_samel_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780484169479_pt_sames_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484169479_pt_sames_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..51ab5c3c
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484169479_pt_sames_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780484169479_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484169479_samel_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..1e178a3a
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484169479_samel_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780484169479_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484169479_samel_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..1cfd3a8b
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484169479_samel_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780484169479_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484169479_sames_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..59b92ff3
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484169479_sames_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780484169479_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484169479_sames_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..a79354bf
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484169479_sames_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780484174384_pt_samel_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484174384_pt_samel_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..c6579601
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484174384_pt_samel_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780484174384_pt_sames_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484174384_pt_sames_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..51ab5c3c
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484174384_pt_sames_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780484174384_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484174384_samel_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..1e178a3a
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484174384_samel_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780484174384_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484174384_samel_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..1cfd3a8b
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484174384_samel_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780484174384_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484174384_sames_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..9e7c33ad
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484174384_sames_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780484174384_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484174384_sames_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..641bf750
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484174384_sames_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780484750885_pt_samel_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484750885_pt_samel_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..c1bc94f8
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484750885_pt_samel_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780484750885_pt_sames_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484750885_pt_sames_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..331eaa4e
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484750885_pt_sames_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780484750885_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484750885_samel_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..2fba4f5f
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484750885_samel_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780484750885_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484750885_samel_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..83503d0d
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484750885_samel_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780484750885_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484750885_sames_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..8f058a5a
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484750885_sames_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780484750885_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484750885_sames_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..59663eff
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484750885_sames_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780484770421_pt_samel_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484770421_pt_samel_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..c1bc94f8
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484770421_pt_samel_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780484770421_pt_sames_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484770421_pt_sames_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..331eaa4e
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484770421_pt_sames_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780484770421_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484770421_samel_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..2fba4f5f
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484770421_samel_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780484770421_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484770421_samel_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..83503d0d
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484770421_samel_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780484770421_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484770421_sames_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..d0d3546d
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484770421_sames_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780484770421_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780484770421_sames_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..488a4043
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780484770421_sames_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780485629288_pt_samel_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780485629288_pt_samel_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..c1bc94f8
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485629288_pt_samel_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780485629288_pt_sames_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780485629288_pt_sames_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..331eaa4e
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485629288_pt_sames_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780485629288_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780485629288_samel_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..2fba4f5f
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485629288_samel_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780485629288_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780485629288_samel_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..83503d0d
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485629288_samel_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780485629288_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780485629288_sames_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..22bd2ec2
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485629288_sames_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780485629288_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780485629288_sames_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..904b0608
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485629288_sames_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780485714220_pt_samel_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780485714220_pt_samel_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..c1bc94f8
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485714220_pt_samel_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780485714220_pt_sames_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780485714220_pt_sames_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..331eaa4e
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485714220_pt_sames_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780485714220_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780485714220_samel_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..2fba4f5f
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485714220_samel_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780485714220_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780485714220_samel_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..83503d0d
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485714220_samel_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780485714220_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780485714220_sames_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..49fd93a4
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485714220_sames_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780485714220_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780485714220_sames_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..0b0d1691
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485714220_sames_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780485720493_pt_samel_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780485720493_pt_samel_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..c1bc94f8
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485720493_pt_samel_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780485720493_pt_sames_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780485720493_pt_sames_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..331eaa4e
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485720493_pt_sames_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780485720493_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780485720493_samel_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..2fba4f5f
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485720493_samel_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780485720493_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780485720493_samel_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..83503d0d
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485720493_samel_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780485720493_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780485720493_sames_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..1a4fb98a
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485720493_sames_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780485720493_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780485720493_sames_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..42c49db8
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485720493_sames_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780485772267_pt_samel_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780485772267_pt_samel_Death_Metal_s2_spec.png
new file mode 100644
index 00000000..c0ce21a8
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485772267_pt_samel_Death_Metal_s2_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780485772267_pt_sames_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780485772267_pt_sames_Death_Metal_s2_spec.png
new file mode 100644
index 00000000..a465cf4f
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485772267_pt_sames_Death_Metal_s2_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780485772267_samel_canon_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780485772267_samel_canon_Death_Metal_s2_spec.png
new file mode 100644
index 00000000..79d4183a
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485772267_samel_canon_Death_Metal_s2_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780485772267_samel_fp32_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780485772267_samel_fp32_Death_Metal_s2_spec.png
new file mode 100644
index 00000000..ed69fabd
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485772267_samel_fp32_Death_Metal_s2_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780485772267_sames_canon_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780485772267_sames_canon_Death_Metal_s2_spec.png
new file mode 100644
index 00000000..546ad02d
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485772267_sames_canon_Death_Metal_s2_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780485772267_sames_fp32_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780485772267_sames_fp32_Death_Metal_s2_spec.png
new file mode 100644
index 00000000..d924da05
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485772267_sames_fp32_Death_Metal_s2_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780485778528_pt_samel_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780485778528_pt_samel_Death_Metal_s2_spec.png
new file mode 100644
index 00000000..c0ce21a8
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485778528_pt_samel_Death_Metal_s2_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780485778528_pt_sames_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780485778528_pt_sames_Death_Metal_s2_spec.png
new file mode 100644
index 00000000..a465cf4f
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485778528_pt_sames_Death_Metal_s2_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780485778528_samel_canon_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780485778528_samel_canon_Death_Metal_s2_spec.png
new file mode 100644
index 00000000..79d4183a
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485778528_samel_canon_Death_Metal_s2_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780485778528_samel_fp32_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780485778528_samel_fp32_Death_Metal_s2_spec.png
new file mode 100644
index 00000000..ed69fabd
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485778528_samel_fp32_Death_Metal_s2_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780485778528_sames_canon_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780485778528_sames_canon_Death_Metal_s2_spec.png
new file mode 100644
index 00000000..a188e08c
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485778528_sames_canon_Death_Metal_s2_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780485778528_sames_fp32_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780485778528_sames_fp32_Death_Metal_s2_spec.png
new file mode 100644
index 00000000..b9485032
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780485778528_sames_fp32_Death_Metal_s2_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486038662_pt_samel_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780486038662_pt_samel_Death_Metal_s2_spec.png
new file mode 100644
index 00000000..d1584165
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486038662_pt_samel_Death_Metal_s2_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486038662_pt_sames_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780486038662_pt_sames_Death_Metal_s2_spec.png
new file mode 100644
index 00000000..7802aa7c
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486038662_pt_sames_Death_Metal_s2_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486038662_samel_canon_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780486038662_samel_canon_Death_Metal_s2_spec.png
new file mode 100644
index 00000000..10876409
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486038662_samel_canon_Death_Metal_s2_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486038662_samel_fp32_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780486038662_samel_fp32_Death_Metal_s2_spec.png
new file mode 100644
index 00000000..778bcc5b
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486038662_samel_fp32_Death_Metal_s2_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486038662_sames_canon_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780486038662_sames_canon_Death_Metal_s2_spec.png
new file mode 100644
index 00000000..2d905d40
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486038662_sames_canon_Death_Metal_s2_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486038662_sames_fp32_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780486038662_sames_fp32_Death_Metal_s2_spec.png
new file mode 100644
index 00000000..eadd2a18
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486038662_sames_fp32_Death_Metal_s2_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486225881_pt_samel_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780486225881_pt_samel_Death_Metal_s2_spec.png
new file mode 100644
index 00000000..673885d6
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486225881_pt_samel_Death_Metal_s2_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486225881_pt_sames_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780486225881_pt_sames_Death_Metal_s2_spec.png
new file mode 100644
index 00000000..057ad46a
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486225881_pt_sames_Death_Metal_s2_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486225881_samel_canon_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780486225881_samel_canon_Death_Metal_s2_spec.png
new file mode 100644
index 00000000..b8d2e10c
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486225881_samel_canon_Death_Metal_s2_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486225881_samel_fp32_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780486225881_samel_fp32_Death_Metal_s2_spec.png
new file mode 100644
index 00000000..8cf678aa
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486225881_samel_fp32_Death_Metal_s2_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486225881_sames_canon_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780486225881_sames_canon_Death_Metal_s2_spec.png
new file mode 100644
index 00000000..e3043ab7
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486225881_sames_canon_Death_Metal_s2_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486225881_sames_fp32_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780486225881_sames_fp32_Death_Metal_s2_spec.png
new file mode 100644
index 00000000..457c9f67
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486225881_sames_fp32_Death_Metal_s2_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486238402_pt_samel_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780486238402_pt_samel_Death_Metal_s2_spec.png
new file mode 100644
index 00000000..673885d6
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486238402_pt_samel_Death_Metal_s2_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486238402_pt_sames_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780486238402_pt_sames_Death_Metal_s2_spec.png
new file mode 100644
index 00000000..057ad46a
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486238402_pt_sames_Death_Metal_s2_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486238402_samel_canon_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780486238402_samel_canon_Death_Metal_s2_spec.png
new file mode 100644
index 00000000..b8d2e10c
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486238402_samel_canon_Death_Metal_s2_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486238402_samel_fp32_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780486238402_samel_fp32_Death_Metal_s2_spec.png
new file mode 100644
index 00000000..8cf678aa
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486238402_samel_fp32_Death_Metal_s2_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486238402_sames_canon_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780486238402_sames_canon_Death_Metal_s2_spec.png
new file mode 100644
index 00000000..774e621f
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486238402_sames_canon_Death_Metal_s2_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486238402_sames_fp32_Death_Metal_s2_spec.png b/optimized/tensorRT/output/ab_gradio/1780486238402_sames_fp32_Death_Metal_s2_spec.png
new file mode 100644
index 00000000..6d840a16
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486238402_sames_fp32_Death_Metal_s2_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486700361_pt_samel_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486700361_pt_samel_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..c1bc94f8
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486700361_pt_samel_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486700361_pt_sames_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486700361_pt_sames_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..331eaa4e
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486700361_pt_sames_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486700361_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486700361_samel_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..2fba4f5f
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486700361_samel_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486700361_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486700361_samel_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..83503d0d
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486700361_samel_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486700361_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486700361_sames_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..22bd2ec2
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486700361_sames_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486700361_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486700361_sames_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..904b0608
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486700361_sames_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486716579_pt_samel_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486716579_pt_samel_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..292b3476
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486716579_pt_samel_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486716579_pt_sames_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486716579_pt_sames_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..ab6f2483
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486716579_pt_sames_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486716579_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486716579_samel_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..a10d99df
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486716579_samel_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486716579_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486716579_samel_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..71feeeea
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486716579_samel_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486716579_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486716579_sames_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..574f211b
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486716579_sames_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486716579_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486716579_sames_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..3603e14c
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486716579_sames_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486723920_pt_samel_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486723920_pt_samel_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..292b3476
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486723920_pt_samel_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486723920_pt_sames_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486723920_pt_sames_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..ab6f2483
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486723920_pt_sames_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486723920_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486723920_samel_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..a10d99df
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486723920_samel_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486723920_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486723920_samel_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..71feeeea
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486723920_samel_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486723920_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486723920_sames_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..db99739b
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486723920_sames_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486723920_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486723920_sames_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..89aac57a
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486723920_sames_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486731811_pt_samel_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486731811_pt_samel_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..c1bc94f8
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486731811_pt_samel_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486731811_pt_sames_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486731811_pt_sames_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..331eaa4e
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486731811_pt_sames_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486731811_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486731811_samel_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..2fba4f5f
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486731811_samel_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486731811_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486731811_samel_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..83503d0d
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486731811_samel_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486731811_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486731811_sames_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..6c0a096b
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486731811_sames_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486731811_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486731811_sames_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..504157dd
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486731811_sames_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486740562_pt_samel_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486740562_pt_samel_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..c1bc94f8
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486740562_pt_samel_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486740562_pt_sames_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486740562_pt_sames_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..331eaa4e
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486740562_pt_sames_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486740562_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486740562_samel_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..2fba4f5f
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486740562_samel_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486740562_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486740562_samel_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..83503d0d
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486740562_samel_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486740562_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486740562_sames_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..68b1d986
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486740562_sames_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486740562_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486740562_sames_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..f8426956
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486740562_sames_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486749056_pt_samel_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486749056_pt_samel_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..c1bc94f8
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486749056_pt_samel_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486749056_pt_sames_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486749056_pt_sames_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..331eaa4e
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486749056_pt_sames_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486749056_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486749056_samel_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..2fba4f5f
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486749056_samel_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486749056_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486749056_samel_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..83503d0d
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486749056_samel_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486749056_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486749056_sames_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..a7463fe5
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486749056_sames_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486749056_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486749056_sames_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..d71afc80
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486749056_sames_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486757188_pt_samel_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486757188_pt_samel_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..c1bc94f8
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486757188_pt_samel_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486757188_pt_sames_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486757188_pt_sames_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..331eaa4e
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486757188_pt_sames_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486757188_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486757188_samel_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..2fba4f5f
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486757188_samel_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486757188_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486757188_samel_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..83503d0d
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486757188_samel_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486757188_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486757188_sames_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..0d31bc64
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486757188_sames_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486757188_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486757188_sames_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..8208246b
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486757188_sames_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486770233_pt_samel_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486770233_pt_samel_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..aaecf14d
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486770233_pt_samel_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486770233_pt_sames_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486770233_pt_sames_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..8fba811c
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486770233_pt_sames_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486770233_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486770233_samel_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..ac55ada0
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486770233_samel_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486770233_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486770233_samel_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..2de5a7a0
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486770233_samel_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486770233_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486770233_sames_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..0b79cc8f
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486770233_sames_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486770233_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486770233_sames_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..fb12b1a3
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486770233_sames_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486778839_pt_samel_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486778839_pt_samel_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..ab77d8a4
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486778839_pt_samel_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486778839_pt_sames_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486778839_pt_sames_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..fff2a513
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486778839_pt_sames_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486778839_samel_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486778839_samel_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..dfe3c931
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486778839_samel_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486778839_samel_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486778839_samel_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..a7faa5f8
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486778839_samel_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486778839_sames_canon_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486778839_sames_canon_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..20b763d7
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486778839_sames_canon_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/output/ab_gradio/1780486778839_sames_fp32_Death_Metal_s1_spec.png b/optimized/tensorRT/output/ab_gradio/1780486778839_sames_fp32_Death_Metal_s1_spec.png
new file mode 100644
index 00000000..c432a3b2
Binary files /dev/null and b/optimized/tensorRT/output/ab_gradio/1780486778839_sames_fp32_Death_Metal_s1_spec.png differ
diff --git a/optimized/tensorRT/quantize/README.md b/optimized/tensorRT/quantize/README.md
new file mode 100644
index 00000000..7aac3f8a
--- /dev/null
+++ b/optimized/tensorRT/quantize/README.md
@@ -0,0 +1,182 @@
+# SA3 Encoder + Decoder — Quantized Tiers (SAME-S / SAME-L)
+
+Train-free quantized variants of the Stable Audio 3 **encoders and decoders**, each a single
+**self-contained ONNX** derived from the published bf16 export. TensorRT builds the engine; quality is
+preserved by GPTQ weight-error compensation and per-op precision placement (details below).
+
+Two tiers, both derived from the published bf16 export:
+
+- **`fp8`** — fp8 **compute** on the FFN GEMMs (fp8 tensor cores) → real speedup **and** a smaller engine, near-transparent.
+- **`fp8_fast`** — fp8 pushed into the attention projections too (SAME-S only) — faster still, lower quality.
+
+For max fidelity, use the bf16 baseline (or `--precision fp32`).
+
+> **Retired — `w8_bf16` (int8 weight-only).** Its `DequantizeLinear` constant-folds to bf16 at build, so
+> the *engine* came out byte-for-byte the size and speed of the bf16 baseline (only the *ONNX* was
+> smaller) with slightly lossier weights — strictly dominated by bf16. The lesson worth keeping: **ONNX
+> size ≠ engine size.** fp8 stays 1-byte all the way through (small ONNX *and* small engine *and* fp8
+> math); an int8-weight-only tier only shrinks the download, then expands back to bf16 in the engine.
+
+---
+
+## Tiers at a glance
+
+The **bf16 baseline is the first row** of each table. Decoder dB is PSNR vs that bf16 baseline; encoder
+dB/cos is vs **eager** encode (the latent). dB is a heuristic — confirm near-transparent tiers by ear;
+`_fast` has audible artifacts. Speed is whole-model median at seq 1292 (~2 min audio).
+
+### SAME-S decoder
+
+| file | quantized | compute | onnx | speed | dB vs bf16 (music / sfx) |
+|---|---|---|---|---|---|
+| `dec_dynamic_bf16.onnx` | — *(bf16 baseline)* | bf16 | 219 MB | 1.00× | ref (7.25 ms @1292 · 23.2 @4096) |
+| `dec_fp8.onnx` | fp8 weights + activations (FFN) | fp8 FFN GEMM | 58 MB | **1.14×** | 30.9 / 41.0 · near-transparent |
+| `dec_fp8_fast.onnx` | fp8 weights + activations (all linears) | fp8 all GEMMs | 58 MB | **1.22×** | 26.1 / 36.5 · lossy |
+
+### SAME-L decoder
+
+| file | quantized | compute | onnx | speed | dB vs bf16 |
+|---|---|---|---|---|---|
+| `dec_dynamic_triton_swa.onnx` | — *(bf16 baseline)* | bf16 | 1193 MB | 1.00× | ref (54.0 ms @1292 · 174.4 @4096) |
+| `dec_fp8.onnx` | fp8 weights + activations (FFN) | fp8 FFN GEMM | 937 MB | **1.15×** | 43.4 · near-transparent |
+
+SAME-L has **no `_fast` tier**: its attention projections are fp32 islands (fp8 there collapses quality — see islands below).
+
+### SAME-S encoder
+
+The **encoder is cheap** (~1 ms SAME-S, ~7 ms SAME-L), so quant mostly buys **size** — `fp8` shrinks both the onnx and the engine.
+
+| file | quantized | compute | onnx | speed | dB / cos vs eager |
+|---|---|---|---|---|---|
+| `enc_dynamic_bf16.onnx` | — *(bf16 baseline)* | bf16 | 216 MB | 1.00× | 36.4 / 0.997 (~1 ms) |
+| `enc_fp8.onnx` | fp8 (FFN) | fp8 FFN GEMM | 54 MB | 1.08× | 29.9 / 0.986 · near-transparent |
+| `enc_fp8_fast.onnx` | fp8 (all linears) | fp8 all GEMMs | 54 MB | 1.14× | 22.3 / 0.924 · lossy |
+
+### SAME-L encoder
+
+| file | quantized | compute | onnx | speed | dB / cos vs eager |
+|---|---|---|---|---|---|
+| `enc_dynamic_triton_swa.onnx` | — *(bf16 baseline)* | bf16 | 1192 MB | 1.00× | 46.0 / 0.9997 (~7 ms) |
+| `enc_fp8.onnx` | fp8 (FFN) | fp8 FFN GEMM | 895 MB | 1.03× | 30.8 / 0.989 · near-transparent |
+
+Encoder input is audio `(1,2,N)`; output latent `(1,256,N/4096)`. **Every encoder onnx carries a
+silence-pad node** (see below), so any `N` is accepted.
+
+### `fp8` vs `fp8_fast`
+
+Both run the **FFN** GEMMs (`ff.0` / `ff.2`) in fp8. The difference is the **attention projections**:
+- **`fp8`** leaves `to_qkv` / `to_out` in bf16 compute (weight-only fp8 for storage) — attention inputs
+ stay full-precision → near-transparent.
+- **`fp8_fast`** runs `to_qkv` / `to_out` in fp8 too — the extra **+0.08×** (1.14 → 1.22×), but it
+ fp8-rounds Q/K/V *before* the still-bf16 attention core, and the softmax path is fp8-sensitive → the
+ quality drop (31 → 26 dB). SAME-S only.
+
+---
+
+## Sequence length, padding & building
+
+The onnx graphs are **fully dynamic** — every model is correct at **any** length (verified L=1→8192
+vs eager, odd/even/prime, no size-dependent error; quant tiers track bf16 identically at all lengths).
+The only bound is the **TensorRT optimization profile** you build with. Two rules:
+
+- **Build wide.** Use `min=1` and a `max` covering your longest track. Reference build uses
+ **latent `[1, 1292, 8192]`** (decoders, ≈12:40) and **audio `[1, 2097152, 33554432]`** (encoders).
+ `L=1–31` then work and match eager (these can't be chunked); long files run **natively, no chunking**
+ (a 12:40 decode / 6:20 encode is a single call). The old `[32,4096]` was just a conservative default.
+- **Encoders auto-pad.** The SAME-L encoder needs the audio length to be a multiple of 4096 (the
+ downsample ratio); the built-in pad node silence-pads the tail up to the next multiple, matching what
+ eager does internally. Callers feed any `N`. (SAME-S never needed it; padded anyway for a uniform API.)
+
+The `encode_chunked` path in the runtime is a **stale workaround** — it predated wide profiles and was
+chasing what turned out to be out-of-profile garbage, not a real divergence. Single-shot is accurate at
+any length (SAME-L enc cos 0.9995 on a real 285 s track). Chunking is unnecessary.
+
+---
+
+## Where the quantization goes, per tier
+
+Every decoder block is `latent_proj → [ attn: to_qkv → (QKᵀ·softmax·V) → to_out ] → [ ff: ff.0 → GELU → ff.2 ]`,
+with RMSNorm + RoPE around attention. Each tier touches a different subset:
+
+### `dec_fp8` — fp8 on the FFN
+- **FFN** (`ff.0` up-proj, `ff.2` down-proj) `+ latent_proj`: **weight fp8 + activation fp8** (per-tensor clipping scale) → **fp8 GEMM**. Weights GPTQ-compensated.
+- **Attention projections** (`to_qkv`, `to_out`): SAME-S → **weight-only fp8** (bf16 compute — kept out of fp8 math because they feed softmax); SAME-L → **fp32 islands** (untouched).
+- **Attention core** (QKᵀ·softmax·V): bf16 flash (unchanged).
+- **RMSNorm / RoPE:** fp32.
+
+### `dec_fp8_fast` — fp8 everywhere it pays (SAME-S only)
+- **All linears** (FFN + `to_qkv` + `to_out` + `latent_proj`): **weight fp8 + activation fp8** → fp8 GEMM.
+- **Attention core** still bf16 — but now it's fed **fp8-rounded Q/K/V** from the projections. *That rounding is the quality cost* (the softmax path is fp8-sensitive), buying +0.08× speed over `dec_fp8`.
+- **RMSNorm / RoPE:** fp32.
+
+---
+
+## The levers (reference)
+
+### What you quantize — the two halves of every GEMM
+
+| lever | format | compute or storage | speed | quality | notes |
+|---|---|---|---|---|---|
+| **Weights** | fp8 (per-tensor*) or int8 (per-channel) | **storage** (dequant→bf16 at build) | none | near-lossless; **GPTQ** compensates rounding | 1 byte either format — fp8 ≠ smaller than int8 |
+| **Activations** | fp8 only (dynamic, per-tensor **clip** scale) | turns the GEMM into fp8 tensor-core compute | **unlocks the speedup** | the **lossy half** (~31 dB floor; GPTQ can't fix it) | calibration-critical — amax scale → collapse; use a clipping scale |
+
+\* fp8 weight scale **must be per-tensor in a strongly-typed build** (per-channel silently collapses the GEMM). int8 is happily per-channel. int8 *activations* are a non-starter — softmax breaks.
+
+### Where you apply it — which op
+
+| lever | touches | fp8? | speed | quality | verdict |
+|---|---|---|---|---|---|
+| **FFN GEMM** | `ff.0`, `ff.2` | ✅ weight+act | 1.14× (both) / 1.07× (ff.0 only) | ff.0 transparent; **ff.2 sensitive** (±outliers) | the main speed win → `dec_fp8` |
+| **Attn projections** | `to_qkv`, `to_out` | ✅ weight+act | +0.08× on top | lossy — fp8-rounds Q/K/V *before* attention | `dec_fp8_fast` (SAME-S) |
+| **Attn core** | QKᵀ·softmax·V | ❌ no fp8 MHA kernel @ hd=64 | ~none | — | **dead at hd=64** (Sage 0.46–0.97×, fully-fp8 softmax slower); the real lever is algorithmic **windowing** (~1.6× on SAME-L long-seq), not quant |
+
+### Leave-alone islands (constraints, not levers)
+
+| op | rule | why |
+|---|---|---|
+| **RMSNorm** | keep fp32/bf16 | cheap; quantizing buys nothing |
+| **RoPE** | **must stay fp32** | bf16 RoPE angle → long-sequence clip bug (≥2 min renders) |
+| **Attn projections (SAME-L)** | fp32 islands | fp8 compute there collapses quality in the strongly-typed graph |
+
+The mental model: **activations = speed (and the quality floor); weights = size (near-free with GPTQ); attn core = neither (use windowing).**
+
+---
+
+## Which to pick
+
+- **Max fidelity** → the bf16 baseline (or `--precision fp32` for bit-exact eager)
+- **Faster + smaller, near-transparent (default quant)** → `dec_fp8`
+- **Fastest, quality-tolerant** → `dec_fp8_fast` (SAME-S)
+
+## Reducing SAME-L size (optional)
+Both SAME-L tiers are 937 MB because the attention projections stay fp32 (680 MB). Weight-only-quantizing
+those islands (int8/fp8 weights, bf16 compute — no speed or quality cost) pulls both to ~500 MB.
+
+## Building the engines
+
+The tier **ONNX files ship on HuggingFace** (`stabilityai/stable-audio-3-optimized`, alongside the
+existing `dec_dynamic_bf16.onnx` / `enc_dynamic_*`). To build an engine with the wide profile:
+
+```bash
+python build_tiers.py --arch {same-s|same-l} --kind {enc|dec} [--fp8]
+# SAME-S fp8 / fp8_fast tiers need --fp8; SAME-L carries fp8 in-graph (strongly-typed) and needs no flag.
+```
+
+`build_tiers.py` bakes the **wide** profile (decoder `latent [1,1292,8192]`, encoder `audio [1,·,33.5M]`),
+so L=1..31 work and long files run natively. SAME-L uses the `diff_attn_swa` plugin (PREFER_JIT).
+
+## Reproducing the tier ONNX (grafters)
+
+The quantized ONNX are produced by grafting QDQ / int8 / fp8-stored weights onto the published bf16 export:
+
+| grafter | produces |
+|---|---|
+| `fp8_gptq.py` | SAME-S `dec_fp8` / `dec_fp8_fast` (fp8-stored + GPTQ, ORT Hessian capture) |
+| `fp8_gptq_samel.py` | SAME-L `dec_fp8` (fp8-stored + GPTQ, eager Hessian capture — SWA blocks ORT) |
+| `gptq_w8.py` / `gptq_samel.py` | int8 weight-only ONNX — the *retired* `w8_bf16` tier (kept for reference) |
+| `enc_quant_sames.py` / `enc_quant_samel.py` | the encoder tiers |
+| `pad_encoder.py` | adds the silence-pad node so any audio length is accepted |
+
+> **Note:** the grafters reference campaign-local calibration data (real-audio latents) and model
+> checkpoints via absolute paths at the top of each file — adjust those to reproduce. Calibration +
+> Hessians are computed from real-audio latents. The ready-made ONNX on HF need none of this.
diff --git a/optimized/tensorRT/quantize/build_tiers.py b/optimized/tensorRT/quantize/build_tiers.py
new file mode 100644
index 00000000..36f18e68
--- /dev/null
+++ b/optimized/tensorRT/quantize/build_tiers.py
@@ -0,0 +1,68 @@
+#!/usr/bin/env python3
+"""Build a SA3 encoder/decoder tier engine with a WIDE dynamic profile.
+
+ python build_tiers.py --arch {same-s,same-l} --kind {enc,dec} [--fp8]
+
+Why "wide": the historical [32,4096]-frame profile was a conservative default, not a model limit.
+With min=1 the models are correct at L=1..31 (verified vs eager — these can't be chunked), and long
+files run natively (a ~12:40 decode / 6:20 encode is a single call, no chunking). See README.
+
+Profiles built here:
+ decoder : latent (1,256,1) .. (1,256,1292) .. (1,256,8192) # 1 frame .. ~12:40
+ encoder : audio (1,2,1) .. (1,2,2097152) .. (1,2,33554432) # 1 samp .. ~12:40
+
+SAME-S -> weakly-typed EXPLICIT_BATCH + BF16 (+ FP8 builder flag for fp8/fp8_fast tiers).
+SAME-L -> strongly-typed + diff_attn_swa plugin + NO FP8 flag (the in-graph fp8 QDQ nodes
+ carry the precision). Plugin impl is chosen by GPU arch: AOT on sm_120 (JIT isn't
+ stream-capturable there), JIT on sm_90 (override with SA3_SWA_PLUGIN=aot|jit).
+`--fp8` is required for fp8 / fp8_fast SAME-S tiers; harmless/ignored for SAME-L and w8/bf16.
+"""
+import argparse, os, sys
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "scripts"))
+import tensorrt as trt
+
+DEC_PROFILE = ((1, 256, 1), (1, 256, 1292), (1, 256, 8192))
+ENC_PROFILE = ((1, 2, 1), (1, 2, 2097152), (1, 2, 33554432))
+
+def build(onnx_path, out_path, arch, kind, fp8):
+ lg = trt.Logger(trt.Logger.ERROR)
+ trt.init_libnvinfer_plugins(lg, "")
+ strong = arch == "same-l"
+ flags = 0
+ if strong:
+ # SWA plugin: sm_120 (Blackwell) needs AOT — the JIT impl isn't stream-capturable there
+ # and silently drops the decode inside the runtime's mega-graph. sm_90 uses JIT. Choose by
+ # the GPU arch (override with SA3_SWA_PLUGIN=aot|jit); set it before importing the plugin.
+ import torch
+ gpu = "sm_%d%d" % torch.cuda.get_device_capability()
+ want_aot = gpu == "sm_120" or os.environ.get("SA3_SWA_PLUGIN") == "aot"
+ os.environ.setdefault("SA3_SWA_PLUGIN", "aot" if want_aot else "jit")
+ import diff_attn_nocast_plugin # registers samel::diff_attn_swa (AOT + JIT impls) # noqa: F401
+ flags |= 1 << int(trt.NetworkDefinitionCreationFlag.STRONGLY_TYPED)
+ flags |= 1 << int(trt.NetworkDefinitionCreationFlag.PREFER_AOT_PYTHON_PLUGINS if want_aot
+ else trt.NetworkDefinitionCreationFlag.PREFER_JIT_PYTHON_PLUGINS)
+ else:
+ flags |= 1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH)
+ b = trt.Builder(lg); net = b.create_network(flags); p = trt.OnnxParser(net, lg)
+ if not p.parse_from_file(onnx_path):
+ for i in range(p.num_errors): print(p.get_error(i))
+ raise SystemExit("parse failed")
+ cfg = b.create_builder_config(); cfg.set_memory_pool_limit(trt.MemoryPoolType.WORKSPACE, 20 << 30)
+ if not strong:
+ cfg.set_flag(trt.BuilderFlag.BF16)
+ if fp8: cfg.set_flag(trt.BuilderFlag.FP8) # SAME-L carries fp8 in-graph; only SAME-S needs the flag
+ name, prof = ("audio", ENC_PROFILE) if kind == "enc" else ("latent", DEC_PROFILE)
+ pr = b.create_optimization_profile(); pr.set_shape(name, *prof); cfg.add_optimization_profile(pr)
+ ser = b.build_serialized_network(net, cfg)
+ if ser is None: raise SystemExit("build failed")
+ with open(out_path, "wb") as f: f.write(ser)
+ print(f"built {out_path} ({ser.nbytes/1e6:.0f} MB) — {arch} {kind} {'fp8 ' if fp8 else ''}wide", flush=True)
+
+if __name__ == "__main__":
+ ap = argparse.ArgumentParser()
+ ap.add_argument("onnx"); ap.add_argument("out")
+ ap.add_argument("--arch", required=True, choices=["same-s", "same-l"])
+ ap.add_argument("--kind", required=True, choices=["enc", "dec"])
+ ap.add_argument("--fp8", action="store_true", help="required for SAME-S fp8/fp8_fast tiers")
+ a = ap.parse_args()
+ build(a.onnx, a.out, a.arch, a.kind, a.fp8)
diff --git a/optimized/tensorRT/quantize/enc_quant_samel.py b/optimized/tensorRT/quantize/enc_quant_samel.py
new file mode 100644
index 00000000..8e021bb6
--- /dev/null
+++ b/optimized/tensorRT/quantize/enc_quant_samel.py
@@ -0,0 +1,124 @@
+#!/usr/bin/env python3
+"""SAME-L ENCODER quant tiers (w8 + fp8). Eager-capture the encoder FFN activations (Hessian + p99.9
+clip scale) via hooks (SWA plugin blocks ORT), map eager<->onnx FFN linears by order+weight-value,
+graft int8 weight-only (w8) and fp8 FFN GEMM (fp8, per-tensor weight, per-linear clip scale). Attention
+proj stay fp32 islands. Self-contained."""
+import sys, os, re, json, glob
+import numpy as np, torch, onnx
+from onnx import TensorProto, helper, numpy_helper
+sys.path.insert(0, "/weka2/cj/clod/sa3s/stable-audio-3/optimized/tensorRT/scripts"); sys.path.insert(0, "/weka2/cj/clod/fp8_calib/build")
+from stable_audio_3.factory import create_autoencoder_from_config
+from stable_audio_3.loading_utils import copy_state_dict
+torch.set_grad_enabled(False)
+WD = "/weka2/cj/clod/sames_fp8"; E4M3 = 448.0
+cfg = json.load(open("/weka2/cj/clod/sa3s/models/SAME-L/SAME-L.json"))
+ae = create_autoencoder_from_config(cfg["model"], cfg["sample_rate"])
+ck = torch.load("/weka2/cj/clod/sa3s/models/SAME-L/SAME-L.ckpt", map_location="cpu", weights_only=False)
+copy_state_dict(ae, ck.get("state_dict", ck) if isinstance(ck, dict) else ck); ae = ae.cuda().eval()
+# eager encoder FFN linears, in module order
+eproj = [m for nm, m in ae.named_modules() if re.search(r"encoder\..*\.ff\.ff\.0\.proj$", nm)]
+eout = [m for nm, m in ae.named_modules() if re.search(r"encoder\..*\.ff\.ff\.2$", nm)]
+print(f"eager encoder FFN: {len(eproj)} proj + {len(eout)} out", flush=True)
+HP = {}; PP = {} # keyed by (role, idx)
+def mk(role, i):
+ def hook(m, inp):
+ A = inp[0].reshape(-1, inp[0].shape[-1]).double(); h = (A.T @ A).cpu().numpy()
+ HP[(role, i)] = h if (role, i) not in HP else HP[(role, i)] + h
+ p = float(np.percentile(np.abs(inp[0].detach().float().cpu().numpy()), 99.9)); PP.setdefault((role, i), []).append(p)
+ return hook
+H = [m.register_forward_pre_hook(mk("proj", i)) for i, m in enumerate(eproj)] + [m.register_forward_pre_hook(mk("out", i)) for i, m in enumerate(eout)]
+AUD = np.load(f"{WD}/calib_audio_samel.npz")["audio"].astype(np.float32)
+for ci in range(AUD.shape[0]):
+ ae.encode(torch.tensor(AUD[ci:ci + 1], device="cuda")); print(f" captured clip {ci}", flush=True)
+for h in H: h.remove()
+ASC = {k: max(float(np.percentile(v, 90)) / E4M3, 1e-4) for k, v in PP.items()} # per-linear clip scale (robust over clips)
+def fp8_gptq(Wnp, Hnp, sc, bs=128, pd=0.05):
+ dev = "cuda"; W = torch.tensor(np.asarray(Wnp, np.float32), device=dev); Hh = torch.tensor(np.asarray(Hnp, np.float32), device=dev)
+ cin = W.shape[1]; idx = torch.arange(cin, device=dev); Hh[~torch.isfinite(Hh)] = 0
+ d = torch.diag(Hh).clone(); dead = d <= 0; d[dead] = 1; Hh[idx, idx] = d; W[:, dead] = 0; Hh = (Hh + Hh.T) / 2; damp = pd * d.mean(); Hinv = None
+ for k in range(6):
+ try: L = torch.linalg.cholesky(Hh + torch.eye(cin, device=dev) * (damp * 4.0 ** k)); Hinv = torch.linalg.cholesky(torch.cholesky_inverse(L), upper=True); break
+ except Exception: continue
+ if Hinv is None: Hinv = torch.diag(1.0 / torch.sqrt(d + damp))
+ Q = torch.zeros_like(W)
+ for i1 in range(0, cin, bs):
+ i2 = min(i1 + bs, cin); W1 = W[:, i1:i2].clone(); Q1 = torch.zeros_like(W1); Err = torch.zeros_like(W1); Hi = Hinv[i1:i2, i1:i2]
+ for j in range(i2 - i1):
+ w = W1[:, j]; dd = Hi[j, j].clamp_min(1e-12); qf = (w / sc).to(torch.float8_e4m3fn).to(torch.float32)
+ q = qf * sc; Q1[:, j] = qf; err = (w - q) / dd; Err[:, j] = err; W1[:, j:] -= err[:, None] * Hi[j, j:][None, :]
+ Q[:, i1:i2] = Q1
+ if i2 < cin: W[:, i2:] -= Err @ Hinv[i1:i2, i2:]
+ return Q
+def warr(inits, prod, n):
+ w = n.input[1]
+ if w in inits: return w, numpy_helper.to_array(inits[w]), False, None
+ p = prod.get(w)
+ if p and p.op_type == "Transpose" and p.input and p.input[0] in inits: return p.input[0], numpy_helper.to_array(inits[p.input[0]]), True, p
+ return None, None, None, None
+SRC = os.environ["ENC_ONNX"]
+def onnx_ffn(g, inits, prod):
+ pr, ou = {}, {}
+ for n in g.node:
+ if n.op_type != "MatMul": continue
+ mp = re.search(r"blocks\.(\d+)/ff/proj", n.name); mo = re.search(r"blocks\.(\d+)/ff/out", n.name)
+ if mp: pr[int(mp.group(1))] = n
+ elif mo: ou[int(mo.group(1))] = n
+ return [pr[i] for i in sorted(pr)], [ou[i] for i in sorted(ou)]
+def build_tier(mode, out):
+ m = onnx.load(SRC, load_external_data=True); g = m.graph
+ inits = {i.name: i for i in g.initializer}; prod = {o: n for n in g.node for o in n.output}
+ oproj, oout = onnx_ffn(g, inits, prod)
+ assert len(oproj) == len(eproj) and len(oout) == len(eout), f"{len(oproj)}/{len(eproj)} {len(oout)}/{len(eout)}"
+ fp8 = mode == "fp8"
+ if fp8: g.initializer.append(helper.make_tensor("fp8_zero", TensorProto.FLOAT8E4M3FN, [], [0.0]))
+ new_nodes, new_inits, drop = [], [], set()
+ for role, elist, olist in [("proj", eproj, oproj), ("out", eout, oout)]:
+ for i, (em, n) in enumerate(zip(elist, olist)):
+ ws, W, vt, tn = warr(inits, prod, n)
+ Wg = W if vt else W.T # [out,in]
+ Weag = em.weight.detach().double().cpu().numpy()
+ assert np.allclose(Wg, Weag, atol=1e-2), f"map mismatch {role}{i} (L2 {np.abs(Wg-Weag).mean():.4f})"
+ pfx = n.name.strip("/").replace("/", "_"); qax = 0 if vt else 1; wdq = f"{pfx}_wdq"
+ if fp8:
+ scale = np.float32(max(np.abs(Wg).max() / E4M3, 1e-8)) # per-tensor
+ Qi = fp8_gptq(Wg, HP[(role, i)], scale); Qs = Qi if vt else Qi.T.contiguous()
+ raw = Qs.to(torch.float8_e4m3fn).view(torch.uint8).cpu().numpy().tobytes()
+ new_inits.append(helper.make_tensor(f"{pfx}_w8", TensorProto.FLOAT8E4M3FN, list(Qs.shape), raw, raw=True))
+ new_inits.append(numpy_helper.from_array(np.array(scale, np.float16), f"{pfx}_wsc"))
+ new_nodes.append(helper.make_node("DequantizeLinear", [f"{pfx}_w8", f"{pfx}_wsc", "fp8_zero"], [wdq], name=f"{pfx}_DQw"))
+ a = ASC[(role, i)]; new_inits.append(numpy_helper.from_array(np.array(a, np.float16), f"{pfx}_asc"))
+ aq, adq = f"{pfx}_aq", f"{pfx}_adq"
+ new_nodes.append(helper.make_node("QuantizeLinear", [n.input[0], f"{pfx}_asc", "fp8_zero"], [aq], name=f"{pfx}_Qa"))
+ new_nodes.append(helper.make_node("DequantizeLinear", [aq, f"{pfx}_asc", "fp8_zero"], [adq], name=f"{pfx}_DQa"))
+ n.input[0] = adq
+ else:
+ scale = np.maximum(np.abs(Wg).max(axis=1) / 127.0, 1e-4).astype(np.float32) # per-channel int8
+ sc_b = scale.reshape(-1, 1) if not vt else scale.reshape(-1, 1)
+ Wq = np.round((Wg / scale.reshape(-1, 1))).clip(-127, 127).astype(np.int8); Qs = Wq if vt else Wq.T
+ new_inits.append(numpy_helper.from_array(Qs, f"{pfx}_w8"))
+ new_inits.append(numpy_helper.from_array(scale.astype(np.float16), f"{pfx}_wsc"))
+ new_inits.append(helper.make_tensor(f"{pfx}_wzp", TensorProto.INT8, [scale.size], [0] * scale.size))
+ new_nodes.append(helper.make_node("DequantizeLinear", [f"{pfx}_w8", f"{pfx}_wsc", f"{pfx}_wzp"], [wdq], name=f"{pfx}_DQw", axis=qax))
+ if vt:
+ for k, inp in enumerate(tn.input):
+ if inp == ws: tn.input[k] = wdq
+ else: n.input[1] = wdq
+ drop.add(ws)
+ g.initializer.extend(new_inits); g.node.extend(new_nodes)
+ keep = [i for i in g.initializer if i.name not in drop]; del g.initializer[:]; g.initializer.extend(keep)
+ avail = {i.name for i in g.initializer} | {i.name for i in g.input} | {""}; order, rem = [], list(g.node)
+ while rem:
+ nx, prog = [], False
+ for nd in rem:
+ if all(i in avail for i in nd.input): order.append(nd); [avail.add(o) for o in nd.output]; prog = True
+ else: nx.append(nd)
+ rem = nx
+ if not prog: raise RuntimeError("topo stuck")
+ del g.node[:]; g.node.extend(order)
+ for f in (out, out + ".data"):
+ if os.path.exists(f): os.remove(f)
+ onnx.save(m, out, save_as_external_data=True, all_tensors_to_one_file=True, location=os.path.basename(out) + ".data", size_threshold=1024)
+ print(f" {mode} -> {out}", flush=True)
+build_tier("w8", f"{WD}/enc_samel_w8.onnx")
+build_tier("fp8", f"{WD}/enc_samel_fp8.onnx")
+print("done", flush=True)
diff --git a/optimized/tensorRT/quantize/enc_quant_sames.py b/optimized/tensorRT/quantize/enc_quant_sames.py
new file mode 100644
index 00000000..562a5500
--- /dev/null
+++ b/optimized/tensorRT/quantize/enc_quant_sames.py
@@ -0,0 +1,122 @@
+#!/usr/bin/env python3
+"""SAME-S ENCODER quant tiers. ORT-capture FFN/attn-proj activations from calibration AUDIO (Hessian +
+p99.9 clip scale), then emit 3 self-contained onnx: w8 (int8 weight-only), fp8 (FFN fp8-compute +
+attn weight-only fp8), fast (all-linears fp8-compute). Per-channel weight scales (weakly-typed OK)."""
+import os, re
+import numpy as np, onnx, torch
+from onnx import TensorProto, helper, numpy_helper
+import onnxruntime as ort
+torch.set_grad_enabled(False)
+WD = "/weka2/cj/clod/sames_fp8"; E4M3 = 448.0
+SRC = os.environ.get("ENC_ONNX")
+if not SRC:
+ from huggingface_hub import hf_hub_download
+ SRC = hf_hub_download("stabilityai/stable-audio-3-optimized", "onnx/same-s/enc_dynamic_bf16.onnx")
+AUD = np.load(f"{WD}/calib_audio_sames.npz")["audio"].astype(np.float32) # [6,2,N]
+def warr(inits, prod, n):
+ w = n.input[1]
+ if w in inits: return w, numpy_helper.to_array(inits[w]), False, None
+ p = prod.get(w)
+ if p and p.op_type == "Transpose" and p.input and p.input[0] in inits: return p.input[0], numpy_helper.to_array(inits[p.input[0]]), True, p
+ return None, None, None, None
+# ---- capture (ORT): Hessian H=AtA + p99.9 abs per linear ----
+m = onnx.load(SRC, load_external_data=True); g = m.graph
+inits = {i.name: i for i in g.initializer}; prod = {o: n for n in g.node for o in n.output}
+tin = {}
+for n in g.node:
+ if n.op_type == "MatMul":
+ ws, W, vt, tn = warr(inits, prod, n)
+ if ws is not None and W.ndim == 2: tin[n.name] = n.input[0]
+n0 = len(g.output); ex = {o.name for o in g.output}
+for t in set(tin.values()):
+ if t not in ex: g.output.append(helper.make_tensor_value_info(t, TensorProto.FLOAT, None))
+for f in (f"{WD}/_enc.onnx", f"{WD}/_enc.onnx.data"):
+ if os.path.exists(f): os.remove(f)
+onnx.save(m, f"{WD}/_enc.onnx", save_as_external_data=True, all_tensors_to_one_file=True, location="_enc.onnx.data", size_threshold=1024)
+so = ort.SessionOptions(); so.graph_optimization_level = ort.GraphOptimizationLevel.ORT_DISABLE_ALL
+sess = ort.InferenceSession(f"{WD}/_enc.onnx", so, providers=["CUDAExecutionProvider", "CPUExecutionProvider"])
+iname = sess.get_inputs()[0].name; ks = list(tin.keys()); ts = [tin[k] for k in ks]
+Hs = {k: None for k in ks}; P99 = {k: [] for k in ks}
+for ci in range(AUD.shape[0]):
+ acts = sess.run(ts, {iname: AUD[ci:ci + 1]})
+ for k, a in zip(ks, acts):
+ A = a.reshape(-1, a.shape[-1]).astype(np.float64); Hs[k] = A.T @ A if Hs[k] is None else Hs[k] + A.T @ A
+ av = np.abs(a).ravel(); P99[k].append(av if av.size < 400000 else np.random.default_rng(ci).choice(av, 400000, replace=False))
+ print(f" captured clip {ci}", flush=True)
+del sess
+ASCALE = {k: max(float(np.percentile(np.concatenate(v), 99.9)) / E4M3, 1e-4) for k, v in P99.items()} # per-linear clip scale
+print(f" clip scales: min {min(ASCALE.values()):.4f} max {max(ASCALE.values()):.4f}", flush=True)
+# ---- GPTQ (int8 or fp8), blocked, per-output-channel ----
+def gptq(Wnp, Hnp, sc_np, fmt, bs=128, pd=0.05):
+ dev = "cuda"; W = torch.tensor(np.asarray(Wnp, np.float32), device=dev); H = torch.tensor(np.asarray(Hnp, np.float32), device=dev)
+ sc = torch.tensor(sc_np.astype(np.float32), device=dev); cin = W.shape[1]; idx = torch.arange(cin, device=dev)
+ H[~torch.isfinite(H)] = 0; d = torch.diag(H).clone(); dead = d <= 0; d[dead] = 1; H[idx, idx] = d; W[:, dead] = 0
+ H = (H + H.T) / 2; damp = pd * d.mean(); Hinv = None
+ for k in range(6):
+ try: L = torch.linalg.cholesky(H + torch.eye(cin, device=dev) * (damp * 4.0 ** k)); Hinv = torch.linalg.cholesky(torch.cholesky_inverse(L), upper=True); break
+ except Exception: continue
+ if Hinv is None: Hinv = torch.diag(1.0 / torch.sqrt(d + damp))
+ Q = torch.zeros_like(W)
+ for i1 in range(0, cin, bs):
+ i2 = min(i1 + bs, cin); W1 = W[:, i1:i2].clone(); Q1 = torch.zeros_like(W1); Err = torch.zeros_like(W1); Hi = Hinv[i1:i2, i1:i2]
+ for j in range(i2 - i1):
+ w = W1[:, j]; dd = Hi[j, j].clamp_min(1e-12)
+ qf = torch.clamp(torch.round(w / sc), -127, 127) if fmt == "int8" else (w / sc).to(torch.float8_e4m3fn).to(torch.float32)
+ q = qf * sc; Q1[:, j] = qf; err = (w - q) / dd; Err[:, j] = err; W1[:, j:] -= err[:, None] * Hi[j, j:][None, :]
+ Q[:, i1:i2] = Q1
+ if i2 < cin: W[:, i2:] -= Err @ Hinv[i1:i2, i2:]
+ return Q
+# ---- graft one tier (fresh load -> self-contained) ----
+def build_tier(mode, out):
+ m = onnx.load(SRC, load_external_data=True); g = m.graph
+ inits = {i.name: i for i in g.initializer}; prod = {o: n for n in g.node for o in n.output}
+ fp8 = mode in ("fp8", "fast"); noact = re.compile(r"to_qkv|to_out") if mode == "fp8" else re.compile(r"$^")
+ if fp8: g.initializer.append(helper.make_tensor("fp8_zero", TensorProto.FLOAT8E4M3FN, [], [0.0]))
+ new_nodes, new_inits, drop = [], [], set()
+ for n in [x for x in g.node if x.op_type == "MatMul" and x.name in Hs]:
+ ws, W, vt, tn = warr(inits, prod, n)
+ if ws is None: continue
+ Wg = W if vt else W.T; div = E4M3 if fp8 else 127.0
+ scale = np.maximum(np.abs(Wg).max(axis=1) / div, 1e-8).astype(np.float32)
+ Qi = gptq(Wg, Hs[n.name], scale, "fp8" if fp8 else "int8"); Qs = Qi if vt else Qi.T.contiguous()
+ pfx = n.name.strip("/").replace("/", "_"); qax = 0 if vt else 1; wdq = f"{pfx}_wdq"
+ do_act = fp8 and not noact.search(n.name)
+ if fp8:
+ raw = Qs.to(torch.float8_e4m3fn).view(torch.uint8).cpu().numpy().tobytes()
+ new_inits.append(helper.make_tensor(f"{pfx}_w8", TensorProto.FLOAT8E4M3FN, list(Qs.shape), raw, raw=True))
+ new_inits.append(numpy_helper.from_array(scale, f"{pfx}_wsc"))
+ new_nodes.append(helper.make_node("DequantizeLinear", [f"{pfx}_w8", f"{pfx}_wsc", "fp8_zero"], [wdq], name=f"{pfx}_DQw", axis=qax))
+ if do_act:
+ a = ASCALE[n.name]; new_inits.append(helper.make_tensor(f"{pfx}_asc", TensorProto.FLOAT, [], [a]))
+ aq, adq = f"{pfx}_aq", f"{pfx}_adq"
+ new_nodes.append(helper.make_node("QuantizeLinear", [n.input[0], f"{pfx}_asc", "fp8_zero"], [aq], name=f"{pfx}_Qa"))
+ new_nodes.append(helper.make_node("DequantizeLinear", [aq, f"{pfx}_asc", "fp8_zero"], [adq], name=f"{pfx}_DQa"))
+ n.input[0] = adq
+ else:
+ new_inits.append(numpy_helper.from_array(Qs.to(torch.int8).cpu().numpy(), f"{pfx}_w8"))
+ new_inits.append(numpy_helper.from_array(scale, f"{pfx}_wsc"))
+ new_inits.append(helper.make_tensor(f"{pfx}_wzp", TensorProto.INT8, [scale.size], [0] * scale.size))
+ new_nodes.append(helper.make_node("DequantizeLinear", [f"{pfx}_w8", f"{pfx}_wsc", f"{pfx}_wzp"], [wdq], name=f"{pfx}_DQw", axis=qax))
+ if vt:
+ for i, inp in enumerate(tn.input):
+ if inp == ws: tn.input[i] = wdq
+ else: n.input[1] = wdq
+ drop.add(ws)
+ g.initializer.extend(new_inits); g.node.extend(new_nodes)
+ keep = [i for i in g.initializer if i.name not in drop]; del g.initializer[:]; g.initializer.extend(keep)
+ avail = {i.name for i in g.initializer} | {i.name for i in g.input} | {""}; order, rem = [], list(g.node)
+ while rem:
+ nx, prog = [], False
+ for nd in rem:
+ if all(i in avail for i in nd.input): order.append(nd); [avail.add(o) for o in nd.output]; prog = True
+ else: nx.append(nd)
+ rem = nx
+ if not prog: raise RuntimeError("topo stuck")
+ del g.node[:]; g.node.extend(order)
+ for f in (out, out + ".data"):
+ if os.path.exists(f): os.remove(f)
+ onnx.save(m, out, save_as_external_data=True, all_tensors_to_one_file=True, location=os.path.basename(out) + ".data", size_threshold=1024)
+ print(f" {mode:5s} -> {out}", flush=True)
+for mode, out in [("w8", f"{WD}/enc_sames_w8.onnx"), ("fp8", f"{WD}/enc_sames_fp8.onnx"), ("fast", f"{WD}/enc_sames_fast.onnx")]:
+ build_tier(mode, out)
+print("done", flush=True)
diff --git a/optimized/tensorRT/quantize/fp8_gptq.py b/optimized/tensorRT/quantize/fp8_gptq.py
new file mode 100644
index 00000000..08854c44
--- /dev/null
+++ b/optimized/tensorRT/quantize/fp8_gptq.py
@@ -0,0 +1,104 @@
+#!/usr/bin/env python3
+"""fp8-STORED + GPTQ decoder tier — self-contained, every linear 8-bit.
+ALL linears: per-output-channel fp8 weight, GPTQ-compensated, STORED as fp8 (small onnx).
+ - FFN / non-attention: ALSO per-tensor fp8 activation QDQ -> fp8 GEMM (speed, 1.14x)
+ - attention proj (to_qkv/to_out): weight-only fp8, DQ to bf16 -> bf16 compute (quality-safe)
+Fresh reload before grafting so the output .data is self-contained. argv: ."""
+import sys, os, json, re
+import numpy as np, onnx, torch
+from onnx import TensorProto, helper, numpy_helper
+import onnxruntime as ort
+torch.set_grad_enabled(False)
+DEC, OUT = sys.argv[1], sys.argv[2]; WD = "/weka2/cj/clod/sames_fp8"
+AS = json.load(open(f"{WD}/dec_fp8_act_scales_real.json")); nsc = AS["node_scale"]; gs = AS["global_scale"]
+LATS = np.load(f"{WD}/calib_latents.npz")["latents"]
+NOACT = re.compile(os.environ.get("NOACT_RE", r"to_qkv|to_out")) # weight-only fp8 (bf16); set "$^" for all-linears fp8
+E4M3 = 448.0
+def warr(inits, prod, n):
+ w = n.input[1]
+ if w in inits: return w, numpy_helper.to_array(inits[w]), False, None
+ p = prod.get(w)
+ if p and p.op_type == "Transpose" and p.input and p.input[0] in inits: return p.input[0], numpy_helper.to_array(inits[p.input[0]]), True, p
+ return None, None, None, None
+# ---- capture Hessians for ALL linears (ORT), on a temp copy ----
+m = onnx.load(DEC, load_external_data=True); g = m.graph
+inits = {i.name: i for i in g.initializer}; prod = {o: n for n in g.node for o in n.output}
+tin = {n.name: n.input[0] for n in g.node if n.op_type == "MatMul" and warr(inits, prod, n)[0] is not None and warr(inits, prod, n)[1].ndim == 2}
+n0 = len(g.output); ex = {o.name for o in g.output}
+for t in set(tin.values()):
+ if t not in ex: g.output.append(helper.make_tensor_value_info(t, TensorProto.FLOAT, None))
+for _f in (f"{WD}/_fpg.onnx", f"{WD}/_fpg.onnx.data"):
+ if os.path.exists(_f): os.remove(_f)
+onnx.save(m, f"{WD}/_fpg.onnx", save_as_external_data=True, all_tensors_to_one_file=True, location="_fpg.onnx.data", size_threshold=1024)
+so = ort.SessionOptions(); so.graph_optimization_level = ort.GraphOptimizationLevel.ORT_DISABLE_ALL
+sess = ort.InferenceSession(f"{WD}/_fpg.onnx", so, providers=["CUDAExecutionProvider", "CPUExecutionProvider"])
+iname = sess.get_inputs()[0].name; ks = list(tin.keys()); ts = [tin[k] for k in ks]; Hs = {k: None for k in ks}
+for li in range(LATS.shape[0]):
+ acts = sess.run(ts, {iname: LATS[li:li + 1, :, :768].astype(np.float32)})
+ for k, a in zip(ks, acts):
+ A = a.reshape(-1, a.shape[-1]).astype(np.float64); Hs[k] = A.T @ A if Hs[k] is None else Hs[k] + A.T @ A
+ print(f" captured {li}", flush=True)
+del sess
+# ---- GPTQ (torch, blocked, per-output-channel fp8 weight) ----
+def fp8_gptq(Wnp, Hnp, sc_np, bs=128, pd=0.05):
+ dev = "cuda"; W = torch.tensor(np.asarray(Wnp, np.float32), device=dev); H = torch.tensor(np.asarray(Hnp, np.float32), device=dev)
+ sc = torch.tensor(sc_np.astype(np.float32), device=dev); cin = W.shape[1]; idx = torch.arange(cin, device=dev)
+ H[~torch.isfinite(H)] = 0; d = torch.diag(H).clone(); dead = d <= 0; d[dead] = 1; H[idx, idx] = d; W[:, dead] = 0
+ H = (H + H.T) / 2; damp = pd * d.mean(); Hinv = None
+ for k in range(6):
+ try: L = torch.linalg.cholesky(H + torch.eye(cin, device=dev) * (damp * 4.0 ** k)); Hinv = torch.linalg.cholesky(torch.cholesky_inverse(L), upper=True); break
+ except Exception: continue
+ if Hinv is None: Hinv = torch.diag(1.0 / torch.sqrt(d + damp))
+ Q = torch.zeros_like(W)
+ for i1 in range(0, cin, bs):
+ i2 = min(i1 + bs, cin); W1 = W[:, i1:i2].clone(); Q1 = torch.zeros_like(W1); Err = torch.zeros_like(W1); Hi = Hinv[i1:i2, i1:i2]
+ for j in range(i2 - i1):
+ w = W1[:, j]; dd = Hi[j, j].clamp_min(1e-12)
+ qf = (w / sc).to(torch.float8_e4m3fn).to(torch.float32); q = qf * sc
+ Q1[:, j] = qf; err = (w - q) / dd; Err[:, j] = err; W1[:, j:] -= err[:, None] * Hi[j, j:][None, :]
+ Q[:, i1:i2] = Q1
+ if i2 < cin: W[:, i2:] -= Err @ Hinv[i1:i2, i2:]
+ return Q
+# ---- fresh reload (all raw_data present -> self-contained output), then graft ----
+m = onnx.load(DEC, load_external_data=True); g = m.graph
+inits = {i.name: i for i in g.initializer}; prod = {o: n for n in g.node for o in n.output}
+g.initializer.append(helper.make_tensor("fp8_zero", TensorProto.FLOAT8E4M3FN, [], [0.0]))
+new_nodes, new_inits, drop, nact = [], [], set(), 0
+for n in [x for x in g.node if x.op_type == "MatMul" and x.name in Hs]:
+ ws, W, vt, tn = warr(inits, prod, n)
+ if ws is None: continue
+ Wg = W if vt else W.T
+ scale = np.maximum(np.abs(Wg).max(axis=1) / E4M3, 1e-8).astype(np.float32)
+ Qs = fp8_gptq(Wg, Hs[n.name], scale); Qs = Qs if vt else Qs.T.contiguous()
+ raw = Qs.to(torch.float8_e4m3fn).view(torch.uint8).cpu().numpy().tobytes()
+ pfx = n.name.strip("/").replace("/", "_"); qax = 0 if vt else 1
+ new_inits.append(helper.make_tensor(f"{pfx}_w8", TensorProto.FLOAT8E4M3FN, list(Qs.shape), raw, raw=True))
+ new_inits.append(numpy_helper.from_array(scale, f"{pfx}_wsc"))
+ wdq = f"{pfx}_wdq"
+ new_nodes.append(helper.make_node("DequantizeLinear", [f"{pfx}_w8", f"{pfx}_wsc", "fp8_zero"], [wdq], name=f"{pfx}_DQw", axis=qax))
+ if not NOACT.search(n.name): # FFN -> fp8 activation too (fp8 GEMM)
+ a = float(max(nsc.get(n.name, gs), 1e-4)); new_inits.append(helper.make_tensor(f"{pfx}_asc", TensorProto.FLOAT, [], [a]))
+ aq, adq = f"{pfx}_aq", f"{pfx}_adq"
+ new_nodes.append(helper.make_node("QuantizeLinear", [n.input[0], f"{pfx}_asc", "fp8_zero"], [aq], name=f"{pfx}_Qa"))
+ new_nodes.append(helper.make_node("DequantizeLinear", [aq, f"{pfx}_asc", "fp8_zero"], [adq], name=f"{pfx}_DQa"))
+ n.input[0] = adq; nact += 1
+ if vt:
+ for i, inp in enumerate(tn.input):
+ if inp == ws: tn.input[i] = wdq
+ else: n.input[1] = wdq
+ drop.add(ws)
+g.initializer.extend(new_inits); g.node.extend(new_nodes)
+_keep = [i for i in g.initializer if i.name not in drop]; del g.initializer[:]; g.initializer.extend(_keep)
+avail = {i.name for i in g.initializer} | {i.name for i in g.input} | {""}; order, rem = [], list(g.node)
+while rem:
+ nx, prog = [], False
+ for nd in rem:
+ if all(i in avail for i in nd.input): order.append(nd); [avail.add(o) for o in nd.output]; prog = True
+ else: nx.append(nd)
+ rem = nx
+ if not prog: raise RuntimeError("topo stuck")
+del g.node[:]; g.node.extend(order)
+for f in (OUT, OUT + ".data"):
+ if os.path.exists(f): os.remove(f)
+onnx.save(m, OUT, save_as_external_data=True, all_tensors_to_one_file=True, location=os.path.basename(OUT) + ".data", size_threshold=1024)
+print(f"fp8-stored: {len(drop)} linears 8-bit ({nact} with fp8 activation/GEMM, {len(drop)-nact} weight-only) -> {OUT}", flush=True)
diff --git a/optimized/tensorRT/quantize/fp8_gptq_samel.py b/optimized/tensorRT/quantize/fp8_gptq_samel.py
new file mode 100644
index 00000000..0b1937e7
--- /dev/null
+++ b/optimized/tensorRT/quantize/fp8_gptq_samel.py
@@ -0,0 +1,119 @@
+#!/usr/bin/env python3
+"""SAME-L fp8-STORED + GPTQ. Eager Hessian+amax capture (ORT can't run the SWA plugin), fp8-GPTQ the
+FFN weights, graft fp8-stored weight + per-tensor fp8 activation QDQ onto the SAME-L onnx. Attention
+proj stay fp32 islands (fp8 compute breaks them). Self-contained. Writes samel_fp8_gptq.onnx."""
+import sys, os, re, json
+from pathlib import Path
+import numpy as np, torch, onnx
+from onnx import TensorProto, helper, numpy_helper
+sys.path.insert(0, "/weka2/cj/clod/sa3s/stable-audio-3/optimized/tensorRT/scripts")
+sys.path.insert(0, "/weka2/cj/clod/fp8_calib/build")
+from stable_audio_3.factory import create_autoencoder_from_config
+from stable_audio_3.loading_utils import copy_state_dict
+torch.set_grad_enabled(False)
+ML = "/weka2/cj/clod/sa3s/models/SAME-L"; AB = "/weka2/cj/clod/sames_fp8/decoder_ab"; E4M3 = 448.0
+cfg = json.load(open(f"{ML}/SAME-L.json"))
+ae = create_autoencoder_from_config(cfg["model"], cfg["sample_rate"])
+ck = torch.load(f"{ML}/SAME-L.ckpt", map_location="cpu", weights_only=False)
+copy_state_dict(ae, ck.get("state_dict", ck) if isinstance(ck, dict) else ck); ae = ae.to("cuda").eval()
+def ekey(nm):
+ m = re.search(r"transformers\.(\d+)\.ff\.ff\.0\.proj$", nm); x = f"b{m.group(1)}.proj" if m else None
+ if x: return x
+ m = re.search(r"transformers\.(\d+)\.ff\.ff\.2$", nm)
+ if m: return f"b{m.group(1)}.out"
+ return "latent_proj" if re.search(r"latent_proj$", nm) else None
+def okey(nm):
+ m = re.search(r"blocks\.(\d+)/ff/proj", nm); x = f"b{m.group(1)}.proj" if m else None
+ if x: return x
+ m = re.search(r"blocks\.(\d+)/ff/out", nm)
+ if m: return f"b{m.group(1)}.out"
+ return "latent_proj" if re.search(r"latent_proj", nm) else None
+mods = {}
+for nm, mod in ae.named_modules():
+ if hasattr(mod, "weight") and getattr(mod.weight, "ndim", 0) == 2:
+ k = ekey(nm)
+ if k: mods[k] = mod
+print(f"hooked {len(mods)} FFN linears", flush=True)
+H = {k: None for k in mods}; AMAX = {k: 0.0 for k in mods}
+def mk_hook(k):
+ def hook(m, inp):
+ A = inp[0].reshape(-1, inp[0].shape[-1]).double(); h = (A.T @ A).cpu().numpy()
+ H[k] = h if H[k] is None else H[k] + h; AMAX[k] = max(AMAX[k], float(inp[0].abs().max()))
+ return hook
+handles = [m.register_forward_pre_hook(mk_hook(k)) for k, m in mods.items()]
+lats = [np.load("/weka2/cj/clod/fp8_listening/latents_2min.npz")["bf16"]]
+lats += [np.load(f) for f in sorted(Path(AB).glob("samel_lat_*.npy"))]
+for i, L in enumerate(lats):
+ ae.decode(torch.tensor(L, device="cuda", dtype=torch.float32)); print(f" captured {i} ({L.shape[-1]})", flush=True)
+for h in handles: h.remove()
+def fp8_gptq(Wnp, Hnp, sc_np, bs=128, pd=0.05):
+ dev = "cuda"; W = torch.tensor(np.asarray(Wnp, np.float32), device=dev); Hh = torch.tensor(np.asarray(Hnp, np.float32), device=dev)
+ sc = torch.tensor(sc_np.astype(np.float32), device=dev); cin = W.shape[1]; idx = torch.arange(cin, device=dev)
+ Hh[~torch.isfinite(Hh)] = 0; d = torch.diag(Hh).clone(); dead = d <= 0; d[dead] = 1; Hh[idx, idx] = d; W[:, dead] = 0
+ Hh = (Hh + Hh.T) / 2; damp = pd * d.mean(); Hinv = None
+ for k in range(6):
+ try: L = torch.linalg.cholesky(Hh + torch.eye(cin, device=dev) * (damp * 4.0 ** k)); Hinv = torch.linalg.cholesky(torch.cholesky_inverse(L), upper=True); break
+ except Exception: continue
+ if Hinv is None: Hinv = torch.diag(1.0 / torch.sqrt(d + damp))
+ Q = torch.zeros_like(W)
+ for i1 in range(0, cin, bs):
+ i2 = min(i1 + bs, cin); W1 = W[:, i1:i2].clone(); Q1 = torch.zeros_like(W1); Err = torch.zeros_like(W1); Hi = Hinv[i1:i2, i1:i2]
+ for j in range(i2 - i1):
+ w = W1[:, j]; dd = Hi[j, j].clamp_min(1e-12)
+ qf = (w / sc).to(torch.float8_e4m3fn).to(torch.float32); q = qf * sc
+ Q1[:, j] = qf; err = (w - q) / dd; Err[:, j] = err; W1[:, j:] -= err[:, None] * Hi[j, j:][None, :]
+ Q[:, i1:i2] = Q1
+ if i2 < cin: W[:, i2:] -= Err @ Hinv[i1:i2, i2:]
+ return Q
+# ---- graft onto the SAME-L onnx (fresh load -> self-contained) ----
+from huggingface_hub import hf_hub_download
+DEC = hf_hub_download("stabilityai/stable-audio-3-optimized", "onnx/same-l/dec_dynamic_triton_swa.onnx")
+m = onnx.load(DEC, load_external_data=True); g = m.graph
+inits = {i.name: i for i in g.initializer}; prod = {o: n for n in g.node for o in n.output}
+def warr(n):
+ w = n.input[1]
+ if w in inits: return w, numpy_helper.to_array(inits[w]), False, None
+ p = prod.get(w)
+ if p and p.op_type == "Transpose" and p.input and p.input[0] in inits: return p.input[0], numpy_helper.to_array(inits[p.input[0]]), True, p
+ return None, None, None, None
+g.initializer.append(helper.make_tensor("fp8_zero", TensorProto.FLOAT8E4M3FN, [], [0.0]))
+new_nodes, new_inits, drop, done = [], [], set(), 0
+for n in [x for x in g.node if x.op_type == "MatMul"]:
+ key = okey(n.name)
+ if key not in mods: continue # skip attn proj (fp32 islands) / non-FFN
+ ws, W, vt, tn = warr(n)
+ if ws is None: continue
+ Wg = mods[key].weight.detach().double().cpu().numpy() # eager [out,in]
+ scale = np.float32(max(np.abs(Wg).max() / E4M3, 1e-8)) # per-TENSOR scalar (strongly-typed fp8 GEMM needs per-tensor weight scale)
+ Qi = fp8_gptq(Wg, H[key], scale); Qstore = Qi if vt else Qi.T.contiguous() # onnx orientation
+ raw = Qstore.to(torch.float8_e4m3fn).view(torch.uint8).cpu().numpy().tobytes()
+ pfx = n.name.strip("/").replace("/", "_")
+ new_inits.append(helper.make_tensor(f"{pfx}_w8", TensorProto.FLOAT8E4M3FN, list(Qstore.shape), raw, raw=True))
+ new_inits.append(numpy_helper.from_array(np.array(scale, np.float16), f"{pfx}_wsc")) # scalar fp16 (fp16 trunk)
+ a = 0.05; new_inits.append(numpy_helper.from_array(np.array(a, np.float16), f"{pfx}_asc")) # repo-calibrated FFN activation CLIPPING scale
+ wdq, aq, adq = f"{pfx}_wdq", f"{pfx}_aq", f"{pfx}_adq"
+ new_nodes.append(helper.make_node("DequantizeLinear", [f"{pfx}_w8", f"{pfx}_wsc", "fp8_zero"], [wdq], name=f"{pfx}_DQw"))
+ new_nodes.append(helper.make_node("QuantizeLinear", [n.input[0], f"{pfx}_asc", "fp8_zero"], [aq], name=f"{pfx}_Qa"))
+ new_nodes.append(helper.make_node("DequantizeLinear", [aq, f"{pfx}_asc", "fp8_zero"], [adq], name=f"{pfx}_DQa"))
+ n.input[0] = adq
+ if vt:
+ for i, inp in enumerate(tn.input):
+ if inp == ws: tn.input[i] = wdq
+ else: n.input[1] = wdq
+ drop.add(ws); done += 1
+g.initializer.extend(new_inits); g.node.extend(new_nodes)
+_keep = [i for i in g.initializer if i.name not in drop]; del g.initializer[:]; g.initializer.extend(_keep)
+avail = {i.name for i in g.initializer} | {i.name for i in g.input} | {""}; order, rem = [], list(g.node)
+while rem:
+ nx, prog = [], False
+ for nd in rem:
+ if all(i in avail for i in nd.input): order.append(nd); [avail.add(o) for o in nd.output]; prog = True
+ else: nx.append(nd)
+ rem = nx
+ if not prog: raise RuntimeError("topo stuck")
+del g.node[:]; g.node.extend(order)
+OUT = "/weka2/cj/clod/sames_fp8/samel_fp8_gptq.onnx"
+for f in (OUT, OUT + ".data"):
+ if os.path.exists(f): os.remove(f)
+onnx.save(m, OUT, save_as_external_data=True, all_tensors_to_one_file=True, location=os.path.basename(OUT) + ".data", size_threshold=1024)
+print(f"SAME-L fp8-stored+GPTQ on {done} FFN linears -> {OUT}", flush=True)
diff --git a/optimized/tensorRT/quantize/gptq_samel.py b/optimized/tensorRT/quantize/gptq_samel.py
new file mode 100644
index 00000000..9baedba5
--- /dev/null
+++ b/optimized/tensorRT/quantize/gptq_samel.py
@@ -0,0 +1,130 @@
+#!/usr/bin/env python3
+"""GPTQ int8 for the SAME-L decoder FFN linears. ORT can't run the SWA plugin, so capture the
+per-linear input activations from the EAGER SAME-L autoencoder (hooks), build Hessians, GPTQ, and
+graft int8 (fp16 scale) onto the SAME-L onnx (attention proj kept fp32). Writes samel_w8_gptq.onnx."""
+import sys, os, re
+from pathlib import Path
+import numpy as np, torch, onnx
+from onnx import TensorProto, helper, numpy_helper
+sys.path.insert(0, "/weka2/cj/clod/sa3s/stable-audio-3/optimized/tensorRT/scripts")
+sys.path.insert(0, "/weka2/cj/clod/fp8_calib/build")
+from stable_audio_3.factory import create_autoencoder_from_config
+from stable_audio_3.loading_utils import copy_state_dict
+import json
+torch.set_grad_enabled(False)
+ML = "/weka2/cj/clod/sa3s/models/SAME-L"; AB = "/weka2/cj/clod/sames_fp8/decoder_ab"
+cfg = json.load(open(f"{ML}/SAME-L.json"))
+ae = create_autoencoder_from_config(cfg["model"], cfg["sample_rate"])
+ck = torch.load(f"{ML}/SAME-L.ckpt", map_location="cpu", weights_only=False)
+copy_state_dict(ae, ck.get("state_dict", ck) if isinstance(ck, dict) else ck)
+ae = ae.to("cuda").eval()
+# hook the decoder FFN linears (exclude attention proj). Normalize eager+onnx names to (block, role).
+def ekey(nm):
+ m = re.search(r"transformers\.(\d+)\.ff\.ff\.0\.proj$", nm)
+ if m: return f"b{m.group(1)}.proj"
+ m = re.search(r"transformers\.(\d+)\.ff\.ff\.2$", nm)
+ if m: return f"b{m.group(1)}.out"
+ if re.search(r"latent_proj$", nm): return "latent_proj"
+ return None
+def okey(nm):
+ m = re.search(r"blocks\.(\d+)/ff/proj", nm)
+ if m: return f"b{m.group(1)}.proj"
+ m = re.search(r"blocks\.(\d+)/ff/out", nm)
+ if m: return f"b{m.group(1)}.out"
+ if re.search(r"latent_proj", nm): return "latent_proj"
+ return None
+mods = {}
+for nm, mod in ae.named_modules():
+ if hasattr(mod, "weight") and getattr(mod.weight, "ndim", 0) == 2:
+ k = ekey(nm)
+ if k: mods[k] = mod
+print(f"hooked {len(mods)} eager FFN linears: {sorted(mods)[:3]} ...", flush=True)
+H = {k: None for k in mods}
+def mk_hook(k):
+ def hook(m, inp):
+ A = inp[0].reshape(-1, inp[0].shape[-1]).double()
+ h = (A.T @ A).cpu().numpy()
+ H[k] = h if H[k] is None else H[k] + h
+ return hook
+handles = [m.register_forward_pre_hook(mk_hook(k)) for k, m in mods.items()]
+# calibration latents: medium 2-min + 380s + the 10 real-song SAME-L latents
+lats = [np.load("/weka2/cj/clod/fp8_listening/latents_2min.npz")["bf16"]] # 1292 samples ...
+lats += [np.load(f) for f in sorted(Path(AB).glob("samel_lat_*.npy"))] # + 10×376 = ample for the Hessian
+for i, L in enumerate(lats):
+ ae.decode(torch.tensor(L, device="cuda", dtype=torch.float32)); print(f" captured latent {i} ({L.shape[-1]})", flush=True)
+for h in handles: h.remove()
+def gptq(Wnp, Hnp, scale_np, pd=0.05, bs=128):
+ dev = "cuda"
+ W = torch.tensor(np.asarray(Wnp, np.float32), device=dev) # [out, in]
+ H = torch.tensor(np.asarray(Hnp, np.float32), device=dev)
+ sc = torch.tensor(scale_np.astype(np.float32), device=dev) # [out]
+ cin = W.shape[1]; idx = torch.arange(cin, device=dev)
+ H[~torch.isfinite(H)] = 0.0
+ d = torch.diag(H).clone(); dead = d <= 0; d[dead] = 1.0; H[idx, idx] = d; W[:, dead] = 0.0
+ H = (H + H.T) / 2; damp = pd * d.mean(); Hinv = None
+ for k in range(6): # damped Cholesky of H⁻¹ (upper)
+ try:
+ L = torch.linalg.cholesky(H + torch.eye(cin, device=dev) * (damp * 4.0 ** k))
+ Hinv = torch.linalg.cholesky(torch.cholesky_inverse(L), upper=True); break
+ except Exception:
+ continue
+ if Hinv is None: Hinv = torch.diag(1.0 / torch.sqrt(d + damp))
+ Q = torch.zeros_like(W)
+ for i1 in range(0, cin, bs): # BLOCKED: small updates in-block, 1 matmul for the rest
+ i2 = min(i1 + bs, cin); W1 = W[:, i1:i2].clone(); Q1 = torch.zeros_like(W1); Err = torch.zeros_like(W1)
+ Hi = Hinv[i1:i2, i1:i2]
+ for j in range(i2 - i1):
+ w = W1[:, j]; dd = Hi[j, j].clamp_min(1e-12)
+ qi = torch.clamp(torch.round(w / sc), -127, 127); err = (w - qi * sc) / dd
+ Q1[:, j] = qi; Err[:, j] = err; W1[:, j:] -= err[:, None] * Hi[j, j:][None, :]
+ Q[:, i1:i2] = Q1
+ if i2 < cin: W[:, i2:] -= Err @ Hinv[i1:i2, i2:]
+ return torch.clamp(torch.round(Q), -127, 127).to(torch.int8).cpu().numpy()
+# graft onto the onnx
+from huggingface_hub import hf_hub_download
+DEC = hf_hub_download("stabilityai/stable-audio-3-optimized", "onnx/same-l/dec_dynamic_triton_swa.onnx")
+m = onnx.load(DEC, load_external_data=True); g = m.graph
+inits = {i.name: i for i in g.initializer}; prod = {o: n for n in g.node for o in n.output}
+def warr(n):
+ w = n.input[1]
+ if w in inits: return w, numpy_helper.to_array(inits[w]), False, None
+ p = prod.get(w)
+ if p and p.op_type == "Transpose" and p.input and p.input[0] in inits: return p.input[0], numpy_helper.to_array(inits[p.input[0]]), True, p
+ return None, None, None, None
+new_nodes, new_inits, done = [], [], 0
+for n in [x for x in g.node if x.op_type == "MatMul"]:
+ key = okey(n.name)
+ if key not in mods: continue # skip attn proj / non-FFN
+ ws, W, vt, tn = warr(n)
+ if ws is None: continue
+ Wg = mods[key].weight.detach().double().cpu().numpy() # eager [out,in]
+ scale = np.maximum(np.abs(Wg).max(axis=1) / 127.0, 1e-4).astype(np.float32)
+ Qi = gptq(Wg, H[key], scale) # [out,in]
+ qax = 0 if vt else 1
+ inits[ws].CopyFrom(numpy_helper.from_array(Qi if vt else Qi.T, ws))
+ pfx = n.name.strip("/").replace("/", "_")
+ new_inits += [numpy_helper.from_array(scale.astype(np.float16), f"{pfx}_wsc"),
+ helper.make_tensor(f"{pfx}_wzp", TensorProto.INT8, [scale.size], [0] * scale.size)]
+ dq = f"{pfx}_wdq"
+ new_nodes.append(helper.make_node("DequantizeLinear", [ws, f"{pfx}_wsc", f"{pfx}_wzp"], [dq], name=f"{pfx}_DQw", axis=qax))
+ if vt:
+ for i, inp in enumerate(tn.input):
+ if inp == ws: tn.input[i] = dq
+ else: n.input[1] = dq
+ done += 1
+g.initializer.extend(new_inits); g.node.extend(new_nodes)
+avail = {i.name for i in g.initializer} | {i.name for i in g.input} | {""}
+order, rem = [], list(g.node)
+while rem:
+ nx, prog = [], False
+ for nd in rem:
+ if all(i in avail for i in nd.input): order.append(nd); [avail.add(o) for o in nd.output]; prog = True
+ else: nx.append(nd)
+ rem = nx
+ if not prog: raise RuntimeError("topo stuck")
+del g.node[:]; g.node.extend(order)
+OUT = "/weka2/cj/clod/sames_fp8/samel_w8_gptq.onnx"
+for f in (OUT, OUT + ".data"):
+ if os.path.exists(f): os.remove(f)
+onnx.save(m, OUT, save_as_external_data=True, all_tensors_to_one_file=True, location=os.path.basename(OUT) + ".data", size_threshold=1024)
+print(f"GPTQ int8 grafted on {done} SAME-L FFN linears -> {OUT}", flush=True)
diff --git a/optimized/tensorRT/quantize/gptq_w8.py b/optimized/tensorRT/quantize/gptq_w8.py
new file mode 100644
index 00000000..5263cc21
--- /dev/null
+++ b/optimized/tensorRT/quantize/gptq_w8.py
@@ -0,0 +1,87 @@
+#!/usr/bin/env python3
+"""GPTQ int8 per-channel weight-only quantization of a decoder onnx. Captures each linear's input
+activations (ORT), builds the Hessian H = AᵀA, and quantizes weights column-by-column with
+inverse-Hessian error compensation (the OBQ/GPTQ update). Writes a W8 onnx. argv:
+[Lslice]."""
+import os, sys
+import numpy as np, onnx
+from onnx import TensorProto, helper, numpy_helper
+import onnxruntime as ort
+DEC, OUT = sys.argv[1], sys.argv[2]
+LSL = int(sys.argv[3]) if len(sys.argv) > 3 else 768
+LATS = np.load("/weka2/cj/clod/sames_fp8/calib_latents.npz")["latents"]
+m = onnx.load(DEC, load_external_data=True); g = m.graph
+inits = {i.name: i for i in g.initializer}; prod = {o: n for n in g.node for o in n.output}
+def warr(n):
+ w = n.input[1]
+ if w in inits: return w, numpy_helper.to_array(inits[w]), False, None
+ p = prod.get(w)
+ if p and p.op_type == "Transpose" and p.input and p.input[0] in inits: return p.input[0], numpy_helper.to_array(inits[p.input[0]]), True, p
+ return None, None, None, None
+linears = []
+for n in g.node:
+ if n.op_type == "MatMul":
+ ws, W, vt, tn = warr(n)
+ if ws is not None and W.ndim == 2: linears.append((n, ws, W, vt, tn))
+print(f"{len(linears)} linears; capturing activations (L={LSL}) ...", flush=True)
+# --- capture inputs via ORT (add as temp outputs, then restore) ---
+in_names = [n.input[0] for n, *_ in linears]
+n0 = len(g.output)
+existing = {o.name for o in g.output}
+for t in in_names:
+ if t not in existing: g.output.append(helper.make_tensor_value_info(t, TensorProto.FLOAT, None))
+onnx.save(m, "/weka2/cj/clod/sames_fp8/_gptq_cap.onnx", save_as_external_data=True, all_tensors_to_one_file=True, location="_gptq_cap.onnx.data", size_threshold=1024)
+del g.output[n0:]
+so = ort.SessionOptions(); so.graph_optimization_level = ort.GraphOptimizationLevel.ORT_DISABLE_ALL
+sess = ort.InferenceSession("/weka2/cj/clod/sames_fp8/_gptq_cap.onnx", so, providers=["CUDAExecutionProvider", "CPUExecutionProvider"])
+iname = sess.get_inputs()[0].name
+Hs = {n.name: None for n, *_ in linears}
+for li in range(LATS.shape[0]):
+ acts = sess.run(in_names, {iname: LATS[li:li + 1, :, :LSL].astype(np.float32)})
+ for (n, *_), a in zip(linears, acts):
+ A = a.reshape(-1, a.shape[-1]).astype(np.float64)
+ Hs[n.name] = A.T @ A if Hs[n.name] is None else Hs[n.name] + A.T @ A
+ print(f" latent {li} done", flush=True)
+# --- GPTQ ---
+def gptq(W, H, scale, pd=0.01):
+ W = W.astype(np.float64).copy(); cin = W.shape[1]; H = H.copy()
+ dead = np.diag(H) == 0; H[dead, dead] = 1.0; W[:, dead] = 0.0
+ H[np.diag_indices(cin)] += pd * np.mean(np.diag(H))
+ Hinv = np.linalg.cholesky(np.linalg.inv(H)).T # upper-tri Cholesky of H⁻¹
+ Qi = np.zeros(W.shape, dtype=np.int8); sc = scale
+ for i in range(cin):
+ w = W[:, i]; d = Hinv[i, i]
+ qi = np.clip(np.round(w / sc), -127, 127); q = qi * sc
+ Qi[:, i] = qi.astype(np.int8)
+ W[:, i:] -= np.outer((w - q) / d, Hinv[i, i:])
+ return Qi
+new_nodes, new_inits = [], []
+for (n, ws, W, vt, tn) in linears:
+ Wg = W if vt else W.T # -> [out, in]
+ scale = np.maximum(np.abs(Wg).max(axis=1) / 127.0, 1e-9).astype(np.float32) # per-output [out]
+ Qi = gptq(Wg, Hs[n.name], scale) # [out, in] int8
+ inits[ws].CopyFrom(numpy_helper.from_array(Qi if vt else Qi.T, ws))
+ pfx = n.name.strip("/").replace("/", "_"); qax = 0 if vt else 1
+ new_inits += [helper.make_tensor(f"{pfx}_wsc", TensorProto.FLOAT, [scale.size], scale.tolist()),
+ helper.make_tensor(f"{pfx}_wzp", TensorProto.INT8, [scale.size], [0] * scale.size)]
+ dq = f"{pfx}_wdq"
+ new_nodes.append(helper.make_node("DequantizeLinear", [ws, f"{pfx}_wsc", f"{pfx}_wzp"], [dq], name=f"{pfx}_DQw", axis=qax))
+ if vt:
+ for i, inp in enumerate(tn.input):
+ if inp == ws: tn.input[i] = dq
+ else: n.input[1] = dq
+g.initializer.extend(new_inits); g.node.extend(new_nodes)
+avail = {i.name for i in g.initializer} | {i.name for i in g.input} | {""}
+order, rem = [], list(g.node)
+while rem:
+ nx, prog = [], False
+ for nd in rem:
+ if all(i in avail for i in nd.input): order.append(nd); [avail.add(o) for o in nd.output]; prog = True
+ else: nx.append(nd)
+ rem = nx
+ if not prog: raise RuntimeError("topo stuck")
+del g.node[:]; g.node.extend(order)
+for f in (OUT, OUT + ".data"):
+ if os.path.exists(f): os.remove(f)
+onnx.save(m, OUT, save_as_external_data=True, all_tensors_to_one_file=True, location=os.path.basename(OUT) + ".data", size_threshold=1024)
+print(f"GPTQ int8 on {len(linears)} linears -> {OUT}", flush=True)
diff --git a/optimized/tensorRT/quantize/pad_encoder.py b/optimized/tensorRT/quantize/pad_encoder.py
new file mode 100644
index 00000000..ec3faf9c
--- /dev/null
+++ b/optimized/tensorRT/quantize/pad_encoder.py
@@ -0,0 +1,43 @@
+#!/usr/bin/env python3
+"""Add a dynamic silence-pad node to an encoder onnx so it accepts ANY audio length: pads the sample
+axis up to the next multiple of 4096 with zeros (matches what eager does internally). argv: ."""
+import sys, os
+import onnx
+from onnx import helper, TensorProto
+IN, OUT = sys.argv[1], sys.argv[2]
+m = onnx.load(IN, load_external_data=True); g = m.graph
+audio = g.input[0].name; adt = g.input[0].type.tensor_type.elem_type
+opset = {o.domain: o.version for o in m.opset_import}.get("", 0)
+print(f"input '{audio}' dtype={adt} opset={opset}", flush=True)
+g.initializer.extend([
+ helper.make_tensor("_pc4096", TensorProto.INT64, [1], [4096]),
+ helper.make_tensor("_pzeros5", TensorProto.INT64, [5], [0, 0, 0, 0, 0]),
+ helper.make_tensor("_pidx2", TensorProto.INT64, [1], [2]),
+ helper.make_tensor("_pc0", adt, [], [0]),
+])
+for n in g.node: # rewire consumers to the padded tensor
+ for i, inp in enumerate(n.input):
+ if inp == audio: n.input[i] = "_audio_pad"
+g.node.extend([ # pad = (4096 - N%4096) % 4096, pad end of axis 2
+ helper.make_node("Shape", [audio], ["_pshape"], name="_pad_shape"),
+ helper.make_node("Gather", ["_pshape", "_pidx2"], ["_pN"], name="_pad_gather", axis=0),
+ helper.make_node("Mod", ["_pN", "_pc4096"], ["_pr"], name="_pad_mod1"),
+ helper.make_node("Sub", ["_pc4096", "_pr"], ["_pt"], name="_pad_sub"),
+ helper.make_node("Mod", ["_pt", "_pc4096"], ["_ppad"], name="_pad_mod2"),
+ helper.make_node("Concat", ["_pzeros5", "_ppad"], ["_ppads"], name="_pad_concat", axis=0),
+ helper.make_node("Pad", [audio, "_ppads", "_pc0"], ["_audio_pad"], name="_pad_pad", mode="constant"),
+])
+avail = {i.name for i in g.initializer} | {i.name for i in g.input} | {""}
+order, rem = [], list(g.node)
+while rem:
+ nx, prog = [], False
+ for nd in rem:
+ if all(i in avail for i in nd.input): order.append(nd); [avail.add(o) for o in nd.output]; prog = True
+ else: nx.append(nd)
+ rem = nx
+ if not prog: raise RuntimeError("topo stuck")
+del g.node[:]; g.node.extend(order)
+for f in (OUT, OUT + ".data"):
+ if os.path.exists(f): os.remove(f)
+onnx.save(m, OUT, save_as_external_data=True, all_tensors_to_one_file=True, location=os.path.basename(OUT) + ".data", size_threshold=1024)
+print(f"padded encoder -> {OUT}", flush=True)
diff --git a/optimized/tensorRT/scripts/_arch.py b/optimized/tensorRT/scripts/_arch.py
new file mode 100644
index 00000000..8bcfcdae
--- /dev/null
+++ b/optimized/tensorRT/scripts/_arch.py
@@ -0,0 +1,13 @@
+"""Current-GPU SM arch as a plain 'sm_XX' string.
+
+Used by swa_mma_aot to retarget the AOT SWA kernel's PTX `.target`. The plain form (no
+architecture-specific 'a' suffix, e.g. sm_90a / sm_120a) is required because TRT's PTX
+loader rejects the arch-specific variants; the m16n8k8 TF32 MMA this kernel uses is
+available from sm_80 up, so the plain target compiles and loads on every arch.
+"""
+
+
+def detect_arch() -> str:
+ import torch
+ major, minor = torch.cuda.get_device_capability()
+ return f"sm_{major}{minor}"
diff --git a/optimized/tensorRT/scripts/pt_inference.py b/optimized/tensorRT/scripts/pt_inference.py
index 3b296268..b14657e7 100644
--- a/optimized/tensorRT/scripts/pt_inference.py
+++ b/optimized/tensorRT/scripts/pt_inference.py
@@ -35,7 +35,7 @@
# Defaults — matches the user's "medium / SAME-L / fp32" scope.
PT_MODEL_DIR = Path("/weka2/cj/clod/sa3s/models/SA3-M-hf")
-T5_TRT_PATH = TRT_REPO / "models" / "sm_90" / "t5gemma" / "t5gemma_fp16mixed.trt"
+T5_TRT_PATH = TRT_REPO / "models" / "sm_90" / "t5gemma" / "t5gemma_fp16.trt"
SAMEL_CKPT_DIR = Path("/weka2/cj/clod/sa3s/models/SAME-L") # local SAME-L = TRT decoder's weight source
SAMES_CKPT_DIR = Path("/weka2/cj/clod/sa3s/models/SAME-S") # local SAME-S = TRT SAME-S decoder's weight source
diff --git a/optimized/tensorRT/scripts/sa3_gradio.py b/optimized/tensorRT/scripts/sa3_gradio.py
index 91654ca2..3e8580e9 100644
--- a/optimized/tensorRT/scripts/sa3_gradio.py
+++ b/optimized/tensorRT/scripts/sa3_gradio.py
@@ -59,7 +59,7 @@ def discover_variants(dit_name: str) -> list[tuple[str, Path]]:
"""Return [(label, path)] of available DiT engine files for this model.
Scans models///dit_*.trt. The canonical engine
- (dit_fp16mixed.trt) is always first if present; other variants follow
+ (dit_fp16.trt) is always first if present; other variants follow
alphabetically. For the medium DiT also appends a pseudo-variant
"pytorch fp32 (GT)" that dispatches to the PyTorch-eager backend.
"""
@@ -70,14 +70,14 @@ def discover_variants(dit_name: str) -> list[tuple[str, Path]]:
if not d.exists():
return []
files = sorted(d.glob("dit_*.trt"))
- canonical_name = "dit_fp16mixed.trt"
+ canonical_name = "dit_fp16.trt"
canonical = [f for f in files if f.name == canonical_name]
others = [f for f in files if f.name != canonical_name]
out = []
for f in canonical + others:
label = f.name[len("dit_"):-len(".trt")] if f.name.startswith("dit_") and f.name.endswith(".trt") else f.name
if f.name == canonical_name:
- label = "fp16mixed (canonical)"
+ label = "fp16 (canonical)"
elif "buggy" in f.name:
label = label + " ← old, broken"
out.append((label, f))
@@ -88,34 +88,25 @@ def discover_variants(dit_name: str) -> list[tuple[str, Path]]:
def discover_decoder_variants(decoder_name: str) -> list[tuple[str, Path]]:
- """Return [(label, path)] of available decoder engine files. Scans
- models///dec_*.trt.
-
- Canonical (what's referenced by DECODER_PATHS) is always first if present.
- Naming convention: dec_dynamic_.trt (e.g.
- dec_dynamic_bf16.trt, dec_dynamic_fp32.trt, dec_dynamic_triton_swa.trt).
+ """Return [(label, path)] of decoder quantization tiers for this decoder —
+ canonical first. Uses the known tier set (canonical / fp8 / fp8_fast;
+ see quantize/README.md); engines auto-download from HF on selection if missing.
+ Any extra local dec_*.trt not in the known set are appended.
"""
- d = MODELS_ROOT / decoder_name
- if not d.exists():
- return []
- files = sorted(d.glob("dec_*.trt"))
- canonical = canon.DECODER_PATHS.get(decoder_name)
- canonical_name = canonical.name if canonical else ""
- canonicals = [f for f in files if f.name == canonical_name]
- others = [f for f in files if f.name != canonical_name]
- out = []
- for f in canonicals + others:
- # Strip 'dec_dynamic_' prefix and '.trt' suffix for the label.
- stem = f.name
- if stem.startswith("dec_dynamic_"):
- label = stem[len("dec_dynamic_"):-len(".trt")]
- elif stem.startswith("dec_"):
- label = stem[len("dec_"):-len(".trt")]
- else:
- label = stem
- if f.name == canonical_name:
- label = f"{label} (canonical)"
- out.append((label, f))
+ out, known = [], set()
+ for tier, fname in canon.DECODER_TIER_FILENAME.get(decoder_name, {}).items():
+ label = f"{tier} (canonical)" if tier == "canonical" else tier
+ out.append((label, canon.ARCH_DIR / decoder_name / fname))
+ known.add(fname)
+ d = canon.ARCH_DIR / decoder_name
+ if d.exists():
+ for f in sorted(d.glob("dec_*.trt")):
+ if f.name in known:
+ continue
+ stem = f.name
+ label = (stem[len("dec_dynamic_"):-len(".trt")] if stem.startswith("dec_dynamic_")
+ else stem[len("dec_"):-len(".trt")])
+ out.append((label, f))
return out
@@ -196,10 +187,12 @@ def get_inference(dit: str, decoder: str, dit_variant_path: str,
# Override the canonical engine path lookups before constructing.
canon.DIT_CHOICES[dit]["engine"] = Path(dit_variant_path)
canon.DECODER_PATHS[decoder] = Path(dec_variant_path)
+ dec_tier = canon.decoder_tier_from_filename(decoder, Path(dec_variant_path).name)
print(f"\n → loading SA3Inference({dit!r}, {decoder!r}, "
f"dit={Path(dit_variant_path).name}, "
- f"dec={Path(dec_variant_path).name})")
+ f"dec={Path(dec_variant_path).name}, tier={dec_tier})")
inf = SA3Inference(dit, decoder,
+ dec_precision=dec_tier,
default_T_lat=default_T_lat,
default_steps=default_steps,
default_seconds=default_seconds,
diff --git a/optimized/tensorRT/scripts/sa3_trt.py b/optimized/tensorRT/scripts/sa3_trt.py
index 4a6c5699..39170d4f 100644
--- a/optimized/tensorRT/scripts/sa3_trt.py
+++ b/optimized/tensorRT/scripts/sa3_trt.py
@@ -445,6 +445,7 @@ class SA3Inference:
def __init__(self, dit: str, decoder: str, *,
precision: str | None = None,
+ dec_precision: str = "canonical",
default_T_lat: int = 324, default_steps: int = 8,
default_seconds: float = 30.0,
models_dir: Path | None = None,
@@ -455,16 +456,16 @@ def __init__(self, dit: str, decoder: str, *,
Args:
dit: one of DIT_CHOICES — "sm-music" / "sm-sfx" / "medium"
decoder: one of DECODER_PATHS — "same-s" / "same-l"
- precision: None (default → "fp16mixed" for every model), or an
- explicit "fp16mixed" (canonical: FP16 trunk, FP32
+ precision: None (default → "fp16" for every model), or an
+ explicit "fp16" (canonical: FP16 trunk, FP32
RMSNorm/RoPE islands, FMHA-fused FP16 attention core),
"fp8" (medium only; max-speed clean tier — fp8 linears
+ bf16 fused FMHA + baked fp32 RoPE, ~1.3× faster than
- fp16mixed and clean at long sequence, capped L≤4096),
- "bf16" (medium only; ~3% faster but its bf16 RoPE
- angle drifts at long sequence, and not seed-
- reproducible vs fp16mixed), or "fp32" (bit-equiv
- PyTorch eager, ~2× slower). Engines auto-download
+ fp16 and clean at long sequence, capped L≤4096),
+ or "fp32" (bit-equiv
+ PyTorch eager, ~2× slower). ("fp16mixed" and the
+ retired medium-only "bf16" alias to "fp16".)
+ Engines auto-download
from HF if the requested file is missing.
default_T_lat: latent length to build the initial graph at
default_steps: pingpong steps for the initial graph
@@ -479,18 +480,16 @@ def __init__(self, dit: str, decoder: str, *,
raise ValueError(f"unknown dit={dit!r}; valid: {list(DIT_CHOICES)}")
if decoder not in DECODER_PATHS:
raise ValueError(f"unknown decoder={decoder!r}; valid: {list(DECODER_PATHS)}")
- # Resolve precision default per model: fp16-mixed everywhere.
- # bf16 is medium-only and selectable.
+ # Resolve precision default per model: fp16 everywhere. Retired tokens
+ # (fp16mixed, bf16) alias to fp16.
if precision is None:
precision = canon.default_precision(dit)
+ precision = canon.normalize_precision(precision)
if precision not in canon.PRECISIONS:
raise ValueError(f"unknown precision={precision!r}; valid: {canon.PRECISIONS}")
- if precision == "bf16" and dit != "medium":
- raise ValueError("precision='bf16' is only available for dit='medium' "
- "(sm-music/sm-sfx already fuse in fp16mixed)")
if precision == "fp8" and dit != "medium":
raise ValueError("precision='fp8' is only available for dit='medium' "
- "(fp8 linears + bf16 FMHA + baked fp32 RoPE; sm-music/sm-sfx ship fp16mixed only)")
+ "(fp8 linears + bf16 FMHA + baked fp32 RoPE; sm-music/sm-sfx ship fp16 only)")
# Quiet: patch canon's stage/sub/_stage_vram to no-ops so loading
# doesn't spam stdout (gradio in particular wants a clean log).
@@ -504,7 +503,7 @@ def __init__(self, dit: str, decoder: str, *,
if models_dir is not None and str(models_dir) != str(canon.MODELS_DIR):
new_root = Path(models_dir).resolve()
new_arch_dir = new_root / ARCH
- canon.T5GEMMA_PATH = new_arch_dir / "t5gemma" / "t5gemma_fp16mixed.trt"
+ canon.T5GEMMA_PATH = new_arch_dir / "t5gemma" / "t5gemma_fp16.trt"
for kk in DIT_CHOICES:
DIT_CHOICES[kk]["engine"] = new_arch_dir / DIT_CHOICES[kk]["engine"].relative_to(canon.ARCH_DIR)
for kk in DECODER_PATHS:
@@ -521,7 +520,7 @@ def __init__(self, dit: str, decoder: str, *,
self.quiet = quiet
# 1. Lazy-download any missing engines (precision-aware).
- needed = canon.get_engine_files(dit, decoder, precision, with_encoder=with_encoder)
+ needed = canon.get_engine_files(dit, decoder, precision, with_encoder=with_encoder, dec_tier=dec_precision)
_ensure_files(needed)
# 2. Heavy imports (torch + tensorrt + plugin).
@@ -547,10 +546,10 @@ def __init__(self, dit: str, decoder: str, *,
engine_specs = {
"t5": canon.T5GEMMA_PATH,
"dit": canon.get_dit_engine_path(dit, precision),
- "dec": canon.get_decoder_engine_path(decoder, precision),
+ "dec": canon.resolve_decoder_engine(decoder, precision, dec_precision),
}
if with_encoder:
- engine_specs["enc"] = ENCODER_PATHS[decoder]
+ engine_specs["enc"] = canon.resolve_encoder_engine(decoder, dec_precision)
t0 = time.time()
with concurrent.futures.ThreadPoolExecutor(max_workers=len(engine_specs)) as ex:
futs = {name: ex.submit(TRTRunner, path) for name, path in engine_specs.items()}
@@ -721,17 +720,20 @@ def main():
ap.add_argument("--inpaint-range", default=None)
ap.add_argument("--dit", choices=list(DIT_CHOICES.keys()), default=None)
ap.add_argument("--decoder", choices=list(DECODER_PATHS.keys()), default=None)
- ap.add_argument("--precision", choices=list(canon.PRECISIONS), default=None,
- help="DiT engine precision. Default is 'fp16mixed' for every model: "
+ ap.add_argument("--dec-precision", choices=list(canon.DECODER_TIERS), default="canonical",
+ help="Decoder/encoder quantization tier (orthogonal to --precision, which is the "
+ "DiT). 'canonical' (bf16, default); 'fp8' (~1.14x, near-transparent); "
+ "'fp8_fast' (SAME-S only; fp8 attention projections too, ~1.22x, lossier). "
+ "Engine auto-downloads from HF. (For max-fidelity decode use --precision fp32.)")
+ ap.add_argument("--precision", choices=list(canon.PRECISIONS) + list(canon._PRECISION_ALIAS), default=None,
+ help="DiT engine precision. Default is 'fp16' for every model: "
"canonical FP16 trunk + FP32 RMSNorm/RoPE islands + FMHA-fused FP16 "
"attention core, fp32-accurate at every length. 'fp8' (medium only) "
"is the max-speed clean tier — fp8 linears + bf16 fused FMHA + baked "
- "fp32 RoPE, ~1.3x faster than fp16mixed at every length and clean at "
- "long sequence (L<=4096). 'bf16' (medium only) is ~3%% faster than "
- "fp16mixed but evaluates RoPE's angle in bf16 and drifts at long "
- "sequence (clips on a 6-min render); not seed-reproducible vs "
- "fp16mixed. 'fp32' = bit-equiv PyTorch eager, slower. "
- "Auto-downloads from HF.")
+ "fp32 RoPE, ~1.3x faster than fp16 at every length and clean at "
+ "long sequence (L<=4096). 'fp32' = bit-equiv PyTorch eager, slower. "
+ "('fp16mixed' and the retired medium-only 'bf16' are accepted as "
+ "deprecated aliases for 'fp16'.) Auto-downloads from HF.")
ap.add_argument("--models-dir", default=str(canon.MODELS_DIR))
ap.add_argument("--seconds", type=float, default=30.0)
ap.add_argument("--steps", type=int, default=8)
@@ -749,6 +751,10 @@ def main():
args = ap.parse_args()
if args.steps < 1:
ap.error(f"--steps must be ≥ 1 (got {args.steps})")
+ if args.precision in canon._PRECISION_ALIAS: # retired token → canonical, with a note
+ _canon_p = canon._PRECISION_ALIAS[args.precision]
+ print(f" note: --precision {args.precision} is retired → using {_canon_p}", file=sys.stderr)
+ args.precision = _canon_p
# Mute display in quiet mode — match sa3_trt's behavior.
if args.quiet:
@@ -766,7 +772,7 @@ def _noop_vram(label): return 0
if args.models_dir != str(canon.MODELS_DIR):
new_root = Path(args.models_dir).resolve()
new_arch_dir = new_root / ARCH
- canon.T5GEMMA_PATH = new_arch_dir / "t5gemma" / "t5gemma_fp16mixed.trt"
+ canon.T5GEMMA_PATH = new_arch_dir / "t5gemma" / "t5gemma_fp16.trt"
for kk in DIT_CHOICES:
DIT_CHOICES[kk]["engine"] = new_arch_dir / DIT_CHOICES[kk]["engine"].relative_to(canon.ARCH_DIR)
for kk in DECODER_PATHS:
@@ -837,7 +843,8 @@ def _noop_vram(label): return 0
if args.negative_prompt:
suffix = "" if args.cfg != 1.0 else dim(" (ignored: --cfg=1.0)")
print(f" {k('neg prompt')} {bold(repr(args.negative_prompt))}{suffix}")
- print(f" {k('dit')} {magenta(v(args.dit))} {k('decoder')} {magenta(v(args.decoder))} {k('precision')} {v(args.precision)}")
+ _dtier = f" {k('dec-tier')} {magenta(v(args.dec_precision))}" if args.dec_precision != "canonical" else ""
+ print(f" {k('dit')} {magenta(v(args.dit))} {k('decoder')} {magenta(v(args.decoder))}{_dtier} {k('precision')} {v(args.precision)}")
print(f" {k('σmax')} {bold(f'{sigma_max:.2f}')}")
print(f" {k('seconds')} {v(f'{args.seconds}s')} {k('steps')} {v(args.steps)} {k('seed')} {args.seed}")
print(f" {k('cfg')} {v(args.cfg)} {k('mega-graph')} {v('on' if use_mega else 'off (fallback)')}")
@@ -853,7 +860,7 @@ def _noop_vram(label): return 0
# Lazy-download
needed = canon.get_engine_files(args.dit, args.decoder, args.precision,
- with_encoder=bool(args.init_audio))
+ with_encoder=bool(args.init_audio), dec_tier=args.dec_precision)
_ensure_files(needed)
# Heavy imports
@@ -879,10 +886,10 @@ def _noop_vram(label): return 0
engine_specs = {
"t5": canon.T5GEMMA_PATH,
"dit": canon.get_dit_engine_path(args.dit, args.precision),
- "dec": canon.get_decoder_engine_path(args.decoder, args.precision),
+ "dec": canon.resolve_decoder_engine(args.decoder, args.precision, args.dec_precision),
}
if args.init_audio:
- engine_specs["enc"] = ENCODER_PATHS[args.decoder]
+ engine_specs["enc"] = canon.resolve_encoder_engine(args.decoder, args.dec_precision)
t0 = time.time()
with concurrent.futures.ThreadPoolExecutor(max_workers=len(engine_specs)) as ex:
futs = {name: ex.submit(TRTRunner, path) for name, path in engine_specs.items()}
diff --git a/optimized/tensorRT/scripts/sa3_trt_core.py b/optimized/tensorRT/scripts/sa3_trt_core.py
index 96df0065..3b0d5dff 100644
--- a/optimized/tensorRT/scripts/sa3_trt_core.py
+++ b/optimized/tensorRT/scripts/sa3_trt_core.py
@@ -95,15 +95,15 @@ def _detect_gpu_arch() -> str:
ARCH = _detect_gpu_arch()
ARCH_DIR = MODELS_DIR / ARCH
HF_SUBDIR = f"tensorRT/{ARCH}"
-T5GEMMA_PATH = ARCH_DIR / "t5gemma" / "t5gemma_fp16mixed.trt"
+T5GEMMA_PATH = ARCH_DIR / "t5gemma" / "t5gemma_fp16.trt"
# What engine files each DiT choice needs (relative to MODELS_DIR, mirroring HF repo layout).
# Each DiT engine bundles its conditioner tensors (padding_embedding + seconds_total
# Linear) as graph Constants, so no sidecar weight files are needed.
DIT_ENGINE_FILES = {
- "sm-music": ["sa3-sm-music/dit_fp16mixed.trt"],
- "sm-sfx": ["sa3-sm-sfx/dit_fp16mixed.trt"],
- "medium": ["sa3-m/dit_fp16mixed.trt"], # fp16mixed is the medium default
+ "sm-music": ["sa3-sm-music/dit_fp16.trt"],
+ "sm-sfx": ["sa3-sm-sfx/dit_fp16.trt"],
+ "medium": ["sa3-m/dit_fp16.trt"], # fp16 is the medium default
}
DECODER_FILES = {
"same-s": [
@@ -121,15 +121,15 @@ def _detect_gpu_arch() -> str:
# T5Gemma engine — downloaded from HF per-arch. (The tokenizer.json is
# arch-agnostic and ships bundled with the repo at scripts/tokenizer.json,
# so it's NOT in this list.)
- "t5gemma/t5gemma_fp16mixed.trt",
+ "t5gemma/t5gemma_fp16.trt",
]
DIT_CHOICES = {
- "sm-music": {"engine": ARCH_DIR / "sa3-sm-music" / "dit_fp16mixed.trt",
+ "sm-music": {"engine": ARCH_DIR / "sa3-sm-music" / "dit_fp16.trt",
"default_decoder": "same-s"},
- "sm-sfx": {"engine": ARCH_DIR / "sa3-sm-sfx" / "dit_fp16mixed.trt",
+ "sm-sfx": {"engine": ARCH_DIR / "sa3-sm-sfx" / "dit_fp16.trt",
"default_decoder": "same-s"},
- "medium": {"engine": ARCH_DIR / "sa3-m" / "dit_fp16mixed.trt", # medium default
+ "medium": {"engine": ARCH_DIR / "sa3-m" / "dit_fp16.trt", # medium default
"default_decoder": "same-l"},
}
DECODER_PATHS = {
@@ -144,78 +144,71 @@ def _detect_gpu_arch() -> str:
# ─── Precision-keyed engine maps ─────────────────────────────────────────
#
-# Three DiT precisions:
-# fp16mixed — canonical AND the default for every DiT, medium included since
-# 2026-07. FP16 trunk with FP32 islands around RMSNorm and the RoPE
-# *generation*; the attention core (QK^T → Softmax → P·V) runs FP16
-# so TRT's FMHA fuser fires (96 fused nodes on medium). Before that
-# island was bounded the whole O(L²) core sat in FP32 unfused, which
-# is what made bf16 look 4.7× faster. Now: teacher-forced velocity
-# cos 1.0000 vs the FP32 engine at every length, and within ~3% of
-# bf16's speed. Requires STRONGLY_TYPED (see build_dit_fp16mixed.py).
-# bf16 — medium ONLY. Same dit.onnx as fp32, built with BuilderFlag.BF16
-# (EXPLICIT_BATCH), which lets the FMHA fuser fire on a uniform bf16
-# trunk. Marginally the fastest engine, but LOW-FIDELITY AT LONG
-# SEQUENCE: it evaluates RoPE's rotation angle in bf16, and that
-# angle reaches ~4155 rad at L=4092 where bf16's spacing is 32 rad
-# (> 2π), so position information for the fast-rotating dims is
-# destroyed. Over 8 sampling steps the latent inflates ~2.5× and the
-# decoder clips 2–3% of samples on a 6-min render. Fine at short
-# lengths (clean at L=256); prefer fp16mixed for anything long.
-# Also not seed-reproducible vs fp16mixed.
+# Three DiT precisions (fp16mixed was renamed → fp16 — every tier is mixed, so the
+# qualifier was noise; the medium-only bf16 tier was retired. Both old tokens alias
+# to fp16, see _PRECISION_ALIAS):
+# fp16 — canonical AND the default for every DiT. FP16 trunk with FP32
+# islands around RMSNorm and the RoPE *generation*; the attention
+# core (QK^T → Softmax → P·V) runs FP16 so TRT's FMHA fuser fires
+# (96 fused nodes on medium). Bounding that island is what lets the
+# core fuse — before it, the whole O(L²) core sat in FP32 unfused
+# (which is what made the retired bf16 tier look 4.7× faster). Now:
+# teacher-forced velocity cos 1.0000 vs the FP32 engine at every
+# length, ~3% off the old bf16 speed, and accurate at long sequence
+# (bf16 drifted — it evaluated RoPE's angle too coarsely past ~2048
+# rad, ~4155 rad @L4092 vs bf16 spacing 32 rad, clipping 2–3% of a
+# 6-min render). Requires STRONGLY_TYPED (see build_dit_fp16.py).
# fp8 — all DiTs, fp8 E4M3 on the linear GEMMs (attention + RoPE kept
# higher precision). On MEDIUM it's a max-speed clean tier (~1.3×
-# over fp16mixed) via the recipe described below. On sm-music /
+# over fp16) via the recipe described below. On sm-music /
# sm-sfx it is a CLEAN WEIGHT-HALVING tier (engine 479 vs 936 MB),
# only marginally faster (~1.10–1.17×): those DiTs' ~5 ms forward is
# overhead-bound at batch 1, so fp8's GEMM-math savings barely show.
-# Their fp8 is an fp8-QDQ graft onto the fp16mixed graph (fp8 linears
-# + fp16 fused attention + the fp16mixed fp32 islands, STRONGLY_TYPED),
-# velocity-cos ~0.99 vs eager, clip% at/below fp16mixed. medium recipe:
+# Their fp8 is an fp8-QDQ graft onto the fp16 graph (fp8 linears
+# + fp16 fused attention + the fp16 fp32 islands, STRONGLY_TYPED),
+# velocity-cos ~0.99 vs eager, clip% at/below fp16. medium recipe:
# MAX-SPEED clean tier. fp8 E4M3 on the 176 linear
# GEMMs + bf16 fused FMHA (96 nodes) + a BAKED fp32 RoPE constant
# table: position cos/sin are computed host-side at build time and
# frozen as a graph Constant (no in-graph trig), so the island is
# precision-policy-robust and cross-runtime-stable — a constant
# can't be re-fused to bf16 the way a runtime pin can. ~1.3× faster
-# than fp16mixed at every length (measured H200, same-run round-
+# than fp16 at every length (measured H200, same-run round-
# robin: 1.40× @L129 / 1.32× @L1292 / 1.30× @L4092) and CLEAN at
# long sequence (latent std 0.86 vs eager 0.95, 0.000% clip @2min &
# full) — the baked RoPE dodges bf16's long-angle drift. It is a
# SPEED tier over an already-good default, NOT a fidelity upgrade:
-# single-step velocity cos vs fp32 is ~0.92–0.97 (below fp16mixed's
+# single-step velocity cos vs fp32 is ~0.92–0.97 (below fp16's
# ~1.0) but the 8-step render stays coherent. Built weakly-typed
# EXPLICIT_BATCH + BF16 + FP8 + OBEY_PRECISION_CONSTRAINTS (the
-# opposite of fp16mixed's STRONGLY_TYPED — see build_from_onnx.py).
+# opposite of fp16's STRONGLY_TYPED — see build_from_onnx.py).
# The baked table is sized to the profile max L=4096 (= the SAME-L
# decoder's own cap), so within the shipped range no re-bake is
# needed; L>4096 is rejected (a re-bake would be required).
# fp32 — pure-FP32, bit-equivalent to PyTorch eager. ~2× size/latency.
#
# The lookup tables below resolve the engine filename per (dit/decoder,
-# precision). The bf16 DiT recipe is a build-time precision change only (no new
-# ONNX): reuse sa3-m/dit.onnx, build with BF16. Decoders/encoders are unchanged
-# by bf16 (it's a DiT-trunk fusion recipe), so decoder "bf16" reuses the
-# canonical decoder engine. Encoders are FP16-mixed only.
+# precision). Decoders/encoders have no DiT-specific recipe, so they reuse their
+# canonical fp16 engine for every precision. Encoders are fp16 only.
DIT_ENGINE_FILENAME = {
- "bf16": "dit_bf16.trt", # medium only; drifts at long sequence
- "fp8": "dit_fp8.trt", # all DiTs; fp8 linears (medium: +baked RoPE; small: graft on fp16mixed)
- "fp16mixed": "dit_fp16mixed.trt",
- "fp32": "dit_fp32.trt",
+ "fp8": "dit_fp8.trt", # all DiTs; fp8 linears (medium: +baked RoPE; small: graft on fp16)
+ "fp16": "dit_fp16.trt", # canonical + default for every DiT (fp16 trunk + fp32 RMSNorm/RoPE islands)
+ "fp32": "dit_fp32.trt", # pure fp32, bit-equivalent to eager
}
-# DiT precisions actually built per model. bf16 is medium-only; fp8 is available
-# for all three (medium via baked-RoPE/bf16-attn; sm-music/sm-sfx via an fp8-QDQ
-# graft onto their fp16mixed graph — fp8 linears + fp16 fused attn + fp32 islands).
+# DiT precisions actually built per model. fp8 is available for all three (medium
+# via baked-RoPE/bf16-attn; sm-music/sm-sfx via an fp8-QDQ graft onto their fp16
+# graph — fp8 linears + fp16 fused attn + fp32 islands). The retired medium-only
+# bf16 tier (drifted at long sequence) now aliases to fp16 — see _PRECISION_ALIAS.
_DIT_PRECISIONS = {
- "sm-music": ("fp16mixed", "fp8", "fp32"),
- "sm-sfx": ("fp16mixed", "fp8", "fp32"),
- "medium": ("bf16", "fp8", "fp16mixed", "fp32"),
+ "sm-music": ("fp16", "fp8", "fp32"),
+ "sm-sfx": ("fp16", "fp8", "fp32"),
+ "medium": ("fp8", "fp16", "fp32"),
}
-# Per-DiT default precision — fp16mixed everywhere. Medium moved off bf16 once
-# fp16mixed's attention core was fused (4.3x faster than the old fp16mixed engine,
+# Per-DiT default precision — fp16 everywhere. Medium moved off bf16 once
+# fp16's attention core was fused (4.3x faster than the old fp16 engine,
# ~3% off bf16, and fp32-accurate at every sequence length).
-DIT_DEFAULT_PRECISION = {"sm-music": "fp16mixed", "sm-sfx": "fp16mixed",
- "medium": "fp16mixed"}
+DIT_DEFAULT_PRECISION = {"sm-music": "fp16", "sm-sfx": "fp16",
+ "medium": "fp16"}
_DIT_SUBDIR = {"sm-music": "sa3-sm-music", "sm-sfx": "sa3-sm-sfx", "medium": "sa3-m"}
# Both SAME decoders' latent profiles start at L=32 (verified against the engines:
# min=(1,256,32) for same-l and same-s; the DiT's starts at 1). Below 32,
@@ -228,31 +221,97 @@ def _detect_gpu_arch() -> str:
DECODER_ENGINE_FILENAME = {
"same-l": {
- # bf16/fp8 are DiT-only recipes → decoder reuses its canonical fp16-mixed engine.
- "bf16": "dec_dynamic_triton_swa.trt",
- "fp8": "dec_dynamic_triton_swa.trt",
- "fp16mixed": "dec_dynamic_triton_swa.trt",
- "fp32": "dec_dynamic_fp32.trt",
+ # fp8 is a DiT-only recipe → decoder reuses its canonical fp16 engine.
+ # (dec_dynamic_bf16.trt keeps its name — it's the same-s canonical engine.)
+ "fp8": "dec_dynamic_triton_swa.trt",
+ "fp16": "dec_dynamic_triton_swa.trt",
+ "fp32": "dec_dynamic_fp32.trt",
},
"same-s": {
- "bf16": "dec_dynamic_bf16.trt",
- "fp8": "dec_dynamic_bf16.trt",
- "fp16mixed": "dec_dynamic_bf16.trt",
- "fp32": "dec_dynamic_fp32.trt",
+ "fp8": "dec_dynamic_bf16.trt",
+ "fp16": "dec_dynamic_bf16.trt",
+ "fp32": "dec_dynamic_fp32.trt",
},
}
-PRECISIONS = ("bf16", "fp8", "fp16mixed", "fp32")
+PRECISIONS = ("fp8", "fp16", "fp32")
+# Back-compat: retired precision tokens resolve to their replacement.
+# fp16mixed → fp16 : every tier is mixed-precision, so the qualifier was noise.
+# bf16 → fp16 : the medium-only bf16 DiT drifted at long sequence; retired.
+_PRECISION_ALIAS = {"fp16mixed": "fp16", "bf16": "fp16"}
+def normalize_precision(precision):
+ """Map a (possibly retired) precision token to its canonical name."""
+ return _PRECISION_ALIAS.get(precision, precision) if precision else precision
+
+# ── Decoder / encoder quantization TIERS (orthogonal to the DiT --precision) ──
+# Train-free fp8 tiers grafted onto the bf16 export (see quantize/README.md).
+# "canonical" = the shipped default decoder engine. The others are downloaded from HF on demand
+# (tensorRT///dec_*.trt) — the same wide-profile engines built by quantize/build_tiers.py.
+# fp8 fp8 FFN GEMMs (near-transparent, ~1.14×) — the speed pick
+# fp8_fast SAME-S only: fp8 on the attention projections too (~1.22×, lossier)
+# (int8-weight-only "w8_bf16" was retired: its DequantizeLinear constant-folds to bf16 at build, so
+# the engine was byte-for-byte the size/speed of the bf16 baseline with slightly lossier weights —
+# strictly dominated. For a max-fidelity decoder use --precision fp32.)
+DECODER_TIER_FILENAME = {
+ "same-s": {"canonical": DECODER_ENGINE_FILENAME["same-s"]["fp16"],
+ "fp8": "dec_fp8.trt", "fp8_fast": "dec_fp8_fast.trt"},
+ "same-l": {"canonical": DECODER_ENGINE_FILENAME["same-l"]["fp16"],
+ "fp8": "dec_fp8.trt"},
+}
+ENCODER_TIER_FILENAME = {
+ "same-s": {"canonical": "enc_dynamic_bf16.trt",
+ "fp8": "enc_fp8.trt", "fp8_fast": "enc_fp8_fast.trt"},
+ "same-l": {"canonical": "enc_dynamic_triton_swa.trt",
+ "fp8": "enc_fp8.trt"},
+}
+DECODER_TIERS = ("canonical", "fp8", "fp8_fast")
+
+
+def get_decoder_tier_path(decoder_name: str, tier: str = "canonical") -> Path:
+ tiers = DECODER_TIER_FILENAME.get(decoder_name)
+ if tiers is None:
+ raise ValueError(f"unknown decoder={decoder_name!r}; valid: {list(DECODER_TIER_FILENAME)}")
+ if tier not in tiers:
+ raise ValueError(f"decoder tier {tier!r} not available for {decoder_name}; valid: {list(tiers)}")
+ return ARCH_DIR / decoder_name / tiers[tier]
+
+
+def get_encoder_tier_path(decoder_name: str, tier: str = "canonical") -> Path:
+ tiers = ENCODER_TIER_FILENAME[decoder_name]
+ return ARCH_DIR / decoder_name / tiers.get(tier, tiers["canonical"]) # enc falls back if a tier is decoder-only
+
+
+def decoder_tier_from_filename(decoder_name: str, filename: str) -> str:
+ """Reverse-map a decoder engine filename (e.g. 'dec_fp8.trt') to its tier name
+ ('fp8'); used by the gradio variant picker. Falls back to 'canonical'."""
+ for tier, fname in DECODER_TIER_FILENAME.get(decoder_name, {}).items():
+ if fname == filename:
+ return tier
+ return "canonical"
+
+
+def resolve_decoder_engine(decoder_name: str, precision: str = None, dec_tier: str = "canonical") -> Path:
+ """dec_tier 'canonical' → the precision-driven canonical/fp32 decoder (back-compat with
+ --precision, e.g. fp32 still yields dec_dynamic_fp32.trt); any other tier → its engine."""
+ if dec_tier == "canonical":
+ return get_decoder_engine_path(decoder_name, precision)
+ return get_decoder_tier_path(decoder_name, dec_tier)
+
+
+def resolve_encoder_engine(decoder_name: str, dec_tier: str = "canonical") -> Path:
+ if dec_tier == "canonical":
+ return ENCODER_PATHS[decoder_name]
+ return get_encoder_tier_path(decoder_name, dec_tier)
def default_precision(dit_name: str) -> str:
- """Default DiT precision: fp16-mixed for every model.
+ """Default DiT precision: fp16 for every model.
- Medium used bf16 until 2026-07, when bounding the fp16-mixed RoPE island let
- its attention core fuse — that made fp16-mixed 4.3x faster than before and
- within ~3% of bf16, while staying accurate at long sequence (bf16 evaluates
- RoPE's angle too coarsely past ~2048 rad and drifts).
+ (fp16 = FP16 trunk + FP32 RMSNorm/RoPE islands + an FMHA-fused FP16 attention
+ core. Medium used the now-retired bf16 tier until 2026-07, when bounding the
+ fp16 RoPE island let its attention core fuse — 4.3x faster than before and
+ accurate at long sequence, where bf16 drifted.)
"""
- return DIT_DEFAULT_PRECISION.get(dit_name, "fp16mixed")
+ return DIT_DEFAULT_PRECISION.get(dit_name, "fp16")
def get_dit_engine_path(dit_name: str, precision: str = None) -> Path:
@@ -260,15 +319,11 @@ def get_dit_engine_path(dit_name: str, precision: str = None) -> Path:
raise ValueError(f"unknown dit={dit_name!r}; valid: {list(_DIT_SUBDIR)}")
if precision is None:
precision = default_precision(dit_name)
+ precision = normalize_precision(precision)
if precision not in DIT_ENGINE_FILENAME:
raise ValueError(f"unknown precision={precision!r}; valid: {PRECISIONS}")
- if precision == "bf16" and dit_name != "medium":
- raise ValueError(
- f"precision='bf16' is only available for --dit medium (FMHA-fused); "
- f"{dit_name} uses standard attention and already fuses in fp16mixed. "
- f"Valid for {dit_name}: {_DIT_PRECISIONS.get(dit_name)}")
# fp8 is available for all three DiTs (medium: baked-RoPE + bf16 attn; sm-music/
- # sm-sfx: fp8-QDQ graft on their fp16mixed graph). Only bf16 stays medium-only.
+ # sm-sfx: fp8-QDQ graft on their fp16 graph).
return ARCH_DIR / _DIT_SUBDIR[dit_name] / DIT_ENGINE_FILENAME[precision]
@@ -276,24 +331,33 @@ def get_decoder_engine_path(decoder_name: str, precision: str = None) -> Path:
if decoder_name not in DECODER_ENGINE_FILENAME:
raise ValueError(f"unknown decoder={decoder_name!r}; valid: {list(DECODER_ENGINE_FILENAME)}")
if precision is None:
- precision = "fp16mixed" # decoders have no bf16-specific engine; canonical
+ precision = "fp16" # decoders track the canonical (fp16) engine
+ precision = normalize_precision(precision)
if precision not in DECODER_ENGINE_FILENAME[decoder_name]:
raise ValueError(f"unknown precision={precision!r}; valid: {PRECISIONS}")
return ARCH_DIR / decoder_name / DECODER_ENGINE_FILENAME[decoder_name][precision]
def get_engine_files(dit_name: str, decoder_name: str, precision: str = None,
- with_encoder: bool = False) -> list[str]:
+ with_encoder: bool = False, dec_tier: str = "canonical") -> list[str]:
"""Relative paths (under ARCH_DIR) needed for the chosen pipeline. Pass this
list to _ensure_files() to auto-download anything missing from HF.
- precision=None resolves to the per-model default (fp16-mixed)."""
+ precision=None resolves to the per-model default (fp16); dec_tier picks the
+ decoder/encoder quantization tier (canonical / fp8 / fp8_fast)."""
if precision is None:
precision = default_precision(dit_name)
+ precision = normalize_precision(precision)
files = list(SHARED_FILES)
files.append(f"{_DIT_SUBDIR[dit_name]}/{DIT_ENGINE_FILENAME[precision]}")
- files.append(f"{decoder_name}/{DECODER_ENGINE_FILENAME[decoder_name][precision]}")
- if with_encoder:
- files.append(f"{decoder_name}/" + ENCODER_PATHS[decoder_name].name)
+ if dec_tier == "canonical": # precision-driven canonical/fp32 decoder (back-compat)
+ files.append(f"{decoder_name}/{DECODER_ENGINE_FILENAME[decoder_name][precision]}")
+ if with_encoder:
+ files.append(f"{decoder_name}/{ENCODER_PATHS[decoder_name].name}")
+ else: # a quantization tier
+ files.append(f"{decoder_name}/{DECODER_TIER_FILENAME[decoder_name][dec_tier]}")
+ if with_encoder:
+ enc = ENCODER_TIER_FILENAME[decoder_name]
+ files.append(f"{decoder_name}/{enc.get(dec_tier, enc['canonical'])}")
return files
# ─── Display helpers (ANSI color when stdout is a TTY) ───────────────────
@@ -1082,14 +1146,14 @@ def prompt_user_if_missing(args):
suggested = DIT_CHOICES[args.dit]["default_decoder"]
args.decoder = _arrow_pick("Choose audio decoder:", list(DECODER_PATHS.keys()), default=suggested)
print(f" → {args.decoder}")
- # Resolve DiT precision default per model: fp16-mixed everywhere. bf16 is
- # medium-only and selectable (sm-music / sm-sfx use standard attention and
- # already fuse in fp16mixed).
+ # Resolve DiT precision: default fp16 for every model. Retired tokens
+ # (fp16mixed, bf16) alias to fp16 with a one-line note.
if getattr(args, "precision", None) is None:
args.precision = default_precision(args.dit)
- if args.precision == "bf16" and args.dit != "medium":
- sys.exit("error: --precision bf16 is only available for --dit medium "
- "(sm-music / sm-sfx already fuse in fp16mixed).")
+ elif args.precision in _PRECISION_ALIAS:
+ _canon = _PRECISION_ALIAS[args.precision]
+ print(f" note: --precision {args.precision} is retired → using {_canon}", file=sys.stderr)
+ args.precision = _canon
if args.seed is None:
args.seed = random.randint(0, 2**31 - 1)
return args
@@ -1137,20 +1201,22 @@ def main():
ap.add_argument("--decoder", choices=list(DECODER_PATHS.keys()), default=None,
help="Audio decoder. 'same-s' pairs with sm-* (110 MB engine). "
"'same-l' pairs with medium (1.2 GB engine). Interactive picker if omitted.")
- ap.add_argument("--precision", choices=list(PRECISIONS), default=None,
- help="DiT engine precision (default is 'fp16mixed' for every model). "
- "'fp16mixed' = FP16 trunk + FP32 RMSNorm/RoPE islands with an FMHA-fused "
+ ap.add_argument("--dec-precision", choices=list(DECODER_TIERS), default="canonical",
+ help="Decoder/encoder quantization tier (orthogonal to --precision, the DiT): "
+ "canonical (bf16) | fp8 (~1.14x, near-transparent) | fp8_fast (SAME-S only, "
+ "~1.22x). Auto-downloads from HF. (For max-fidelity decode use --precision fp32.)")
+ ap.add_argument("--precision", choices=list(PRECISIONS) + list(_PRECISION_ALIAS), default=None,
+ help="DiT engine precision (default 'fp16' for every model). "
+ "'fp16' = FP16 trunk + FP32 RMSNorm/RoPE islands with an FMHA-fused "
"FP16 attention core (canonical; fp32-accurate at every length). "
"'fp8' = fp8 E4M3 linears (attention + RoPE kept higher precision). On MEDIUM "
"it's a max-speed clean tier (bf16 fused FMHA + baked fp32 RoPE, ~1.3× faster, "
"clean at long sequence, capped at L<=4096). On sm-music/sm-sfx it's a clean "
"weight-halving tier (479 vs 936 MB engine, velocity-cos ~0.99 vs eager) that is "
"only marginally faster (~1.1×; their small forward is overhead-bound at batch 1). "
- "'bf16' (MEDIUM ONLY) = ~3%% faster still, but it evaluates RoPE's angle "
- "in bf16 and drifts at long sequence (clips 2-3%% of samples on a 6-min "
- "render); fine for short clips, not seed-reproducible vs fp16mixed. "
"'fp32' = pure FP32, matches PyTorch eager bit-for-bit but ~2× slower "
- "and ~2× the VRAM. Engines auto-download from HF if missing.")
+ "and ~2× the VRAM. ('fp16mixed' and the retired medium-only 'bf16' are accepted "
+ "as deprecated aliases for 'fp16'.) Engines auto-download from HF if missing.")
ap.add_argument("--models-dir", default=str(MODELS_DIR),
help=f"Directory containing the TRT engines. Default: {MODELS_DIR}")
# ── Sampling ──
@@ -1207,7 +1273,7 @@ def _stage_vram(label): return 0
if args.models_dir != str(MODELS_DIR):
new_root = Path(args.models_dir).resolve()
new_arch_dir = new_root / ARCH
- T5GEMMA_PATH = new_arch_dir / "t5gemma" / "t5gemma_fp16mixed.trt"
+ T5GEMMA_PATH = new_arch_dir / "t5gemma" / "t5gemma_fp16.trt"
for kk in DIT_CHOICES:
DIT_CHOICES[kk]["engine"] = new_arch_dir / DIT_CHOICES[kk]["engine"].relative_to(ARCH_DIR)
for kk in DECODER_PATHS:
@@ -1309,7 +1375,7 @@ def _stage_vram(label): return 0
# ── Lazy-download any missing engine files for the chosen (dit, decoder) combo ──
needed = get_engine_files(args.dit, args.decoder, args.precision,
- with_encoder=bool(args.init_audio))
+ with_encoder=bool(args.init_audio), dec_tier=args.dec_precision)
_ensure_files(needed)
# ── Heavy imports (torch + tensorrt + plugin) ──
@@ -1337,10 +1403,10 @@ def _stage_vram(label): return 0
engine_specs = {
"t5": T5GEMMA_PATH,
"dit": get_dit_engine_path(args.dit, args.precision),
- "dec": get_decoder_engine_path(args.decoder, args.precision),
+ "dec": resolve_decoder_engine(args.decoder, args.precision, args.dec_precision),
}
if args.init_audio:
- engine_specs["enc"] = ENCODER_PATHS[args.decoder]
+ engine_specs["enc"] = resolve_encoder_engine(args.decoder, args.dec_precision)
t0 = time.time()
with concurrent.futures.ThreadPoolExecutor(max_workers=len(engine_specs)) as ex:
futs = {name: ex.submit(TRTRunner, path) for name, path in engine_specs.items()}
diff --git a/optimized/tflite/README.md b/optimized/tflite/README.md
index 9fd20034..97f8ac98 100644
--- a/optimized/tflite/README.md
+++ b/optimized/tflite/README.md
@@ -72,7 +72,7 @@ lazy-download from HuggingFace on first use.
| `--precision` | legacy name | size (sm DiT / medium DiT / codecs) | quality | CPU speed |
|---------------|-------------|-------------------------------------|---------|-----------|
| `fp32` (default) | — | 1.8 GB / 5.8 GB / 0.2–1.8 GB | reference | 1× — on CPU this is *also* the fast choice |
-| `w16a32` | fp16mixed | 0.9 / 2.9 / 0.1–0.9 GB | ≈lossless (fp16 weights, fp32 activations; 62–75 dB per-forward) | 1.5–3× slower, model-dependent (XNNPACK dequantizes per-matmul) |
+| `w16a32` | fp16 | 0.9 / 2.9 / 0.1–0.9 GB | ≈lossless (fp16 weights, fp32 activations; 62–75 dB per-forward) | 1.5–3× slower, model-dependent (XNNPACK dequantizes per-matmul) |
| `w8a32` | woint8 | 0.45 / 1.5 / 0.05–0.5 GB | GPTQ int8 weights — codecs transparent (40–46 dB); DiT gives a *different but plausible* sample | ≈fp32 |
| `w8a8-dyn` | dynint8 | 0.45 / 1.5 / 0.05–0.5 GB | lowest (int8 weights + activations, per-invoke dynamic scales) | fastest (~1.2–1.3×) |
diff --git a/optimized/tflite/scripts/sa3_tflite.py b/optimized/tflite/scripts/sa3_tflite.py
index 9c020ae3..f022e1a2 100644
--- a/optimized/tflite/scripts/sa3_tflite.py
+++ b/optimized/tflite/scripts/sa3_tflite.py
@@ -75,7 +75,7 @@
# fp32 reference models (default — on CPU fp16 is SLOWER than fp32, so unlike
# TRT the fastest-and-accurate choice is fp32)
# w16a32 fp16 weights / fp32 activations — half size, ≈lossless, 1.5-3× slower
-# on an M4 Pro (legacy name: fp16mixed; ≈ TRT's fp16mixed in spirit,
+# on an M4 Pro (legacy name: fp16; ≈ TRT's fp16 in spirit,
# storage-only here)
# w8a32 GPTQ weight-only int8 — ¼ size at fp32 speed (legacy: woint8)
# w8a8-dyn GPTQ dynamic int8 — fastest (~1.2-1.3×), lowest quality (legacy: dynint8)