FP8 Sol-Attn with Examples of Dense&Sol-sparse Wan2.1 and Minimax-H3

docs/en/attention.md

@@ -71,6 +71,9 @@ class SparseAttentionConfig:
     sol_tau: float = 1.0                    # Sol-Attn routing threshold
     sol_threshold_type: str = "diag"        # "diag" or "exact"
     sol_kv_splits: int | str = "auto"       # "auto", 1, 2, or 4
+    sol_fp8: bool = False                    # FP8 Q/K/V Sol-Attn on SM90
+    sol_fp8_layer_start: int = 0             # First layer using FP8 Sol-Attn
+    sol_fp8_layer_end: int | None = None     # Exclusive end; None means all remaining layers
 ```
 
 ## Calling Flow
@@ -163,6 +166,7 @@ else:
 | Pipeline | Dense Attention | Radial | Sol-Attn | Notes |
 |----------|-----------------|--------|----------|-------|
 | `Wan21VideoPipeline` | Yes | Yes | Experimental | Sol-Attn covers eligible self-attention calls |
+| `MiniMaxH3Pipeline` | Yes | No | Experimental | FL2VA supports exact prefix sinks and FP8 Q/K/V on SM90 |
 | `Wan22VideoPipeline` | Yes | Yes | No | Sol-Attn is not wired into Wan2.2 yet |
 | `QwenImagePipeline` | Yes | No | No | Image generation doesn't need temporal sparse attention |
 | `ZImagePipeline` | Yes | No | No | Image generation doesn't need temporal sparse attention |
@@ -197,10 +201,23 @@ config = AttentionConfig.sol_attention()
 pipe_config.dit_config.attention_config = config
 ```
 
-Sol-Attn is used only for contiguous, noncausal BF16 self-attention with equal Q/K/V
-shapes and head dimension 128. Unsupported calls, dense warmup layers or timesteps,
-and kernel runtime failures fall back to the existing dense attention path. Ring/USP
-also remains dense because its online merge requires log-sum-exp output.
+Sol-Attn is used for contiguous, noncausal self-attention with equal Q/K/V shapes
+and head dimension 128. BF16 is supported by the architecture-specific kernels;
+SM90 additionally supports E4M3 Q/K/V with FP32 accumulation. Unsupported calls,
+dense warmup layers or timesteps, and kernel runtime failures fall back to the
+existing dense attention path. Ring/USP remains dense because its online merge
+requires log-sum-exp output.
+
+`sol_fp8_layer_start` and `sol_fp8_layer_end` restrict E4M3 Q/K/V to a half-open
+transformer-layer range. Sparse layers outside that range continue to use BF16
+Sol-Attn. This controls accumulated FP8 routing error in diffusion models.
+Setting `dense_timesteps=0`, `dense_layers=0`, and a negative `tau` forces all
+KV blocks onto the exact route. The Wan optimized example exposes this as
+`--attention fp8-dense`; `--attention fp8-sol` enables centroid routing with the
+same FP8 Q/K/V and QK/PV kernel.
+For the kernel data flow, precision boundaries, and H100 ablations, see the
+[FP8 Sol-Attn technical article](blog/fp8_sol_attention.md).
+
 
 ### QwenImagePipeline / ZImagePipeline
 

docs/en/blog/index.md

@@ -13,6 +13,7 @@ alternatives, implementation tradeoffs, and hardware-specific results behind tha
 
 | Date | Article | Status | Validated platform |
 |---|---|---|---|
+| 2026-08-19 | [FP8 Sol-Attn: Quantized Sparse Attention for Video DiTs on H100](fp8_sol_attention.md) | Validated | 1 x H100 80 GB |
 | 2026-08-06 | [CUDA IPC Ulysses: Overlapping Attention Communication on H100](cuda_ipc_ulysses.md) | Validated | 4 x H100 80 GB |
 
 ## Publication Contract

docs/en/quantization.md

@@ -94,7 +94,7 @@ quant_config = QuantConfig(
 )
 ```
 
-For MiniMax H3, use `quantization="tf-kernel-fp8"` with
+For MiniMax H3, use `quantization="fp8"` (`"tf-kernel-fp8"` remains an alias) with
 `examples/minimax_h3/minimax_h3_fl2va_h100.py`. This backend is single-GPU only and keeps the FP8
 weights resident after first-use conversion. It is distinct from the scaled-FP8 checkpoint path below: the latter
 expects weights and scales already serialized in the checkpoint.

docs/zh/attention.md

@@ -71,6 +71,9 @@ class SparseAttentionConfig:
     sol_tau: float = 1.0                    # Sol-Attn 路由阈值
     sol_threshold_type: str = "diag"        # "diag" 或 "exact"
     sol_kv_splits: int | str = "auto"       # "auto"、1、2 或 4
+    sol_fp8: bool = False                    # SM90 FP8 Q/K/V Sol-Attn
+    sol_fp8_layer_start: int = 0             # 启用 FP8 Sol-Attn 的首层
+    sol_fp8_layer_end: int | None = None     # 结束层（不包含）；None 表示其余所有层
 ```
 
 ## 调用流程
@@ -163,6 +166,7 @@ else:
 | Pipeline | 密集注意力 | Radial | Sol-Attn | 说明 |
 |----------|-----------|--------|----------|------|
 | `Wan21VideoPipeline` | 支持 | 支持 | 实验性 | Sol-Attn 用于满足约束的 self-attention |
+| `MiniMaxH3Pipeline` | 支持 | 不支持 | 实验性 | FL2VA 支持 exact prefix sink 与 SM90 FP8 Q/K/V |
 | `Wan22VideoPipeline` | 支持 | 支持 | 不支持 | 尚未接入 Wan2.2 |
 | `QwenImagePipeline` | 支持 | 不支持 | 不支持 | 图像生成不需要时序稀疏注意力 |
 | `ZImagePipeline` | 支持 | 不支持 | 不支持 | 图像生成不需要时序稀疏注意力 |
@@ -197,10 +201,22 @@ config = AttentionConfig.sol_attention()
 pipe_config.dit_config.attention_config = config
 ```
 
-Sol-Attn 仅用于连续、非因果、BF16、Q/K/V 形状相同且 head dimension 为 128 的
-self-attention。其他调用、dense 预热层/时间步以及内核运行失败都会回退到现有密集路径。
+Sol-Attn 用于连续、非因果、Q/K/V 形状相同且 head dimension 为 128 的
+self-attention。各架构内核支持 BF16，SM90 还支持使用 FP32 累加的 E4M3 Q/K/V。
+其他调用、dense 预热层/时间步以及内核运行失败都会回退到现有密集路径。
 Ring/USP 需要 LSE 做在线合并，因此仍使用支持 LSE 的密集后端。
 
+`sol_fp8_layer_start` 和 `sol_fp8_layer_end` 用半开区间限制使用 E4M3 Q/K/V
+的 transformer 层，区间外的稀疏层继续使用 BF16 Sol-Attn，以控制扩散模型中
+逐层累积的 FP8 路由误差。
+
+设置 `dense_timesteps=0`、`dense_layers=0` 和负数 `tau` 会强制所有 KV block
+走 exact 路径。Wan 优化示例将其暴露为 `--attention fp8-dense`；
+`--attention fp8-sol` 使用相同的 FP8 Q/K/V 与 QK/PV kernel 并启用质心路由。
+关于 kernel 数据流、精度边界与 H100 消融结果，参见
+[FP8 Sol-Attn 技术文章](blog/fp8_sol_attention.md)。
+
+
 ### QwenImagePipeline / ZImagePipeline
 
 仅支持密集注意力（图像生成没有时序维度）：