updated KB

Author: adhikjoshi

knowledgebase/skills/skill-fp8-serving-eval.md

@@ -4,7 +4,7 @@ kind: skill
 title: FP8 Serving Evaluation
 type: ""
 category: precision
-summary: Validate mature FP8 deployment branches before investing in custom tensor-core kernel work on Hopper or Blackwell.
+summary: Validate packaged or synthesized FP8 deployment branches before investing in custom tensor-core kernel work on Hopper or Blackwell.
 support_level: stable
 reliability: ""
 review_status: ""
@@ -61,7 +61,9 @@ required_tools:
   - search_knowledge_base
 steps:
   - measure the higher-precision control path first
-  - enable the strongest runtime-supported FP8 branch for the same benchmark suite
+  - if the model already has a packaged FP8 checkpoint or runtime flavor, test that first
+  - if no packaged FP8 checkpoint exists, try synthesizing one with TensorRT Model Optimizer, Transformer Engine, torchao float8 flows, or llm-compressor using the same benchmark suite
+  - record calibration data, quantization configuration, and any higher-precision fallback modules
   - track quality drift and throughput together before any custom kernel work
   - only move beyond the FP8 path if the hot operator remains dominant after the runtime baseline
 verification:
@@ -73,6 +75,7 @@ benchmark_rubric:
   - store the exact runtime and calibration configuration with the result
 failure_recovery:
   - fallback to bf16 or fp16 for unstable layers
+  - fallback to weight-only or mixed-precision quantization if full FP8 activation paths regress quality or fail in the runtime
   - switch to AWQ or KV-cache optimization if the remaining bottleneck is mostly memory traffic
 artifacts_to_save:
   - benchmark_json
@@ -87,6 +90,9 @@ reference_source_ids:
   - nvidia-transformer-engine
   - tensorrt-llm-docs
   - nvidia-blackwell-cutlass
+  - nvidia-tensorrt-model-optimizer
+  - llm-compressor
+  - torchao
 backend: ""
 runtimes:
   - tensorrt-llm
@@ -102,7 +108,9 @@ path: ""
 ## Steps
 
 - measure the higher-precision control path first
-- enable the strongest runtime-supported FP8 branch for the same benchmark suite
+- if the model already has a packaged FP8 checkpoint or runtime flavor, test that first
+- if no packaged FP8 checkpoint exists, try synthesizing one with TensorRT Model Optimizer, Transformer Engine, torchao float8 flows, or llm-compressor using the same benchmark suite
+- record calibration data, quantization configuration, and any higher-precision fallback modules
 - track quality drift and throughput together before any custom kernel work
 - only move beyond the FP8 path if the hot operator remains dominant after the runtime baseline
 
@@ -120,4 +128,5 @@ path: ""
 ## Failure Recovery
 
 - fallback to bf16 or fp16 for unstable layers
+- fallback to weight-only or mixed-precision quantization if full FP8 activation paths regress quality or fail in the runtime
 - switch to AWQ or KV-cache optimization if the remaining bottleneck is mostly memory traffic

knowledgebase/sources/llm-compressor.md

@@ -0,0 +1,24 @@
+---
+id: llm-compressor
+kind: source
+title: LLM Compressor
+type: official-doc
+category: quantization
+summary: vLLM's LLM Compressor supports FP8, FP4, INT8, and INT4 quantization, calibration, and model export flows for deployment on vLLM-compatible runtimes.
+reliability: official
+review_status: reviewed
+url: https://docs.vllm.ai/usage/quantization/llm_compressor/
+tags:
+  - vllm
+  - llm-compressor
+  - quantization
+  - fp8
+  - fp4
+  - int4
+---
+
+## Key Features
+
+- One-shot quantization and calibration for vLLM-ready model artifacts
+- FP8, FP4, INT8, and INT4 support with runtime-oriented compression schemes
+- Useful when a model does not ship a packaged FP8 checkpoint but the target runtime can consume a converted artifact

knowledgebase/sources/nvidia-tensorrt-model-optimizer.md

@@ -0,0 +1,25 @@
+---
+id: nvidia-tensorrt-model-optimizer
+kind: source
+title: NVIDIA TensorRT Model Optimizer
+type: official-doc
+category: quantization
+summary: NVIDIA Model Optimizer supports post-training quantization and checkpoint conversion flows, including FP8 export paths for TensorRT-LLM and related serving stacks.
+reliability: official
+review_status: reviewed
+url: https://github.com/NVIDIA/TensorRT-Model-Optimizer
+tags:
+  - nvidia
+  - modelopt
+  - tensorrt-llm
+  - quantization
+  - fp8
+  - nvfp4
+---
+
+## Key Features
+
+- Offline quantization from bf16 or fp16 checkpoints when no packaged low-precision checkpoint exists
+- Calibration and export flows for TensorRT-LLM deployment
+- FP8, FP4, and other post-training quantization recipes on NVIDIA GPU stacks
+- Useful as a search branch before custom kernel work on Hopper or Blackwell

knowledgebase/strategies/fp8-transformer-engine.md

@@ -24,6 +24,10 @@ source_ids:
   - nvidia-h100
   - nvidia-blackwell-architecture
   - tensorrt-llm-docs
+  - nvidia-tensorrt-model-optimizer
+  - llm-compressor
+  - nvidia-transformer-engine
+  - torchao
 workloads:
   - prefill
   - decode
@@ -53,7 +57,9 @@ preconditions:
   - the model tolerates FP8 calibration and validation
   - runtime stack supports the chosen FP8 serving path
 actions:
-  - benchmark an FP8 path in TensorRT-LLM or an equivalent mature serving runtime first
+  - benchmark a packaged FP8 path in TensorRT-LLM, vLLM, SGLang, or an equivalent mature serving runtime first when one already exists
+  - if no packaged FP8 checkpoint exists, synthesize one with NVIDIA Model Optimizer, Transformer Engine, torchao float8 flows, or llm-compressor when the runtime and GPU support it
+  - run calibration, save the quantization recipe, and validate output quality before spending time on custom kernels
   - validate accuracy before investing in more custom kernel work
   - only hand-write kernels after the FP8 baseline is known
 metrics:
@@ -63,6 +69,7 @@ metrics:
 tradeoffs:
   - calibration quality matters
   - some model components may need fallback higher-precision paths
+  - FP8 conversion can improve throughput and memory use, but unsupported operators or unstable layers may force mixed-precision fallbacks
 preferred_backends: []
 required_tools: []
 steps: []
@@ -83,14 +90,17 @@ path: ""
 
 ## Actions
 
-- benchmark an FP8 path in TensorRT-LLM or an equivalent mature serving runtime first
+- benchmark a packaged FP8 path in TensorRT-LLM, vLLM, SGLang, or an equivalent mature serving runtime first when one already exists
+- if no packaged FP8 checkpoint exists, synthesize one with NVIDIA Model Optimizer, Transformer Engine, torchao float8 flows, or llm-compressor when the runtime and GPU support it
+- run calibration, save the quantization recipe, and validate output quality before spending time on custom kernels
 - validate accuracy before investing in more custom kernel work
 - only hand-write kernels after the FP8 baseline is known
 
 ## Tradeoffs
 
 - calibration quality matters
 - some model components may need fallback higher-precision paths
+- FP8 conversion can improve throughput and memory use, but unsupported operators or unstable layers may force mixed-precision fallbacks
 
 ## Metrics