feat: spark tts use TensorRT-LLM + tritonserver deploy on the modal

[weedge]

feat:

• spark tts use TensorRT-LLM + tritonserver deploy on the modal
  • spark tts python backend: Combine 「 audio_tokenizer -> tensorrt-llm engineer -> vocoder 」 services, use Qwen2.5 text tokenizer (BPE Byte-Pair Encoding)
  • audio_tokenizer python backend: use Wav2Vec2 extract features from audio, then BiCodec decode the audio features to tokens(semantic_tokens + global_tokens)
  • tensorrt-llm engineer TensorRT-LLM backend: use Qwen2.5 0.5B as backbone
    • TensorRT-LLM support model+arch: https://nvidia.github.io/TensorRT-LLM/reference/support-matrix.html
    • TensorRT-LLM support qwen: https://github.com/NVIDIA/TensorRT-LLM/blob/v0.17.0/examples/qwen/README.md
  • vocoder python backend: use BiCodec decode tokens (semantic_tokens + global_tokens) to audio

• add http client and grpc client support with tunnel tcp socket
• add http client sdk tts test
• add spark tts bench cli(http/grpc) with aio
• add bench hf dataset with (wenetspeech4tts 26 rows) concurrency min(num_tasks,26) task to bench used by http client

fix:

• use tensorrt-llm==0.17.0.post1 with tensorrt==10.8.0.43 to compile qwen2.5 0.5 LLM engine

Note

• nvcr.io/nvidia/tritonserver:25.02-trtllm-python-py3 use python3.12.3,
• modal python3.12 use python3.12.1, need building-custom-python-backend-stub:
• https://github.com/triton-inference-server/python_backend?#building-custom-python-backend-stub
• modal now support http api, so test/bench all use http api to run
• if want to support gRPC, use tunnel tcp socket to establish connect

---

infer test:

• first and second spark tts inference api cost (L4 GPU), need warmup LLM when inference serve is running (You may change model_warmup in the model configuration file to achieve this. )
  | container setup to run cost: 44.1 s | first infer cost: 8.52 s | second infer cost: 3.04 s | (generate duration 5s audio)

• the first inference cost statics:

  • rtf:
  
  • service module process cost statics:
  

• the second inference cost statics:

  • rtf:
  
  • service module process cost statics:
  

ps: other such as Perf Analyzer etc.. be continued

todo: streaming infer

---

bench data:

• use yuekai/seed_tts dataset wenetspeech4tts (26 rows)

infer bench (tritornserver deploy on the L4 GPU , use first Inference to warmup):

2 concurrency task do 26 tts | scaling to 2 containers

• rtf:

• service module process cost statics:

4 concurrency task do 26 tts | scaling to 4 containers

• rtf:

8 concurrency task do 26 tts | scaling to 8 containers

• rtf:

16 concurrency task do 26 tts | scaling to 10 containers

• rtf:

• service module process cost statics:

TTS result audio (wav):

---

模型结构见：#130

---

tritonserver (triton-inference-server) arch：
from https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/index.html

---

reference:

• ⭐️ https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/architecture.html
• Add Nvidia Triton Server for GPU Deployment SparkAudio/Spark-TTS#92
• https://github.com/triton-inference-server/server/blob/main/docs/user_guide/metrics.md
• Need help in understanding triton client "get_inference_statistics" output. triton-inference-server/server#5374
• https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md
• https://github.com/triton-inference-server/server/blob/main/docs/user_guide/performance_tuning.md
• https://github.com/triton-inference-server/server/blob/main/docs/user_guide/optimization.md
• https://github.com/triton-inference-server/perf_analyzer/blob/main/README.md

[gemini-code-assist]

Hello @weedge, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

Summary of Changes

This pull request focuses on integrating SparkTTS with TensorRT-LLM and deploying it using Triton server on Modal. It includes adding SparkTTS as a submodule, setting up the necessary environment and dependencies for TensorRT-LLM and Triton server within a Modal image, and creating functions to compile and serve the SparkTTS model using Triton server. The changes also involve replacing container_idle_timeout with scaledown_window in several Modal function definitions.

Highlights

• Submodule Integration: SparkTTS is added as a submodule to the repository.
• TensorRT-LLM and Triton Server Setup: A Modal image is configured with the necessary dependencies (CUDA, TensorRT-LLM, Triton server) to run SparkTTS.
• Model Compilation and Serving: Functions are implemented to compile the SparkTTS model using TensorRT-LLM and serve it via Triton server on Modal.
• Modal Configuration Updates: The container_idle_timeout parameter is replaced with scaledown_window in multiple Modal function definitions for better resource management.

Changelog

Click here to see the changelog

• .gitmodules
  • Added SparkTTS as a submodule.
• deploy/modal/src/bench/run_flash_mla.py
  • Replaced container_idle_timeout with scaledown_window in the @app.function decorator.
• deploy/modal/src/download_assets.py
  • Replaced container_idle_timeout with scaledown_window in the @app.function decorator.
• deploy/modal/src/download_models.py
  • Added hf-transfer to the pip installs for faster downloads.
  • Set environment variable HF_HUB_ENABLE_HF_TRANSFER to enable hf-transfer.
  • Renamed MODEL_DIR to HF_MODEL_DIR and model_dir to hf_model_vol.
  • Replaced container_idle_timeout with scaledown_window in the @app.function decorator.
  • Updated the model download path to use HF_MODEL_DIR.
  • Added example commands for running the download function.
• deploy/modal/src/fastapi_serve.py
  • Replaced container_idle_timeout with scaledown_window in the @app.cls decorator.
• deploy/modal/src/fastapi_webrtc_audio_bot_serve.py
  • Replaced container_idle_timeout with scaledown_window in the @app.cls decorator.
• deploy/modal/src/fastapi_webrtc_freeze_omni_voice_bot_serve.py
  • Replaced container_idle_timeout with scaledown_window in the @app.cls decorator.
• deploy/modal/src/fastapi_webrtc_glm_voice_bot_serve.py
  • Replaced container_idle_timeout with scaledown_window in the @app.cls decorator.
• deploy/modal/src/fastapi_webrtc_minicpmo_vision_voice_bot_serve.py
  • Replaced container_idle_timeout with scaledown_window in the @app.cls decorator.
• deploy/modal/src/fastapi_webrtc_step_voice_bot_serve.py
  • Replaced container_idle_timeout with scaledown_window in the @app.cls decorator.
• deploy/modal/src/fastapi_webrtc_vision_bot_serve.py
  • Replaced container_idle_timeout with scaledown_window in the @app.cls decorator.
• deploy/modal/src/fastapi_ws_moshi_voice_bot_serve.py
  • Replaced container_idle_timeout with scaledown_window in the @app.cls decorator.
• deploy/modal/src/llm/trtllm/bench/tritonserver.py
  • Created a new file to define a Modal app for serving models with Triton server.
  • Configured a Modal image with Triton server and TensorRT-LLM dependencies.
  • Implemented a serve function to start the Triton server with a specified model repository.
• deploy/modal/src/llm/trtllm/bench/tts_spark/client_grpc.py
  • Created a new file to define a Modal app for compiling and serving SparkTTS models using TensorRT-LLM and Triton.
  • Configured a Modal image with the necessary dependencies, including PyTorch, audio libraries, and Triton client.
  • Implemented a compile_model function to clone the SparkTTS repository and build the model.
  • Implemented a serve function to start the Triton server.
• deploy/modal/src/llm/trtllm/bench/tts_spark/client_http.py
  • Created a new file to define a Modal app for compiling and serving SparkTTS models using TensorRT-LLM and Triton.
  • Configured a Modal image with the necessary dependencies, including PyTorch, audio libraries, and Triton client.
  • Implemented a compile_model function to clone the SparkTTS repository and build the model.
  • Implemented a serve function to start the Triton server.
• deploy/modal/src/llm/trtllm/bench/tts_spark/ready_model.py
  • Created a new file to define a Modal app for preparing the SparkTTS model for Triton server.
  • Configured a Modal image with git.
  • Implemented a ready_model function to clone the SparkTTS repository, copy the model repository, and fill the template with pbtext file for api params.
• deploy/modal/src/llm/trtllm/compile_model.py
  • Created a new file to define a Modal app for compiling models using TensorRT-LLM.
  • Configured a Modal image with CUDA, TensorRT-LLM, and other dependencies.
  • Implemented a trtllm_build function to convert and compile the model.
• deploy/modal/src/llm/vllm/bench/client.py
  • Replaced container_idle_timeout with scaledown_window in the @app.function decorator.
• deploy/modal/src/llm/vllm/bench/download_dataset.py
  • Replaced container_idle_timeout with scaledown_window in the @app.function decorator.
• deploy/modal/src/llm/vllm/bench/serve.py
  • Replaced container_idle_timeout with scaledown_window in the @app.function decorator.
• deploy/modal/src/llm/vllm/examples/offline_inference/audio_language.py
  • Replaced container_idle_timeout with scaledown_window in the @app.function decorator.
• deploy/modal/src/llm/vllm/examples/offline_inference/disaggregated_prefill.py
  • Replaced container_idle_timeout with scaledown_window in the @app.function decorator.
• deploy/modal/src/llm/vllm/inference_test.py
  • Replaced container_idle_timeout with scaledown_window in the @app.function decorator.
• deploy/modal/src/llm/vllm/intra_node_inference_test.py
  • Replaced container_idle_timeout with scaledown_window in the @app.function decorator.
• deploy/modal/src/llm/vllm/profile_test.py
  • Replaced container_idle_timeout with scaledown_window in the @app.function decorator.
• deploy/modal/src/run_step_tts_inference.py
  • Replaced container_idle_timeout with scaledown_window in the @app.function decorator.
• deploy/modal/src/train/demo/cp.py
  • Replaced container_idle_timeout with scaledown_window in the @app.function decorator.
• deploy/modal/src/train/demo/ddp.py
  • Replaced container_idle_timeout with scaledown_window in the @app.function decorator.
• deploy/modal/src/train/demo/device_mesh_sp.py
  • Replaced container_idle_timeout with scaledown_window in the @app.function decorator.
• deploy/modal/src/train/demo/device_mesh_tp.py
  • Replaced container_idle_timeout with scaledown_window in the @app.function decorator.
• deploy/modal/src/train/demo/dp.py
  • Replaced container_idle_timeout with scaledown_window in the @app.function decorator.
• deploy/modal/src/train/demo/ep.py
  • Replaced container_idle_timeout with scaledown_window in the @app.function decorator.
• deploy/modal/src/train/demo/fsdp.py
  • Replaced container_idle_timeout with scaledown_window in the @app.function decorator.
• deploy/modal/src/train/demo/pp.py
  • Replaced container_idle_timeout with scaledown_window in the @app.function decorator.
• deploy/modal/src/train/demo/sp.py
  • Replaced container_idle_timeout with scaledown_window in the @app.function decorator.
• deploy/modal/src/train/demo/tp.py
  • Replaced container_idle_timeout with scaledown_window in the @app.function decorator.
• deploy/modal/src/train/fsdp_tp_llama2.py
  • Replaced container_idle_timeout with scaledown_window in the @app.function decorator.
• deps/SparkTTS
  • Added a file to track the SparkTTS submodule commit.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

---

From text to speech, a digital art,
With TensorRT's speed, a brand new start.
Triton serves fast,
While models are cast,
On Modal's cloud, playing a vital part.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces the deployment of Spark TTS using TensorRT-LLM and Triton server on Modal. It includes changes to submodule configurations, benchmarking scripts, download scripts, and server configurations. The changes aim to optimize and scale the TTS service. Overall, the changes seem well-structured, but there are a few areas that could benefit from closer examination and potential refinement.

Summary of Findings

• Missing error handling in serve functions: The serve functions in tritonserver.py and client_grpc.py/client_http.py lack proper error handling for subprocess calls. If any of the commands fail, the function should catch the exception and log an error message, otherwise the process will continue without the user knowing that the command failed.
• Hardcoded paths: Several files contain hardcoded paths, such as /Spark-TTS/runtime/triton_trtllm in client_grpc.py and client_http.py, and /Spark-TTS/runtime/triton_trtllm/scripts/convert_checkpoint.py in compile_model.py. These paths should be configurable via environment variables to improve flexibility and maintainability.
• Inconsistent use of scaledown_window: The pull request replaces container_idle_timeout with scaledown_window in several files. While this change is consistent across most files, it's important to ensure that this replacement is the correct configuration parameter for all use cases and that the values are appropriate.

Merge Readiness

The pull request introduces significant new functionality and touches several core components. While the changes appear to be well-structured, the identified issues regarding error handling and hardcoded paths should be addressed before merging. I am unable to approve this pull request, and recommend that the author address the comments before merging. It's also important to have other reviewers examine the changes to ensure they align with the project's overall architecture and goals.

[weedge]

tritonserver --help

Usage: tritonserver [options]
  --help
        Print usage

Server:
  --id <string>
        Identifier for this server.
  --exit-timeout-secs <integer>
        Timeout (in seconds) when exiting to wait for in-flight
        inferences to finish. After the timeout expires the server exits even if
        inferences are still in flight.

Logging:
  --log-verbose <integer>
        Set verbose logging level. Zero (0) disables verbose logging
        and values >= 1 enable verbose logging.
  --log-info <boolean>
        Enable/disable info-level logging.
  --log-warning <boolean>
        Enable/disable warning-level logging.
  --log-error <boolean>
        Enable/disable error-level logging.
  --log-format <string>
        Set the logging format. Options are "default" and "ISO8601".
        The default is "default". For "default", the log severity (L) and
        timestamp will be logged as "LMMDD hh:mm:ss.ssssss". For "ISO8601",
        the log format will be "YYYY-MM-DDThh:mm:ssZ L".
  --log-file <string>
        Set the name of the log output file. If specified, log
        outputs will be saved to this file. If not specified, log outputs will
        stream to the console.

Model Repository:
  --model-store <string>
        Equivalent to --model-repository.
  --model-repository <string>
        Path to model repository directory. It may be specified
        multiple times to add multiple model repositories. Note that if a model
        is not unique across all model repositories at any time, the model
        will not be available.
  --exit-on-error <boolean>
        Exit the inference server if an error occurs during
        initialization.
  --disable-auto-complete-config
        If set, disables the triton and backends from auto
        completing model configuration files. Model configuration files must be
        provided and all required configuration settings must be specified.
  --strict-readiness <boolean>
        If true /v2/health/ready endpoint indicates ready if the
        server is responsive and all models are available. If false
        /v2/health/ready endpoint indicates ready if server is responsive even if
        some/all models are unavailable.
  --model-control-mode <string>
        Specify the mode for model management. Options are "none",
        "poll" and "explicit". The default is "none". For "none", the server
        will load all models in the model repository(s) at startup and will
        not make any changes to the load models after that. For "poll", the
        server will poll the model repository(s) to detect changes and will
        load/unload models based on those changes. The poll rate is
        controlled by 'repository-poll-secs'. For "explicit", model load and unload
        is initiated by using the model control APIs, and only models
        specified with --load-model will be loaded at startup.
  --repository-poll-secs <integer>
        Interval in seconds between each poll of the model
        repository to check for changes. Valid only when --model-control-mode=poll is
        specified.
  --load-model <string>
        Name of the model to be loaded on server startup. It may be
        specified multiple times to add multiple models. To load ALL models
        at startup, specify '*' as the model name with --load-model=* as the
        ONLY --load-model argument, this does not imply any pattern
        matching. Specifying --load-model=* in conjunction with another
        --load-model argument will result in error. Note that this option will only
        take effect if --model-control-mode=explicit is true.
  --model-config-name <string>
        The custom configuration name for models to load.The name
        should not contain any space character.For example:
        --model-config-name=h100. If --model-config-name is not set, Triton will use the
        default config.pbtxt.
  --model-load-thread-count <integer>
        The number of threads used to concurrently load models in
        model repositories. Default is 4.
  --model-load-retry-count <integer>
        The number of retry to load a model in model repositories.
        Default is 0.
  --model-namespacing <boolean>
        Whether model namespacing is enable or not. If true, models
        with the same name can be served if they are in different namespace.
  --enable-peer-access <boolean>
        Whether the server tries to enable peer access or not. Even
        when this options is set to true,  peer access could still be not
        enabled because the underlying system doesn't support it. The server
        will log a warning in this case. Default is true.

HTTP:
  --allow-http <boolean>
        Allow the server to listen for HTTP requests.
  --http-address <string>
        The address for the http server to bind to. Default is
        0.0.0.0
  --http-port <integer>
        The port for the server to listen on for HTTP requests.
        Default is 8000.
  --reuse-http-port <boolean>
        Allow multiple servers to listen on the same HTTP port when
        every server has this option set. If you plan to use this option as
        a way to load balance between different Triton servers, the same
        model repository or set of models must be used for every server.
  --http-header-forward-pattern <string>
        The regular expression pattern that will be used for
        forwarding HTTP headers as inference request parameters.
  --http-thread-count <integer>
        Number of threads handling HTTP requests.
  --http-restricted-api <<string>:<string>=<string>>
        Specify restricted HTTP api setting. The format of this flag
        is --http-restricted-api=<apis>,<key>=<value>. Where <api> is a
        comma-separated list of apis to be restricted. <key> will be additional
        header key to be checked when a HTTP request is received, and
        <value> is the value expected to be matched. Allowed APIs: health,
        metadata, inference, shared-memory, model-config, model-repository,
        statistics, trace, logging

GRPC:
  --allow-grpc <boolean>
        Allow the server to listen for GRPC requests.
  --grpc-address <string>
        The address for the grpc server to binds to. Default is
        0.0.0.0
  --grpc-port <integer>
        The port for the server to listen on for GRPC requests.
        Default is 8001.
  --reuse-grpc-port <boolean>
        Allow multiple servers to listen on the same GRPC port when
        every server has this option set. If you plan to use this option as
        a way to load balance between different Triton servers, the same
        model repository or set of models must be used for every server.
  --grpc-header-forward-pattern <string>
        The regular expression pattern that will be used for
        forwarding GRPC headers as inference request parameters.
  --grpc-infer-allocation-pool-size <integer>
        The maximum number of inference request/response objects
        that remain allocated for reuse. As long as the number of in-flight
        requests doesn't exceed this value there will be no
        allocation/deallocation of request/response objects.
  --grpc-use-ssl <boolean>
        Use SSL authentication for GRPC requests. Default is false.
  --grpc-use-ssl-mutual <boolean>
        Use mututal SSL authentication for GRPC requests. This
        option will preempt '--grpc-use-ssl' if it is also specified. Default is
        false.
  --grpc-server-cert <string>
        File holding PEM-encoded server certificate. Ignored unless
        --grpc-use-ssl is true.
  --grpc-server-key <string>
        File holding PEM-encoded server key. Ignored unless
        --grpc-use-ssl is true.
  --grpc-root-cert <string>
        File holding PEM-encoded root certificate. Ignore unless
        --grpc-use-ssl is false.
  --grpc-infer-response-compression-level <string>
        The compression level to be used while returning the infer
        response to the peer. Allowed values are none, low, medium and high.
        By default, compression level is selected as none.
  --grpc-keepalive-time <integer>
        The period (in milliseconds) after which a keepalive ping is
        sent on the transport. Default is 7200000 (2 hours).
  --grpc-keepalive-timeout <integer>
        The period (in milliseconds) the sender of the keepalive
        ping waits for an acknowledgement. If it does not receive an
        acknowledgment within this time, it will close the connection. Default is
        20000 (20 seconds).
  --grpc-keepalive-permit-without-calls <boolean>
        Allows keepalive pings to be sent even if there are no calls
        in flight (0 : false; 1 : true). Default is 0 (false).
  --grpc-http2-max-pings-without-data <integer>
        The maximum number of pings that can be sent when there is
        no data/header frame to be sent. gRPC Core will not continue sending
        pings if we run over the limit. Setting it to 0 allows sending pings
        without such a restriction. Default is 2.
  --grpc-http2-min-recv-ping-interval-without-data <integer>
        If there are no data/header frames being sent on the
        transport, this channel argument on the server side controls the minimum
        time (in milliseconds) that gRPC Core would expect between receiving
        successive pings. If the time between successive pings is less than
        this time, then the ping will be considered a bad ping from the peer.
        Such a ping counts as a ‘ping strike’. Default is 300000 (5
        minutes).
  --grpc-http2-max-ping-strikes <integer>
        Maximum number of bad pings that the server will tolerate
        before sending an HTTP2 GOAWAY frame and closing the transport.
        Setting it to 0 allows the server to accept any number of bad pings.
        Default is 2.
  --grpc-max-connection-age <integer>
        Maximum time that a channel may exist in
        milliseconds.Default is undefined.
  --grpc-max-connection-age-grace <integer>
        Grace period after the channel reaches its max age. Default
        is undefined.
  --grpc-restricted-protocol <<string>:<string>=<string>>
        Specify restricted GRPC protocol setting. The format of this
        flag is --grpc-restricted-protocol=<protocols>,<key>=<value>. Where
        <protocol> is a comma-separated list of protocols to be restricted.
        <key> will be additional header key to be checked when a GRPC
        request is received, and <value> is the value expected to be matched.
        Allowed protocols: health, metadata, inference, shared-memory,
        model-config, model-repository, statistics, trace, logging

Sagemaker:
  --allow-sagemaker <boolean>
        Allow the server to listen for Sagemaker requests. Default
        is false.
  --sagemaker-port <integer>
        The port for the server to listen on for Sagemaker requests.
        Default is 8080.
  --sagemaker-safe-port-range <<integer>-<integer>>
        Set the allowed port range for endpoints other than the
        SageMaker endpoints.
  --sagemaker-thread-count <integer>
        Number of threads handling Sagemaker requests. Default is 8.

Vertex:
  --allow-vertex-ai <boolean>
        Allow the server to listen for Vertex AI requests. Default
        is true if AIP_MODE=PREDICTION, false otherwise.
  --vertex-ai-port <integer>
        The port for the server to listen on for Vertex AI requests.
        Default is AIP_HTTP_PORT if set, 8080 otherwise.
  --vertex-ai-thread-count <integer>
        Number of threads handling Vertex AI requests. Default is 8.
  --vertex-ai-default-model <string>
        The name of the model to use for single-model inference
        requests.

Metrics:
  --allow-metrics <boolean>
        Allow the server to provide prometheus metrics.
  --allow-gpu-metrics <boolean>
        Allow the server to provide GPU metrics. Ignored unless
        --allow-metrics is true.
  --allow-cpu-metrics <boolean>
        Allow the server to provide CPU metrics. Ignored unless
        --allow-metrics is true.
  --metrics-address <string>
        The address for the metrics server to bind to. Default is
        the same as --http-address if built with HTTP support. Otherwise,
        default is 0.0.0.0
  --metrics-port <integer>
        The port reporting prometheus metrics. Default is 8002.
  --metrics-interval-ms <float>
        Metrics will be collected once every <metrics-interval-ms>
        milliseconds. Default is 2000 milliseconds.
  --metrics-config <<string>=<string>>
        Specify a metrics-specific configuration setting. The format
        of this flag is --metrics-config=<setting>=<value>. It can be
        specified multiple times.

Tracing:
  --trace-config <<string>,<string>=<string>>
        Specify global or trace mode specific configuration setting.
        The format of this flag is --trace-config <mode>,<setting>=<value>.
        Where <mode> is either "triton" or "opentelemetry". The default is
        "triton". To specify global trace settings (level, rate, count, or
        mode), the format would be --trace-config <setting>=<value>. For
        "triton" mode, the server will use Triton's Trace APIs. For
        "opentelemetry" mode, the server will use OpenTelemetry's APIs to generate,
        collect and export traces for individual inference requests.

Backend:
  --backend-directory <string>
        The global directory searched for backend shared libraries.
        Default is '/opt/tritonserver/backends'.
  --backend-config <<string>,<string>=<string>>
        Specify a backend-specific configuration setting. The format
        of this flag is --backend-config=<backend_name>,<setting>=<value>.
        Where <backend_name> is the name of the backend, such as 'tensorrt'.

Repository Agent:
  --repoagent-directory <string>
        The global directory searched for repository agent shared
        libraries. Default is '/opt/tritonserver/repoagents'.

Response Cache:
  --cache-config <<string>,<string>=<string>>
        Specify a cache-specific configuration setting. The format
        of this flag is --cache-config=<cache_name>,<setting>=<value>. Where
        <cache_name> is the name of the cache, such as 'local' or 'redis'.
        Example: --cache-config=local,size=1048576 will configure a 'local'
        cache implementation with a fixed buffer pool of size 1048576 bytes.
  --cache-directory <string>
        The global directory searched for cache shared libraries.
        Default is '/opt/tritonserver/caches'. This directory is expected to
        contain a cache implementation as a shared library with the name
        'libtritoncache.so'.

Rate Limiter:
  --rate-limit <string>
        Specify the mode for rate limiting. Options are
        "execution_count" and "off". The default is "off". For "execution_count", the
        server will determine the instance using configured priority and the
        number of time the instance has been used to run inference. The
        inference will finally be executed once the required resources are
        available. For "off", the server will ignore any rate limiter config and
        run inference as soon as an instance is ready.
  --rate-limit-resource <<string>:<integer>:<integer>>
        The number of resources available to the server. The format
        of this flag is
        --rate-limit-resource=<resource_name>:<count>:<device>. The <device> is optional and if not listed will be applied to
        every device. If the resource is specified as "GLOBAL" in the model
        configuration the resource is considered shared among all the devices
        in the system. The <device> property is ignored for such resources.
        This flag can be specified multiple times to specify each resources
        and their availability. By default, the max across all instances
        that list the resource is selected as its availability. The values for
        this flag is case-insensitive.

Memory/Device Management:
  --pinned-memory-pool-byte-size <integer>
        The total byte size that can be allocated as pinned system
        memory. If GPU support is enabled, the server will allocate pinned
        system memory to accelerate data transfer between host and devices
        until it exceeds the specified byte size. If 'numa-node' is configured
        via --host-policy, the pinned system memory of the pool size will be
        allocated on each numa node. This option will not affect the
        allocation conducted by the backend frameworks. Default is 256 MB.
  --cuda-memory-pool-byte-size <<integer>:<integer>>
        The total byte size that can be allocated as CUDA memory for
        the GPU device. If GPU support is enabled, the server will allocate
        CUDA memory to minimize data transfer between host and devices
        until it exceeds the specified byte size. This option will not affect
        the allocation conducted by the backend frameworks. The argument
        should be 2 integers separated by colons in the format <GPU device
        ID>:<pool byte size>. This option can be used multiple times, but only
        once per GPU device. Subsequent uses will overwrite previous uses for
        the same GPU device. Default is 64 MB.
  --cuda-virtual-address-size <<integer>:<integer>>
        The total CUDA virtual address size that will be used for
        each implicit state when growable memory is used. This value
        determines the maximum size of each implicit state. The state size cannot go
        beyond this value. The argument should be 2 integers separated by
        colons in the format <GPU device ID>:<CUDA virtual address size>. This
        option can be used multiple times, but only once per GPU device.
        Subsequent uses will overwrite previous uses for the same GPU device.
        Default is 1 GB.
  --min-supported-compute-capability <float>
        The minimum supported CUDA compute capability. GPUs that
        don't support this compute capability will not be used by the server.
  --buffer-manager-thread-count <integer>
        The number of threads used to accelerate copies and other
        operations required to manage input and output tensor contents.
        Default is 0.
  --host-policy <<string>,<string>=<string>>
        Specify a host policy setting associated with a policy name.
        The format of this flag is
        --host-policy=<policy_name>,<setting>=<value>. Currently supported settings are 'numa-node', 'cpu-cores'.
        Note that 'numa-node' setting will affect pinned memory pool behavior,
        see --pinned-memory-pool for more detail.
  --model-load-gpu-limit <<device_id>:<fraction>>
        Specify the limit on GPU memory usage as a fraction. If
        model loading on the device is requested and the current memory usage
        exceeds the limit, the load will be rejected. If not specified, the
        limit will not be set.

DEPRECATED:
  --strict-model-config <boolean>
        DEPRECATED: If true model configuration files must be
        provided and all required configuration settings must be specified. If
        false the model configuration may be absent or only partially specified
        and the server will attempt to derive the missing required
        configuration.
  --response-cache-byte-size <integer>
        DEPRECATED: Please use --cache-config instead.
  --trace-file <string>
        DEPRECATED: Please use --trace-config
        triton,file=<path/to/your/file> Set the file where trace output will be saved. If
        --trace-log-frequency is also specified, this argument value will be the
        prefix of the files to save the trace output. See --trace-log-frequency
        for detail.
  --trace-level <string>
        DEPRECATED: Please use --trace-config
        level=<OFF|TIMESTAMPS|TENSORS>Specify a trace level. OFF to disable tracing, TIMESTAMPS to
        trace timestamps, TENSORS to trace tensors. It may be specified
        multiple times to trace multiple information. Default is OFF.
  --trace-rate <integer>
        DEPRECATED: Please use --trace-config rate=<rate value>Set
        the trace sampling rate. Default is 1000.
  --trace-count <integer>
        DEPRECATED: Please use --trace-config count=<count value>Set
        the number of traces to be sampled. If the value is -1, the number
        of traces to be sampled will not be limited. Default is -1.
  --trace-log-frequency <integer>
        DEPRECATED: Please use --trace-config
        triton,log-frequency=<value>Set the trace log frequency. If the value is 0, Triton will
        only log the trace output to <trace-file> when shutting down.
        Otherwise, Triton will log the trace output to <trace-file>.<idx> when it
        collects the specified number of traces. For example, if the log
        frequency is 100, when Triton collects the 100-th trace, it logs the
        traces to file <trace-file>.0, and when it collects the 200-th trace, it
        logs the 101-th to the 200-th traces to file <trace-file>.1.
        Default is 0.

[weedge]

trtllm-build

• https://nvidia.github.io/TensorRT-LLM/commands/trtllm-build.html
• https://nvidia.github.io/TensorRT-LLM/commands/trtllm-serve.html (Running an OpenAI API compatible server)

usage: trtllm-build [-h] [--checkpoint_dir CHECKPOINT_DIR]
                    [--model_config MODEL_CONFIG]
                    [--build_config BUILD_CONFIG]
                    [--model_cls_file MODEL_CLS_FILE]
                    [--model_cls_name MODEL_CLS_NAME]
                    [--output_dir OUTPUT_DIR]
                    [--max_batch_size MAX_BATCH_SIZE]
                    [--max_input_len MAX_INPUT_LEN]
                    [--max_seq_len MAX_SEQ_LEN]
                    [--max_beam_width MAX_BEAM_WIDTH]
                    [--max_num_tokens MAX_NUM_TOKENS]
                    [--opt_num_tokens OPT_NUM_TOKENS]
                    [--max_encoder_input_len MAX_ENCODER_INPUT_LEN]
                    [--max_prompt_embedding_table_size MAX_PROMPT_EMBEDDING_TABLE_SIZE]
                    [--kv_cache_type KV_CACHE_TYPE]
                    [--paged_kv_cache PAGED_KV_CACHE]
                    [--input_timing_cache INPUT_TIMING_CACHE]
                    [--output_timing_cache OUTPUT_TIMING_CACHE]
                    [--profiling_verbosity {layer_names_only,detailed,none}]
                    [--strip_plan] [--weight_sparsity] [--weight_streaming]
                    [--fast_build] [--workers WORKERS]
                    [--log_level {internal_error,error,warning,info,verbose,debug}]
                    [--enable_debug_output]
                    [--visualize_network VISUALIZE_NETWORK] [--dry_run]
                    [--monitor_memory] [--logits_dtype {float16,float32}]
                    [--gather_context_logits] [--gather_generation_logits]
                    [--gather_all_token_logits]
                    [--lora_dir LORA_DIR [LORA_DIR ...]]
                    [--lora_ckpt_source {hf,nemo}]
                    [--lora_target_modules {attn_qkv,attn_q,attn_k,attn_v,attn_dense,mlp_h_to_4h,mlp_4h_to_h,mlp_gate,cross_attn_qkv,cross_attn_q,cross_attn_k,cross_attn_v,cross_attn_dense,moe_h_to_4h,moe_4h_to_h,moe_gate,moe_router,mlp_router} [{attn_qkv,attn_q,attn_k,attn_v,attn_dense,mlp_h_to_4h,mlp_4h_to_h,mlp_gate,cross_attn_qkv,cross_attn_q,cross_attn_k,cross_attn_v,cross_attn_dense,moe_h_to_4h,moe_4h_to_h,moe_gate,moe_router,mlp_router} ...]]
                    [--max_lora_rank MAX_LORA_RANK]
                    [--speculative_decoding_mode {draft_tokens_external,lookahead_decoding,medusa,explicit_draft_tokens,eagle}]
                    [--max_draft_len MAX_DRAFT_LEN]
                    [--auto_parallel AUTO_PARALLEL]
                    [--gpus_per_node GPUS_PER_NODE]
                    [--cluster_key {A100-SXM-80GB,A100-SXM-40GB,A100-PCIe-80GB,A100-PCIe-40GB,H100-SXM,H100-PCIe,H20,H200-SXM,H200-NVL,V100-PCIe-16GB,V100-PCIe-32GB,V100-SXM-16GB,V100-SXM-32GB,V100S-PCIe,A40,A30,A10,A10G,L40S,L40,L20,L4,L2}]
                    [--bert_attention_plugin {auto,float16,float32,bfloat16,int32,disable}]
                    [--gpt_attention_plugin {auto,float16,float32,bfloat16,int32,disable}]
                    [--gemm_plugin {auto,float16,float32,bfloat16,int32,fp8,nvfp4,disable}]
                    [--gemm_swiglu_plugin {fp8,disable}]
                    [--fp8_rowwise_gemm_plugin {auto,float16,float32,bfloat16,int32,disable}]
                    [--nccl_plugin {auto,float16,float32,bfloat16,int32,disable}]
                    [--lora_plugin {auto,float16,float32,bfloat16,int32,disable}]
                    [--moe_plugin {auto,float16,float32,bfloat16,int32,disable}]
                    [--mamba_conv1d_plugin {auto,float16,float32,bfloat16,int32,disable}]
                    [--low_latency_gemm_plugin {fp8,disable}]
                    [--low_latency_gemm_swiglu_plugin {fp8,disable}]
                    [--context_fmha {enable,disable}]
                    [--bert_context_fmha_fp32_acc {enable,disable}]
                    [--remove_input_padding {enable,disable}]
                    [--reduce_fusion {enable,disable}]
                    [--user_buffer {enable,disable}]
                    [--tokens_per_block TOKENS_PER_BLOCK]
                    [--use_paged_context_fmha {enable,disable}]
                    [--use_fp8_context_fmha {enable,disable}]
                    [--multiple_profiles {enable,disable}]
                    [--paged_state {enable,disable}]
                    [--streamingllm {enable,disable}]
                    [--use_fused_mlp {enable,disable}]
                    [--pp_reduce_scatter {enable,disable}]

options:
  -h, --help            show this help message and exit
  --checkpoint_dir CHECKPOINT_DIR
                        The directory path that contains TensorRT-LLM
                        checkpoint. (default: None)
  --model_config MODEL_CONFIG
                        The file path that saves TensorRT-LLM checkpoint
                        config. (default: None)
  --build_config BUILD_CONFIG
                        The file path that saves TensorRT-LLM build config.
                        (default: None)
  --model_cls_file MODEL_CLS_FILE
                        The file path that defines customized TensorRT-LLM
                        model. (default: None)
  --model_cls_name MODEL_CLS_NAME
                        The customized TensorRT-LLM model class name.
                        (default: None)
  --output_dir OUTPUT_DIR
                        The directory path to save the serialized engine files
                        and engine config file. (default: engine_outputs)
  --max_batch_size MAX_BATCH_SIZE
                        Maximum number of requests that the engine can
                        schedule. (default: 2048)
  --max_input_len MAX_INPUT_LEN
                        Maximum input length of one request. (default: 1024)
  --max_seq_len MAX_SEQ_LEN, --max_decoder_seq_len MAX_SEQ_LEN
                        Maximum total length of one request, including prompt
                        and outputs. If unspecified, the value is deduced from
                        the model config. (default: None)
  --max_beam_width MAX_BEAM_WIDTH
                        Maximum number of beams for beam search decoding.
                        (default: 1)
  --max_num_tokens MAX_NUM_TOKENS
                        Maximum number of batched input tokens after padding
                        is removed in each batch. Currently, the input padding
                        is removed by default; you may explicitly disable it
                        by specifying ``--remove_input_padding disable``.
                        (default: 8192)
  --opt_num_tokens OPT_NUM_TOKENS
                        Optimal number of batched input tokens after padding
                        is removed in each batch It equals to ``max_batch_size
                        * max_beam_width`` by default, set this value as close
                        as possible to the actual number of tokens on your
                        workload. Note that this argument might be removed in
                        the future. (default: None)
  --max_encoder_input_len MAX_ENCODER_INPUT_LEN
                        Maximum encoder input length for enc-dec models. Set
                        ``max_input_len`` to 1 to start generation from
                        decoder_start_token_id of length 1. (default: 1024)
  --max_prompt_embedding_table_size MAX_PROMPT_EMBEDDING_TABLE_SIZE, --max_multimodal_len MAX_PROMPT_EMBEDDING_TABLE_SIZE
                        Maximum prompt embedding table size for prompt tuning,
                        or maximum multimodal input size for multimodal
                        models. Setting a value > 0 enables prompt tuning or
                        multimodal input. (default: 0)
  --kv_cache_type KV_CACHE_TYPE
                        Set KV cache type (continuous, paged, or disabled).
                        For disabled case, KV cache is disabled and only
                        context phase is allowed.
  --paged_kv_cache PAGED_KV_CACHE
                        Deprecated. Enabling this option is equvilient to
                        ``--kv_cache_type paged`` for transformer based
                        models.
  --input_timing_cache INPUT_TIMING_CACHE
                        The file path to read the timing cache. This option is
                        ignored if the file does not exist. (default: None)
  --output_timing_cache OUTPUT_TIMING_CACHE
                        The file path to write the timing cache. (default:
                        model.cache)
  --profiling_verbosity {layer_names_only,detailed,none}
                        The profiling verbosity for the generated TensorRT
                        engine. Setting to detailed allows inspecting tactic
                        choices and kernel parameters. (default:
                        layer_names_only)
  --strip_plan          Enable stripping weights from the final TensorRT
                        engine under the assumption that the refit weights are
                        identical to those provided at build time. (default:
                        False)
  --weight_sparsity     Enable weight sparsity. (default: False)
  --weight_streaming    Enable offloading weights to CPU and streaming loading
                        at runtime. (default: False)
  --fast_build          Enable features for faster engine building. This may
                        cause some performance degradation and is currently
                        incompatible with int8/int4 quantization without
                        plugin. (default: False)
  --workers WORKERS     The number of workers for building in parallel.
                        (default: 1)
  --log_level {internal_error,error,warning,info,verbose,debug}
                        The logging level. (default: info)
  --enable_debug_output
                        Enable debug output. (default: False)
  --visualize_network VISUALIZE_NETWORK
                        The directory path to export TensorRT Network as ONNX
                        prior to Engine build for debugging. (default: None)
  --dry_run             Run through the build process except the actual Engine
                        build for debugging. (default: False)
  --monitor_memory      Enable memory monitor during Engine build. (default:
                        False)

Logits arguments:
  --logits_dtype {float16,float32}
                        The data type of logits. (default: None)
  --gather_context_logits
                        Enable gathering context logits. (default: False)
  --gather_generation_logits
                        Enable gathering generation logits. (default: False)
  --gather_all_token_logits
                        Enable both ``gather_context_logits`` and
                        ``gather_generation_logits``. (default: False)

LoRA arguments:
  --lora_dir LORA_DIR [LORA_DIR ...]
                        The directory of LoRA weights. If multiple directories
                        are provided, the first one is used for configuration.
                        (default: None)
  --lora_ckpt_source {hf,nemo}
                        The source type of LoRA checkpoint. (default: hf)
  --lora_target_modules {attn_qkv,attn_q,attn_k,attn_v,attn_dense,mlp_h_to_4h,mlp_4h_to_h,mlp_gate,cross_attn_qkv,cross_attn_q,cross_attn_k,cross_attn_v,cross_attn_dense,moe_h_to_4h,moe_4h_to_h,moe_gate,moe_router,mlp_router} [{attn_qkv,attn_q,attn_k,attn_v,attn_dense,mlp_h_to_4h,mlp_4h_to_h,mlp_gate,cross_attn_qkv,cross_attn_q,cross_attn_k,cross_attn_v,cross_attn_dense,moe_h_to_4h,moe_4h_to_h,moe_gate,moe_router,mlp_router} ...]
                        The target module names that LoRA is applied. Only
                        effective when ``lora_plugin`` is enabled. (default:
                        None)
  --max_lora_rank MAX_LORA_RANK
                        Maximum LoRA rank for different LoRA modules. It is
                        used to compute the workspace size of LoRA plugin.
                        (default: 64)

Speculative decoding arguments:
  --speculative_decoding_mode {draft_tokens_external,lookahead_decoding,medusa,explicit_draft_tokens,eagle}
                        Mode of speculative decoding. (default: None)
  --max_draft_len MAX_DRAFT_LEN
                        Maximum lengths of draft tokens for speculative
                        decoding target model. (default: 0)

Auto parallel arguments:
  --auto_parallel AUTO_PARALLEL
                        MPI world size for auto parallel. (default: 1)
  --gpus_per_node GPUS_PER_NODE
                        Number of GPUs each node has in a multi-node setup.
                        This is a cluster spec and can be greater/smaller than
                        world size. This option is only used for auto parallel
                        specified with ``--auto_parallel``. (default: 8)
  --cluster_key {A100-SXM-80GB,A100-SXM-40GB,A100-PCIe-80GB,A100-PCIe-40GB,H100-SXM,H100-PCIe,H20,H200-SXM,H200-NVL,V100-PCIe-16GB,V100-PCIe-32GB,V100-SXM-16GB,V100-SXM-32GB,V100S-PCIe,A40,A30,A10,A10G,L40S,L40,L20,L4,L2}
                        Unique name for target GPU type. Inferred from current
                        GPU type if not specified. This option is only used
                        for auto parallel specified with ``--auto_parallel``.
                        (default: None)

Plugin config arguments:
  --bert_attention_plugin {auto,float16,float32,bfloat16,int32,disable}
                        The plugin that uses efficient kernels and enables an
                        in-place update of the KV cache for attention layer of
                        BERT-like encoder models. (default: auto)
  --gpt_attention_plugin {auto,float16,float32,bfloat16,int32,disable}
                        The plugin that uses efficient kernels and enables an
                        in-place update of the KV cache for attention layer of
                        GPT-like decoder models. (default: auto)
  --gemm_plugin {auto,float16,float32,bfloat16,int32,fp8,nvfp4,disable}
                        The GEMM plugin that utilizes NVIDIA cuBLASLt to
                        perform GEMM operations. Note: it's only affective for
                        non-quantized gemm operations (except FP8).Note: For
                        FP8, it also requires same calibration in checkpoint.
                        (default: None)
  --gemm_swiglu_plugin {fp8,disable}
                        The GEMM + SwiGLU fusion in Gated-MLP combines two
                        Matmul operations and one SwiGLU operation into a
                        single kernel. Currently this is only supported for
                        FP8 precision on Hopper. (default: disable)
  --fp8_rowwise_gemm_plugin {auto,float16,float32,bfloat16,int32,disable}
                        The quantized GEMM for fp8, which uses per token
                        dynamic scales for activation and per channel static
                        scales for weights.Note: It also requires same
                        calibration in checkpoint. (default: disable)
  --nccl_plugin {auto,float16,float32,bfloat16,int32,disable}
                        The NCCL plugin wraps NCCL operators to support multi-
                        GPU and even multi-nodes. (default: auto)
  --lora_plugin {auto,float16,float32,bfloat16,int32,disable}
                        Enable LoRA. (default: disable)
  --moe_plugin {auto,float16,float32,bfloat16,int32,disable}
                        Enable some customized kernels to speed up the MoE
                        layer of MoE models. (default: auto)
  --mamba_conv1d_plugin {auto,float16,float32,bfloat16,int32,disable}
                        Enable customized kernels to speed up conv1d operator
                        for Mamba. (default: auto)
  --low_latency_gemm_plugin {fp8,disable}
                        The GEMM plugin that optimized specially for low
                        latency scenarios. (default: disable)
  --low_latency_gemm_swiglu_plugin {fp8,disable}
                        The GEMM + SwiGLU fusion plugin that optimized
                        specially for low latency scenarios. (default:
                        disable)
  --context_fmha {enable,disable}
                        Enable the fused multi-head attention during the
                        context phase, will trigger a kernel that performs the
                        MHA/MQA/GQA block using a single kernel. (default:
                        enable)
  --bert_context_fmha_fp32_acc {enable,disable}
                        Enable the FP32 accumulator for context FMHA in the
                        bert_attention_plugin. If disabled, FP16 is used,
                        better performance but potentially worse accuracy is
                        expected. (default: disable)
  --remove_input_padding {enable,disable}
                        Pack different tokens together, which reduces both the
                        amount of computations and memory consumption.
                        (default: enable)
  --reduce_fusion {enable,disable}
                        Fuse the ResidualAdd and LayerNorm kernels after
                        AllReduce into a single kernel, resulting in improved
                        end-to-end performance. (default: disable)
  --user_buffer {enable,disable}
                        Eliminate extra copies from the local buffer to the
                        shared buffer in the communication kernel, leading to
                        improved end-to-end performance. This feature must be
                        enabled with `--reduce_fusion enable` and is currently
                        only supported for the FP8 LLAMA model. (default:
                        disable)
  --tokens_per_block TOKENS_PER_BLOCK
                        Define how many tokens are contained in each paged kv
                        cache block. (default: 64)
  --use_paged_context_fmha {enable,disable}
                        Allow advanced features like KV cache reuse and
                        chunked context. (default: disable)
  --use_fp8_context_fmha {enable,disable}
                        When FP8 quantization is activated, the attention can
                        be further accelerated by enabling FP8 Context FMHA
                        (default: disable)
  --multiple_profiles {enable,disable}
                        Enables multiple TensorRT optimization profiles in the
                        built engines, will benefits the performance
                        especially when GEMM plugin is disabled, because more
                        optimization profiles help TensorRT have more chances
                        to select better kernels. Note: This feature increases
                        engine build time but no other adverse effects are
                        expected. (default: disable)
  --paged_state {enable,disable}
                        Enable paged state, which helps manage memory for the
                        RNN state more efficiently. (default: enable)
  --streamingllm {enable,disable}
                        Enable
                        [StreamingLLM](https://arxiv.org/abs/2309.17453),
                        which uses a window attention to perform efficient and
                        stable LLM on long texts. (default: disable)
  --use_fused_mlp {enable,disable}
                        Enable horizontal fusion in Gated-MLP that combines
                        two Matmul operations into a single one followed by a
                        separate SwiGLU kernel. (default: enable)
  --pp_reduce_scatter {enable,disable}
                        Enable a pipeline parallelism optimization with
                        ReduceScatter + AllGather targeting large MoE models.
                        (default: disable)

[weedge]

https://github.com/NVIDIA/TensorRT-LLM/blob/v0.17.0/examples/qwen/convert_checkpoint.py

usage: convert.py [-h] --model_dir MODEL_DIR [--tp_size TP_SIZE]
                  [--pp_size PP_SIZE]
                  [--dtype {auto,float16,bfloat16,float32}]
                  [--use_weight_only] [--disable_weight_only_quant_plugin]
                  [--weight_only_precision [{int8,int4,int4_gptq}]]
                  [--calib_dataset CALIB_DATASET] [--smoothquant SMOOTHQUANT]
                  [--per_channel] [--per_token] [--int8_kv_cache]
                  [--per_group] [--group_size GROUP_SIZE]
                  [--load_model_on_cpu] [--use_parallel_embedding]
                  [--embedding_sharding_dim {0,1}] [--output_dir OUTPUT_DIR]
                  [--workers WORKERS] [--moe_tp_size MOE_TP_SIZE]
                  [--moe_ep_size MOE_EP_SIZE]

options:
  -h, --help            show this help message and exit
  --model_dir MODEL_DIR
  --tp_size TP_SIZE     N-way tensor parallelism size
  --pp_size PP_SIZE     N-way pipeline parallelism size
  --dtype {auto,float16,bfloat16,float32}
                        The data type for the model weights and activations if
                        not quantized. If 'auto', the data type is
                        automatically inferred from the source model; however,
                        if the source dtype is float32, it is converted to
                        float16.
  --use_weight_only     Quantize weights for the various GEMMs to
                        INT4/INT8.See --weight_only_precision to set the
                        precision
  --disable_weight_only_quant_plugin
                        By default, using plugin implementation for weight
                        quantization. Enabling
                        disable_weight_only_quant_plugin flag will use ootb
                        implementation instead of plugin.You must also use
                        --use_weight_only for that argument to have an impact.
  --weight_only_precision [{int8,int4,int4_gptq}]
                        Define the precision for the weights when using
                        weight-only quantization.You must also use
                        --use_weight_only for that argument to have an impact.
  --calib_dataset CALIB_DATASET
                        The huggingface dataset name or the local directory of
                        the dataset for calibration.
  --smoothquant SMOOTHQUANT, -sq SMOOTHQUANT
                        Set the α parameter (see
                        https://arxiv.org/pdf/2211.10438.pdf) to Smoothquant
                        the model, and output int8 weights. A good first try
                        is 0.5. Must be in [0, 1]
  --per_channel         By default, we use a single static scaling factor for
                        the GEMM's result. per_channel instead uses a
                        different static scaling factor for each channel. The
                        latter is usually more accurate, but a little slower.
  --per_token           By default, we use a single static scaling factor to
                        scale activations in the int8 range. per_token chooses
                        at run time, and for each token, a custom scaling
                        factor. The latter is usually more accurate, but a
                        little slower.
  --int8_kv_cache       By default, we use dtype for KV cache. int8_kv_cache
                        chooses int8 quantization for KV
  --per_group           By default, we use a single static scaling factor to
                        scale weights in the int4 range. per_group chooses at
                        run time, and for each group, a custom scaling factor.
                        The flag is built for GPTQ/AWQ quantization.
  --group_size GROUP_SIZE
                        Group size used in GPTQ quantization.
  --load_model_on_cpu
  --use_parallel_embedding
                        By default embedding parallelism is disabled. By
                        setting this flag, embedding parallelism is enabled
  --embedding_sharding_dim {0,1}
                        By default the embedding lookup table is sharded along
                        vocab dimension (embedding_sharding_dim=0). To shard
                        it along hidden dimension, set
                        embedding_sharding_dim=1Note: embedding sharing is
                        only enabled when embedding_sharding_dim = 0
  --output_dir OUTPUT_DIR
                        The path to save the TensorRT-LLM checkpoint
  --workers WORKERS     The number of workers for converting checkpoint in
                        parallel
  --moe_tp_size MOE_TP_SIZE
                        N-way tensor parallelism size for MOE, default is
                        tp_size, which will do tp-only for MoE
  --moe_ep_size MOE_EP_SIZE
                        N-way expert parallelism size for MOE, default is 1,
                        which will do tp-only for MoE

[weedge]

# https://nvidia.github.io/TensorRT-LLM/index.html (nice doc)
"""
NeMo -------------
                  |
HuggingFace ------
                  |   convert                             build                    load
Modelopt ---------  ----------> TensorRT-LLM Checkpoint --------> TensorRT Engine ------> TensorRT-LLM ModelRunner
                  |
JAX --------------
                  |
DeepSpeed --------
"""
# https://nvidia.github.io/TensorRT-LLM/performance/perf-benchmarking.html

1. download model hf SparkAudio/Spark-TTS-0.5B ckpt (Bicodec(audio tokenizer) + Wav2Vec2(features extractor) + Qwen2.5 0.5B LLM)

modal run src/download_models.py --repo-ids "SparkAudio/Spark-TTS-0.5B"

2. convert hf SparkAudio/Spark-TTS-0.5B LLM(Qwen2.5 0.5B) to TensorRT-LLM Checkpoint with defined quantization and model parallel mapping, then static build TensorRT Engine with build compile params

# covert to dtype bfloat16
modal run src/llm/trtllm/tts_spark/compile_model.py \
    --app-name "tts-spark" \
    --hf-repo-dir "SparkAudio/Spark-TTS-0.5B/LLM" \
    --trt-dtype "bfloat16" \
    --convert-other-args "" \
    --compile-other-args "--max_batch_size 16 --max_num_tokens 32768"

# use convert from convert-script-url,
modal run src/llm/trtllm/tts_spark/compile_model.py \
    --app-name "tts-spark" \
    --hf-repo-dir "SparkAudio/Spark-TTS-0.5B/LLM" \
    --trt-dtype "bfloat16" \
    --convert-script-url "https://raw.githubusercontent.com/SparkAudio/Spark-TTS/refs/heads/main/runtime/triton_trtllm/scripts/convert_checkpoint.py" \
    --convert-other-args "" \
    --compile-other-args "--max_batch_size 16 --max_num_tokens 32768"

see trtllm-build build TensorRT Engine config(include convert to TensorRT-LLM Checkpoint config)

{
    "version": "0.17.0.post1",
    "pretrained_config": {
        "mlp_bias": false,
        "attn_bias": true,
        "rotary_base": 1000000,
        "rotary_scaling": null,
        "disable_weight_only_quant_plugin": false,
        "num_labels": 1,
        "use_logn_attn": false,
        "moe": {
            "num_experts": 0,
            "shared_expert_intermediate_size": 0,
            "top_k": 0,
            "normalization_mode": 0,
            "sparse_mixer_epsilon": 0.01,
            "tp_mode": 0,
            "device_limited_n_group": 0,
            "device_limited_topk_group": 0,
            "device_limited_routed_scaling_factor": 1
        },
        "architecture": "Qwen2ForCausalLM",
        "dtype": "bfloat16",
        "vocab_size": 166000,
        "hidden_size": 896,
        "num_hidden_layers": 24,
        "num_attention_heads": 14,
        "hidden_act": "silu",
        "logits_dtype": "float32",
        "norm_epsilon": 0.000001,
        "runtime_defaults": null,
        "position_embedding_type": "rope_gpt_neox",
        "num_key_value_heads": 2,
        "intermediate_size": 4864,
        "max_position_embeddings": 32768,
        "mapping": {
            "world_size": 1,
            "gpus_per_node": 8,
            "cp_size": 1,
            "tp_size": 1,
            "pp_size": 1,
            "moe_tp_size": 1,
            "moe_ep_size": 1,
            "auto_parallel": false
        },
        "quantization": {
            "quant_algo": null,
            "kv_cache_quant_algo": null,
            "group_size": 128,
            "smoothquant_val": 0.5,
            "clamp_val": null,
            "use_meta_recipe": false,
            "has_zero_point": false,
            "pre_quant_scale": false,
            "exclude_modules": null
        },
        "use_parallel_embedding": false,
        "embedding_sharding_dim": 0,
        "head_size": 64,
        "qk_layernorm": false,
        "rotary_embedding_dim": 64,
        "seq_length": 8192,
        "qwen_type": "qwen2",
        "moe_intermediate_size": 0,
        "moe_shared_expert_intermediate_size": 0,
        "tie_word_embeddings": true
    },
    "build_config": {
        "max_input_len": 1024,
        "max_seq_len": 32768,
        "opt_batch_size": 8,
        "max_batch_size": 16,
        "max_beam_width": 1,
        "max_num_tokens": 32768,
        "opt_num_tokens": 16,
        "max_prompt_embedding_table_size": 0,
        "kv_cache_type": "PAGED",
        "gather_context_logits": false,
        "gather_generation_logits": false,
        "strongly_typed": true,
        "force_num_profiles": null,
        "profiling_verbosity": "layer_names_only",
        "enable_debug_output": false,
        "max_draft_len": 0,
        "speculative_decoding_mode": 1,
        "use_refit": false,
        "input_timing_cache": null,
        "output_timing_cache": "model.cache",
        "lora_config": {
            "lora_dir": [],
            "lora_ckpt_source": "hf",
            "max_lora_rank": 64,
            "lora_target_modules": [],
            "trtllm_modules_to_hf_modules": {}
        },
        "auto_parallel_config": {
            "world_size": 1,
            "gpus_per_node": 8,
            "cluster_key": "L4",
            "cluster_info": null,
            "sharding_cost_model": "alpha_beta",
            "comm_cost_model": "alpha_beta",
            "enable_pipeline_parallelism": false,
            "enable_shard_unbalanced_shape": false,
            "enable_shard_dynamic_shape": false,
            "enable_reduce_scatter": true,
            "builder_flags": null,
            "debug_mode": false,
            "infer_shape": true,
            "validation_mode": false,
            "same_buffer_io": {
                "past_key_value_(\\d+)": "present_key_value_\\1"
            },
            "same_spec_io": {},
            "sharded_io_allowlist": [
                "past_key_value_\\d+",
                "present_key_value_\\d*"
            ],
            "fill_weights": false,
            "parallel_config_cache": null,
            "profile_cache": null,
            "dump_path": null,
            "debug_outputs": []
        },
        "weight_sparsity": false,
        "weight_streaming": false,
        "plugin_config": {
            "dtype": "bfloat16",
            "bert_attention_plugin": "auto",
            "gpt_attention_plugin": "auto",
            "gemm_plugin": "bfloat16",
            "explicitly_disable_gemm_plugin": false,
            "gemm_swiglu_plugin": null,
            "fp8_rowwise_gemm_plugin": null,
            "qserve_gemm_plugin": null,
            "identity_plugin": null,
            "nccl_plugin": null,
            "lora_plugin": null,
            "weight_only_groupwise_quant_matmul_plugin": null,
            "weight_only_quant_matmul_plugin": null,
            "smooth_quant_plugins": true,
            "smooth_quant_gemm_plugin": null,
            "layernorm_quantization_plugin": null,
            "rmsnorm_quantization_plugin": null,
            "quantize_per_token_plugin": false,
            "quantize_tensor_plugin": false,
            "moe_plugin": "auto",
            "mamba_conv1d_plugin": "auto",
            "low_latency_gemm_plugin": null,
            "low_latency_gemm_swiglu_plugin": null,
            "context_fmha": true,
            "bert_context_fmha_fp32_acc": false,
            "paged_kv_cache": true,
            "remove_input_padding": true,
            "reduce_fusion": false,
            "user_buffer": false,
            "tokens_per_block": 64,
            "use_paged_context_fmha": false,
            "use_fp8_context_fmha": false,
            "multiple_profiles": false,
            "paged_state": false,
            "streamingllm": false,
            "manage_weights": false,
            "use_fused_mlp": true,
            "pp_reduce_scatter": false
        },
        "use_strip_plan": false,
        "max_encoder_input_len": 1024,
        "monitor_memory": false,
        "use_mrope": false
    }
}

3. ready model repository
   • dev python bacckend (spark_tts, audio_tokenizer, vocoder) like lambda function, u can see example/add_sub to do
   • tensorrtllm backend don't to dev, use qwen as backbone,use Triton backend tensorrtllm_backend + TensorRT-LLM to Prepare the Model Repository, use inflight_batcher_llm(Using the C++ TensorRT-LLM backend with the executor API, which includes the latest features including inflight batching) tensorrt_llm(This model is a wrapper of your TensorRT-LLM model and is used for inferencing.)
   • fill params (dynamic_batching, parameters, input/output etc..):
     • ⭐️ tensorrtllm backend model_config
     • tensorrtllm backend Modify the Model Configuration to fill api params which used fill_template.py
     • python backends like tensorrtllm backend which used fill_template.py to do

there are use developed python backend (spark_tts, audio_tokenizer, vocoder) in https://github.com/SparkAudio/Spark-TTS/tree/main/runtime/triton_trtllm/model_repo and just fill grpc pb stub params like this:

# fill template with pbtext file for api params
modal run src/llm/trtllm/tts_spark/ready_model.py \
    --tag-or-hash "main" \
    --trt-dtype "bfloat16" \
    --spark-tts-params "bls_instance_num:4,triton_max_batch_size:16,max_queue_delay_microseconds:0" \
    --audio-tokenizer-params "triton_max_batch_size:16,max_queue_delay_microseconds:0" \
    --tensorrt-llm-params "triton_backend:tensorrtllm,triton_max_batch_size:16,decoupled_mode:False,max_beam_width:1,max_tokens_in_paged_kv_cache:2560,max_attention_window_size:2560,kv_cache_free_gpu_mem_fraction:0.5,exclude_input_in_output:True,enable_kv_cache_reuse:False,batching_strategy:inflight_fused_batching,max_queue_delay_microseconds:0,encoder_input_features_data_type:TYPE_FP16,logits_datatype:TYPE_FP32" \
    --vocoder-params "triton_max_batch_size:16,max_queue_delay_microseconds:0" 

4. run tritonserver

Note

nvcr.io/nvidia/tritonserver:25.02-trtllm-python-py3 use python3.12.3,
modal python3.12 use python3.12.1, need building-custom-python-backend-stub:
https://github.com/triton-inference-server/python_backend?#building-custom-python-backend-stub
more:

• https://github.com/triton-inference-server/backend/blob/main/docs/python_based_backends.md
• https://github.com/triton-inference-server/python_backend/blob/main/README.md

# run tritonserver
APP_NAME=tts-spark modal serve src/llm/trtllm/tts_spark/tritonserver.py 

# curl health to startup tritonserver (as cold-starting)
curl -vv -X GET "https://weedge--tritonserver-serve-dev.modal.run/v2/health/ready" -H  "accept: application/json"

5. http client test (modal don't support grpc,so don't to test, u can test in local docker)

modal run src/llm/trtllm/tts_spark/client_http.py \
    --action health_meta_statics \
    --server-url "weedge--tritonserver-serve-dev.modal.run"

modal run src/llm/trtllm/tts_spark/client_http.py \
    --action cli_sdk_health_meta_statics \
    --server-url "weedge--tritonserver-serve-dev.modal.run"

modal run src/llm/trtllm/tts_spark/client_http.py \
    --action tts \
    --server-url "weedge--tritonserver-serve-dev.modal.run"

modal run src/llm/trtllm/tts_spark/client_http.py \
    --action cli_sdk_tts \
    --output-audio cli_sdk_tts_output.wav \
    --server-url "weedge--tritonserver-serve-dev.modal.run"

modal run src/llm/trtllm/tts_spark/client_http.py \
    --action cli_sdk_tts \
    --output-audio cli_sdk_tts_output.wav \
    --target-text "你好，请将一个儿童故事，文字不少于200字，故事充满童趣和奇妙幻想。" \
    --output-audio "child_story.wav" \
    --server-url "weedge--tritonserver-serve-dev.modal.run"

6. bench

• bench data: use yuekai/seed_tts dataset wenetspeech4tts (26 rows)
• infer bench (tritornserver deploy on the L4 GPU , use first Inference to warmup):
• see bench result: feat: spark tts use TensorRT-LLM + tritonserver deploy on the modal #129 (comment)

# single test
CLI_MODE=http modal run src/llm/trtllm/tts_spark/bench.py \
    --server-url "weedge--tritonserver-serve-dev.modal.run" \
    --reference-audio "/prompt_audio.wav" \
    --huggingface-dataset ""

# hf dataset with (wenetspeech4tts 26 rows) with concurrency 1 task to bench
CLI_MODE=http modal run src/llm/trtllm/tts_spark/bench.py \
    --server-url "weedge--tritonserver-serve-dev.modal.run" \
    --reference-audio "" \
    --huggingface-dataset "yuekai/seed_tts" \
    --split-name wenetspeech4tts \
    --num-tasks 1 

# single test used by http client sdk
CLI_MODE=cli_sdk_http modal run src/llm/trtllm/tts_spark/bench.py \
    --server-url "weedge--tritonserver-serve-dev.modal.run" \
    --reference-audio "/prompt_audio.wav" \
    --huggingface-dataset ""

# hf dataset with (wenetspeech4tts 26 rows) with concurrency 1 task to bench used by http client sdk
CLI_MODE=cli_sdk_http modal run src/llm/trtllm/tts_spark/bench.py \
    --server-url "weedge--tritonserver-serve-dev.modal.run" \
    --reference-audio "" \
    --huggingface-dataset "yuekai/seed_tts" \
    --split-name wenetspeech4tts \
    --num-tasks 1 

# hf dataset with (wenetspeech4tts 26 rows) concurrency min(num_tasks,26) task to bench used by http client sdk
# num-tasks: 2->4->8->16->26
CLI_MODE=cli_sdk_http modal run src/llm/trtllm/tts_spark/bench.py \
    --server-url "weedge--tritonserver-serve-dev.modal.run" \
    --reference-audio "" \
    --huggingface-dataset "yuekai/seed_tts" \
    --split-name wenetspeech4tts \
    --num-tasks 2

todo:

• other GPU cumput arch (Ampere+) bench
• change params to do Perf Analyzer

reference

• TensorRT-LLM:
  • ⭐️ https://nvidia.github.io/TensorRT-LLM/overview.html
  • https://nvidia.github.io/TensorRT-LLM/reference/support-matrix.html
  • https://nvidia.github.io/TensorRT-LLM/performance/perf-benchmarking.html
  • ⭐️ tensorrtllm backend model_config
• triton-inference-server(tritonserver):
  • ⭐️ https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/index.html
  • ⭐️⭐️ https://github.com/triton-inference-server/server/blob/main/docs/README.md
• pb_stub:
  • InferenceRequest
  • InferenceResponse

雁过留声

[weedge]

⭐️ python backend BLS(business-logic-scripting) decoupled(stream):

• https://github.com/triton-inference-server/python_backend/blob/r25.02/README.md#using-bls-with-decoupled-models
• https://github.com/triton-inference-server/python_backend/tree/r25.02/examples/bls_decoupled