Before you start
Applies to
Engine
What problem does this solve
FreeToken currently has no official container image published by the project.
Users who want to run ft serve in Docker must assemble their own CUDA, Python,
build-toolchain, and FreeToken environment. This makes deployments less
reproducible and leaves users without a project-maintained image tag to use in
automation.
I checked the Roadmap, FAQ, and existing issues before opening this request.
Proposed solution
Add a project-maintained CUDA development container image and a GitHub Actions
workflow that:
- builds the image for pull requests and manual workflow runs without publishing;
- publishes only from the upstream repository on pushes to
main and version tags;
- publishes to
ghcr.io/flashml-org/freetoken;
- provides
latest for the default branch plus semantic-version and commit-SHA tags;
- uses GitHub Actions cache for Buildx layers;
- documents how to mount a model directory and run
ft serve with Docker.
The image should retain nvcc, because FreeToken can compile CUDA kernels at
first use.
Alternatives considered
- Users maintain local Dockerfiles: flexible, but duplicates setup work and
produces inconsistent CUDA and dependency combinations.
- Publish only Python wheels: already useful, but does not provide a
reproducible deployment image.
- Use a runtime-only CUDA image: smaller, but unsuitable when JIT compilation
of FreeToken CUDA kernels is needed.
Before you start
Applies to
Engine
What problem does this solve
FreeToken currently has no official container image published by the project.
Users who want to run
ft servein Docker must assemble their own CUDA, Python,build-toolchain, and FreeToken environment. This makes deployments less
reproducible and leaves users without a project-maintained image tag to use in
automation.
I checked the Roadmap, FAQ, and existing issues before opening this request.
Proposed solution
Add a project-maintained CUDA development container image and a GitHub Actions
workflow that:
mainand version tags;ghcr.io/flashml-org/freetoken;latestfor the default branch plus semantic-version and commit-SHA tags;ft servewith Docker.The image should retain
nvcc, because FreeToken can compile CUDA kernels atfirst use.
Alternatives considered
produces inconsistent CUDA and dependency combinations.
reproducible deployment image.
of FreeToken CUDA kernels is needed.