Repository navigation
kern-py: a plugin reads weights from the engine's model id, a directory or a Hugging Face repo - #61
Conversation
…ry or a Hugging Face repo `sglang serve --model-path Qwen/Qwen3.8-27B` and `vllm serve Qwen/Qwen3.8-27B`, the commands the docs give, name the checkpoint by its hub id; both plugins handed that id to kern.Runtime as a directory, and the load failed with "weights Qwen/Qwen3.8-27B: No such file or directory". Every earlier gate passed a local path. `kern.host.checkpoint` resolves an id to its snapshot (the engine's own cache, so nothing is fetched twice), at vLLM's revision and at the commit SGLang's config came from. Gate: tray04 GPU0, lmsysorg/sglang nightly f70e8c68, pip install kern-llm 0.3.0 with this wheel over it, hf download of Pegainfer/kern-qwen38-sm103 and Qwen/Qwen3.8-27B, the docs' serve command with qwen3.8-27b-rsi-sglang.json: greedy 48/48 tokens equal kern run, twice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: JinYan Su <751080330@qq.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 619cbd22b5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| self.group = get_tp_group().cpu_group if m.get("topology") else None | ||
| self.rt = kern.Runtime(os.environ["KERN_MANIFEST"], os.environ["KERN_KERNELS"], | ||
| [vllm_config.model_config.model], torch.cuda.current_device(), | ||
| [checkpoint(vllm_config.model_config.model, vllm_config.model_config.revision)], |
There was a problem hiding this comment.
When a gated/private repository is accessed with a raw token supplied through vLLM's hf_token configuration rather than an environment variable or hf auth login, vLLM can resolve the model config but this call drops the token before snapshot_download, so loading the actual weights fails with an authorization error. vLLM explicitly defines hf_token as the bearer token for remote files; pass it through to checkpoint and then snapshot_download. vLLM ModelConfig documentation
Useful? React with 👍 / 👎.
| return name | ||
| from huggingface_hub import snapshot_download | ||
|
|
||
| return snapshot_download(name, revision=revision, allow_patterns=["*.safetensors", "*.json"]) |
There was a problem hiding this comment.
When vLLM is launched with --download-dir, this unqualified snapshot_download ignores vllm_config.load_config.download_dir and uses the default Hugging Face cache instead. Consequently, an offline deployment whose model exists only in the configured cache cannot start, while an online deployment may download a second multi-gigabyte copy and exhaust the default filesystem. Thread the configured directory through as cache_dir; vLLM documents download_dir as the directory used to download and load weights. vLLM LoadConfig documentation
Useful? React with 👍 / 👎.
1 participant
The vLLM and SGLang serve commands in the docs name the checkpoint by its hub id (
Qwen/Qwen3.8-27B). Both plugins passed that id tokern.Runtimeas a directory, so the documented command failed at load time withweights Qwen/Qwen3.8-27B: No such file or directory. Every earlier gate had used a local path.kern.host.checkpointreturns the name unchanged when it is a directory. Otherwise it resolves the id to its Hugging Face snapshot, using the engine's own cache so nothing is downloaded twice. The revision is vLLM'smodel_config.revision, and for SGLang the commit its config was loaded from (config._commit_hash). SGLang's global server args are not set in the scheduler process.Gate, as a user would run it: tray04 GPU0,
lmsysorg/sglang:nightly-dev-cu13-20261005-f70e8c68,pip install kern-llm==0.3.0with this branch's wheel installed over it,hf downloadofPegainfer/kern-qwen38-sm103(now schema 7) andQwen/Qwen3.8-27B, then the serve command fromdocs/sglang.mdwithqwen3.8-27b-rsi-sglang.json. Greedy output matchedkern runon 48 of 48 tokens, in both of two runs.Ships as 0.3.1.
🤖 Generated with Claude Code