feat: offline cache probe paired with download helper#12
Merged
Conversation
Adds `download::is_repo_cached(repo_id, files)` and the convenience method `ModelPreset::is_cached()` so consumers can decide between rendering a "Download" affordance and a "Ready" indicator without risking a silent download. The motivating gap: `hf-hub` 0.5 has no `HF_HUB_OFFLINE` support and no public lookup-only API. A naïve "try to construct the backend and see if it succeeds" probe ends up downloading on every cold-cache call. The new probe walks `hf-hub`'s on-disk snapshot layout (`<cache_root>/models--<owner>--<repo>/snapshots/<commit>/<file>`) without touching the network, mirroring how `hf-hub` itself resolves the cache root (`HF_HOME` or `\$HOME/.cache/huggingface/hub`). Pairs with `download_files_with_progress`: the same `(repo_id, files)` pair you pass there answers "do I still need to download?" here. Includes unit tests covering missing repo, partial files, multiple snapshots, and slash-flattening in repo ids. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an offline, pure-filesystem cache probe so consumers can answer "is this preset already on disk?" without touching the network.
download::is_repo_cached(repo_id, files) -> bool— generic, mirrors the existingdownload_files_with_progress(repo_id, files, dest_dir, on_progress)signature.ModelPreset::is_cached(&self) -> bool— convenience method for the sherpa-onnx backend's preset constants.Why
hf-hub0.5 has noHF_HUB_OFFLINEsupport and no public lookup-only download API. The natural "try to construct the backend and see if it succeeds" probe therefore ends up firing a full HuggingFace metadata + download cycle on every cold-cache call — a UI that uses it as a quick "is the model ready?" check ends up silently downloading ~100 MB in the background and never showing its "Download model" CTA. Consumers that want to stay offline during a UI render need a pure-filesystem alternative.How
The probe walks the on-disk layout
hf-hubwrites:…and returns
truewhen at least one snapshot dir contains every requested file. The cache root is resolved exactly the wayhf-hub0.5'sCache::from_envdoes: honorHF_HOME(appendinghub), otherwise default to\$HOME/.cache/huggingface/hub.USERPROFILEis also accepted as a Windows fallback forHOME.The implementation is split into:
pub fn is_repo_cached— public entry point, looks upcache_rootfrom the environment.fn hf_hub_cache_root— env resolution, factored out for clarity.fn snapshot_satisfies_files(cache_root, repo_id, files)— pure logic, factored out so the unit test can drive it against a temp dir without touching the real cache or polluting process-wide env vars.ModelPreset::is_cachedis a one-liner overis_repo_cached(self.model_id, &self.files()), paralleling the existingdownload_preset_with_progresswrapper.Tests
Two new unit tests in
download::tests:snapshot_probe_matches_hf_hub_layout— builds the cache layout step by step in a temp dir and asserts (a)falsefor missing repo / empty snapshot dirs / partial files and (b)trueonce every requested file is present in at least one snapshot. Also covers a second, half-complete snapshot not invalidating a first complete one.snapshot_probe_flattens_slashes_in_repo_id— guards against a silent regression where a slash-bearing repo id (the common case) wouldn't be flattened to<owner>--<repo>and the probe would always look in a non-existent directory.Pure-fs, no network, no real models needed — runs under the existing
make cimatrix.Test plan
make test(9 passed, 0 failed)make lint(clippy clean)make ci(fmt + clippy + test + doc-test all green)🤖 Generated with Claude Code