Repository navigation
fix(repository): keep HuggingFace registration off the full cache walk - #120
Merged
Merged
Conversation
Register 的注释写明"registration cost stays independent of how many unrelated repositories a node accumulates",但 conflictScanRoots 对 HuggingFace 命名空间返回的是整个 api/<repoType>,也就是镜像节点上存放 全部上游缓存的那棵树。其余三个命名空间都收敛到了各自子目录,只有它例外。 后果在大缓存节点上是致命的:listConflicts 由 Register 在持有进程级 registrationMu 的情况下调用,于是每次首访一个未注册的 HF 仓库,都会在 锁内遍历整个缓存(含每个仓库的 paths-info 子树),期间所有其他请求排队。 hd-04 现网实测:49T 缓存 / 822 个仓库下,离线实例升级后任何文件读取都 超过 60 秒不返回,而同机未升级的实例为 0.7 秒。goroutine dump 显示卡在 repository.listIn ← listConflicts,并有多个 goroutine 阻塞在 sync.runtime_SemacquireMutex 上。 改为沿候选自身路径定向查找。registrationConflicts 只在两者存储路径存在 大小写无关的前缀关系、仓库路径互为祖先/后代、或共享路径段仅大小写不同时 才判冲突,这些都要求既有仓库位于候选路径沿线,因此只需读取该路径各级目录, 兄弟子树永远不必打开。仅大小写不同的路径段按整棵子树收集,与 listBareLocalRepositories 处理命名空间别名的做法一致。 新增回归测试:缓存内无关仓库从 2 个增加到 200 个时,读取目录数恒为 2, 冲突候选集不变;祖先、后代、大小写孪生、父目录大小写四类冲突仍被检出; 仅共享前缀字符串的无关仓库不会误报。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mangoknight
enabled auto-merge
September 23, 2026 12:28
chengebj
approved these changes
Sep 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Registerpromises that conflict detection cost is independent of how many unrelated repositories a node holds, butconflictScanRootsreturned the wholeapi/<type>tree for HuggingFace.listConflictsruns underregistrationMu, so the first access to an unregistered HF repository on a large cache walked every cached repository while all other requests queued behind the lock.Fix
Resolve HF conflict candidates by walking only the candidate's own path. Every conflict rule (case-insensitive storage prefix, ancestor/descendant, case-only component difference) requires the existing repository to lie on that path, so sibling subtrees never need to be read. A case-folded matching component is still collected whole, as for local namespace aliases.
Tests
TestHFRegistrationCostIsIndependentOfCacheSize: directories read stay constant with 2 vs 200 unrelated cached repositories.TestHFRegistrationStillDetectsConflictsandTestHFRegistrationAllowsUnrelated.go test ./pkg/repository/passes.🤖 Generated with Claude Code