Skip to content

feat(repository): add inventory integration and cache management - #119

Merged
mangoknight merged 2 commits into
dingodb:mainfrom
dongzhenyangofficial-ctrl:feat/repository-inventory-and-cache-management
Sep 23, 2026
Merged

mangoknight merged 2 commits into
dingodb:mainfrom
dongzhenyangofficial-ctrl:feat/repository-inventory-and-cache-management

Conversation

@dongzhenyangofficial-ctrl

@dongzhenyangofficial-ctrl dongzhenyangofficial-ctrl commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

This change adds namespace-aware repository management for uploaded models and remote caches, and connects Speed's local upload inventory to Scheduler without making local operations depend on Scheduler availability.

Changes:

  • Add repository identity/lifecycle handling, node registration and fixed-content access for official revision workflows.
  • Replace periodic full inventory polling with a durable repository outbox, write-ahead mutation intents, proactive HTTP reports, acknowledgements and retries. First registration and explicit reconciliation establish baselines; epoch/sequence handling preserves changes made while reports are in flight. Baselines reuse published manifest hashes and check metadata, existence, size and completion markers without rehashing file contents.
  • Extend ModelScope metadata, file access and preheat support; add persisted standalone cache jobs, task reuse, pause/resume and transfer settings. Retry failed downloads and cache jobs, and keep upload cleanup scoped to uploaded content.
  • Add read-only HF repository views and Linux FUSE model directories, including consumer/sidecar integration; bound repository scans and registration work and preserve repository conflicts and storage failures. Change the default disk check interval to 24 hours.
  • Improve cache completion/reference handling, peer fallback, dependency observations and Prometheus alerts; update the shared gRPC protocol and regression tests.

Validation of the current submitted snapshot in an isolated Linux directory, Go 1.24.1, CGO_ENABLED=0:

  • go build ./...
  • go test ./... -count=1 -timeout=180s
  • git diff --cached --check

This validation covers builds and automated tests; live multi-node deployment and browser acceptance were not rerun for this update. The external management UI is maintained separately.

Companion Scheduler PR: dingodb/dingoscheduler#28

@mangoknight
mangoknight added this pull request to the merge queue Sep 23, 2026
Merged via the queue into dingodb:main with commit 45fef1c Sep 23, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants