Skip to content

feat(repository): add upload inventory and official revision management - #28

Merged
mangoknight merged 2 commits into
dingodb:mainfrom
dongzhenyangofficial-ctrl:feat/upload-inventory-and-revision-management
Sep 23, 2026
Merged

mangoknight merged 2 commits into
dingodb:mainfrom
dongzhenyangofficial-ctrl:feat/upload-inventory-and-revision-management

Conversation

@dongzhenyangofficial-ctrl

@dongzhenyangofficial-ctrl dongzhenyangofficial-ctrl commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

This change gives Scheduler persistent visibility into uploaded files and the nodes that hold them, while keeping upload inventory separate from remote-cache records and explicit official revision definitions.

Changes:

  • Add validated upload inventory ingestion, repository/file/node queries and last-known holdings that survive node unavailability.
  • Accept proactive HTTP inventory reports with transactional epoch/sequence checks, idempotent acknowledgements, durable node report state and explicit baseline reconciliation. Ordinary list refreshes and heartbeats no longer perform full node inventory scans. Failed reconciliation preserves the last confirmed inventory.
  • Add node endpoint registration, health/repository observation APIs and the node health page; provide authenticated internal official revision management with fixed manifests, source validation and concurrency checks.
  • Complete ModelScope remote repository handling: provider-specific default revisions and credentials, pinned-commit registration, per-file identity/size validation, idempotent in-place updates and rejection of stale completed-job notifications. Registration failures remain retryable without changing download completion status.
  • Add namespace filters before pagination, consistent repository identity fields and cache task disposition forwarding; isolate hosted identities from legacy remote paths and update protocol/schema/API documentation and regression tests.

Validation of the current submitted snapshot in an isolated Linux directory, Go 1.24.1, CGO_ENABLED=0:

  • go build ./...
  • go test ./... -count=1 -timeout=180s
  • git diff --cached --check

The system Go 1.26 build is incompatible with the existing sonic dependency; the CI-configured Go 1.24.1 build and full test suite pass. Automated DAO/HTTP tests use controlled fixtures and temporary databases; live multi-node deployment, private ModelScope access and browser acceptance were not rerun for this update. Official revision endpoints are internal gateway APIs; the external management UI is maintained separately.

Companion Speed PR: dingodb/dingospeed#119

@mangoknight
mangoknight merged commit 7c2954b into dingodb:main Sep 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants