Skip to content

fix(chromadb): persist the Rust server data directory - #4989

Closed
tang-vu wants to merge 2 commits into
Osmantic:public-betafrom
tang-vu:fix/beta-chromadb-persistent-storage-20260914
Closed

tang-vu wants to merge 2 commits into
Osmantic:public-betafrom
tang-vu:fix/beta-chromadb-persistent-storage-20260914

Conversation

@tang-vu

@tang-vu tang-vu commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

Ready for independent review (2026-09-15). Candidate 821a0df7624a8c7d4058a407390a7b223e5e210c has a successful GitHub check rollup and no base-branch conflict at this review snapshot. This supersedes earlier draft-status wording only. All validation gaps, migration requirements, live gates and independent human review requirements below remain in force; this is not a merge-ready approval.

ChromaDB 1.5.3 starts with /config.yaml specifying persist_path: /data, but the ODS recipe mounts data/chromadb at /chromadb. A successful collection/vector insert therefore disappears when its container is recreated.

Why this matters

An extension update or recreation can erase a user's vector database even though data/chromadb appears to be configured for persistence. Mount the existing host directory at the path the shipped server actually writes, and pin that same 1.5.3 image by its verified multi-architecture digest. Preserve the existing HTTP health probe, loopback binding and API port.

The README explicitly requires stopping and copying the old container's complete /data directory before recreation, retaining the backup, avoiding merges of populated databases, and checking known collection contents afterward. It does not silently replace or auto-migrate operator data. If the old container is already gone, this fix cannot recover its deleted writable layer. The documented collection example now uses the exact v2 tenant/database route needed to verify the stored data.

Overlap check

Searched open and closed PRs for chromadb and Chroma persistence, plus changed production files under extensions/library/services/chromadb. Merged #4890 fixes the Rust image's missing healthcheck executable and explicitly leaves data paths unchanged; #1066/#635/#609 are older health/API probe changes. #554 changes manifest metadata. No existing PR corrects the persistent mount or verifies collection survival. This PR retains #4890's health behavior and adds a separate persistence boundary test.

Validation

  • Exact old recipe/image: create collection, insert a supplied 3D vector/document/metadata, stop/remove/recreate ? HTTP 404: collection does not exist. The regression fails at this public boundary.
  • Corrected recipe: 2 real-image tests passed in 97 seconds. Fresh vector data survives recreation, and data copied from the stopped legacy container into the host directory survives activation with the new mount. Both return the original collection ID, document, metadata and nearest-neighbor distance 0. No embedding model is downloaded.
  • Existing exact-image health tests: 2 passed, including heartbeat success, missing-route failure and closed-port failure. Fixture containers and networks were removed.
  • Path-filtered GitHub workflow repeats both persistence cases. Scoped pre-commit and git diff --check pass. make gate retains the baseline root/WSL no-sudo fixture failure; the complete local gate is not claimed green.
  • The first GitHub run passed fresh persistence but found a fixture ownership mistake in legacy recovery: Docker had created the empty destination as root, so the runner's docker cp could not write there. Follow-up 821a0df7 creates the fixture destination as the calling user before Compose, matching the README's operator-created backup directory. Both local live cases passed again; the replacement GitHub head passed 33 checks with 4 intentional skips, including both live persistence cases.

Still requires independent review of the operational data-recovery instructions. Live evidence is Linux amd64 with a fresh fixture and a stopped legacy fixture; no user database, hardware fleet or native ARM/macOS/Windows activation was used. Backups must retain the complete database directory and matching image. Reverting only the mount fix reintroduces data loss, so preserve the corrected mount when rolling back unrelated changes.

Final candidate 821a0df7624a8c7d4058a407390a7b223e5e210c: 33 successful CI checks, 4 conditionally skipped; no pending or failed checks.

Batch integration receipt

This is one of 30 independently branched public-beta PRs. The published synthetic integration head is 61f4c6fccc81bea697f4941683fefb5089ba77f4, built from baseline d49ad4bbc760be319de7f4c9b9a93b1f18fe3922 in this reproducible order:

#4946 → #4948 → #4951 → #4954 → #4972 → #4960 → #4961 → #4957 → #4964 → #4967 → #4968 → #4971 → #4975 → #4976 → #4984 → #4982 → #4990 → #4988 → #4989 → #4997 → #5003 → #5006 → #5007 → #5009 → #5012 → #5011 → #5013 → #5014 → #5020 → #5021

Reconciliation preserved all 146 files owned by a single PR byte-for-byte against their exact candidate heads. The eight shared paths preserve both CLI STT/TTS commands and help, every added Makefile test, both environment schema additions and all Extensions UI behavior. LanguageTool and Kroki both remain headless API entries. Canonical regeneration yields 76 catalog entries and 47 library manifests; all README rows are retained. Shared-file changes need these additive resolutions when merged independently.

Combined validation: 2,852 API tests passed / 1 skipped; 1,082 UI tests passed; 47 manifests passed; all 12 added Makefile commands passed; 12 catalog boundary tests passed / 12 opt-in runtime tests skipped in this combined run. Individual exact-image results remain documented above. UI lint/build, scoped secret hooks, diff checks, four platform smoke tests and installer simulation passed. Compose plans for NVIDIA/AMD/Apple each contain 20 services and no duplicate published bindings across the core plus 12 new recipes; this is static plan evidence, not simultaneous startup or GPU inference.

The first unrestricted UI run timed out in two tests under concurrent load; the complete rerun used --maxWorkers=2 with unchanged assertions and timeouts and passed. ESLint reports zero errors and 603 warnings, including the existing JSX usage-detection pattern. Full local make gate still fails the no-sudo Docker baseline covered by #3158. BATS records 416 passed / 5 failed on this root/WSL host (platform detection, root preflight and a 15 GiB versus 20 GiB free-space expectation). Neither full local gate nor BATS is described as green.

This integration receipt supplements each PR's own regression and does not waive its independent review, live hardware/browser/upgrade gates or rollback requirements. The PRs remain drafts; none was merged upstream.

This was referenced Sep 14, 2026
@tang-vu
tang-vu marked this pull request as ready for review September 15, 2026 01:31
@Lightheartdevs

Copy link
Copy Markdown
Collaborator

Thanks for this contribution. public-beta was promoted into main on 2026-09-24 and no longer receives changes, so we're closing pull requests that target it. This isn't a judgment on the change itself. If it's still needed, please rebase onto main and open a focused PR. See #7253 for details and the contribution policy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants