Skip to content

fix(milvus): persist embedded etcd metadata with vector data - #3836

Closed
tang-vu wants to merge 2 commits into
Osmantic:public-betafrom
tang-vu:fix/milvus-persist-embedded-etcd-20260908
Closed

tang-vu wants to merge 2 commits into
Osmantic:public-betafrom
tang-vu:fix/milvus-persist-embedded-etcd-20260908

Conversation

@tang-vu

@tang-vu tang-vu commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

Why this matters

The shipped Milvus standalone extension enables embedded etcd, but its default metadata directory is outside the persisted vector-data mount. Recreating a container can therefore lose collection metadata despite retaining vector files. This is release/data-durability correctness, not an untrusted-input security claim.

Fix and migration boundary

Set ETCD_DATA_DIR=/var/lib/milvus/etcd inside the existing bind mount. Verified against the v2.4.5 configuration and launcher at that tag. The launcher itself names a v2.4.0 image; ODS ships v2.4.5 and this PR does not change its image.

Existing installations must stop writes, preserve the old metadata/data pair and migrate the complete etcd directory before applying this change. The README documents validation and recovery; there is no automatic live-data migration.

Overlap check

Searched all-state Milvus PRs and the compose path. This original #3836 is the canonical persistence change. Compared #5228 by Vishaaallll: it checks manifest/compose existence and shape, but neither changes nor verifies etcd persistence. Those fixtures are complementary, not a stronger replacement; no contributor PR was closed. Historical #647 enables embedded mode but does not persist its metadata.

Validation

  • Linux Python 3.11: python -m pytest -q tests/test-milvus-persistence.py: passed; exact shipped image plus metadata/vector/queue path containment asserted at the compose boundary.
  • docker compose -f extensions/library/services/milvus/compose.yaml config --quiet: passed.
  • Focused pre-commit and diff check passed; test is in make test.
  • No live Milvus collection, recreation or existing-data migration was executed. These remain explicit operator rollout gates, not covered by a static test or CI. Baseline smoke/simulation passed; full make gate/BATS have existing environment/fixture failures.

Rollback and review

Do not simply remove the variable after the new instance accepts writes. Stop the new instance and restore the saved matching metadata/vector-data pair with the original configuration. High-risk data-path change requires independent human review and deployment-specific migration validation before rollout. Ready for code review, not approval to merge or deploy.

Final beta-batch receipt — 2026-09-16

Candidate: 557219d6ca09b88205f56ea5df85772d00fa5ebf. GitHub check audit at 11:07:47 UTC: 37 successful, 4 skipped, 0 failed, 0 pending, matching this exact head; OPEN, public-beta base, Ready (not Draft). Skipped checks are not counted as passed tests.

The combined integration head, 5b9877184ceab0751c6c9fe3564c73303e78f0a3, contains all 30 exact candidate heads on public-beta 3c186f3f12ba21dad7b7266ec5449393529abb00. At this exact final SHA: full dashboard 171 files / 1,274 tests passed; full dashboard API 3,124 passed, 2 skipped; advice boundary suite 23 passed; Bark request/legacy-empty cases 9 passed; changed-file secret/key/size hooks and diff check passed. Shell-only hooks had no matching changed files.

Additional combined checks ran at predecessor 28adf1aa5c0710d3c0de13df7870629e24977fec: Token Spy 130 passed, 1 skipped; CLI contracts 16 passed; real backup/restore round trip passed; Open Interpreter HTTP/runner cases 4 passed; make lint smoke simulate, frontend lint (zero errors, 624 warnings) and production build passed. The only final-head differences are two test files (advice dialog fixtures and Bark's empty-text expectation); production files are identical. These results are not live GPU/model or service rollout qualification.

Qualified merge order: #5490 → #5494 → #5495 → #3848 → #3846 → #3845 → #3844 → #3843 → #3841 → #3838 → #3837 → #3836 → #3835 → #3834 → #3809 → #3808 → #3807 → #3806 → #3804 → #3803 → #3802 → #3801 → #3800 → #3712 → #3711 → #3710 → #3709 → #3708 → #3852 → #3849. Branches are independently useful. Shared-file merging required only preserving the union of four adjacent Makefile registrations: provider-probe redirects, healthcheck error-body, Milvus persistence and healthcheck latency. Production code in these 30 merged without manual conflict resolution. External advice/Bark overlaps have separate pairwise receipts in #5494/#3808. Recheck the combined tree if other shared-file PRs land first; do not drop test registrations.

Full release qualification remains not green: baseline make gate fails a no-sudo Docker fallback fixture; 5 of 428 BATS cases fail in this environment; all-files private-key scanning flags two pre-existing dummy-key fixtures. No checks were disabled. CI success and Ready status do not replace independent human review, especially for auth backups and Milvus data migration. No upstream merge or deployment was performed.

@tang-vu
tang-vu marked this pull request as ready for review September 15, 2026 01:33
@tang-vu
tang-vu changed the base branch from main to public-beta September 16, 2026 10:26
This was referenced Sep 16, 2026
@Lightheartdevs

Copy link
Copy Markdown
Collaborator

Thanks for this contribution. public-beta was promoted into main on 2026-09-24 and no longer receives changes, so we're closing pull requests that target it. This isn't a judgment on the change itself. If it's still needed, please rebase onto main and open a focused PR. See #7253 for details and the contribution policy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants