Skip to content

fix(macos): stop running dashboard-api writers before private-store migration on launch - #8076

Merged
Lightheartdevs merged 3 commits into
Osmantic:mainfrom
vaibhavsrv:fix/macos-private-store-launch-stop-writers
Oct 24, 2026
Merged

Lightheartdevs merged 3 commits into
Osmantic:mainfrom
vaibhavsrv:fix/macos-private-store-launch-stop-writers

Conversation

@vaibhavsrv

@vaibhavsrv vaibhavsrv commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Summary

When rerunning install-macos.sh or executing ods restart/ods update on a running macOS installation upgrading from v3.0.0 (or pre-994cf6475 commits), installers/macos/lib/dashboard-private-state.py attempts to migrate legacy SQLite stores (data/pixel-images and data/pixel-chat-results) onto Docker volumes. Because dashboard-api is still actively running from the pre-existing deployment, Stores.prepare() invokes self.consumers(stopped=True) and raises Private dashboard migration/restore needs stopped writers. Run ods stop, then ods start for migration; retry restore after stopping.. This causes automated installer reruns, updates, and restarts to fail during the final compose launch step after configuration and source files have already been updated.

This patch restricts writer stopping strictly to the migration path after store validation:

  1. Stores.consumers() now validates ownership of all consumers across the inventory before issuing any container stops, ensuring foreign consumers abort immediately with private-store-foreign-consumer without halting owned containers.
  2. Stores.prepare(directory, stop=args.action == "launch") validates that no unfinished restore exists (image-restore-active.json), checks consumer custody, and runs completed-store checks before executing writer stops.
  3. Fully migrated stores and active restore guards preserve running API containers with zero downtime and zero stops. Only incomplete stores needing legacy migration stop running writers before copying. Standalone prepare and restore calls continue to reject active writers.

Fixes #8020.

AI Assistance

Antigravity provided investigation, with autonomous corrections, independent tests, and macOS lifecycle simulation verification.

Release Lane

  • Stable hotfix targeting release/2.6.x
  • Mainline change targeting main
  • Next-minor work targeting the next feature/minor release
  • Not sure; reviewer should help classify

Stable hotfix reason:

Changed Surface

  • Docs only
  • Tests only
  • Dashboard UI
  • Dashboard API / host agent
  • Installer / bootstrap / lifecycle
  • Docker Compose / service manifests
  • Model routing / Hermes / capabilities
  • Network exposure / auth / proxy
  • Dependencies / runtime wiring

Risk And Validation

  • Risk level: Low
  • Validation run:
    • git diff --check
    • Markdown/link sanity for docs
    • Focused tests listed below
    • Dashboard lint/test/build
    • Extension audit / compose validation
    • Release-grade fleet or scoped hardware validation
    • Stable-lane patch validation, if targeting release/2.6.x
    • Not required because:

Commands/results:

$ bash ods/tests/test-macos-dashboard-private-state-launch.sh
All 6 assertion checks passed:
1. launch stops running writers under the lock and completes migration + launch
2. completed stores start without stopping an already running API
3. unfinished-restore refusal preserves running state without stopping writers
4. foreign consumer validation refuses before issuing any container stop
5. explicit prepare without launch still refuses when writers are running
6. launch when already stopped completes without needing docker stop

$ python3 -m unittest ods/tests/test_macos_dashboard_private_state.py
Ran 39 tests in 16.0s
OK

$ python3 .github/scripts/list-unwired-tests.py --check
[PASS] 1359 tests: 1340 executed by CI (312 through ci-suite.txt), 19 recorded as not run

$ git diff --check
Passed with 0 whitespace or formatting issues.

Operational Change Check

If this PR touches installer phases, bootstrap logic, compose stack generation,
service manifests, dashboard API control flows, Hermes, model routing, GPU or
runtime detection, lifecycle commands, host mutation, or network exposure, it
requires release-grade fleet validation before release unless the PR explains a
narrower equivalent.

  • This is not an operational change.
  • This is an operational change and validation is recorded above.
  • This is an operational change and validation is intentionally deferred for:

Notes For Reviewers

This update addresses advisory review feedback for exact head 79fca4789:

  • Restricts writer stops strictly to launches needing migration inside Stores.prepare().
  • Validates all consumer identities for project ownership before any docker stop is issued.
  • Idempotent launch on completed stores and active restore guards preserve running API containers.
  • Extended regression coverage with 6 launch test scenarios and 4 new unit test cases.

Independent installed validation

Head 7d89de9b preserves the author fix and includes main 7e9d2ed6. The 39 Python tests and six shell scenarios pass. An official Mac Mini install completed with exit 0; 4,608 source files and 15 disabled recipes matched, with no missing files, and the generated Hermes configuration matched its expected installer transformation.

A disposable real-Docker project using the installed API image and synthetic SQLite data passed six paths: current-main launch refusal with a running consumer; standalone prepare refusal; candidate migration and launch; idempotent completed-store start; unfinished-restore refusal; and foreign-consumer refusal. Values survived migration, completed stores and refusal paths retained their metadata, and production containers/configuration stayed unchanged. Only the test containers and volumes were removed. The test API sleeps: this proves running-consumer handling, not concurrent SQLite writing.

After reinstall, the existing Portal conversation executed Python through local Qwen 3.5 9B and returned actual output 684 with exit 0 in 105 seconds. This is scoped acceptance; fleet green remains pending.

@Lightheartdevs

Copy link
Copy Markdown
Collaborator

Review of exact head 79fca4789207d36fbba51361f321717b9d423cd1: I reproduced a stop-order regression using the actual Stores.consumers method with a simulated Docker inventory. If the set iteration visits an owned running dashboard-api first and a foreign consumer second, the candidate issues docker stop for the owned container, then raises private-store-foreign-consumer; main refuses without stopping anything. All consumers need to be inspected and ownership-validated before the first stop.

The launch call also stops writers before prepare checks whether both stores are already complete. That adds downtime to an otherwise idempotent ods start. Stopping should be limited to the migration path, while preserving standalone prepare/restore refusal behavior and checking stopped state before copying.

I am preparing a narrow repair and regression coverage. This is source-level reproduction with fake Docker, not live Mac acceptance; the Mac target is currently behind a Tailscale reauthentication gate.

@Lightheartdevs

Copy link
Copy Markdown
Collaborator

Thanks for addressing migration failures when an older macOS dashboard API is still running.

Advisory status: not ready yet

Quickest path to merge:

  • In ods/installers/macos/lib/dashboard-private-state.py:443, restrict the writer stop to launches that actually need migration, after validating the migration/restore state. The unconditional consumers(stop=True) currently precedes prepare()'s completed-store fast path and refusal checks. I reproduced an unnecessary docker stop on a fully migrated launch --operation start; with image-restore-active.json present, launch stops the API, refuses, and never reaches Compose. Main leaves the API running in both cases.
  • Extend ods/tests/test-macos-dashboard-private-state-launch.sh or ods/tests/test_macos_dashboard_private_state.py with those two running-API cases: completed stores must start without stopping the API, and an unfinished-restore refusal must preserve the running state. Keep the legacy migration stop and standalone prepare/restore rejection coverage.

What I checked:

  • Exact-head diff, lifecycle callers, store-custody checks, and CI wiring; git diff --check passed.
  • New launch shell test and all 35 existing private-state unit tests passed on Ubuntu 24.04/WSL with an isolated Pillow dependency.
  • Additional probes exercised the actual helper with real SQLite stores and a fake Docker command interface against both main and this head; only this head issued the unwanted stops. These are fixture results, not live macOS acceptance.
  • GitHub reports 79 passing checks with no pending checks at review time.

Moving the stop behind the migration decision should keep the intended fix without interrupting an already-ready API.

…igration on launch

When rerunning the macOS installer or performing an update/restart on an existing
deployment upgrading from v3.0.0 (or pre-Oct-15 commits), dashboard-private-state.py
attempts to migrate legacy SQLite stores to Docker volumes. Because dashboard-api is
still running from the pre-existing deployment, Stores.prepare() aborts with
'Private dashboard migration/restore needs stopped writers', breaking automated
install, update, and restart flows.

Restrict writer stopping to launch operations that actually require migration:
- Inspect and validate ownership of all consumers in Stores.consumers() before
  issuing any container stops, ensuring foreign consumers abort immediately without
  halting owned containers.
- Pass stop=True from launch into Stores.prepare() so stops only execute once
  unfinished restore and consumer custody checks pass and an incomplete store is
  identified. Fully migrated stores and active restore guards preserve running API
  containers without unnecessary downtime or premature stops.
- Verify stopped container state before copying SQLite stores to Docker volumes.
  Manual restore and standalone prepare continue to refuse running writers.

Fixes Osmantic#8020
@vaibhavsrv
vaibhavsrv force-pushed the fix/macos-private-store-launch-stop-writers branch from 79fca47 to 94c44da Compare October 24, 2026 17:42
@vaibhavsrv

Copy link
Copy Markdown
Contributor Author

Thanks @Lightheartdevs for the sharp review and reproduction details!

Updated head 94c44da46 implements the exact requested repair:

  1. Restricted stop path: Moved writer stopping inside Stores.prepare(directory, stop=args.action == "launch"). The stop is only reached if a store actually requires migration after passing unfinished-restore (image-restore-active.json) and consumer-custody checks. Launches on already completed stores fast-path without stopping the running API.
  2. Stop-order safety: In Stores.consumers(), all consumers are now inspected and ownership-validated across the inventory before any container is stopped. If a foreign consumer is present, private-store-foreign-consumer is raised immediately with 0 container stops.
  3. Running state preservation: An unfinished restore refusal preserves the running API state without issuing stops. Standalone prepare and restore continue to refuse running writers.
  4. Extended regressions:
    • ods/tests/test-macos-dashboard-private-state-launch.sh now covers 6 assertions (legacy migration stop, running API on completed stores preserved with 0 stops, unfinished restore refusal preserved with 0 stops, foreign consumer refusal preserved with 0 stops, standalone prepare refusal, and stopped-writer completion).
    • ods/tests/test_macos_dashboard_private_state.py adds unit tests verifying consumer validation order before stopping, completed store start without stopping, and active restore refusal preserving running state (39 tests passed).

@Lightheartdevs

Copy link
Copy Markdown
Collaborator

Reviewed the author revision 94c44da4: both reported stop-order issues are addressed. Its 39 Python tests and six launch scenarios pass independently. I preserved those source/test bytes and merged current main 7e9d2ed6 into the branch; the synchronized head is 7d89de9b. The official Mac Mini candidate installer is now running. Real Docker migration/idempotent-start validation and post-install Portal continuity remain pending; this is not yet live acceptance.

@Lightheartdevs
Lightheartdevs merged commit cb034c3 into Osmantic:main Oct 24, 2026
79 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants