[BUG] CoW versioned-snapshot publish fails (HTTP 400 PATH_NOT_ALLOWED): daemon_storage_path null -> dest path not translated from NFS mount to daemon-local

[jsbattig]

Bug Description

add_golden_repo_index index_type=scip (and any operation that publishes a CoW versioned snapshot) FAILS at the snapshot-publish step with HTTP 400 PATH_NOT_ALLOWED from the CoW daemon, because cow_daemon.daemon_storage_path is null in ~/.cidx-server/config.json on all cluster nodes. SCIP generation itself SUCCEEDS (scip-python produced a valid 11MB index.scip.db and scip queries return real results); only the post-generation immutable-snapshot publish fails.

Root Cause (proven via daemon-payload replay)

src/code_indexer/server/storage/shared/clone_backend.py: the constructor sets self._daemon_storage_path = (config.daemon_storage_path or "").rstrip("/") -> "" when daemon_storage_path is null. _translate_to_daemon_path then early-returns the CIDX path UNCHANGED (if not self._daemon_storage_path: return cidx_path, ~line 370-371). So the destination is sent to the daemon as the NFS mount path /mnt/cow-storage/.versioned/<repo>/v_... instead of the daemon-local /home/jsbattig/cow-storage/.versioned/.... The daemon's _validate_dest_path (core/clone_manager.py) does resolved.relative_to(<daemon base_path>), which fails for the mount path -> PathNotAllowedError -> api/routes.py returns HTTP 400 PATH_NOT_ALLOWED.

Replay proof (on node .23): POST /api/v1/clones with the untranslated /mnt/cow-storage/... dest -> HTTP 400; with the daemon-local /home/jsbattig/cow-storage/... dest (what translation should produce) -> HTTP 202 accepted. (Probe clone deleted immediately, non-destructive.)

Environment

• Staging cluster v11.27.0, 3 nodes. cow_daemon.daemon_storage_path = null in config.json on .20/.22/.23. CoW daemon on .23:8081, base_path /home/jsbattig/cow-storage, NFS-mounted at /mnt/cow-storage on the nodes.

Expected Behavior

CoW versioned-snapshot publish should translate the CIDX/NFS-mount path to the daemon-local storage path so _validate_dest_path accepts it, and the immutable snapshot publishes (HTTP 202).

Fix Direction

Either: (a) populate cow_daemon.daemon_storage_path = "/home/jsbattig/cow-storage" on cluster nodes so _translate_to_daemon_path maps /mnt/cow-storage -> the daemon base_path — wired durably via the auto-updater idempotent _ensure_X_config pattern (config bootstrap); OR (b) harden clone_backend.py so an empty daemon_storage_path still translates when mount_point differs from the daemon storage root (derive the daemon path from mount_point). Prefer wiring the config for correctness + a code guard so a null value fails loud rather than silently sending an untranslatable path. Then re-verify a versioned-snapshot publish returns 202 and the SCIP add-index job completes.

Impact

Severity: priority-2. Blocks CoW versioned-snapshot publishing (SCIP add-index jobs report FAILED; likely affects immutable-snapshot creation for other index types/activations too). SCIP generation + querying works (mutable golden-repo index), so not a total SCIP outage, but add-index JOBS face-plant and no immutable snapshot is produced. Independent of the v11.27.0 Node.js/timeout changes. Surfaced during #1318 SCIP verification.

[jsbattig]

Fixed on development via /implement-backlog (full tdd -> code-review -> gates). Two commits:

Part A 38cf5bb6 — fail-loud guard: CowDaemonBackend._translate_to_daemon_path now RAISES a clear ValueError (naming cow_daemon.daemon_storage_path + the offending path) when a CIDX path is under mount_point but daemon_storage_path is empty/null, instead of silently emitting the untranslatable NFS path the daemon rejects with HTTP 400. Genuine no-op cases pass through unchanged; solo/local never enters this path.

Part B 7ed406d1 — populate daemon_storage_path durably in BOTH lifecycles (per request, most robust):

• Installer (install-cidx-server.sh): --cow-daemon-storage-path flag + CIDX_COW_DAEMON_STORAGE_PATH env + auto-detect from co-located /etc/cow-storage-daemon/config.json base_path; writes the field into the bootstrap cow_daemon block only when resolved (never null); value-aware merge preserves an existing setting.
• Auto-updater (deployment_executor.py): _ensure_daemon_storage_path (Step 14.6), value-aware idempotent — writes only when missing/empty, no-op when correct, NEVER clobbers a validly-set different value; resolves env -> daemon-config base_path -> leave unset + WARNING (never a bogus value). Self-gates to clone_backend == cow-daemon.

No hardcoded env-specific path (only literal is the fixed, env-overridable daemon-config FILE path).

Gates: fast-automation 12392/0; server-fast all 6 chunks PASS/0; lint clean; code review APPROVED. 9 new unit tests + 3 installer-validator assertions.

Remaining (staging, operator-driven): live re-verification that a versioned-snapshot publish now returns 202 and the SCIP add-index job completes — requires the auto-updater to deploy this to the cluster nodes (no CoW daemon locally). Closing as code-complete; reopen if staging validation surfaces anything.