fix(cua): harden /v1/predict profile handling on the Modal volume (stale-view EPERM + executor boto3)

[mercurialsolo]

Summary

Two related reliability fixes for the /v1/predict profile/snapshot path, both surfaced by the same investigation (intermittent 500s on the canary + degraded snapshotting on the holo3 executor):

1. Modal-volume stale-view mkdir EPERM → intermittent POST /v1/predict -> 500

A long-lived container's /data mount can lag the backend: a tenant dir another container committed under /data/tenants isn't visible in this container's cached view yet, so mkdir(parents=True, exist_ok=True) tries to create it and the backend rejects the conflicting create with EPERM ("Operation not permitted"). exist_ok can't rescue it because is_dir() reads the same stale view.

Fix: paths._ensure_dir() catches the EPERM, calls vol.reload() to resync the mount, and retries once. The Modal entrypoint registers vol.reload via set_volume_reload_hook, so every container (API + executors) recovers. Off Modal (no hook) the original error re-raises — no masking.

2. boto3 + zstandard missing from the executor images → snapshot lifecycle degraded

ProfileSnapshotter.from_env runs in the executors, not just api_image — but executor_image, claude_executor_image, and run_holo3's own inline image lacked the deps. So snapshot save/restore failed with requires boto3 on every holo3 run, and named reusable profiles (login reuse) silently didn't persist. Added as trailing pip layers so the heavy vllm/torch + llama.cpp layers stay cached (per feedback_modal_pip_layer_split).

Tests

tests/test_paths_volume_reload_retry.py — 4 tests, all passing:

• EPERM → reload hook fires → retry succeeds
• no hook (off Modal) → original error re-raises
• persistent EPERM → reload attempted once, then re-raises (not masked)
• happy path → no reload when mkdir succeeds first try

Verification

• Unit tests green (4 passed).
• Deployed to mantis-cua-server; snapshot capture now succeeds on the holo3 path (profile_snapshot capture ... outcome=captured size=266MB), and 3/3 real /v1/predict submits on fresh containers returned 200 with no EPERM.

🤖 Generated with Claude Code