feat(catalog-ui): explore bucket-backed catalogs

[VARUN3WARE]

Summary

• Allow hflow catalog ui --catalog to open existing S3, GCS, and Azure catalogs.
• Validate the remote catalog marker without creating or changing bucket objects.
• Sync append-only Parquet files into the existing local mirror before DuckDB opens.
• Poll for first and later completed appends without restarting the UI.
• Keep DuckDB on loopback and restrict catalog reads to the local mirror.
• Add behavioral tests and document bucket usage, mirroring, and SSH forwarding.

Closes #305.

Why

hflow catalog ui previously rejected every bucket URL even though HFlow's catalog connection already supported bucket-backed catalogs through its local mirror.

This change reuses the existing catalog marker, append, and mirror semantics instead of adding direct object-store access to DuckDB or introducing a new catalog format. Local catalog behavior remains unchanged, while bucket browsing stays read-only.

Validation

• uv run ruff check
  • Passed
• uv run ruff format --check
  • Passed; 206 files already formatted
• uv run ty check
  • Passed
• uv run pytest tests/test_catalog_ui.py tests/test_cli.py tests/test_storage.py -q
  • 92 passed, 1 skipped
• uv run pytest -q
  • 1402 passed, 6 skipped
• Pre-commit hooks
  • ruff check passed
  • ruff format passed

Checklist

• I added or updated outcome-focused tests for changed business logic.
• I updated documentation for changed behavior, flags, formats, or requirements.
• I ran uv run ruff check --fix, uv run ruff format, and uv run ty check.
• I ran the relevant pytest suite.
• I did not add recordings, generated media, credentials, private URLs, or runtime artifacts.
• I preserved stored-data compatibility or documented an explicit version change.

[github-actions]

👋 Hi @VARUN3WARE — thank you so much for your first contribution to HFlow!

A maintainer will review your pull request as soon as possible. In the meantime:

• ⭐ Star the repo: https://github.com/Hebbian-Robotics/hflow
• 💬 Join our Discord: https://discord.gg/vacepQvjmg
• 📖 Check CONTRIBUTING.md for the validation commands CI will run
• 📣 Share the project with your network

💡 Tip: one open pull request per contributor at a time. Issues with an assignee are taken; everything else is fair game.

We are excited to have you here and appreciate your help making the project better! 🙌

[kstonekuan]

Strong first contribution, and an advanced issue for a first PR at that. The design decision is the right one: reusing _local_query_root's existing mirror sync rather than giving DuckDB object-store credentials means SQL never gets a write surface, which is DoD's loopback requirement satisfied by construction rather than by a check.

I verified the read-only claim harder than your test does. Yours compares key names; the DoD says "creates or changes", so I fingerprinted every object by sha256 and size before and after a browsing session:

objects before browsing: 7
objects after browsing:  7
created: []   removed: []   changed in place: []
VERDICT: unchanged

And the startup line prints Catalog: file:///.../bucket/catalog, not the mirror path, so that DoD item holds too.

One thing to fix, and it is about the tests rather than the code.

The two mirror syncs are the mechanism for "visible without a restart", and nothing holds them. Both can be deleted with the whole suite still green:

post-append mirror sync removed        -> 48 passed, nothing noticed
first-append detection stops syncing   -> 48 passed, nothing noticed

The cause is that _append_remote_episode appends through the same BucketStorageRoot the UI reads, so the writer and the reader share one mirror and the append primes it directly. The test then observes a real row-count change and passes, but it cannot distinguish "the UI synced the bucket" from "the writer already put the file in the UI's mirror". In production the ingest is a different process with its own mirror, and the sync is the only path.

Giving the writer its own mirror over the same bucket makes the difference visible. I ran your later-append flow that way:

                              episodes visible after a separate-process append
as submitted                  (2,)   sync worked
post-append sync removed      (1,)   NOT VISIBLE

So the code is right and only the fixture is too generous. Something like this is enough:

writer_root = BucketStorageRoot(f"file://{remote_dir}", mirror=tmp_path / "writer-mirror")
Catalog(writer_root.child("catalog")).append_episode(...)

with the UI on its own BucketStorageRoot. Worth doing for both the first-append and later-append tests, since neither currently holds its sync.

Three smaller notes, none blocking. FileNotFoundError in the except tuple is already covered by OSError. Importing obstore.exceptions inside the handler means a bucket URL that fails for some unrelated reason without the bucket extra installed raises ModuleNotFoundError from the handler and masks the original error; importing it at the top of the branch where you already know the extra is present would avoid that. And a 0.5-second sync cadence against a real bucket is a LIST plus GETs every half second for as long as the UI is open, which is fine for a laptop session and worth a word in docs/CATALOG.md next to the mirror behaviour you documented.

Gate is clean otherwise: ruff check, ruff format --check, ty check, 1420 passed / 6 skipped with current main merged in.

[VARUN3WARE]

Thanks for the careful review, especially the fingerprint check and catching the shared-mirror fixture.

Addressed in 7b4770c:

• First- and later-append tests now append through a separate BucketStorageRoot with its own writer mirror over the same bucket, so the UI sync path is what makes new rows visible.
• Dropped the redundant FileNotFoundError catch.
• Import obstore.exceptions before the handler so a missing extra can’t mask an unrelated failure.
• Documented the 0.5s poll LIST/GET cadence next to the mirror behavior in docs/CATALOG.md.