feat: reconcile the ingest manifest in BOTH directions

[mspinola]

Stacked on #40 (databento-ingest-hardening), branched from ce63560.

The gap

reconcile_manifest() already fixes a manifest that has fallen behind the disk: an interrupted run leaves fetched tables unrecorded and a restart re-downloads them.

The mirror image was unhandled, and it is the worse one. ingest() derives each table's start from the manifest and skips it when already current:

rec = manifest.get(key, {})
start = rec["last_date"] + 1 day if rec else cold_start
if start > end: continue          # "already current"

It never checks the parquet exists. A manifest entry whose file is gone (partial copy, store move, deleted file) is therefore treated as current and never fetched again. No error, no rows, a permanent hole in a paid dataset.

The asymmetry matters: manifest-behind costs money re-downloading data you already have. Manifest-ahead leaves you believing you have data you do not.

Change

reconcile_manifest() now prunes entries with no parquet on disk, so the next run re-fetches them. Returns {"recorded": {...}, "pruned": [...]}. prune=False keeps the old behaviour.

--reconcile-databento reports the prune separately and states what would otherwise have happened:

databento reconcile: PRUNED 3 manifest entries with no parquet on disk, across 2
symbol(s). The next --ingest-databento will re-fetch these; without the prune it
would have skipped them as 'already current'.

Important: this is a between-runs repair

The manifest legitimately runs ahead of the disk while a batch ingest is in flight, because the batch API completes and then lands its files in bulk. Observed on a live run: 11 files at one moment, 145 ten minutes later.

So do not run this against a live ingest. It would prune entries whose files are still landing, and the running process holds the manifest in memory and would overwrite the result anyway.

Tests

Three new, covering both directions plus the opt-out. Suite 121 passed, ruff clean.

Note the fixture had to write ts_event as the index, not a column, matching what _append_raw produces. A column-based fixture passes construction and then fails in _date_bounds.

Development note

Built in a git worktree rather than by checking out this branch, because a --ingest-databento run has been live in the main checkout for 19 hours and cotdata is an editable install there, so a branch switch would have changed the code under a running paid job.