feat: stop writing the legacy manifest, add a one-shot migration

[mspinola]

Completes the second half of ADR-0007 step 1.

Why now rather than after a clean week

The legacy manifest.json held both producer halves in one file, which was unsafe two ways:

• The update is a read-modify-write, so two producers lose each other's entries.
• A file-level sync between two stores resolves it last-writer-wins and silently discards one side.

The second stopped being theoretical when the Windows producer moved from a \\tsclient\ redirected drive to a local store. That turned one shared store into two copies merged by a sync, and a sync pushing a manifest.json containing only price entries would drop every COT entry on arrival.

The per-half files are disjoint, so a file sync merges them correctly by construction.

What changed

Nothing writes manifest.json. _touch_manifest writes only the owning half.

cotdata-update --migrate-manifests splits an existing aggregate into the per-half files. Idempotent, entries already in a half file win so a re-run cannot resurrect stale bookkeeping, and it never touches data.

Until a store is migrated, a domain absent from the per-half files is still read from the aggregate, with a warning naming the domains and the command to run.

reconcile_manifest rewritten to prune each manifest file in place rather than the merged view. It previously wrote its result to the legacy aggregate, which would now be a write to a file nothing reads. It still prunes the aggregate when present, so an un-migrated store can be cleaned without migrating first.

A bug caught by dry-running against the real store

My first version made the fallback per half. Dry-run on a copy of the live store:

BEFORE: {'prices': 94, 'cot_disagg': 27, 'cot_legacy': 95, 'cot_tff': 24}
                                                          ^ metadata MISSING

manifests/prices.json held prices but not metadata, because the price producer had run on the new code while the metadata producer had not. A half file existing does not mean the half is complete, so treating it as migrated hid metadata entirely until the migration ran.

The fallback is now per domain. Same store, after the fix:

BEFORE: {'prices': 94, 'cot_disagg': 27, 'cot_legacy': 95, 'cot_tff': 24, 'metadata': 1}
AFTER : {'prices': 94, 'cot_disagg': 27, 'cot_legacy': 95, 'cot_tff': 24, 'metadata': 1}

241 entries either side. There is a regression test for it.

Verification

Migration dry-run on a copy of the live store moved 147 entries (cot +146, prices +1) with nothing lost. Suite 134 passed, ruff clean.

Operator note

Run once per store after upgrading:

cotdata-update --migrate-manifests

Two stores means running it on both. manifest.json can be deleted once every consumer of that store is on this version.