fix(storage): persist the pre-compression neuron snapshot instead of dropping it

[RobertSigmundsson]

Summary

• save_neuron_snapshot now sends compressed_at as the string the schema declares, so the pre-compression snapshot actually reaches the database instead of being refused.
• Adds a live regression test covering both write paths (first insert, and the merge used on a second write for the same neuron).

Why

neuron_snapshots.compressed_at is defined TYPE string, but the storage layer wrote a datetime into it. On a SCHEMAFULL table SurrealDB refuses the write:

InternalError: Couldn't coerce value for field `compressed_at` of `neuron_snapshots:…`:
Expected `string` but found `d'2026-08-19T18:34:06.419075Z'`

The only caller - CompressionEngine, saving originals before the destructive tiers - wraps the call in a fail-soft except that logs and then compresses anyway. So the failure was invisible in normal operation: tier 3-4 compression replaced a neuron's content while the snapshot meant to protect it silently never existed, and recover_neuron_content had nothing to restore from. The counterpart table compression_backups.compressed_at is TYPE datetime and receives a datetime; only this one column disagreed with its writer.

Sending the declared type is the smaller of the two possible fixes and the one that works everywhere. The alternative - redefining the column as datetime, which would read more naturally and match compression_backups - cannot be delivered by editing the schema alone: ensure_schema swallows already exists on a re-DEFINE, so an existing database would keep the string column and stay broken. That route needs a numbered migration and a SCHEMA_VERSION bump, which is a larger change than this fix and touches every deployment; I have deliberately left it out rather than fold it in here. Happy to follow up with it if you would prefer the column normalised.

Two details worth noting for review:

• No data migration is needed. The field definition and its writer were introduced in the same commit (1f6fe80d, Orphan-prune: pinned fixed (PR #17), but should isolated neurons get the same age/access grace as dead ones? #28) and neither line has changed since, so no neuron_snapshots row has ever been written on a SurrealDB backend. There is no legacy value in the column to be compatible with.
• The read path is untouched. get_neuron_snapshot already parses through _parse_datetime, which accepts both a string and a datetime, so callers keep receiving a datetime exactly as before.
• One inherited quirk, now observable. _parse_datetime drops tzinfo without converting to UTC, so an offset-bearing input would be stored as local wall time labelled UTC. That is pre-existing behaviour and the only in-tree caller passes utcnow().isoformat(), which is already naive UTC - but this value reaches the database for the first time with this fix, so it is worth stating rather than leaving to be discovered.

Test plan

• pytest tests/ -m "not stress" -n auto passes locally: 7079 passed, 148 skipped, 1 xfailed. The skip count is +2 against the same run on main - precisely the two new tests, which skip when SURREALDB_URL is unset.
• ruff check src/ tests/ clean; ruff format --check clean.
• mypy src/ --ignore-missing-imports - Success: no issues found in 353 source files.
• The new tests were proven to fail without the fix: with the one-line change stashed, both fail against a live engine with the coercion error quoted above; with it applied, both pass.
• Verified end to end against a real engine (surrealdb:v3.2.4, ephemeral instance, real ensure_schema): before the fix get_neuron_snapshot returned None and SELECT count() FROM neuron_snapshots GROUP ALL returned 0; after it, the snapshot reads back with its content and tier intact and the count is 1.
• The test is placed with the other live-engine tests and gated on SURREALDB_URL, so it runs in the integration job and skips elsewhere; its throwaway brain name is registered with the shared cleanup helper so it does not accumulate rows.

A note on why the test needs a live engine: this failure is a schema coercion refusal, and the in-memory backend stores snapshots in a plain dict with no schema, so the mismatch cannot surface there. That is also why nothing caught it earlier - before this change, save_neuron_snapshot had no test coverage at all.

Verified by

@RobertSigmundsson

---

Rebased onto v3.8.0 (ae8e8743) and re-tested there. #186 landed in the same area - the
neuron_state write - but the paths are disjoint: this one is the snapshot row in
neuron_snapshots, which #186 does not touch. Thanks for the quick turnaround on both
v3.7.0 and v3.8.0.

[acidkill]

Reviewed against main@ae8e8743, verifying the claims independently rather than taking the description at face value.

Diagnosis confirmed. neuron_snapshots.compressed_at is TYPE string DEFAULT '' (schema.py:396), while the sibling compression_backups.compressed_at is TYPE datetime (schema.py:385) and its writer still correctly sends a datetime. The fix is scoped to the one column whose writer disagreed with it, with no sibling regression.

Checks performed:

• Both write paths covered. insert and merge share the same record_data dict, so .isoformat() applies to the upsert branch too, not only the first insert. Worth stating because a fix applied to just one branch would have looked identical in the diff.
• Read path unchanged. get_neuron_snapshot still routes the value through _parse_datetime, which accepts str and datetime alike, so callers keep receiving a datetime.
• No other writer. Nothing else in the tree writes that column, and no query orders or compares on it, so storing an ISO string has no downstream effect.
• Fail-soft caller confirmed. The only caller wraps the call in except Exception: logger.error(...) and compresses anyway, which is exactly why the refusal was invisible.
• Test hygiene. The throwaway brain name is registered in LIVE_TEST_BRAIN_NAMES, so the live test cleans up after itself.

The "no data migration needed" argument holds: the write never landed, so there is no legacy value in the column to stay compatible with.

The reasoning for preferring the writer-side fix over redefining the column is also correct — ensure_schema swallowing already exists on a re-DEFINE means a schema edit alone would leave existing databases broken. Agreed that normalising the column belongs in a numbered migration rather than folded in here.

No findings. Approving.