fix(memory): preserve case-sensitive engineering meaning in deduplication identities

[joshyorko]

Existing-corpus repair amendment — September 8, 2026

Future identity behavior and already-collapsed records are both required scope. Use #234's existing-corpus preview/reconciliation contract; no separate repair framework and no silent live migration.

Core guarantees and repair classification

• Audit supported pre-change DBs and historical Codex/Hermes/import/bundle shapes for legacy normalized-hash groups, retained originals and source mappings. A group is a potential collision, not proof that every member is damaged. Classify unchanged, confirmed distinct-and-recoverable, potentially conflated, conflicting, and irrecoverable/unknown with bounded counts and references.
• Recover distinct case-sensitive commands, paths and structured whitespace only from retained originals or explicitly authorized source artifacts that actually support them. Preserve the exact available source representation, speaker/producer, source time and import/bundle origin separately from current claim status. A summary is not a recovered verbatim original; neither a hash nor an LLM can recreate lost text reliably.
• Missing lineage/times do not disqualify useful old content or establish verification. Preserve the existing legacy object and its stable reference/meaning. For a reviewed split, create supported versioned successor objects/links and disclose ambiguous old references; never repoint one historical reference silently to a guessed recovered variant.
• Version exact-identity rules and define legacy/new dual lookup so new distinct content does not hit the old lossy equality fallback. Exact retries still dedupe. Source reconciliation does not count as new corroboration, fresh verification or user adoption.
• Apply existing corrections/withdrawals to proven equivalent recovered material via feat(memory): make corrections and forgetting survive derived views, replay and import #234. Recovered evidence is not automatically current truth, and unknown equivalence is not a license for broad deletion or promotion.

Preview/apply and rollout requirements

Preview must show affected records and stable references, proposed identity mappings/splits, recovered-versus-missing originals, proposed status changes, conflicts and unresolved cases before apply. No silent bulk re-keying, overwrite or removal of an old record. Preserve origins across bundle re-export/import, backup/restore and supported readers. Bind the plan to DB/schema/policy and affected revisions; retry/resume is idempotent. Stale plans conflict. Any rollback must preserve later legitimate writes and known withdrawals rather than restore the whole DB over them.

Adapters consume supported old/new identities and expose uncertainty without reconstructing provenance. Consumers first run the audit on an isolated pre-change snapshot with verified backup, approve a scoped plan, validate references/recall after apply, and report what could not be repaired. No claim that installing the new binary fixes the live corpus.

Additional acceptance gates

• Build a sanitized DB using the pre-change normalization/upsert behavior, including -d/-D, case-sensitive paths and indentation collisions; retain both originals for some groups, only one for others, and no recoverable originals for others.
• Include transferred origins, historical assistant imports, missing metadata and useful other records. Audit differentiates proven loss from potential ambiguity and preserves otherwise permitted legacy recall.
• Preview/cancel is non-mutating; apply/safe retry/restart preserves old references and truthful successor mappings. Concurrent correction invalidates conflicting plans.
• Recover only source-supported variants; hashes-only cases remain explicitly unresolved, not invented or silently dropped.
• Exercise new writes into legacy hash groups, old archive reimport, bundle round trip, correction and pre/post-change restore reconciliation. No new collapse, duplicated independent evidence, false freshness or known-withdrawn resurrection.
• Report recovery coverage and limits; pristine-record tests alone cannot close this issue.

---

Outcome

Distinct commands, paths, identifiers, and structured snippets must not be silently collapsed into one memory merely because prose-normalization makes them look alike.

Priority: P1. Source-proven identity collision; end-to-end impact and migration must be demonstrated in a regression test before changing storage.

Evidence at current master

Baseline 25d617b5f08593c76d336f90b8ffd1e78f77afbd.

src/ids.rs normalize_for_hash() lowercases all content and collapses every whitespace run. content_hash() uses that normalized text for every record type. Its test explicitly asserts case-insensitive equivalence.

src/store.rs documents upsert_record() as keyed on this hash: an existing match returns Skipped and merges source IDs.

Holding profile/workspace/repo/type/scope constant, these have the same current identity:

• Use /srv/Alpha/config.yaml versus Use /srv/alpha/config.yaml.
• Use git branch -d feature versus Use git branch -D feature.
  Whitespace normalization can also erase meaningful indentation/newlines in structured examples. This is a semantic identity collision, not a cryptographic SHA-256 collision.

Design constraints

• Separate exact content/event identity from optional search/prose-similarity normalization.
• Version any canonicalization change. Do not silently re-key all existing data, break bundle origins, or change immutable public handles.
• Preserve source attribution: similarity is not independent corroboration; repeated upserts are not fresh verification.
• Choose a small backwards-compatible migration/dual-lookup policy after examining current store, import, Dreamer, bundle, and recall usage.
• Existing already-collapsed information cannot be reconstructed from hashes alone. Report the uncertainty; do not invent historical source contents.
• Avoid an LLM-based deduplicator or a new storage backend.

Acceptance

1. Service/store round trip preserves both case-sensitive examples as distinct engineering records with correct provenance.
2. Identical safe retries remain idempotent.
3. Unicode, case-folding, Markdown code spans/fences, structured whitespace, and path cases have explicit fixtures.
4. Intentional plain-prose normalization behavior is documented and tested separately from immutable identity.
5. Open old DB, preview migration, cancel, apply, restart, and rollback/restore preserve records and references.
6. Bundle re-export/import preserves origin mappings; exact-id replay cannot become duplicate independent evidence.
7. A source-only duplicate import must not make an old fact appear newly verified.

Touchpoints

src/ids.rs, src/store.rs, callers in ingestion/service/Dreamer, portable bundle conformance, identity/migration tests. Inspect before expanding changes. Link migration decisions into the existing spec, not a second identity framework.

[joshyorko]

Independent minimal reproduction

Executed an ASCII-only Python reproduction of the exact src/ids.rs normalization/hash construction at 25d617b5f08593c76d336f90b8ffd1e78f77afbd (not the Rust service/store). With identical identity fields, all three distinct payload pairs produced equal hashes:

• /srv/Alpha/config.yaml versus /srv/alpha/config.yaml in the same sentence.
• git branch -d feature versus git branch -D feature in the same sentence.
• A two-statement Python example with the second statement outside versus inside an if block.

Minimal formula for these ASCII fixtures:

normalized = " ".join(content.strip().lower().split())
fields = [profile, workspace, repo_id, record_type, scope, normalized]
digest = hashlib.sha256("\x1f".join(fields).encode()).hexdigest()

This confirms the identity-function collision for the listed cases. The next implementation step remains a Rust service/store RED test demonstrating actual upsert behavior and a separately reviewed migration plan. It is not a claim that production records were inspected or that a fix has been applied.