Data: preserve embedded email metadata evidence in dedupe decisions

[seonghobae]

Current authority — 2026-09-18

Email identity/deduplication must keep sender-authored metadata separate from collection-time observation. An RFC 5322 Date fallback used for storage must never become strong duplicate evidence, filename dates remain weak display hints, and provider UIDL is collection identity rather than a replacement for Message-ID/source fingerprints.

Canonical safety contract

• persisted/source vocabulary remains date_provenance = parsed | missing | invalid | unknown;
• metadata auto-link requires genuine parsed Date plus complete sender/recipients/subject/body evidence;
• incomplete metadata falls back to raw RFC822 bytes or deterministic canonical-source identity and does not create an automatic metadata link;
• direct import without metadata-strong evidence returns deterministic dedupe_review_required behavior rather than a silent merge;
• Message-ID equality remains an independent strong signal; missing direct-import Message-ID uses deterministic raw-source identity;
• auto_link / review_required / distinct stay separate decision zones;
• import, IMAP and POP3 use one provenance/dedupe boundary;
• POP3 UIDL stays transport/collection identity and does not alter the message-provenance vocabulary;
• migration lineage must converge to one Alembic head.

Canonical owner — #1195

Broad email identity/dedupe/POP3 collection owner is now exact #1195@630d5461d1eb8b59b392e4170dec66ce1edf034c, base #1623@509be4c1d9b6c7ba239a108656e2382681a85341, tree 5ae39dbe676ba3c84b647010182ec9eb0410ef35. It remains Draft/open/mergeable with 26 effective email provenance/dedupe/collection files.

The earlier provenance repairs are preserved in this lineage: zone-less/invalid/missing Date classification, complete-metadata gating, source-bound direct import, shared IMAP/POP3 persistence, deterministic Message-ID/source fallback, conservative auto_link/review_required/distinct decisions, and doctoring. Later POP3 work also fixes imported-count semantics, first-window starvation, per-message negative RETR continuation, and cross-poll starvation through durable owner-scoped UIDL progress (observed | retryable, nullable retry_after, never-attempted-before-due-retry scheduling).

The POP3 collection additions do not introduce a second dedupe vocabulary. Provider UIDL remains transport identity; raw reconstructed message bytes continue to feed the canonical source-provenance/dedupe boundary.

#1656 reconciliation — zero-delta provenance

All still-valid #1656 domain delta is either represented by #1195 or explicitly rejected with an invariant reason. #1656 is ordinary/non-force converged as exact 61920344e987f3405d4963168726be1274215e82 directly on #1195 and currently has 0 changed files / 0 additions / 0 deletions. Keep it Draft provenance until canonical successor evidence is complete; historical approval/checks do not transfer.

The proposed persisted message_id_evidence field remains rejected. Parser/input provenance plus deterministic source identity already covers the current decision consumers; a future persisted field requires a concrete invariant/consumer and a successor migration on the canonical lineage, not a sibling head.

Alembic ownership collision

Source reconciliation is not the same as migration integration. #1195 currently carries branch-local 0018_email_date_provenance → 0019_pop3_observed_uidl, while canonical workspace/database owner #1503 exact 9151c75568c582c8147cfee6757cd00a9b4d60b7 owns 0018_workspace_registry → 0019_email_read_state_repair → 0020_workspace_organization_binding → 0021_workspace_personal_owner_binding. Both descend from historical 0017, so independent merge would create parallel Alembic heads.

Required order is #1503 protected integration first, then ordinary/non-force adoption by #1195 and rechain/renumber of email/POP3 revisions behind the then-current protected head. Fresh and historical PostgreSQL upgrades must end with exactly one Alembic head.

Remaining acceptance gaps

The issue remains open because exact integrated evidence is incomplete:

• current feat(email): deterministic dedupe provenance — gate strong fingerprints on genuine Date (naruon#1086) #1195 630d5461... has zero repository-local pull-request workflow receipts under the stacked-admission defect tracked by fix(ci): make stacked PR validation a develop prerequisite #1691;
• the latest formal CodeRabbit APPROVED review was submitted at 2026-09-17 05:43 UTC and predates the later POP3 retry/progress generation, so it is not qualifying current-head review evidence;
• no exact-current PostgreSQL fresh/historical migration, security, warning-clean full test, coverage, or restart-durability GREEN is claimed;
• the import API/result surface still needs exact integrated verification for review/match reason behavior;
• fix(deps): patch frontend audit security floors #1623, fix(db): make fresh Alembic bootstrap tolerate retired emails schema #1694, fix(ci): make stacked PR validation a develop prerequisite #1691 and fix: create workspace_entities/workspace_documents registry and provision Workspace rows #1503 prerequisite paths must integrate normally before downstream acceptance can be trusted.

Completion criteria

• valid/missing/invalid/unknown Date provenance is deterministic and RFC 5322 edge cases are covered;
• incomplete sender/recipients/subject/body cannot produce metadata-strong auto-link evidence;
• Message-ID-less byte-identical EML uses source identity without promoting synthetic Date;
• import/IMAP/POP3 obey one provenance vocabulary and dedupe policy;
• provider UIDL collection progress remains separate from message identity and survives restart/backlog conditions without starvation;
• API-facing review/match reason behavior is verified against exact current code;
• migration graph is exactly one head and both fresh PostgreSQL and historical upgrade paths execute GREEN;
• exact-current test/security/warning/coverage evidence is GREEN;
• exact-current independent review is qualifying;
• feat(email): deterministic dedupe provenance — gate strong fingerprints on genuine Date (naruon#1086) #1195/fix(email): persist import provenance evidence for dedupe review #1656 valid source/test/doctoring delta is represented only by canonical feat(email): deterministic dedupe provenance — gate strong fingerprints on genuine Date (naruon#1086) #1195 plus zero-delta provenance history.

This is deterministic provenance/dedupe logic. It does not require a new LLM writer, model fallback, or parallel external owner.

[seonghobae]

Implementation successor update: #1656 now carries the provenance slice on canonical dependency-security ancestry instead of stale direct-develop ancestry.

Exact current #1656 head 53cdd97871f558aabdb8d6c826fb977b58969d24 is an ordinary two-parent adoption of predecessor 5891174065ab2540405bb38af76823e916023975 and #1623 17a7618eda2b212b691f08fa936e042b34258fc9. Fresh compare over #1623 is ahead 3 / behind 0 with exactly nine email-provenance files.

Implemented in this slice: Date evidence parsed/missing/invalid, Message-ID evidence embedded/missing, persistence columns/migration, strong fingerprint refusal for incomplete provenance, raw-content fallback identity preservation, dedupe_review_required reason code, filename-date non-promotion, and a PostgreSQL-marked live-schema regression. The broader issue remains open because current stacked-base executable evidence is not yet trustworthy and the issue's complete API review/match-reason contract must still be evaluated as a whole before closure.

A fresh event-time CI provenance defect was also exposed during the retarget and handed to canonical .github#2073; no predecessor run is being promoted as current-base acceptance. #1656 remains Draft and is not a completion claim for #1086.

[seonghobae]

Additional #1086 repair at current #1656 exact head 538f4334ae43c5bb97e2fb76c4e2a9556890c530.

Fresh RFC 5322 review found a provenance hole in the first implementation: email.utils.parsedate_to_datetime() returns a naive datetime both for valid -0000 and for a non-conforming Date that omits the required zone. The prior parser assigned UTC to every naive result while keeping date_evidence=parsed, allowing a zone-less header to become strong fingerprint evidence.

Source-order RED 05e35827ca91a15c21c2f6e3be2498e5a3091fcc requires zone-less Date evidence to be invalid with no strong fingerprint while preserving -0000; causal fix 8ed225a941defc985aa54bbe91e359abc1d5654f distinguishes a present RFC 5322/obsolete zone token before UTC binding. docs/doctoring/email-date-zone-provenance.md at 538f433... records RFC 5322 §§3.3/4.3 and the acceptance boundary.

Fresh compare over canonical #1623 is ahead 6 / behind 0 with 11 effective files. Because the substantive repair was pushed after #1656's feature-base retarget, the current exact head also provides a clean .github#2073 canary: zero PR-triggered product workflow runs. Issue #1086 remains open pending complete API contract/current executable evidence rather than being closed on source-only progress.

[seonghobae]

#1656 successor evidence update: fresh exact-range review found and the branch repaired a remaining provenance gap.

At predecessor 538f4334..., _email_fingerprint() correctly returned None when a required metadata field was blank even if date_evidence="parsed", but the import result only returned dedupe_review_required when Date evidence itself was non-parsed. Thus an imported record that bypassed strong metadata dedupe could be presented without a review reason.

RED c278b1a7e5ae2b404aaab86046da52169090e30b adds a parsed-Date import with a blank required fingerprint field and requires dedupe_review_required; it also independently locks nullable=True for both provenance migration columns. Minimal production fix a6ad38a76c8c5544eb313a485e78c1c4c8310189 classifies review from fingerprint is None. Doctoring head e8ce0c731631bf8816f13eb873ec34aecc950842 records the lineage. The two CodeRabbit findings are resolved.

Issue remains open: #1656 exact head still has zero stacked-base product workflow runs, the PostgreSQL migration path lacks trustworthy current-base/head execution, current-head qualifying approval is pending, and broader API review-state/match-reason acceptance in this issue is not yet complete.

[seonghobae]

Retry-owner evidence update from #1654: exact head is now cf1e6220bcc5ad7f2927d7f0fb86b155dd91a0b3 on #1623@17a7618.... Fresh review found the unit fake ignored the production due-time predicate and could immediately reselect a row that had just been rescheduled into the future. Harness repair 050e5f0... now preserves next_retry_at <= current_time and due ordering; the transient-reschedule regression explicitly requires one processed row, one reschedule, zero exhaustion, and one commit. Production FOR UPDATE SKIP LOCKED / per-item claim source is unchanged by this repair.

This does not close #1086: exact feature-base workflow evidence remains absent, the PostgreSQL concurrency tests need hosted execution on the final base/head, and crash-after-remote-side-effect exactly-once/idempotency remains outside the current claim contract.