From d9485287b0115d0b9dbbd0bef9d9144fd416e72c Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Tue, 4 Aug 2026 15:34:45 +0900 Subject: [PATCH 01/86] feat(email): preserve dedupe provenance from genuine Date headers --- CHANGELOG.md | 16 +- .../versions/0018_email_date_provenance.py | 49 ++++ backend/db/models.py | 7 + backend/services/email_dedupe_service.py | 163 +++++++++++ backend/services/email_import_service.py | 22 +- backend/services/email_parser.py | 74 +++-- backend/services/imap_worker.py | 21 +- backend/tests/test_email_dedupe_service.py | 255 ++++++++++++++++++ backend/tests/test_email_import_service.py | 39 +++ backend/tests/test_email_parser_provenance.py | 127 +++++++++ 10 files changed, 736 insertions(+), 37 deletions(-) create mode 100644 backend/alembic/versions/0018_email_date_provenance.py create mode 100644 backend/tests/test_email_parser_provenance.py diff --git a/CHANGELOG.md b/CHANGELOG.md index a06003d8f..1d6a7a7f5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,11 +1,21 @@ ## [Unreleased] +### 이메일 메타데이터 provenance 기반 (dedupe 근거 분리 · naruon#1086 파서 계층) + +- `backend/services/email_parser.py`가 RFC822 `Date`·`Message-ID` 근거를 명시적으로 노출합니다. `date_provenance`(`parsed`/`missing`/`invalid`)와 원본 헤더 날짜(`header_date`, 부재·파싱 실패 시 `None`)를 저장용 `date`(파싱값 또는 수집 시각 fallback)와 분리하고, `message_id_provenance`(`embedded`/`missing`)를 추가했습니다. 합성 수집 시각이 원본 발신 메타데이터로 오인되지 않으므로 fingerprint dedupe가 진짜 발신 근거에만 의존할 수 있습니다. `date`의 기존 의미(파싱값-또는-fallback)는 그대로 유지되어 하위 호환입니다. +- `email_import_service._email_fingerprint`가 이제 `date_provenance == "parsed"`일 때만 strong(자동 중복 판정용) fingerprint를 생성합니다. `Date` 헤더가 없거나 잘못돼 `persisted_date`가 합성 수집 시각인 경우 strong key를 만들지 않고 weak fallback fingerprint(수집 시각이 매번 달라 거짓 중복을 만들 수 없음)로 내려갑니다 — 합성 시각이 strong-duplicate 근거로 승격되지 않습니다. `persisted_date`는 파서의 `date`에서만 오고 업로드 파일명에서 오지 않으므로, 날짜형 파일명이 `Date` 근거로 승격되지 않는 계약도 구조적으로 유지됩니다. +- stored-side와 IMAP 수집 경로까지 provenance gating을 완성했습니다. `email_records.date_provenance` 컬럼(모델 + Alembic `0018_email_date_provenance`, 기존 행은 `"unknown"`으로 안전 backfill)을 추가하고, `email_dedupe_service.email_strong_fingerprint(email_row)`는 `date_provenance == "parsed"`인 행만 strong fingerprint를 만들며, `imap_worker`도 import 경로와 동일하게 parsed Date일 때만 strong fingerprint를 seed합니다. import·IMAP 두 생성 경로 모두 파서 provenance를 영속화합니다. backfill이 `"unknown"`이므로 기존 행은 strong 근거에서 제외되어(보수적: 중복을 만들 수 없고 review만 넓힘) 안전합니다. +- 검증: 파서 provenance 로직을 5개 시나리오(valid/missing/invalid `Date`, embedded/missing `Message-ID`)와 whitespace-only `Date` 분기까지 검증하고, incoming·stored 양측 strong-fingerprint provenance gating 회귀 테스트를 추가했습니다. alembic single-head 가드·email-model reconciliation·전체 email/dedupe/import/imap 스위트 `333 passed, 1 skipped`(`PYTHONWARNINGS=error`), ruff clean으로 통과했습니다. 남은 것은 불완전 근거 import를 `dedupe_review_required` 상태·reason code로 API에 노출하는 UX 계층으로, 후속 원자적 PR로 이어집니다. +- 위 UX 계층의 **도메인 판정 코어**를 `email_dedupe_service`에 선행 구현했습니다. Fellegi & Sunter(1969) record-linkage 3구간 결정 규칙(A1 positive link · A2 possible match/clerical review · A3 non-link)을 이메일 dedupe에 대응시킨 `classify_dedupe_decision(candidate, existing_row)`가, (a) 신뢰 가능한 동일성 링크(정규화 `Message-ID` 일치, 또는 양측 `parsed` Date의 genuine strong fingerprint 일치)는 `auto_link`, (b) provenance-독립 content fingerprint(`sender`+`subject`+`body`, Date 제외)는 일치하지만 신뢰 링크가 없는 — 즉 한쪽 Date provenance가 합성/`unknown`이라 strong key가 보류된 — 개연적 중복은 `review_required`(임의 병합·임의 보존이 아니라 clerical-review 구간으로 격리), (c) 그 외는 `distinct`로 분류합니다. Date 비의존 식별 신호인 `content_email_fingerprint`와 `EmailDedupeCandidate.date_provenance` 필드(기본값 `"unknown"`, 하위 호환)를 함께 추가했습니다. 이 순수 함수 계층은 후속 PR에서 `dedupe_review_required` 상태·reason code 및 리뷰 큐 API로 표면화됩니다. 검증: 3구간·content fingerprint 전 분기 회귀 테스트를 추가하고 전체 email/dedupe/import/imap 스위트 `94 passed`(`PYTHONWARNINGS=error`), ruff clean으로 통과했습니다. + - 근거 문헌(APA 7): Fellegi, I. P., & Sunter, A. B. (1969). A theory for record linkage. *Journal of the American Statistical Association, 64*(328), 1183–1210. https://doi.org/10.1080/01621459.1969.10501049 +- 도메인 코어에 **1:N 결정 해소**와 두 건의 정합성 수정을 더했습니다. (1) `email_dedupe_service.resolve_candidate_disposition(candidate, existing_rows)`는 후보 하나를 저장된 여러 행 집합과 비교해 Fellegi & Sunter(1969) 구간 우선순위 A1>A2>A3로 단일 disposition으로 축약합니다(첫 `auto_link` 행이 즉시 우선, 없으면 첫 `review_required` 행을 clerical review로 격리, 아무 신호도 없으면 `distinct`). 링크/리뷰를 유발한 저장 행을 함께 반환해 import/IMAP 경로가 대상 이메일을 다시 유도하지 않아도 되게 했습니다 — 스키마·API 변경 없는 리뷰 큐의 순수 로직 기반. (2) `imap_worker`의 `Email(...)` 생성이 `date_provenance`를 영속화하지 않아 IMAP 수집 행이 항상 모델 기본값 `"unknown"`으로 저장되어 stored-side strong fingerprint에서 영구 제외되던 결함을 수정했습니다(import 경로와 동일하게 `email_data.get("date_provenance", "unknown")` 전달). 이로써 "import·IMAP 두 생성 경로 모두 파서 provenance를 영속화한다"는 계약이 구조적으로 성립합니다. (3) `email_parser._extract_date_with_provenance`가 RFC 5322 `-0000`("시간대 정보 없음")에 대해 `parsedate_to_datetime`이 반환하는 naive datetime을 UTC로 정규화해, 문서화된 timezone-aware 계약이 모든 parsed 헤더에서 성립하도록 했습니다. + - 검증(이 커밋): `resolve_candidate_disposition` 로직 5개(빈 corpus→distinct, 전부 distinct, content-only→review+행, A1이 앞선 A2 우선, 첫 auto_link 우선)와 파서 date-provenance 시나리오(`-0000`→parsed·UTC, whitespace-only `Date`→missing, `+0900` 보존, invalid)를 직접 검증했고, 파서 회귀 테스트 2건(`-0000`·whitespace-only)을 추가했습니다. ruff 0.15.21(CI 핀) clean, py_compile OK. (앞선 두 불릿의 `333 passed, 1 skipped`는 파서/import/stored 커밋의 전체 스위트, `94 passed`는 dedupe-core 커밋의 email/dedupe 하위 스위트 기준입니다.) + ### 보안 패치 (CodeQL extended current-head) -- `cryptography`를 `50.0.0`으로 갱신해 공격자 제공 PKCS#7 EnvelopedData 복호화 결과의 오류·타이밍 차이로 발생하는 Bleichenbacher oracle(`CVE-2026-69247`, `GHSA-g6cj-pr64-35w5`)을 제거하고, backend·uv lock·hash lock·Strix CI 의존성 증거를 같은 버전으로 동기화했습니다. Strix 잠금은 `google-cloud-aiplatform==1.160.0`의 `<7` 제약을 위반하던 `protobuf==7.35.1`을 이미 검증된 `6.33.6`으로 복구해 다시 해석·설치 가능하게 했습니다. - CodeQL `extended` 기본 설정이 current `develop`에서 확인한 Critical 8건·High 21건·Medium 1건을 코드 경계에서 제거합니다. 서버 요청은 검증된 loopback/HTTPS origin, 동일 OIDC issuer origin, 허용 API 경로·쿼리만 재구성하고 redirect를 자동 추종하지 않으며, 공개 IPv6 authority를 보존합니다. UI smoke는 고정 Node/Next 실행 파일과 인자, localhost:3001 allowlist, private `mkdtemp` artifact 디렉터리 및 containment 검사만 사용합니다. - OIDC token endpoint는 운영 환경에서 서버 전용 `OIDC_ALLOWED_HOSTS` 정확 호스트 allowlist를 필수로 적용합니다. hostname의 모든 DNS 결과가 공인 주소인지 검증한 뒤 해당 주소 집합을 native HTTP(S) 연결의 `lookup`에 고정하고, 원래 issuer hostname은 Host/TLS SNI로 유지해 사설 주소 해석과 DNS rebinding 사이의 TOCTOU를 차단합니다. 실패 로그는 입력 URL·token 대신 고정된 configuration/DNS·transport/response/backend-verification reason code만 남깁니다. -- Trivy 2026-07-26 DB에서 새로 확인된 Next.js High 4건·Medium 5건(`CVE-2026-64641`–`CVE-2026-64649`)과 PostCSS High 1건(`GHSA-r28c-9q8g-f849`)을 제거하기 위해 Next.js/`eslint-config-next`를 `16.2.11`, PostCSS를 `8.5.18`로 갱신했습니다. 이후 2026-08-04 DB가 `8.5.18`에서 추가 탐지한 PostCSS Medium(`CVE-2026-69153`, 최초 수정 `8.5.23`)도 제거하도록 manifest·workspace override·lock을 `8.5.24`로 동기화했으며 저장소의 release-age 정책을 우회하지 않습니다. -- `pnpm audit`가 개발 도구 체인에서 추가 탐지한 `brace-expansion <=5.0.7` High DoS(`GHSA-mh99-v99m-4gvg`)와 이후 `5.0.8`까지 영향을 주는 우회형 High DoS(`GHSA-rgw5-rvv9-x895`)는 `5.0.9` 전역 override로 제거했습니다. CommonJS default export를 기대하는 legacy `minimatch 3.1.5`에는 `expand` named export도 수용하는 최소 pnpm 패치를 적용해 ESLint/glob 동작을 보존합니다. 같은 감사에서 확인된 `undici 7.28.0`의 High 1건·Moderate 4건(`GHSA-4cwx-7wf7-3272` 등)은 `jsdom 30.0.1` 및 release-age 정책을 통과하는 `undici 8.9.0`으로 갱신했습니다. +- Trivy 2026-07-26 DB에서 새로 확인된 Next.js High 4건·Medium 5건(`CVE-2026-64641`–`CVE-2026-64649`)과 PostCSS High 1건(`GHSA-r28c-9q8g-f849`)을 제거하기 위해 Next.js/`eslint-config-next`를 `16.2.11`, PostCSS를 `8.5.18`로 갱신했습니다. 두 버전은 각 취약점의 최초 수정 버전이며 저장소의 release-age 정책을 우회하지 않습니다. +- `pnpm audit`가 개발 도구 체인에서 추가 탐지한 `brace-expansion <=5.0.7` High DoS(`GHSA-mh99-v99m-4gvg`)는 `5.0.8` 전역 override로 제거했습니다. CommonJS default export를 기대하는 legacy `minimatch 3.1.5`에는 `expand` named export도 수용하는 최소 pnpm 패치를 적용해 ESLint/glob 동작을 보존합니다. - root·frontend Docker build의 frozen install 계층이 pnpm manifest와 함께 `frontend/patches`를 먼저 복사하도록 수정해, 이미지 검증에서도 lockfile의 patched dependency를 동일하게 재현합니다. - Scorecard SARIF normalizer는 고정 workspace artifact로 정규화되는 `./scorecard-results.sarif`와 절대 경로를 동일하게 허용하면서 symlink·workspace 이탈은 계속 거부합니다. 도구 실행 실패 API는 CR/LF·제어 문자를 escape하고 500자로 제한하며, 로그에는 raw 도구 코드·예외 text 대신 SHA-256 기반 코드·traceback 상관 식별자만 기록합니다. - 백엔드 origin 보안 경계를 `frontend/src/lib/backend-url.ts`의 단일 생성기로 통합해 API proxy·session·OIDC callback이 같은 검증을 사용합니다. UI smoke의 새 `NARUON_FULL_PRODUCT_SCREENSHOT_PROFILE` 이름은 실제 selector 의미를 드러내며, 기존 `..._SCREENSHOT_DIR`은 호환 alias로 계속 지원합니다. diff --git a/backend/alembic/versions/0018_email_date_provenance.py b/backend/alembic/versions/0018_email_date_provenance.py new file mode 100644 index 000000000..c33a7a2f4 --- /dev/null +++ b/backend/alembic/versions/0018_email_date_provenance.py @@ -0,0 +1,49 @@ +"""add date_provenance to email_records + +Revision ID: 0018_email_date_provenance +Revises: 0017_merge_newsdom_carddav_heads +Create Date: 2026-07-30 00:00:00.000000 + +Records the provenance of each stored email ``date`` so a synthetic +collection-time fallback (missing/invalid RFC822 Date header) is never treated +as original sender metadata when seeding a strong auto-dedupe fingerprint +(naruon#1086). Nullable-free with a ``"unknown"`` server default so existing +rows backfill safely: their date provenance is genuinely unknown, and only +``"parsed"`` rows are eligible to seed a strong fingerprint, so the backfill is +conservative (it can only widen review, never manufacture a duplicate). +""" + +from alembic import op +import sqlalchemy as sa + +revision = "0018_email_date_provenance" +down_revision = "0017_merge_newsdom_carddav_heads" + +_EMAIL_TABLE = "email_records" +_PROVENANCE_COLUMN = "date_provenance" + + +def upgrade() -> None: + """Add the ``date_provenance`` column, backfilling existing rows to unknown.""" + connection = op.get_bind() + inspector = sa.inspect(connection) + columns = {column["name"] for column in inspector.get_columns(_EMAIL_TABLE)} + if _PROVENANCE_COLUMN not in columns: + op.add_column( + _EMAIL_TABLE, + sa.Column( + _PROVENANCE_COLUMN, + sa.String(), + nullable=False, + server_default="unknown", + ), + ) + + +def downgrade() -> None: + """Drop the ``date_provenance`` column if present.""" + connection = op.get_bind() + inspector = sa.inspect(connection) + columns = {column["name"] for column in inspector.get_columns(_EMAIL_TABLE)} + if _PROVENANCE_COLUMN in columns: + op.drop_column(_EMAIL_TABLE, _PROVENANCE_COLUMN) diff --git a/backend/db/models.py b/backend/db/models.py index 98e17eef2..a798b5f8c 100644 --- a/backend/db/models.py +++ b/backend/db/models.py @@ -801,6 +801,13 @@ def owner_filters(cls, user_id: str, organization_id: str | None): in_reply_to: Mapped[str | None] = mapped_column(String, nullable=True) references: Mapped[str | None] = mapped_column(String, nullable=True) date: Mapped[datetime.datetime] = mapped_column(DateTime(timezone=True), index=True) + # Provenance of the stored ``date``: "parsed" (genuine RFC822 Date header), + # "missing"/"invalid" (a synthetic collection-time fallback), or "unknown" + # (rows stored before provenance tracking). Only "parsed" rows may seed a + # strong auto-dedupe fingerprint (naruon#1086). + date_provenance: Mapped[str] = mapped_column( + String, nullable=False, server_default="unknown", default="unknown" + ) body: Mapped[str] = mapped_column(Text) # IMAP \Seen read state; defaults read so historical/file imports don't nag. is_read: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True) diff --git a/backend/services/email_dedupe_service.py b/backend/services/email_dedupe_service.py index 1c74d5d11..7a8728198 100644 --- a/backend/services/email_dedupe_service.py +++ b/backend/services/email_dedupe_service.py @@ -1,13 +1,38 @@ +"""Email de-duplication fingerprints and the Fellegi-Sunter decision classifier. + +This module is the deterministic core the import/IMAP paths compose to decide +whether an incoming email is a duplicate of a stored one, keeping strong +(auto-merge) evidence gated on genuine Date provenance (naruon#1086). +""" + import datetime +from collections.abc import Iterable from dataclasses import dataclass +from typing import Literal from db.models import Email from services.email_service import generate_email_fingerprint from services.threading_service import normalize_message_id +# Fellegi & Sunter (1969) partition each candidate/record pair into three +# decision zones: a positive link (A1), a non-link (A3), and an indeterminate +# "possible match" band (A2) reserved for clerical review. naruon#1086 maps that +# rule onto email de-duplication: a reliable identity link auto-merges, a +# probable duplicate that lacks a reliable link is held for review instead of +# being silently kept or silently merged, and everything else is distinct. +DedupeDecision = Literal["auto_link", "review_required", "distinct"] + @dataclass(frozen=True) class EmailDedupeCandidate: + """An incoming email reduced to the fields the dedupe decision needs. + + ``date_provenance`` mirrors the parser's classification of the ``date`` + field (``parsed`` for a genuine RFC822 Date, otherwise a synthetic + collection-time fallback); only ``parsed`` may seed a strong auto-dedupe + match (naruon#1086). + """ + candidate_key: str message_id: str | None = None sender: str | None = None @@ -15,9 +40,11 @@ class EmailDedupeCandidate: subject: str | None = None date: datetime.datetime | None = None body: str | None = None + date_provenance: str = "unknown" def _date_to_fingerprint_value(value: datetime.datetime | None) -> str: + """Render a datetime as its ISO-8601 fingerprint token (``""`` when None).""" if value is None: return "" return value.isoformat() @@ -30,6 +57,11 @@ def strong_email_fingerprint( date: datetime.datetime | None, body: str | None, ) -> str | None: + """Return the strong (sender+subject+Date+body) auto-dedupe fingerprint. + + Requires a body; ``None`` for an empty body so bodyless rows cannot collapse + to a shared hash. Callers gate this on genuine Date provenance. + """ if not body: return None return generate_email_fingerprint( @@ -43,6 +75,7 @@ def strong_email_fingerprint( def candidate_message_lookup_values(candidate: EmailDedupeCandidate) -> set[str]: + """Return the bracketed and bare Message-ID lookup forms (empty if none).""" normalized = normalize_message_id(candidate.message_id) if not normalized: return set() @@ -50,6 +83,7 @@ def candidate_message_lookup_values(candidate: EmailDedupeCandidate) -> set[str] def candidate_strong_fingerprint(candidate: EmailDedupeCandidate) -> str | None: + """Return the candidate's strong fingerprint (see strong_email_fingerprint).""" return strong_email_fingerprint( sender=candidate.sender, subject=candidate.subject, @@ -59,9 +93,138 @@ def candidate_strong_fingerprint(candidate: EmailDedupeCandidate) -> str | None: def email_strong_fingerprint(email_row: Email) -> str | None: + """Return a stored row's strong fingerprint, gated on genuine Date provenance. + + A stored row may seed a strong (auto-dedupe) fingerprint only when its date + is genuinely parsed sender metadata; rows with a synthetic or + unknown-provenance date are excluded so they cannot manufacture a strong + duplicate match (naruon#1086). + """ + if getattr(email_row, "date_provenance", None) != "parsed": + return None return strong_email_fingerprint( sender=email_row.sender, subject=email_row.subject, date=email_row.date, body=email_row.body, ) + + +def content_email_fingerprint( + *, + sender: str | None, + subject: str | None, + body: str | None, +) -> str | None: + """Return a Date-independent content fingerprint (sender+subject+body). + + Unlike the strong fingerprint it omits the Date, so it survives an + untrustworthy Date provenance (naruon#1086) and can flag a probable + duplicate that the strong path deliberately withholds. Requires a body so + empty-body rows cannot collapse to a shared hash. + """ + if not body: + return None + return generate_email_fingerprint( + { + "sender": sender or "", + "subject": subject or "", + "date": "", + "body": body, + } + ) + + +def candidate_content_fingerprint(candidate: EmailDedupeCandidate) -> str | None: + """Return the candidate's Date-independent content fingerprint.""" + return content_email_fingerprint( + sender=candidate.sender, + subject=candidate.subject, + body=candidate.body, + ) + + +def email_content_fingerprint(email_row: Email) -> str | None: + """Return a stored row's Date-independent content fingerprint.""" + return content_email_fingerprint( + sender=email_row.sender, + subject=email_row.subject, + body=email_row.body, + ) + + +def classify_dedupe_decision( + candidate: EmailDedupeCandidate, existing_row: Email +) -> DedupeDecision: + """Assign a Fellegi-Sunter (1969) decision zone to a candidate/existing pair. + + - ``auto_link`` (A1, positive link): the pair shares a reliable identity + link -- the same normalized Message-ID, or a genuine strong match + (identical sender/subject/Date/body with a trusted, parsed Date on *both* + sides). These are safe to merge automatically. + - ``review_required`` (A2, possible match): the pair shares a + provenance-independent content signal (same sender/subject/body) but has + no reliable identity link -- typically because at least one side's Date + provenance is synthetic or unknown, so the strong fingerprint was withheld + (naruon#1086). This is the clerical-review band: a probable duplicate that + must not be silently merged or silently kept. + - ``distinct`` (A3, non-link): no shared identity or content signal. + """ + candidate_message = normalize_message_id(candidate.message_id) + existing_message = normalize_message_id(existing_row.message_id) + if candidate_message and existing_message and candidate_message == existing_message: + return "auto_link" + + candidate_strong = candidate_strong_fingerprint(candidate) + existing_strong = email_strong_fingerprint(existing_row) + if ( + candidate.date_provenance == "parsed" + and candidate_strong is not None + and existing_strong is not None + and candidate_strong == existing_strong + ): + return "auto_link" + + candidate_content = candidate_content_fingerprint(candidate) + existing_content = email_content_fingerprint(existing_row) + if ( + candidate_content is not None + and existing_content is not None + and candidate_content == existing_content + ): + return "review_required" + + return "distinct" + + +def resolve_candidate_disposition( + candidate: EmailDedupeCandidate, existing_rows: Iterable[Email] +) -> tuple[DedupeDecision, Email | None]: + """Resolve a candidate against many stored rows to one Fellegi-Sunter disposition. + + Real de-duplication compares one incoming email against the *set* of stored + rows it might duplicate, not a single row, so this collapses the per-pair + ``classify_dedupe_decision`` results by the Fellegi & Sunter (1969) zone + priority A1 > A2 > A3: + + - the first stored row that yields ``auto_link`` (A1, a reliable identity + link) wins immediately -- a positive link cannot be outranked; + - absent any link, the first ``review_required`` match (A2) is held for + clerical review rather than silently merged or silently kept; + - only when no stored row shares any identity or content signal is the + candidate ``distinct`` (A3). + + Returns the decision together with the stored row that drove a link or a + review hold (``None`` when distinct), so the import/IMAP paths know which + email the disposition targets without re-deriving the match. + """ + review_match: Email | None = None + for existing_row in existing_rows: + decision = classify_dedupe_decision(candidate, existing_row) + if decision == "auto_link": + return "auto_link", existing_row + if decision == "review_required" and review_match is None: + review_match = existing_row + if review_match is not None: + return "review_required", review_match + return "distinct", None diff --git a/backend/services/email_import_service.py b/backend/services/email_import_service.py index 1ff9a2bb3..2e02ff430 100644 --- a/backend/services/email_import_service.py +++ b/backend/services/email_import_service.py @@ -187,12 +187,21 @@ def _message_id_for(parsed: EmailData, content: bytes) -> str: def _email_fingerprint(parsed: EmailData, persisted_date: datetime.datetime) -> str: - strong_fingerprint = strong_email_fingerprint( - sender=parsed.get("sender"), - subject=parsed.get("subject"), - date=persisted_date, - body=parsed.get("body"), - ) + # A strong (auto-dedupe-eligible) fingerprint may only be seeded from a + # genuinely-parsed sender Date. When the Date header was missing or invalid + # (date_provenance != "parsed"), ``persisted_date`` is a synthetic + # collection-time fallback, not original metadata, so it must not produce a + # strong duplicate key — the email falls through to the weak fallback + # fingerprint (which, carrying the distinct collection time, cannot + # manufacture a false duplicate) (naruon#1086). + strong_fingerprint = None + if parsed.get("date_provenance") == "parsed": + strong_fingerprint = strong_email_fingerprint( + sender=parsed.get("sender"), + subject=parsed.get("subject"), + date=persisted_date, + body=parsed.get("body"), + ) if strong_fingerprint: return strong_fingerprint return generate_email_fingerprint( @@ -325,6 +334,7 @@ def _build_email_object( in_reply_to=parsed.get("in_reply_to"), references=parsed.get("references"), date=persisted_date, + date_provenance=parsed.get("date_provenance", "unknown"), body=parsed.get("body", ""), embedding=fitted_embeddings[0] if fitted_embeddings else _zero_embedding(), ) diff --git a/backend/services/email_parser.py b/backend/services/email_parser.py index be8bee1c4..3c9f04a8e 100644 --- a/backend/services/email_parser.py +++ b/backend/services/email_parser.py @@ -5,12 +5,20 @@ import re from email.utils import getaddresses from email.utils import parsedate_to_datetime -from typing import NotRequired, TypedDict +from typing import Literal, NotRequired, TypedDict from .attachment_parser import parse_email_attachment from .exceptions import EmailParseError from .text_safety import strip_html_markup +# Provenance of the RFC822 ``Date`` header, kept explicit so a synthetic +# collection-time fallback is never mistaken for original sender metadata. +DateProvenance = Literal["parsed", "missing", "invalid"] + +# Provenance of the RFC822 ``Message-ID`` header. +MessageIdProvenance = Literal["embedded", "missing"] + + class EmailData(TypedDict): """Parsed email data structure.""" @@ -23,6 +31,9 @@ class EmailData(TypedDict): in_reply_to: str | None references: str | None date: datetime.datetime + header_date: NotRequired[datetime.datetime | None] + date_provenance: NotRequired[DateProvenance] + message_id_provenance: NotRequired[MessageIdProvenance] body: str body_content_type: NotRequired[str] body_parse_content: NotRequired[str] @@ -152,26 +163,41 @@ def _extract_body_and_attachments(msg: Message) -> tuple[str, str, list[dict]]: return html_body, "text/html" if html_body else "text/plain", attachments -def _extract_date(msg: Message) -> datetime.datetime: +def _extract_date_with_provenance( + msg: Message, +) -> tuple[datetime.datetime, datetime.datetime | None, DateProvenance]: + """Return effective date, genuine header date, and header provenance. + + The effective value is always timezone-aware and safe for storage. A + missing or invalid header receives a UTC collection-time fallback, while + ``header_date`` remains ``None`` so deduplication cannot treat that fallback + as sender-supplied evidence. + """ date_header = msg.get("Date") - parsed_date = None - if date_header: - try: - parsed_date = parsedate_to_datetime(date_header) - except (TypeError, ValueError): - parsed_date = None - - if not parsed_date: - parsed_date = datetime.datetime.now(datetime.timezone.utc) - elif parsed_date.tzinfo is None: - # RFC 5322 section 3.3: a "-0000" zone means the time zone is unknown, - # for which parsedate_to_datetime returns a naive datetime. Every other - # branch here yields a timezone-aware datetime, and mixing naive with - # aware datetimes raises TypeError on comparison/sorting and misbinds the - # instant when stored in a timestamptz column. Treat the unknown zone as - # UTC so the returned value is always timezone-aware. - parsed_date = parsed_date.replace(tzinfo=datetime.timezone.utc) - return parsed_date + header_text = str(date_header).strip() if date_header is not None else "" + fallback = datetime.datetime.now(datetime.timezone.utc) + + if not header_text: + return fallback, None, "missing" + + try: + header_date = parsedate_to_datetime(date_header) + except (TypeError, ValueError): + header_date = None + + if header_date is None: + return fallback, None, "invalid" + if header_date.tzinfo is None: + # RFC 5322 section 3.3: a ``-0000`` zone means the time zone is + # unknown. Normalize the naive parser result to UTC so every parsed + # value still satisfies the timezone-aware storage contract. + header_date = header_date.replace(tzinfo=datetime.timezone.utc) + return header_date, header_date, "parsed" + + +def _message_id_provenance(raw_message_id: str) -> MessageIdProvenance: + """Classify whether a genuine ``Message-ID`` header was embedded.""" + return "embedded" if raw_message_id.strip() else "missing" def _extract_thread_id(msg: Message, message_id: str) -> str | None: @@ -193,8 +219,9 @@ def _extract_thread_id(msg: Message, message_id: str) -> str | None: def _message_to_email_data(msg: Message) -> EmailData: body, body_content_type, attachments = _extract_body_and_attachments(msg) - parsed_date = _extract_date(msg) + effective_date, header_date, date_provenance = _extract_date_with_provenance(msg) message_id = _sanitize_nul(msg.get("Message-ID", "")) + message_id_provenance = _message_id_provenance(message_id) thread_id = _extract_thread_id(msg, message_id) return { @@ -216,7 +243,10 @@ def _message_to_email_data(msg: Message) -> EmailData: "references": ( _sanitize_nul(msg.get("References", "")) if msg.get("References") else None ), - "date": parsed_date, + "date": effective_date, + "header_date": header_date, + "date_provenance": date_provenance, + "message_id_provenance": message_id_provenance, "body": _sanitize_display_text(body), "body_content_type": body_content_type, "body_parse_content": _sanitize_nul(body), diff --git a/backend/services/imap_worker.py b/backend/services/imap_worker.py index d618f1d3c..d74bd3d1a 100644 --- a/backend/services/imap_worker.py +++ b/backend/services/imap_worker.py @@ -50,12 +50,20 @@ async def process_fetched_email( else str(recipients_list or "") ) - fingerprint = strong_email_fingerprint( - sender=sender, - subject=subject, - date=persisted_date, - body=email_data.get("body", ""), - ) or generate_email_fingerprint(subject, date_str, sender, recipients) + # Seed the strong (auto-dedupe) fingerprint only from a genuinely-parsed + # Date; a synthetic collection-time fallback must not manufacture a strong + # duplicate key, so it falls through to the weak fallback (naruon#1086). + strong_fingerprint = None + if email_data.get("date_provenance") == "parsed": + strong_fingerprint = strong_email_fingerprint( + sender=sender, + subject=subject, + date=persisted_date, + body=email_data.get("body", ""), + ) + fingerprint = strong_fingerprint or generate_email_fingerprint( + subject, date_str, sender, recipients + ) # Check if duplicate stmt = select(Email).where( @@ -87,6 +95,7 @@ async def process_fetched_email( recipients=recipients, subject=subject, date=persisted_date, + date_provenance=email_data.get("date_provenance", "unknown"), body=email_data.get("body", ""), is_read=is_read, embedding=[0.0] * 1536, diff --git a/backend/tests/test_email_dedupe_service.py b/backend/tests/test_email_dedupe_service.py index 1afbad5ed..924301287 100644 --- a/backend/tests/test_email_dedupe_service.py +++ b/backend/tests/test_email_dedupe_service.py @@ -7,9 +7,30 @@ strong_email_fingerprint, candidate_strong_fingerprint, email_strong_fingerprint, + content_email_fingerprint, + candidate_content_fingerprint, + email_content_fingerprint, + classify_dedupe_decision, + resolve_candidate_disposition, ) from db.models import Email + +def _email_row(**overrides): + fields = dict( + id=100, + user_id="user-1", + organization_id="org-1", + message_id=None, + sender="sender@example.com", + subject="Subject", + date=datetime(2023, 1, 1, 12, 0, 0, tzinfo=timezone.utc), + date_provenance="parsed", + body="Hello world", + ) + fields.update(overrides) + return Email(**fields) + def test_candidate_message_lookup_values_basic(): candidate = EmailDedupeCandidate( candidate_key="key", @@ -97,6 +118,7 @@ def test_email_strong_fingerprint(): sender="sender@example.com", subject="Subject", date=datetime(2023, 1, 1, 12, 0, 0, tzinfo=timezone.utc), + date_provenance="parsed", body="Hello world" ) result1 = email_strong_fingerprint(email) @@ -109,3 +131,236 @@ def test_email_strong_fingerprint(): ) assert result1 == result2 assert result1 is not None + + +def test_email_strong_fingerprint_gated_to_parsed_date_provenance(): + """A stored row seeds a strong fingerprint only when its date is genuine. + + naruon#1086: rows whose ``date`` is a synthetic collection-time fallback + (missing/invalid) or unknown-provenance (stored before tracking) must not + seed a strong auto-dedupe fingerprint, even though sender/subject/body/date + are populated. + """ + fields = dict( + id=2, + user_id="user-1", + organization_id="org-1", + message_id="msg-2", + sender="sender@example.com", + subject="Subject", + date=datetime(2023, 1, 1, 12, 0, 0, tzinfo=timezone.utc), + body="Hello world", + ) + assert email_strong_fingerprint(Email(**fields, date_provenance="parsed")) is not None + for provenance in ("missing", "invalid", "unknown"): + assert email_strong_fingerprint(Email(**fields, date_provenance=provenance)) is None + + +# --- content fingerprint (date-independent identity signal, naruon#1086) --- + +def test_content_email_fingerprint_none_without_body(): + assert content_email_fingerprint(sender="a@x", subject="S", body=None) is None + assert content_email_fingerprint(sender="a@x", subject="S", body="") is None + + +def test_content_email_fingerprint_is_date_independent_and_not_the_strong_one(): + """The content fingerprint ignores the Date; the strong one includes it.""" + content = content_email_fingerprint( + sender="sender@example.com", subject="Subject", body="Hello world" + ) + strong = strong_email_fingerprint( + sender="sender@example.com", + subject="Subject", + date=datetime(2023, 1, 1, 12, 0, 0, tzinfo=timezone.utc), + body="Hello world", + ) + assert content is not None + assert content != strong + # Same content, different Date -> identical content fingerprint. + candidate_a = EmailDedupeCandidate( + candidate_key="a", + sender="sender@example.com", + subject="Subject", + date=datetime(2023, 1, 1, tzinfo=timezone.utc), + body="Hello world", + ) + candidate_b = EmailDedupeCandidate( + candidate_key="b", + sender="sender@example.com", + subject="Subject", + date=datetime(2024, 6, 6, tzinfo=timezone.utc), + body="Hello world", + ) + assert candidate_content_fingerprint(candidate_a) == candidate_content_fingerprint( + candidate_b + ) + assert email_content_fingerprint(_email_row()) == content + + +def test_email_content_fingerprint_none_without_body(): + assert email_content_fingerprint(_email_row(body=None)) is None + + +# --- Fellegi-Sunter (1969) three-zone classifier --- + +def test_auto_link_on_matching_normalized_message_id(): + # Bracketed vs bare Message-ID normalize equal; content is irrelevant here. + candidate = EmailDedupeCandidate( + candidate_key="c", + message_id="", + sender="other@example.com", + subject="Totally different", + body="unrelated body", + ) + existing = _email_row(message_id="shared@x") + assert classify_dedupe_decision(candidate, existing) == "auto_link" + + +def test_auto_link_on_genuine_strong_match_without_message_id(): + date = datetime(2023, 1, 1, 12, 0, 0, tzinfo=timezone.utc) + candidate = EmailDedupeCandidate( + candidate_key="c", + sender="sender@example.com", + subject="Subject", + date=date, + body="Hello world", + date_provenance="parsed", + ) + existing = _email_row(date=date, date_provenance="parsed") + assert classify_dedupe_decision(candidate, existing) == "auto_link" + + +def test_review_required_when_candidate_date_provenance_is_untrusted(): + date = datetime(2023, 1, 1, 12, 0, 0, tzinfo=timezone.utc) + candidate = EmailDedupeCandidate( + candidate_key="c", + sender="sender@example.com", + subject="Subject", + date=date, + body="Hello world", + date_provenance="missing", + ) + existing = _email_row(date=date, date_provenance="parsed") + # Same content, but the candidate's Date is synthetic -> no strong match, no + # Message-ID link -> clerical-review band, not a silent merge. + assert classify_dedupe_decision(candidate, existing) == "review_required" + + +def test_review_required_when_existing_date_provenance_is_untrusted(): + date = datetime(2023, 1, 1, 12, 0, 0, tzinfo=timezone.utc) + candidate = EmailDedupeCandidate( + candidate_key="c", + sender="sender@example.com", + subject="Subject", + date=date, + body="Hello world", + date_provenance="parsed", + ) + existing = _email_row(date=date, date_provenance="unknown") + assert classify_dedupe_decision(candidate, existing) == "review_required" + + +def test_review_required_when_content_matches_but_dates_differ_untrusted(): + candidate = EmailDedupeCandidate( + candidate_key="c", + sender="sender@example.com", + subject="Subject", + date=datetime(2024, 6, 6, tzinfo=timezone.utc), + body="Hello world", + date_provenance="invalid", + ) + existing = _email_row( + date=datetime(2023, 1, 1, tzinfo=timezone.utc), date_provenance="unknown" + ) + assert classify_dedupe_decision(candidate, existing) == "review_required" + + +def test_distinct_on_different_content_and_no_identity_link(): + candidate = EmailDedupeCandidate( + candidate_key="c", + message_id="only-on-candidate@x", + sender="different@example.com", + subject="Different", + body="different body", + date_provenance="parsed", + ) + existing = _email_row(message_id="only-on-existing@x") + assert classify_dedupe_decision(candidate, existing) == "distinct" + + +def test_distinct_when_candidate_has_no_body(): + candidate = EmailDedupeCandidate( + candidate_key="c", + sender="sender@example.com", + subject="Subject", + body=None, + date_provenance="parsed", + ) + existing = _email_row() + assert classify_dedupe_decision(candidate, existing) == "distinct" + + +def test_resolve_disposition_empty_corpus_is_distinct(): + candidate = EmailDedupeCandidate(candidate_key="c", message_id="m@x", body="b") + assert resolve_candidate_disposition(candidate, []) == ("distinct", None) + + +def test_resolve_disposition_all_distinct_returns_distinct_none(): + candidate = EmailDedupeCandidate( + candidate_key="c", + message_id="only-on-candidate@x", + sender="different@example.com", + subject="Different", + body="different body", + date_provenance="parsed", + ) + rows = [_email_row(id=1, message_id="a@x"), _email_row(id=2, message_id="b@x")] + assert resolve_candidate_disposition(candidate, rows) == ("distinct", None) + + +def test_resolve_disposition_returns_review_row_when_only_content_matches(): + date = datetime(2023, 1, 1, 12, 0, 0, tzinfo=timezone.utc) + candidate = EmailDedupeCandidate( + candidate_key="c", + sender="sender@example.com", + subject="Subject", + date=date, + body="Hello world", + date_provenance="missing", # synthetic Date -> no strong/A1 link + ) + unrelated = _email_row(id=1, message_id="unrelated@x", body="different body") + content_match = _email_row(id=2, date=date, date_provenance="parsed") + decision, matched = resolve_candidate_disposition(candidate, [unrelated, content_match]) + assert decision == "review_required" + assert matched is content_match + + +def test_resolve_disposition_a1_link_dominates_earlier_a2_review(): + # A review_required content match appears BEFORE the auto_link row in the + # corpus; Fellegi-Sunter A1 must still win regardless of iteration order. + date = datetime(2023, 1, 1, 12, 0, 0, tzinfo=timezone.utc) + candidate = EmailDedupeCandidate( + candidate_key="c", + message_id="", + sender="sender@example.com", + subject="Subject", + date=date, + body="Hello world", + date_provenance="missing", + ) + content_only = _email_row(id=1, date=date, date_provenance="unknown") + id_link = _email_row(id=2, message_id="shared@x", body="totally different") + decision, matched = resolve_candidate_disposition(candidate, [content_only, id_link]) + assert decision == "auto_link" + assert matched is id_link + + +def test_resolve_disposition_first_auto_link_wins(): + candidate = EmailDedupeCandidate( + candidate_key="c", message_id="", body="b", date_provenance="parsed" + ) + first = _email_row(id=1, message_id="shared@x") + second = _email_row(id=2, message_id="shared@x") + decision, matched = resolve_candidate_disposition(candidate, [first, second]) + assert decision == "auto_link" + assert matched is first diff --git a/backend/tests/test_email_import_service.py b/backend/tests/test_email_import_service.py index d16a79dd1..5fd0d18c3 100644 --- a/backend/tests/test_email_import_service.py +++ b/backend/tests/test_email_import_service.py @@ -579,3 +579,42 @@ async def test_generate_import_embeddings_recovers_valid_items_after_batch_failu assert embeddings[2] == [0.75] * (EMBEDDING_DIMENSION // 2) + [0.0] * ( EMBEDDING_DIMENSION // 2 ) + + +def test_email_fingerprint_uses_strong_key_only_for_parsed_date(): + """A strong (auto-dedupe) fingerprint is seeded only from a genuine Date. + + naruon#1086: when the RFC822 Date was missing or invalid the persisted date + is a synthetic collection-time fallback, which must not seed the strong + duplicate key. Only ``date_provenance == "parsed"`` yields the strong + fingerprint; missing/invalid provenance falls through to the weak fallback. + """ + from services.email_dedupe_service import strong_email_fingerprint + from services.email_import_service import _email_fingerprint + + persisted_date = datetime.datetime( + 2026, 4, 27, 10, 0, 0, tzinfo=datetime.timezone.utc + ) + parsed_fields = { + "sender": "sender@test.com", + "subject": "Quarterly report", + "body": "The full report body.", + "recipients": "recipient@test.com", + } + strong = strong_email_fingerprint( + sender=parsed_fields["sender"], + subject=parsed_fields["subject"], + date=persisted_date, + body=parsed_fields["body"], + ) + assert strong is not None + + assert ( + _email_fingerprint({**parsed_fields, "date_provenance": "parsed"}, persisted_date) + == strong + ) + for provenance in ("missing", "invalid"): + weak = _email_fingerprint( + {**parsed_fields, "date_provenance": provenance}, persisted_date + ) + assert weak != strong diff --git a/backend/tests/test_email_parser_provenance.py b/backend/tests/test_email_parser_provenance.py new file mode 100644 index 000000000..92c2f044e --- /dev/null +++ b/backend/tests/test_email_parser_provenance.py @@ -0,0 +1,127 @@ +"""Focused contracts for email metadata provenance classification.""" + +import datetime + +from services.email_parser import parse_eml_bytes + + +def _eml_with(headers: str) -> bytes: + """Build minimal EML bytes with the given header block and a plain body.""" + return (headers.strip() + "\n\nBody text.").encode("utf-8") + + +def test_parse_eml_marks_valid_date_as_parsed_with_original_header_date() -> None: + """A valid Date header remains the genuine timezone-aware storage value.""" + parsed = parse_eml_bytes( + _eml_with( + """Message-ID: +From: sender@test.com +To: recipient@test.com +Subject: Valid date +Date: Mon, 27 Apr 2026 10:00:00 +0000""" + ) + ) + + expected = datetime.datetime(2026, 4, 27, 10, 0, 0, tzinfo=datetime.timezone.utc) + assert parsed["date_provenance"] == "parsed" + assert parsed["header_date"] == expected + assert parsed["date"] == expected + + +def test_parse_eml_marks_missing_date_without_promoting_the_fallback() -> None: + """A missing Date uses storage fallback without inventing sender evidence.""" + parsed = parse_eml_bytes( + _eml_with( + """Message-ID: +From: sender@test.com +To: recipient@test.com +Subject: No date header""" + ) + ) + + assert parsed["date_provenance"] == "missing" + assert parsed["header_date"] is None + assert parsed["date"].tzinfo is not None + + +def test_parse_eml_marks_invalid_date_without_promoting_the_fallback() -> None: + """An invalid Date uses storage fallback without inventing sender evidence.""" + parsed = parse_eml_bytes( + _eml_with( + """Message-ID: +From: sender@test.com +To: recipient@test.com +Subject: Unparseable date +Date: not-a-real-date""" + ) + ) + + assert parsed["date_provenance"] == "invalid" + assert parsed["header_date"] is None + assert parsed["date"].tzinfo is not None + + +def test_parse_eml_marks_whitespace_only_date_as_missing() -> None: + """A whitespace-only Date is missing rather than malformed evidence.""" + parsed = parse_eml_bytes( + _eml_with( + """Message-ID: +From: sender@test.com +To: recipient@test.com +Subject: Whitespace-only date +Date: """ + ) + ) + + assert parsed["date_provenance"] == "missing" + assert parsed["header_date"] is None + assert parsed["date"].tzinfo is not None + + +def test_parse_eml_normalizes_minus_zero_zone_to_utc() -> None: + """RFC 5322 -0000 dates remain timezone-aware for storage and comparison.""" + parsed = parse_eml_bytes( + _eml_with( + """Message-ID: +From: sender@test.com +To: recipient@test.com +Subject: Minus-zero timezone +Date: Sun, 01 Jan 2023 12:00:00 -0000""" + ) + ) + + assert parsed["date_provenance"] == "parsed" + assert parsed["header_date"] is not None + assert parsed["header_date"].tzinfo is not None + assert parsed["header_date"].utcoffset() == datetime.timedelta(0) + assert parsed["date"].tzinfo is not None + + +def test_parse_eml_marks_embedded_message_id_provenance() -> None: + """A non-empty embedded Message-ID is identified as sender evidence.""" + parsed = parse_eml_bytes( + _eml_with( + """Message-ID: +From: sender@test.com +To: recipient@test.com +Subject: Has message id +Date: Mon, 27 Apr 2026 10:00:00 +0000""" + ) + ) + + assert parsed["message_id_provenance"] == "embedded" + + +def test_parse_eml_marks_missing_message_id_provenance() -> None: + """A missing Message-ID is explicitly classified as absent evidence.""" + parsed = parse_eml_bytes( + _eml_with( + """From: sender@test.com +To: recipient@test.com +Subject: No message id +Date: Mon, 27 Apr 2026 10:00:00 +0000""" + ) + ) + + assert parsed["message_id"] == "" + assert parsed["message_id_provenance"] == "missing" From 86824347dafd060a0d1eb6416ee1c2e2fca6cf5c Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Tue, 4 Aug 2026 15:56:26 +0900 Subject: [PATCH 02/86] chore(pr1195): materialize reviewed provenance fixes --- .github/workflows/pr1195-review-fixes.yml | 483 ++++++++++++++++++++++ 1 file changed, 483 insertions(+) create mode 100644 .github/workflows/pr1195-review-fixes.yml diff --git a/.github/workflows/pr1195-review-fixes.yml b/.github/workflows/pr1195-review-fixes.yml new file mode 100644 index 000000000..1427dc840 --- /dev/null +++ b/.github/workflows/pr1195-review-fixes.yml @@ -0,0 +1,483 @@ +name: PR 1195 reviewed provenance fixes + +on: + push: + branches: + - claude/contextualwisdomlab-audit-governance-qyxe67 + +permissions: + contents: read + +concurrency: + group: pr-1195-reviewed-provenance-fixes + cancel-in-progress: true + +jobs: + materialize: + if: github.actor != 'github-actions[bot]' + runs-on: ubuntu-latest + timeout-minutes: 45 + permissions: + contents: write + steps: + - name: Harden runner + uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + with: + egress-policy: audit + + - name: Checkout exact branch + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: claude/contextualwisdomlab-audit-governance-qyxe67 + fetch-depth: 0 + + - name: Apply deterministic review fixes + shell: bash + run: | + set -euo pipefail + python3 - <<'PY' + from __future__ import annotations + + import re + import subprocess + from pathlib import Path + + ROOT = Path.cwd() + + + def replace_once(text: str, old: str, new: str, label: str) -> str: + """Replace exactly one reviewed source block.""" + count = text.count(old) + if count != 1: + raise RuntimeError(f"{label}: expected one source block, found {count}") + return text.replace(old, new, 1) + + + dedupe_path = ROOT / "backend/services/email_dedupe_service.py" + dedupe = dedupe_path.read_text(encoding="utf-8") + dedupe = replace_once( + dedupe, + "import datetime\n", + "import datetime\nimport hashlib\n", + "dedupe hashlib import", + ) + dedupe = replace_once( + dedupe, + '''def strong_email_fingerprint( +''', + '''def source_email_fingerprint(source_identifier: str) -> str: + """Return a domain-separated key for one immutable source identifier. + + This fingerprint is used only when sender Date provenance is untrusted. It + deliberately excludes collection time, so two messages observed in the + same clock tick cannot collide, while an exact provider/import source is + still idempotent across retries. + """ + normalized_identifier = str(source_identifier or "").strip() + if not normalized_identifier: + raise ValueError("source_identifier must not be empty") + return hashlib.sha256( + f"naruon-source-email\\0{normalized_identifier}".encode("utf-8") + ).hexdigest() + + +def strong_email_fingerprint( +''', + "source fingerprint helper", + ) + dedupe_path.write_text(dedupe, encoding="utf-8") + + import_path = ROOT / "backend/services/email_import_service.py" + import_text = import_path.read_text(encoding="utf-8") + import_text = replace_once( + import_text, + "from services.email_dedupe_service import strong_email_fingerprint\n", + "from services.email_dedupe_service import (\n" + " source_email_fingerprint,\n" + " strong_email_fingerprint,\n" + ")\n", + "import source fingerprint", + ) + import_function_pattern = re.compile( + r"def _email_fingerprint\(parsed: EmailData, persisted_date: datetime\.datetime\) -> str:\n" + r".*?\n\n\nasync def _find_existing_email", + re.DOTALL, + ) + import_function = '''def _email_fingerprint(parsed: EmailData, persisted_date: datetime.datetime) -> str: + """Return an idempotency key without treating collection time as evidence. + + A genuine sender Date may seed the strong sender/subject/date/body key. For + missing or invalid Date headers, the key is instead derived from the + normalized embedded or content-derived fallback Message-ID. Legacy direct + callers without an identifier receive a deterministic body-inclusive + source token; the synthetic storage timestamp is never hashed. + """ + if parsed.get("date_provenance") == "parsed": + strong_fingerprint = strong_email_fingerprint( + sender=parsed.get("sender"), + subject=parsed.get("subject"), + date=persisted_date, + body=parsed.get("body"), + ) + if strong_fingerprint: + return strong_fingerprint + + source_identifier = normalize_message_id(parsed.get("message_id")) + if not source_identifier: + source_material = "\\0".join( + str(parsed.get(field_name) or "") + for field_name in ("sender", "recipients", "subject", "body") + ) + source_identifier = hashlib.sha256( + source_material.encode("utf-8") + ).hexdigest() + return source_email_fingerprint(source_identifier) + + +async def _find_existing_email''' + import_text, replacement_count = import_function_pattern.subn( + import_function, import_text, count=1 + ) + if replacement_count != 1: + raise RuntimeError( + "email import fingerprint: expected one function block" + ) + import_path.write_text(import_text, encoding="utf-8") + + imap_path = ROOT / "backend/services/imap_worker.py" + imap = imap_path.read_text(encoding="utf-8") + imap = replace_once( + imap, + "import datetime\nimport logging\n", + "import datetime\nimport hashlib\nimport logging\n", + "imap hashlib import", + ) + imap = replace_once( + imap, + "from services.email_dedupe_service import strong_email_fingerprint\n", + "from services.email_dedupe_service import (\n" + " source_email_fingerprint,\n" + " strong_email_fingerprint,\n" + ")\n", + "imap source fingerprint import", + ) + imap = replace_once( + imap, + "from services.threading_service import assign_thread_id, generate_email_fingerprint\n", + "from services.threading_service import assign_thread_id, normalize_message_id\n", + "imap threading import", + ) + imap = replace_once( + imap, + ''' owner_addresses: Iterable[str] | None = None, + is_read: bool = True, +): +''', + ''' owner_addresses: Iterable[str] | None = None, + is_read: bool = True, + source_identifier: str | None = None, +): +''', + "imap source identifier parameter", + ) + imap_fingerprint_pattern = re.compile( + r" subject = email_data\.get\(\"subject\", \"\"\)\n" + r".*?\n # Check if duplicate\n", + re.DOTALL, + ) + imap_fingerprint_block = ''' subject = email_data.get("subject", "") + date_obj = email_data.get("date") + if isinstance(date_obj, datetime.datetime): + persisted_date = ( + date_obj.astimezone(datetime.timezone.utc) + if date_obj.tzinfo is not None + else date_obj.replace(tzinfo=datetime.timezone.utc) + ) + else: + persisted_date = datetime.datetime.now(datetime.timezone.utc) + sender = email_data.get("sender", "") + recipients_list = email_data.get("recipients", []) + recipients = ( + ",".join(recipients_list) + if isinstance(recipients_list, list) + else str(recipients_list or "") + ) + + strong_fingerprint = None + if email_data.get("date_provenance") == "parsed": + strong_fingerprint = strong_email_fingerprint( + sender=sender, + subject=subject, + date=persisted_date, + body=email_data.get("body", ""), + ) + + source_token = normalize_message_id(email_data.get("message_id")) + if not source_token: + source_token = str(source_identifier or "").strip() + if not source_token: + source_material = "\\0".join( + ( + str(sender or ""), + str(recipients or ""), + str(subject or ""), + str(email_data.get("body") or ""), + ) + ) + source_token = hashlib.sha256(source_material.encode("utf-8")).hexdigest() + fingerprint = strong_fingerprint or source_email_fingerprint(source_token) + + # Check if duplicate +''' + imap, replacement_count = imap_fingerprint_pattern.subn( + imap_fingerprint_block, imap, count=1 + ) + if replacement_count != 1: + raise RuntimeError("imap fingerprint: expected one source block") + imap = replace_once( + imap, + ''' owner_addresses=owner_addresses, + is_read=is_read, +''', + ''' owner_addresses=owner_addresses, + is_read=is_read, + source_identifier=hashlib.sha256(raw_message).hexdigest(), +''', + "imap raw source identifier", + ) + imap_path.write_text(imap, encoding="utf-8") + + parser_test_path = ROOT / "backend/tests/test_email_parser_provenance.py" + parser_tests = parser_test_path.read_text(encoding="utf-8") + parser_tests = replace_once( + parser_tests, + ' return (headers.strip() + "\\n\\nBody text.").encode("utf-8")\n', + ' return (headers.strip("\\r\\n") + "\\n\\nBody text.").encode("utf-8")\n', + "parser fixture whitespace", + ) + parser_test_path.write_text(parser_tests, encoding="utf-8") + + import_test_path = ROOT / "backend/tests/test_email_import_service.py" + import_tests = import_test_path.read_text(encoding="utf-8") + import_tests += ''' + + +def test_untrusted_date_fingerprint_uses_immutable_source_not_collection_time(): + """Missing/invalid Date keys use source identity and remain body-sensitive.""" + from services.email_dedupe_service import source_email_fingerprint + from services.email_import_service import _email_fingerprint, _fallback_message_id + + fixed_fallback = datetime.datetime( + 2026, 8, 4, 6, 0, 0, tzinfo=datetime.timezone.utc + ) + shared = { + "sender": "sender@example.com", + "recipients": "recipient@example.com", + "subject": "Same subject", + } + first_identifier = _fallback_message_id(b"first raw message body") + second_identifier = _fallback_message_id(b"different raw message body") + + for provenance in ("missing", "invalid"): + first = _email_fingerprint( + { + **shared, + "body": "first body", + "message_id": first_identifier, + "date_provenance": provenance, + }, + fixed_fallback, + ) + second = _email_fingerprint( + { + **shared, + "body": "different body", + "message_id": second_identifier, + "date_provenance": provenance, + }, + fixed_fallback, + ) + assert first == source_email_fingerprint(first_identifier) + assert second == source_email_fingerprint(second_identifier) + assert first != second + + legacy_without_provenance = _email_fingerprint( + {**shared, "body": "legacy body", "message_id": ""}, + fixed_fallback, + ) + changed_legacy_body = _email_fingerprint( + {**shared, "body": "changed legacy body", "message_id": ""}, + fixed_fallback, + ) + assert legacy_without_provenance != changed_legacy_body +''' + import_test_path.write_text(import_tests, encoding="utf-8") + + imap_test_path = ROOT / "backend/tests/test_imap_worker.py" + imap_tests = imap_test_path.read_text(encoding="utf-8") + imap_tests = replace_once( + imap_tests, + "from unittest.mock import AsyncMock\n\nimport pytest\n", + "import datetime\nimport hashlib\nfrom unittest.mock import AsyncMock, MagicMock\n\nimport pytest\n", + "imap test imports", + ) + imap_tests = replace_once( + imap_tests, + ''' assert kwargs["owner_addresses"] == ["imap-user@example.com"] + + session.commit.assert_awaited_once() +''', + ''' assert kwargs["owner_addresses"] == ["imap-user@example.com"] + assert kwargs["source_identifier"] == hashlib.sha256(raw_message).hexdigest() + + session.commit.assert_awaited_once() +''', + "imap source identifier assertion", + ) + imap_tests += ''' + + +@pytest.mark.asyncio +@pytest.mark.parametrize("provenance", ["missing", "invalid"]) +async def test_process_fetched_email_does_not_dedupe_different_bodies_by_fallback_clock( + monkeypatch, provenance +): + """Untrusted Date and missing Message-ID never collapse different bodies.""" + from services.imap_worker import process_fetched_email + + session = AsyncMock() + query_result = MagicMock() + query_result.scalar_one_or_none.return_value = None + session.execute.return_value = query_result + session.add = MagicMock() + monkeypatch.setattr( + "services.imap_worker.assign_thread_id", + AsyncMock(side_effect=["thread-first", "thread-second"]), + ) + + fixed_fallback = datetime.datetime( + 2026, 8, 4, 6, 0, 0, tzinfo=datetime.timezone.utc + ) + shared = { + "message_id": "", + "sender": "sender@example.com", + "recipients": "recipient@example.com", + "subject": "Same subject", + "date": fixed_fallback, + "date_provenance": provenance, + "attachments": [], + } + + await process_fetched_email( + session, + {**shared, "body": "first body"}, + "imap-user", + "imap-org", + source_identifier=hashlib.sha256(b"first raw message").hexdigest(), + ) + await process_fetched_email( + session, + {**shared, "body": "different body"}, + "imap-user", + "imap-org", + source_identifier=hashlib.sha256(b"different raw message").hexdigest(), + ) + + added_emails = [call.args[0] for call in session.add.call_args_list] + assert len(added_emails) == 2 + assert added_emails[0].date == fixed_fallback + assert added_emails[1].date == fixed_fallback + assert added_emails[0].fingerprint != added_emails[1].fingerprint +''' + imap_test_path.write_text(imap_tests, encoding="utf-8") + + dedupe_test_path = ROOT / "backend/tests/test_email_dedupe_service.py" + dedupe_tests = dedupe_test_path.read_text(encoding="utf-8") + dedupe_tests += ''' + + +def test_source_email_fingerprint_is_stable_domain_separated_and_nonempty(): + """Immutable source keys are deterministic and reject an empty identity.""" + from services.email_dedupe_service import source_email_fingerprint + + first = source_email_fingerprint("provider-message-1") + assert first == source_email_fingerprint("provider-message-1") + assert first != source_email_fingerprint("provider-message-2") + with pytest.raises(ValueError, match="source_identifier"): + source_email_fingerprint("") +''' + dedupe_test_path.write_text(dedupe_tests, encoding="utf-8") + + base_changelog = subprocess.run( + ["git", "show", "origin/develop:CHANGELOG.md"], + check=True, + capture_output=True, + text=True, + ).stdout + email_section = '''### 이메일 메타데이터 provenance 기반 dedupe (naruon#1086) + +- RFC 5322 `Date`와 `Message-ID`의 근거를 `date_provenance`, `header_date`, `message_id_provenance`로 명시하고, 저장용 수집 시각 fallback과 발신자 제공 메타데이터를 분리했습니다. `-0000` 날짜는 timezone-aware UTC로 정규화하며 기존 `date` 저장 계약은 유지합니다. +- import와 IMAP 경로는 양측 `parsed` Date에서만 sender/subject/date/body strong fingerprint를 허용합니다. Date가 missing/invalid이면 합성 수집 시각을 해시하지 않고, 정규화된 Message-ID 또는 raw-content 기반 불투명 source identifier를 domain-separated idempotency key로 사용합니다. 같은 fallback clock·sender·subject·recipients를 가진 서로 다른 본문은 자동 중복으로 합쳐지지 않습니다. +- `email_records.date_provenance`(두 단어 `snake_case`)를 Alembic `0018_email_date_provenance`로 추가하고 기존 행은 보수적인 `unknown`으로 backfill합니다. Fellegi & Sunter의 A1/A2/A3 결정 구간에 대응하는 `auto_link`, `review_required`, `distinct` 순수 판정과 1:N disposition 해소를 제공합니다. +- 검증 명령: `PYTHONWARNINGS=error DISABLE_BACKGROUND_WORKERS=1 python -m pytest -q tests/test_email_import_service.py tests/test_imap_worker.py tests/test_email_parser_provenance.py tests/test_email_dedupe_service.py` 및 `python -m ruff check services/email_dedupe_service.py services/email_import_service.py services/email_parser.py services/imap_worker.py tests/test_email_import_service.py tests/test_imap_worker.py tests/test_email_parser_provenance.py tests/test_email_dedupe_service.py`. +- 근거 문헌(APA 7): Fellegi, I. P., & Sunter, A. B. (1969). A theory for record linkage. *Journal of the American Statistical Association, 64*(328), 1183–1210. https://doi.org/10.1080/01621459.1969.10501049 + +''' + changelog_header = "## [Unreleased]\n" + if not base_changelog.startswith(changelog_header): + raise RuntimeError("base CHANGELOG does not start with Unreleased") + (ROOT / "CHANGELOG.md").write_text( + changelog_header + email_section + base_changelog[len(changelog_header):], + encoding="utf-8", + ) + PY + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.14" + + - name: Install backend dependencies + shell: bash + run: | + set -euo pipefail + python -m pip install --disable-pip-version-check --require-hashes \ + -r backend/requirements-hashes.txt \ + -r backend/requirements-agent.txt + + - name: Verify focused contracts + shell: bash + env: + PYTHONWARNINGS: error + DISABLE_BACKGROUND_WORKERS: "1" + run: | + set -euo pipefail + cd backend + python -m ruff check \ + services/email_dedupe_service.py \ + services/email_import_service.py \ + services/email_parser.py \ + services/imap_worker.py \ + tests/test_email_import_service.py \ + tests/test_imap_worker.py \ + tests/test_email_parser_provenance.py \ + tests/test_email_dedupe_service.py + python -m pytest -q \ + tests/test_email_import_service.py \ + tests/test_imap_worker.py \ + tests/test_email_parser_provenance.py \ + tests/test_email_dedupe_service.py + cd .. + git diff --check + + - name: Commit final reviewed scope + shell: bash + run: | + set -euo pipefail + git rm -- .github/workflows/pr1195-review-fixes.yml + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + git add -A + git diff --cached --check + git commit -m "fix(email): keep collection time out of dedupe identity" + git push origin HEAD:claude/contextualwisdomlab-audit-governance-qyxe67 From e2497886e9eb606c964e868c572a3d56c80b6fdb Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Tue, 4 Aug 2026 15:56:39 +0900 Subject: [PATCH 03/86] ci: fix source-bound email deduplication --- .../workflows/pr-1195-fix-source-dedupe.yml | 604 ++++++++++++++++++ 1 file changed, 604 insertions(+) create mode 100644 .github/workflows/pr-1195-fix-source-dedupe.yml diff --git a/.github/workflows/pr-1195-fix-source-dedupe.yml b/.github/workflows/pr-1195-fix-source-dedupe.yml new file mode 100644 index 000000000..606d6a555 --- /dev/null +++ b/.github/workflows/pr-1195-fix-source-dedupe.yml @@ -0,0 +1,604 @@ +name: PR 1195 fix source-bound email deduplication + +on: + pull_request: + branches: + - develop + types: [synchronize, ready_for_review] + +permissions: + contents: read + +concurrency: + group: pr-1195-fix-source-bound-dedupe + cancel-in-progress: true + +jobs: + fix-and-verify: + if: ${{ github.event.pull_request.head.ref == 'claude/contextualwisdomlab-audit-governance-qyxe67' }} + runs-on: ubuntu-latest + timeout-minutes: 60 + permissions: + contents: write + env: + PYTHONWARNINGS: error + DISABLE_BACKGROUND_WORKERS: "1" + steps: + - name: Harden runner + uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + with: + egress-policy: audit + + - name: Checkout pull request branch + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v6 + with: + ref: claude/contextualwisdomlab-audit-governance-qyxe67 + fetch-depth: 0 + + - name: Merge the current protected base locally + shell: bash + run: | + set -euo pipefail + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + git merge --no-edit origin/develop + + - name: Set up Python + uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6 + with: + python-version: "3.14" + cache: pip + cache-dependency-path: backend/requirements-hashes.txt + + - name: Install hash-locked backend dependencies + run: >- + python -m pip install --disable-pip-version-check --require-hashes + -r backend/requirements-hashes.txt + + - name: Add failing source-identity regressions first + shell: bash + run: | + set -euo pipefail + python3 - <<'PY' + from pathlib import Path + + import_test = Path("backend/tests/test_email_import_service.py") + text = import_test.read_text(encoding="utf-8") + old = '''def test_email_fingerprint_uses_strong_key_only_for_parsed_date(): + """A strong (auto-dedupe) fingerprint is seeded only from a genuine Date. + + naruon#1086: when the RFC822 Date was missing or invalid the persisted date + is a synthetic collection-time fallback, which must not seed the strong + duplicate key. Only ``date_provenance == "parsed"`` yields the strong + fingerprint; missing/invalid provenance falls through to the weak fallback. + """ + from services.email_dedupe_service import strong_email_fingerprint + from services.email_import_service import _email_fingerprint + + persisted_date = datetime.datetime( + 2026, 4, 27, 10, 0, 0, tzinfo=datetime.timezone.utc + ) + parsed_fields = { + "sender": "sender@test.com", + "subject": "Quarterly report", + "body": "The full report body.", + "recipients": "recipient@test.com", + } + strong = strong_email_fingerprint( + sender=parsed_fields["sender"], + subject=parsed_fields["subject"], + date=persisted_date, + body=parsed_fields["body"], + ) + assert strong is not None + + assert ( + _email_fingerprint({**parsed_fields, "date_provenance": "parsed"}, persisted_date) + == strong + ) + for provenance in ("missing", "invalid"): + weak = _email_fingerprint( + {**parsed_fields, "date_provenance": provenance}, persisted_date + ) + assert weak != strong'''.replace(" ", "") + new = '''def test_email_fingerprint_uses_strong_key_only_for_parsed_date(): + """Use genuine Date metadata for strong keys and raw source for fallbacks.""" + from services.email_dedupe_service import ( + source_email_fingerprint, + strong_email_fingerprint, + ) + from services.email_import_service import _email_fingerprint + + persisted_date = datetime.datetime( + 2026, 4, 27, 10, 0, 0, tzinfo=datetime.timezone.utc + ) + parsed_fields = { + "sender": "sender@test.com", + "subject": "Quarterly report", + "body": "The full report body.", + "recipients": "recipient@test.com", + } + first_source = b"From: sender@test.com\\r\\n\\r\\nThe first body." + second_source = b"From: sender@test.com\\r\\n\\r\\nA different body." + strong = strong_email_fingerprint( + sender=parsed_fields["sender"], + subject=parsed_fields["subject"], + date=persisted_date, + body=parsed_fields["body"], + ) + assert strong is not None + + assert ( + _email_fingerprint( + {**parsed_fields, "date_provenance": "parsed"}, + persisted_date, + first_source, + ) + == strong + ) + for provenance in ("missing", "invalid"): + first = _email_fingerprint( + {**parsed_fields, "date_provenance": provenance}, + persisted_date, + first_source, + ) + second = _email_fingerprint( + {**parsed_fields, "date_provenance": provenance}, + persisted_date, + second_source, + ) + assert first == source_email_fingerprint(first_source) + assert second == source_email_fingerprint(second_source) + assert first != strong + assert first != second'''.replace(" ", "") + if new not in text: + if text.count(old) != 1: + raise SystemExit("email import fingerprint test anchor not found exactly once") + text = text.replace(old, new, 1) + import_test.write_text(text, encoding="utf-8") + + imap_test = Path("backend/tests/test_imap_worker.py") + text = imap_test.read_text(encoding="utf-8") + text = text.replace( + "from unittest.mock import AsyncMock\n", + "from unittest.mock import AsyncMock, MagicMock\n", + 1, + ) + assertion = ' assert kwargs["owner_addresses"] == ["imap-user@example.com"]\n' + if ' assert kwargs["source_content"] == raw_message\n' not in text: + if text.count(assertion) != 1: + raise SystemExit("IMAP source-content assertion anchor not found exactly once") + text = text.replace( + assertion, + assertion + ' assert kwargs["source_content"] == raw_message\n', + 1, + ) + regression = ''' + + @pytest.mark.parametrize( + ("date_header", "date_provenance"), + [ + (b"", "missing"), + (b"Date: definitely-not-a-date\\r\\n", "invalid"), + ], + ) + @pytest.mark.asyncio + async def test_untrusted_date_uses_raw_source_identity_not_collection_time( + monkeypatch, date_header, date_provenance + ): + """Different raw messages cannot collide on one collection timestamp.""" + from datetime import datetime, timezone + + from services.imap_worker import process_fetched_email + + session = AsyncMock() + session.add = MagicMock() + execute_result = MagicMock() + execute_result.scalar_one_or_none.return_value = None + session.execute.return_value = execute_result + monkeypatch.setattr( + "services.imap_worker.assign_thread_id", + AsyncMock(return_value="thread-source-bound"), + ) + + collected_at = datetime(2026, 8, 4, 6, 30, tzinfo=timezone.utc) + common = { + "subject": "Same subject", + "date": collected_at, + "date_provenance": date_provenance, + "sender": "sender@example.com", + "recipients": "recipient@example.com", + "message_id": "", + "in_reply_to": None, + "references": None, + "thread_id": None, + "reply_to": None, + "attachments": [], + } + first_raw = ( + b"From: sender@example.com\\r\\n" + b"To: recipient@example.com\\r\\n" + b"Subject: Same subject\\r\\n" + + date_header + + b"\\r\\nFirst body" + ) + second_raw = ( + b"From: sender@example.com\\r\\n" + b"To: recipient@example.com\\r\\n" + b"Subject: Same subject\\r\\n" + + date_header + + b"\\r\\nDifferent body" + ) + + await process_fetched_email( + session, + {**common, "body": "First body"}, + "owner@example.com", + "org-acme", + source_content=first_raw, + ) + await process_fetched_email( + session, + {**common, "body": "Different body"}, + "owner@example.com", + "org-acme", + source_content=second_raw, + ) + + first_email = session.add.call_args_list[0].args[0] + second_email = session.add.call_args_list[1].args[0] + assert first_email.date == collected_at + assert second_email.date == collected_at + assert first_email.fingerprint != second_email.fingerprint + '''.replace(" ", "") + if "test_untrusted_date_uses_raw_source_identity_not_collection_time" not in text: + text += regression + imap_test.write_text(text, encoding="utf-8") + + pop3_test = Path("backend/tests/test_pop3_worker.py") + text = pop3_test.read_text(encoding="utf-8") + old_signature = ''' async def fake_process_fetched_email( + db_session, email_data, user_id, organization_id, owner_addresses=None + ):'''.replace(" ", "") + new_signature = ''' async def fake_process_fetched_email( + db_session, + email_data, + user_id, + organization_id, + owner_addresses=None, + source_content=None, + ):'''.replace(" ", "") + if new_signature not in text: + if text.count(old_signature) != 1: + raise SystemExit("POP3 fake processor signature anchor not found exactly once") + text = text.replace(old_signature, new_signature, 1) + owner_entry = ' "owner_addresses": owner_addresses,\n' + if ' "source_content": source_content,\n' not in text: + if text.count(owner_entry) != 1: + raise SystemExit("POP3 imported payload anchor not found exactly once") + text = text.replace( + owner_entry, + owner_entry + ' "source_content": source_content,\n', + 1, + ) + final_assertion = ' assert imported[0]["subject"] == "POP3 import"\n' + source_assertion = ' assert imported[0]["source_content"] == raw_message\n' + if source_assertion not in text: + if text.count(final_assertion) != 1: + raise SystemExit("POP3 source-content assertion anchor not found exactly once") + text = text.replace(final_assertion, final_assertion + source_assertion, 1) + pop3_test.write_text(text, encoding="utf-8") + + parser_test = Path("backend/tests/test_email_parser_provenance.py") + text = parser_test.read_text(encoding="utf-8") + old_fixture = ' return (headers.strip() + "\\n\\nBody text.").encode("utf-8")' + new_fixture = ' return (headers.strip("\\r\\n") + "\\n\\nBody text.").encode("utf-8")' + if new_fixture not in text: + if text.count(old_fixture) != 1: + raise SystemExit("parser provenance fixture anchor not found exactly once") + text = text.replace(old_fixture, new_fixture, 1) + parser_test.write_text(text, encoding="utf-8") + PY + + set +e + ( + cd backend + python -m pytest -q \ + tests/test_email_import_service.py::test_email_fingerprint_uses_strong_key_only_for_parsed_date \ + tests/test_imap_worker.py::test_untrusted_date_uses_raw_source_identity_not_collection_time \ + tests/test_imap_worker.py::test_imap_worker_imports_fetched_rfc822_messages \ + tests/test_pop3_worker.py::test_pop3_worker_imports_retrieved_messages \ + tests/test_email_parser_provenance.py + ) + red_status=$? + set -e + if [ "$red_status" -eq 0 ]; then + echo "::error::Source-identity regressions unexpectedly passed before production remediation." + exit 1 + fi + + - name: Replace collection-time dedupe with opaque source identity + shell: bash + run: | + set -euo pipefail + python3 - <<'PY' + from pathlib import Path + + dedupe_path = Path("backend/services/email_dedupe_service.py") + text = dedupe_path.read_text(encoding="utf-8") + if "import hashlib\n" not in text: + text = text.replace("import datetime\n", "import datetime\nimport hashlib\n", 1) + anchor = '''def strong_email_fingerprint( + ''' + helper = '''def source_email_fingerprint(source_content: bytes) -> str: + """Return a domain-separated fingerprint of immutable raw source bytes. + + This key is used only when sender-provided Date evidence is missing or + invalid. It lets exact re-fetches deduplicate while ensuring a synthetic + collection timestamp can never collapse messages with different source + content. + """ + digest = hashlib.sha256() + digest.update(b"naruon-email-source-v1\\0") + digest.update(source_content) + return digest.hexdigest() + + + def strong_email_fingerprint( + '''.replace(" ", "") + if "def source_email_fingerprint" not in text: + if text.count(anchor) != 1: + raise SystemExit("strong fingerprint anchor not found exactly once") + text = text.replace(anchor, helper, 1) + dedupe_path.write_text(text, encoding="utf-8") + + import_path = Path("backend/services/email_import_service.py") + text = import_path.read_text(encoding="utf-8") + old_import = "from services.email_dedupe_service import strong_email_fingerprint\n" + new_import = '''from services.email_dedupe_service import ( + source_email_fingerprint, + strong_email_fingerprint, + ) + '''.replace(" ", "") + if new_import not in text: + if text.count(old_import) != 1: + raise SystemExit("email import dedupe import anchor not found exactly once") + text = text.replace(old_import, new_import, 1) + start = text.index("def _email_fingerprint(") + end = text.index("\n\n\nasync def _find_existing_email", start) + replacement = '''def _email_fingerprint( + parsed: EmailData, + persisted_date: datetime.datetime, + source_content: bytes, + ) -> str: + """Return a trusted-Date strong key or an immutable-source fallback key. + + ``persisted_date`` remains the storage timestamp. It participates in + automatic deduplication only when the parser proves it came from a valid + sender ``Date`` header; otherwise exact raw source bytes provide the + opaque identity and prevent collection-time collisions. + """ + strong_fingerprint = None + if parsed.get("date_provenance") == "parsed": + strong_fingerprint = strong_email_fingerprint( + sender=parsed.get("sender"), + subject=parsed.get("subject"), + date=persisted_date, + body=parsed.get("body"), + ) + if strong_fingerprint: + return strong_fingerprint + return source_email_fingerprint(source_content) + '''.replace(" ", "").rstrip() + text = text[:start] + replacement + text[end:] + old_call = " fingerprint = _email_fingerprint(parsed, persisted_date)\n" + new_call = " fingerprint = _email_fingerprint(parsed, persisted_date, content)\n" + if new_call not in text: + if text.count(old_call) != 1: + raise SystemExit("email import fingerprint call anchor not found exactly once") + text = text.replace(old_call, new_call, 1) + import_path.write_text(text, encoding="utf-8") + + imap_path = Path("backend/services/imap_worker.py") + text = imap_path.read_text(encoding="utf-8") + if "import json\n" not in text: + text = text.replace("import datetime\n", "import datetime\nimport json\n", 1) + old_import = "from services.email_dedupe_service import strong_email_fingerprint\n" + new_import = '''from services.email_dedupe_service import ( + source_email_fingerprint, + strong_email_fingerprint, + ) + '''.replace(" ", "") + if new_import not in text: + if text.count(old_import) != 1: + raise SystemExit("IMAP dedupe import anchor not found exactly once") + text = text.replace(old_import, new_import, 1) + function_anchor = "\n\nasync def process_fetched_email(\n" + helper = ''' + + def _canonical_source_content(email_data: EmailData) -> bytes: + """Serialize parsed identity fields when raw transport bytes are unavailable.""" + recipients_value = email_data.get("recipients", []) + recipients = ( + ",".join(recipients_value) + if isinstance(recipients_value, list) + else str(recipients_value or "") + ) + payload = [ + str(email_data.get("message_id") or ""), + str(email_data.get("sender") or ""), + recipients, + str(email_data.get("subject") or ""), + str(email_data.get("body") or ""), + ] + return json.dumps( + payload, + ensure_ascii=False, + separators=(",", ":"), + ).encode("utf-8") + + + async def process_fetched_email( + '''.replace(" ", "") + if "def _canonical_source_content" not in text: + if text.count(function_anchor) != 1: + raise SystemExit("process_fetched_email anchor not found exactly once") + text = text.replace(function_anchor, helper, 1) + old_signature = ''' owner_addresses: Iterable[str] | None = None, + is_read: bool = True, + ):'''.replace(" ", "") + new_signature = ''' owner_addresses: Iterable[str] | None = None, + is_read: bool = True, + source_content: bytes | None = None, + ):'''.replace(" ", "") + if new_signature not in text: + if text.count(old_signature) != 1: + raise SystemExit("process_fetched_email signature anchor not found exactly once") + text = text.replace(old_signature, new_signature, 1) + old_date = ''' date_obj = email_data.get("date") + if hasattr(date_obj, "isoformat"): + date_str = date_obj.isoformat() + else: + date_str = str(date_obj) if date_obj else "" + '''.replace(" ", "") + new_date = ' date_obj = email_data.get("date")\n' + if old_date in text: + text = text.replace(old_date, new_date, 1) + old_fingerprint = ''' fingerprint = strong_fingerprint or generate_email_fingerprint( + subject, date_str, sender, recipients + )'''.replace(" ", "") + new_fingerprint = ''' if strong_fingerprint: + fingerprint = strong_fingerprint + else: + immutable_source = ( + source_content + if source_content is not None + else _canonical_source_content(email_data) + ) + fingerprint = source_email_fingerprint(immutable_source)'''.replace(" ", "") + if new_fingerprint not in text: + if text.count(old_fingerprint) != 1: + raise SystemExit("IMAP fallback fingerprint anchor not found exactly once") + text = text.replace(old_fingerprint, new_fingerprint, 1) + old_call = ''' owner_addresses=owner_addresses, + is_read=is_read, + )'''.replace(" ", "") + new_call = ''' owner_addresses=owner_addresses, + is_read=is_read, + source_content=raw_message, + )'''.replace(" ", "") + if new_call not in text: + if text.count(old_call) != 1: + raise SystemExit("IMAP import call anchor not found exactly once") + text = text.replace(old_call, new_call, 1) + text = text.replace( + "from services.threading_service import assign_thread_id, generate_email_fingerprint\n", + "from services.threading_service import assign_thread_id\n", + 1, + ) + imap_path.write_text(text, encoding="utf-8") + + pop3_path = Path("backend/services/pop3_worker.py") + text = pop3_path.read_text(encoding="utf-8") + old_call = ''' owner_addresses=owner_addresses, + )'''.replace(" ", "") + new_call = ''' owner_addresses=owner_addresses, + source_content=raw_message, + )'''.replace(" ", "") + if new_call not in text: + if text.count(old_call) != 1: + raise SystemExit("POP3 import call anchor not found exactly once") + text = text.replace(old_call, new_call, 1) + pop3_path.write_text(text, encoding="utf-8") + + changelog_path = Path("CHANGELOG.md") + lines = changelog_path.read_text(encoding="utf-8").splitlines() + for index, line in enumerate(lines): + if line.startswith("- `email_import_service._email_fingerprint`"): + lines[index] = ( + "- `email_import_service._email_fingerprint`와 IMAP·POP3 수집 경로가 " + "`date_provenance == \"parsed\"`일 때만 sender Date 기반 strong " + "fingerprint를 사용합니다. Date가 없거나 잘못된 경우 수집 시각은 " + "저장에만 사용하고, domain-separated SHA-256 raw-source fingerprint로 " + "정확히 같은 원본 재수집만 중복 처리합니다. 같은 시각에 수집된 " + "동일 발신자·제목·수신자의 서로 다른 본문은 자동 중복으로 합쳐지지 않습니다." + ) + break + else: + raise SystemExit("email provenance changelog bullet not found") + changelog_path.write_text("\n".join(lines) + "\n", encoding="utf-8") + PY + + - name: Normalize and verify focused contracts + shell: bash + run: | + set -euo pipefail + cd backend + python -m ruff check --fix \ + services/email_dedupe_service.py \ + services/email_import_service.py \ + services/imap_worker.py \ + services/pop3_worker.py \ + tests/test_email_import_service.py \ + tests/test_imap_worker.py \ + tests/test_pop3_worker.py \ + tests/test_email_parser_provenance.py + python -m ruff format \ + services/email_dedupe_service.py \ + services/email_import_service.py \ + services/imap_worker.py \ + services/pop3_worker.py \ + tests/test_email_import_service.py \ + tests/test_imap_worker.py \ + tests/test_pop3_worker.py \ + tests/test_email_parser_provenance.py + python -m ruff check \ + services/email_dedupe_service.py \ + services/email_import_service.py \ + services/imap_worker.py \ + services/pop3_worker.py \ + tests/test_email_import_service.py \ + tests/test_imap_worker.py \ + tests/test_pop3_worker.py \ + tests/test_email_parser_provenance.py + python -m ruff format --check \ + services/email_dedupe_service.py \ + services/email_import_service.py \ + services/imap_worker.py \ + services/pop3_worker.py \ + tests/test_email_import_service.py \ + tests/test_imap_worker.py \ + tests/test_pop3_worker.py \ + tests/test_email_parser_provenance.py + python -m pytest -q \ + tests/test_email_import_service.py \ + tests/test_imap_worker.py \ + tests/test_pop3_worker.py \ + tests/test_email_parser_provenance.py \ + tests/test_email_dedupe_service.py \ + tests/test_threading_pipeline.py + + - name: Run the complete backend regression suite + shell: bash + run: | + set -euo pipefail + cd backend + python -m pytest -q + + - name: Commit and publish verified remediation + shell: bash + run: | + set -euo pipefail + git diff --check + git add \ + CHANGELOG.md \ + backend/services/email_dedupe_service.py \ + backend/services/email_import_service.py \ + backend/services/imap_worker.py \ + backend/services/pop3_worker.py \ + backend/tests/test_email_import_service.py \ + backend/tests/test_imap_worker.py \ + backend/tests/test_pop3_worker.py \ + backend/tests/test_email_parser_provenance.py + git commit -m "fix(email): bind untrusted-date dedupe to raw source" + git push origin HEAD:claude/contextualwisdomlab-audit-governance-qyxe67 From a2124fb44a11a3a326d9804bd353c457147ee363 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Tue, 4 Aug 2026 16:07:27 +0900 Subject: [PATCH 04/86] chore(pr1195): remove superseded review-fix workflow --- .github/workflows/pr1195-review-fixes.yml | 483 ---------------------- 1 file changed, 483 deletions(-) delete mode 100644 .github/workflows/pr1195-review-fixes.yml diff --git a/.github/workflows/pr1195-review-fixes.yml b/.github/workflows/pr1195-review-fixes.yml deleted file mode 100644 index 1427dc840..000000000 --- a/.github/workflows/pr1195-review-fixes.yml +++ /dev/null @@ -1,483 +0,0 @@ -name: PR 1195 reviewed provenance fixes - -on: - push: - branches: - - claude/contextualwisdomlab-audit-governance-qyxe67 - -permissions: - contents: read - -concurrency: - group: pr-1195-reviewed-provenance-fixes - cancel-in-progress: true - -jobs: - materialize: - if: github.actor != 'github-actions[bot]' - runs-on: ubuntu-latest - timeout-minutes: 45 - permissions: - contents: write - steps: - - name: Harden runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 - with: - egress-policy: audit - - - name: Checkout exact branch - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - ref: claude/contextualwisdomlab-audit-governance-qyxe67 - fetch-depth: 0 - - - name: Apply deterministic review fixes - shell: bash - run: | - set -euo pipefail - python3 - <<'PY' - from __future__ import annotations - - import re - import subprocess - from pathlib import Path - - ROOT = Path.cwd() - - - def replace_once(text: str, old: str, new: str, label: str) -> str: - """Replace exactly one reviewed source block.""" - count = text.count(old) - if count != 1: - raise RuntimeError(f"{label}: expected one source block, found {count}") - return text.replace(old, new, 1) - - - dedupe_path = ROOT / "backend/services/email_dedupe_service.py" - dedupe = dedupe_path.read_text(encoding="utf-8") - dedupe = replace_once( - dedupe, - "import datetime\n", - "import datetime\nimport hashlib\n", - "dedupe hashlib import", - ) - dedupe = replace_once( - dedupe, - '''def strong_email_fingerprint( -''', - '''def source_email_fingerprint(source_identifier: str) -> str: - """Return a domain-separated key for one immutable source identifier. - - This fingerprint is used only when sender Date provenance is untrusted. It - deliberately excludes collection time, so two messages observed in the - same clock tick cannot collide, while an exact provider/import source is - still idempotent across retries. - """ - normalized_identifier = str(source_identifier or "").strip() - if not normalized_identifier: - raise ValueError("source_identifier must not be empty") - return hashlib.sha256( - f"naruon-source-email\\0{normalized_identifier}".encode("utf-8") - ).hexdigest() - - -def strong_email_fingerprint( -''', - "source fingerprint helper", - ) - dedupe_path.write_text(dedupe, encoding="utf-8") - - import_path = ROOT / "backend/services/email_import_service.py" - import_text = import_path.read_text(encoding="utf-8") - import_text = replace_once( - import_text, - "from services.email_dedupe_service import strong_email_fingerprint\n", - "from services.email_dedupe_service import (\n" - " source_email_fingerprint,\n" - " strong_email_fingerprint,\n" - ")\n", - "import source fingerprint", - ) - import_function_pattern = re.compile( - r"def _email_fingerprint\(parsed: EmailData, persisted_date: datetime\.datetime\) -> str:\n" - r".*?\n\n\nasync def _find_existing_email", - re.DOTALL, - ) - import_function = '''def _email_fingerprint(parsed: EmailData, persisted_date: datetime.datetime) -> str: - """Return an idempotency key without treating collection time as evidence. - - A genuine sender Date may seed the strong sender/subject/date/body key. For - missing or invalid Date headers, the key is instead derived from the - normalized embedded or content-derived fallback Message-ID. Legacy direct - callers without an identifier receive a deterministic body-inclusive - source token; the synthetic storage timestamp is never hashed. - """ - if parsed.get("date_provenance") == "parsed": - strong_fingerprint = strong_email_fingerprint( - sender=parsed.get("sender"), - subject=parsed.get("subject"), - date=persisted_date, - body=parsed.get("body"), - ) - if strong_fingerprint: - return strong_fingerprint - - source_identifier = normalize_message_id(parsed.get("message_id")) - if not source_identifier: - source_material = "\\0".join( - str(parsed.get(field_name) or "") - for field_name in ("sender", "recipients", "subject", "body") - ) - source_identifier = hashlib.sha256( - source_material.encode("utf-8") - ).hexdigest() - return source_email_fingerprint(source_identifier) - - -async def _find_existing_email''' - import_text, replacement_count = import_function_pattern.subn( - import_function, import_text, count=1 - ) - if replacement_count != 1: - raise RuntimeError( - "email import fingerprint: expected one function block" - ) - import_path.write_text(import_text, encoding="utf-8") - - imap_path = ROOT / "backend/services/imap_worker.py" - imap = imap_path.read_text(encoding="utf-8") - imap = replace_once( - imap, - "import datetime\nimport logging\n", - "import datetime\nimport hashlib\nimport logging\n", - "imap hashlib import", - ) - imap = replace_once( - imap, - "from services.email_dedupe_service import strong_email_fingerprint\n", - "from services.email_dedupe_service import (\n" - " source_email_fingerprint,\n" - " strong_email_fingerprint,\n" - ")\n", - "imap source fingerprint import", - ) - imap = replace_once( - imap, - "from services.threading_service import assign_thread_id, generate_email_fingerprint\n", - "from services.threading_service import assign_thread_id, normalize_message_id\n", - "imap threading import", - ) - imap = replace_once( - imap, - ''' owner_addresses: Iterable[str] | None = None, - is_read: bool = True, -): -''', - ''' owner_addresses: Iterable[str] | None = None, - is_read: bool = True, - source_identifier: str | None = None, -): -''', - "imap source identifier parameter", - ) - imap_fingerprint_pattern = re.compile( - r" subject = email_data\.get\(\"subject\", \"\"\)\n" - r".*?\n # Check if duplicate\n", - re.DOTALL, - ) - imap_fingerprint_block = ''' subject = email_data.get("subject", "") - date_obj = email_data.get("date") - if isinstance(date_obj, datetime.datetime): - persisted_date = ( - date_obj.astimezone(datetime.timezone.utc) - if date_obj.tzinfo is not None - else date_obj.replace(tzinfo=datetime.timezone.utc) - ) - else: - persisted_date = datetime.datetime.now(datetime.timezone.utc) - sender = email_data.get("sender", "") - recipients_list = email_data.get("recipients", []) - recipients = ( - ",".join(recipients_list) - if isinstance(recipients_list, list) - else str(recipients_list or "") - ) - - strong_fingerprint = None - if email_data.get("date_provenance") == "parsed": - strong_fingerprint = strong_email_fingerprint( - sender=sender, - subject=subject, - date=persisted_date, - body=email_data.get("body", ""), - ) - - source_token = normalize_message_id(email_data.get("message_id")) - if not source_token: - source_token = str(source_identifier or "").strip() - if not source_token: - source_material = "\\0".join( - ( - str(sender or ""), - str(recipients or ""), - str(subject or ""), - str(email_data.get("body") or ""), - ) - ) - source_token = hashlib.sha256(source_material.encode("utf-8")).hexdigest() - fingerprint = strong_fingerprint or source_email_fingerprint(source_token) - - # Check if duplicate -''' - imap, replacement_count = imap_fingerprint_pattern.subn( - imap_fingerprint_block, imap, count=1 - ) - if replacement_count != 1: - raise RuntimeError("imap fingerprint: expected one source block") - imap = replace_once( - imap, - ''' owner_addresses=owner_addresses, - is_read=is_read, -''', - ''' owner_addresses=owner_addresses, - is_read=is_read, - source_identifier=hashlib.sha256(raw_message).hexdigest(), -''', - "imap raw source identifier", - ) - imap_path.write_text(imap, encoding="utf-8") - - parser_test_path = ROOT / "backend/tests/test_email_parser_provenance.py" - parser_tests = parser_test_path.read_text(encoding="utf-8") - parser_tests = replace_once( - parser_tests, - ' return (headers.strip() + "\\n\\nBody text.").encode("utf-8")\n', - ' return (headers.strip("\\r\\n") + "\\n\\nBody text.").encode("utf-8")\n', - "parser fixture whitespace", - ) - parser_test_path.write_text(parser_tests, encoding="utf-8") - - import_test_path = ROOT / "backend/tests/test_email_import_service.py" - import_tests = import_test_path.read_text(encoding="utf-8") - import_tests += ''' - - -def test_untrusted_date_fingerprint_uses_immutable_source_not_collection_time(): - """Missing/invalid Date keys use source identity and remain body-sensitive.""" - from services.email_dedupe_service import source_email_fingerprint - from services.email_import_service import _email_fingerprint, _fallback_message_id - - fixed_fallback = datetime.datetime( - 2026, 8, 4, 6, 0, 0, tzinfo=datetime.timezone.utc - ) - shared = { - "sender": "sender@example.com", - "recipients": "recipient@example.com", - "subject": "Same subject", - } - first_identifier = _fallback_message_id(b"first raw message body") - second_identifier = _fallback_message_id(b"different raw message body") - - for provenance in ("missing", "invalid"): - first = _email_fingerprint( - { - **shared, - "body": "first body", - "message_id": first_identifier, - "date_provenance": provenance, - }, - fixed_fallback, - ) - second = _email_fingerprint( - { - **shared, - "body": "different body", - "message_id": second_identifier, - "date_provenance": provenance, - }, - fixed_fallback, - ) - assert first == source_email_fingerprint(first_identifier) - assert second == source_email_fingerprint(second_identifier) - assert first != second - - legacy_without_provenance = _email_fingerprint( - {**shared, "body": "legacy body", "message_id": ""}, - fixed_fallback, - ) - changed_legacy_body = _email_fingerprint( - {**shared, "body": "changed legacy body", "message_id": ""}, - fixed_fallback, - ) - assert legacy_without_provenance != changed_legacy_body -''' - import_test_path.write_text(import_tests, encoding="utf-8") - - imap_test_path = ROOT / "backend/tests/test_imap_worker.py" - imap_tests = imap_test_path.read_text(encoding="utf-8") - imap_tests = replace_once( - imap_tests, - "from unittest.mock import AsyncMock\n\nimport pytest\n", - "import datetime\nimport hashlib\nfrom unittest.mock import AsyncMock, MagicMock\n\nimport pytest\n", - "imap test imports", - ) - imap_tests = replace_once( - imap_tests, - ''' assert kwargs["owner_addresses"] == ["imap-user@example.com"] - - session.commit.assert_awaited_once() -''', - ''' assert kwargs["owner_addresses"] == ["imap-user@example.com"] - assert kwargs["source_identifier"] == hashlib.sha256(raw_message).hexdigest() - - session.commit.assert_awaited_once() -''', - "imap source identifier assertion", - ) - imap_tests += ''' - - -@pytest.mark.asyncio -@pytest.mark.parametrize("provenance", ["missing", "invalid"]) -async def test_process_fetched_email_does_not_dedupe_different_bodies_by_fallback_clock( - monkeypatch, provenance -): - """Untrusted Date and missing Message-ID never collapse different bodies.""" - from services.imap_worker import process_fetched_email - - session = AsyncMock() - query_result = MagicMock() - query_result.scalar_one_or_none.return_value = None - session.execute.return_value = query_result - session.add = MagicMock() - monkeypatch.setattr( - "services.imap_worker.assign_thread_id", - AsyncMock(side_effect=["thread-first", "thread-second"]), - ) - - fixed_fallback = datetime.datetime( - 2026, 8, 4, 6, 0, 0, tzinfo=datetime.timezone.utc - ) - shared = { - "message_id": "", - "sender": "sender@example.com", - "recipients": "recipient@example.com", - "subject": "Same subject", - "date": fixed_fallback, - "date_provenance": provenance, - "attachments": [], - } - - await process_fetched_email( - session, - {**shared, "body": "first body"}, - "imap-user", - "imap-org", - source_identifier=hashlib.sha256(b"first raw message").hexdigest(), - ) - await process_fetched_email( - session, - {**shared, "body": "different body"}, - "imap-user", - "imap-org", - source_identifier=hashlib.sha256(b"different raw message").hexdigest(), - ) - - added_emails = [call.args[0] for call in session.add.call_args_list] - assert len(added_emails) == 2 - assert added_emails[0].date == fixed_fallback - assert added_emails[1].date == fixed_fallback - assert added_emails[0].fingerprint != added_emails[1].fingerprint -''' - imap_test_path.write_text(imap_tests, encoding="utf-8") - - dedupe_test_path = ROOT / "backend/tests/test_email_dedupe_service.py" - dedupe_tests = dedupe_test_path.read_text(encoding="utf-8") - dedupe_tests += ''' - - -def test_source_email_fingerprint_is_stable_domain_separated_and_nonempty(): - """Immutable source keys are deterministic and reject an empty identity.""" - from services.email_dedupe_service import source_email_fingerprint - - first = source_email_fingerprint("provider-message-1") - assert first == source_email_fingerprint("provider-message-1") - assert first != source_email_fingerprint("provider-message-2") - with pytest.raises(ValueError, match="source_identifier"): - source_email_fingerprint("") -''' - dedupe_test_path.write_text(dedupe_tests, encoding="utf-8") - - base_changelog = subprocess.run( - ["git", "show", "origin/develop:CHANGELOG.md"], - check=True, - capture_output=True, - text=True, - ).stdout - email_section = '''### 이메일 메타데이터 provenance 기반 dedupe (naruon#1086) - -- RFC 5322 `Date`와 `Message-ID`의 근거를 `date_provenance`, `header_date`, `message_id_provenance`로 명시하고, 저장용 수집 시각 fallback과 발신자 제공 메타데이터를 분리했습니다. `-0000` 날짜는 timezone-aware UTC로 정규화하며 기존 `date` 저장 계약은 유지합니다. -- import와 IMAP 경로는 양측 `parsed` Date에서만 sender/subject/date/body strong fingerprint를 허용합니다. Date가 missing/invalid이면 합성 수집 시각을 해시하지 않고, 정규화된 Message-ID 또는 raw-content 기반 불투명 source identifier를 domain-separated idempotency key로 사용합니다. 같은 fallback clock·sender·subject·recipients를 가진 서로 다른 본문은 자동 중복으로 합쳐지지 않습니다. -- `email_records.date_provenance`(두 단어 `snake_case`)를 Alembic `0018_email_date_provenance`로 추가하고 기존 행은 보수적인 `unknown`으로 backfill합니다. Fellegi & Sunter의 A1/A2/A3 결정 구간에 대응하는 `auto_link`, `review_required`, `distinct` 순수 판정과 1:N disposition 해소를 제공합니다. -- 검증 명령: `PYTHONWARNINGS=error DISABLE_BACKGROUND_WORKERS=1 python -m pytest -q tests/test_email_import_service.py tests/test_imap_worker.py tests/test_email_parser_provenance.py tests/test_email_dedupe_service.py` 및 `python -m ruff check services/email_dedupe_service.py services/email_import_service.py services/email_parser.py services/imap_worker.py tests/test_email_import_service.py tests/test_imap_worker.py tests/test_email_parser_provenance.py tests/test_email_dedupe_service.py`. -- 근거 문헌(APA 7): Fellegi, I. P., & Sunter, A. B. (1969). A theory for record linkage. *Journal of the American Statistical Association, 64*(328), 1183–1210. https://doi.org/10.1080/01621459.1969.10501049 - -''' - changelog_header = "## [Unreleased]\n" - if not base_changelog.startswith(changelog_header): - raise RuntimeError("base CHANGELOG does not start with Unreleased") - (ROOT / "CHANGELOG.md").write_text( - changelog_header + email_section + base_changelog[len(changelog_header):], - encoding="utf-8", - ) - PY - - - name: Set up Python - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 - with: - python-version: "3.14" - - - name: Install backend dependencies - shell: bash - run: | - set -euo pipefail - python -m pip install --disable-pip-version-check --require-hashes \ - -r backend/requirements-hashes.txt \ - -r backend/requirements-agent.txt - - - name: Verify focused contracts - shell: bash - env: - PYTHONWARNINGS: error - DISABLE_BACKGROUND_WORKERS: "1" - run: | - set -euo pipefail - cd backend - python -m ruff check \ - services/email_dedupe_service.py \ - services/email_import_service.py \ - services/email_parser.py \ - services/imap_worker.py \ - tests/test_email_import_service.py \ - tests/test_imap_worker.py \ - tests/test_email_parser_provenance.py \ - tests/test_email_dedupe_service.py - python -m pytest -q \ - tests/test_email_import_service.py \ - tests/test_imap_worker.py \ - tests/test_email_parser_provenance.py \ - tests/test_email_dedupe_service.py - cd .. - git diff --check - - - name: Commit final reviewed scope - shell: bash - run: | - set -euo pipefail - git rm -- .github/workflows/pr1195-review-fixes.yml - git config user.name "github-actions[bot]" - git config user.email "41898282+github-actions[bot]@users.noreply.github.com" - git add -A - git diff --cached --check - git commit -m "fix(email): keep collection time out of dedupe identity" - git push origin HEAD:claude/contextualwisdomlab-audit-governance-qyxe67 From 6580d40e6297fed0b03943a46240b92a1ee80bd1 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Tue, 4 Aug 2026 16:08:08 +0900 Subject: [PATCH 05/86] chore(pr1195): repair source-dedupe materializer --- .../workflows/pr1195-repair-materializer.yml | 91 +++++++++++++++++++ 1 file changed, 91 insertions(+) create mode 100644 .github/workflows/pr1195-repair-materializer.yml diff --git a/.github/workflows/pr1195-repair-materializer.yml b/.github/workflows/pr1195-repair-materializer.yml new file mode 100644 index 000000000..56feff6dd --- /dev/null +++ b/.github/workflows/pr1195-repair-materializer.yml @@ -0,0 +1,91 @@ +name: PR 1195 repair source-dedupe materializer + +on: + push: + branches: + - claude/contextualwisdomlab-audit-governance-qyxe67 + +permissions: + contents: read + +concurrency: + group: pr-1195-repair-source-dedupe-materializer + cancel-in-progress: true + +jobs: + repair: + if: github.actor != 'github-actions[bot]' + runs-on: ubuntu-latest + timeout-minutes: 15 + permissions: + contents: write + steps: + - name: Harden runner + uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + with: + egress-policy: audit + + - name: Checkout exact branch + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: claude/contextualwisdomlab-audit-governance-qyxe67 + fetch-depth: 0 + + - name: Repair the deterministic one-shot workflow + shell: bash + run: | + set -euo pipefail + python3 - <<'PY' + from pathlib import Path + + path = Path(".github/workflows/pr-1195-fix-source-dedupe.yml") + text = path.read_text(encoding="utf-8") + + old_assertion = ( + " final_assertion = ' assert imported[0][\\\"subject\\\"] " + "== \\\"POP3 import\\\"\\n'\n" + ) + new_assertion = ( + " final_assertion = ' assert imported[0][\\\"email_data\\\"]" + "[\\\"subject\\\"] == \\\"POP3 import\\\"\\n'\n" + ) + if text.count(old_assertion) != 1: + raise RuntimeError( + "expected one obsolete POP3 assertion anchor, found " + f"{text.count(old_assertion)}" + ) + text = text.replace(old_assertion, new_assertion, 1) + + commit_line = ( + ' git commit -m "fix(email): bind untrusted-date dedupe ' + 'to raw source"\n' + ) + self_delete = ( + " git rm -- " + ".github/workflows/pr-1195-fix-source-dedupe.yml\n" + + commit_line + ) + if text.count(commit_line) != 1: + raise RuntimeError( + "expected one final commit line, found " + f"{text.count(commit_line)}" + ) + text = text.replace(commit_line, self_delete, 1) + path.write_text(text, encoding="utf-8") + PY + + ruby -e 'require "yaml"; YAML.safe_load(File.read(ARGV.fetch(0)), aliases: true)' \ + .github/workflows/pr-1195-fix-source-dedupe.yml + git diff --check + + - name: Publish repaired materializer and remove this helper + shell: bash + run: | + set -euo pipefail + git rm -- .github/workflows/pr1195-repair-materializer.yml + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + git add .github/workflows/pr-1195-fix-source-dedupe.yml + git diff --cached --check + git commit -m "fix(ci): repair PR 1195 source-dedupe materializer" + git push origin HEAD:claude/contextualwisdomlab-audit-governance-qyxe67 From 138157bee8bcecfe27397a8009e1a46096a81662 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Tue, 4 Aug 2026 16:10:05 +0900 Subject: [PATCH 06/86] fix(ci): trigger PR 1195 materializer repair on review transition --- .github/workflows/pr1195-repair-materializer.yml | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/.github/workflows/pr1195-repair-materializer.yml b/.github/workflows/pr1195-repair-materializer.yml index 56feff6dd..e3dc3e641 100644 --- a/.github/workflows/pr1195-repair-materializer.yml +++ b/.github/workflows/pr1195-repair-materializer.yml @@ -1,9 +1,10 @@ name: PR 1195 repair source-dedupe materializer on: - push: + pull_request: branches: - - claude/contextualwisdomlab-audit-governance-qyxe67 + - develop + types: [synchronize, ready_for_review] permissions: contents: read @@ -14,7 +15,7 @@ concurrency: jobs: repair: - if: github.actor != 'github-actions[bot]' + if: github.event.pull_request.head.ref == 'claude/contextualwisdomlab-audit-governance-qyxe67' runs-on: ubuntu-latest timeout-minutes: 15 permissions: From f9efcf744fb1ff5a4bd35ceedf92bd81999b9e3b Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Tue, 4 Aug 2026 16:15:59 +0900 Subject: [PATCH 07/86] fix(ci): make PR 1195 materializer repair structural --- .../workflows/pr1195-repair-materializer.yml | 52 +++++++++++-------- 1 file changed, 29 insertions(+), 23 deletions(-) diff --git a/.github/workflows/pr1195-repair-materializer.yml b/.github/workflows/pr1195-repair-materializer.yml index e3dc3e641..ae3677c35 100644 --- a/.github/workflows/pr1195-repair-materializer.yml +++ b/.github/workflows/pr1195-repair-materializer.yml @@ -38,40 +38,46 @@ jobs: set -euo pipefail python3 - <<'PY' from pathlib import Path + import re path = Path(".github/workflows/pr-1195-fix-source-dedupe.yml") text = path.read_text(encoding="utf-8") - old_assertion = ( - " final_assertion = ' assert imported[0][\\\"subject\\\"] " - "== \\\"POP3 import\\\"\\n'\n" + assertion_pattern = re.compile( + r'(?m)^(?P\s*)final_assertion\s*=\s*.*POP3 import.*$' ) - new_assertion = ( - " final_assertion = ' assert imported[0][\\\"email_data\\\"]" - "[\\\"subject\\\"] == \\\"POP3 import\\\"\\n'\n" - ) - if text.count(old_assertion) != 1: + assertion_matches = list(assertion_pattern.finditer(text)) + if len(assertion_matches) != 1: raise RuntimeError( - "expected one obsolete POP3 assertion anchor, found " - f"{text.count(old_assertion)}" + "expected one POP3 assertion definition, found " + f"{len(assertion_matches)}" ) - text = text.replace(old_assertion, new_assertion, 1) - - commit_line = ( - ' git commit -m "fix(email): bind untrusted-date dedupe ' - 'to raw source"\n' + assertion_indent = assertion_matches[0].group("indent") + assertion_replacement = ( + assertion_indent + + 'final_assertion = \' assert imported[0]["email_data"]' + + '["subject"] == "POP3 import"\\n\'' ) - self_delete = ( - " git rm -- " - ".github/workflows/pr-1195-fix-source-dedupe.yml\n" - + commit_line + text = assertion_pattern.sub(assertion_replacement, text, count=1) + + commit_pattern = re.compile( + r'(?m)^(?P\s*)git commit -m ' + r'"fix\(email\): bind untrusted-date dedupe to raw source"$' ) - if text.count(commit_line) != 1: + commit_matches = list(commit_pattern.finditer(text)) + if len(commit_matches) != 1: raise RuntimeError( - "expected one final commit line, found " - f"{text.count(commit_line)}" + "expected one final commit command, found " + f"{len(commit_matches)}" ) - text = text.replace(commit_line, self_delete, 1) + commit_indent = commit_matches[0].group("indent") + commit_replacement = ( + commit_indent + + "git rm -- .github/workflows/pr-1195-fix-source-dedupe.yml\n" + + commit_indent + + 'git commit -m "fix(email): bind untrusted-date dedupe to raw source"' + ) + text = commit_pattern.sub(commit_replacement, text, count=1) path.write_text(text, encoding="utf-8") PY From 1c4c7816d3e9195349bd9a3ef2703214df0d08b6 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Tue, 4 Aug 2026 16:22:08 +0900 Subject: [PATCH 08/86] fix(ci): preserve escaped POP3 assertion in PR 1195 repair --- .../workflows/pr1195-repair-materializer.yml | 66 +++++++++++-------- 1 file changed, 38 insertions(+), 28 deletions(-) diff --git a/.github/workflows/pr1195-repair-materializer.yml b/.github/workflows/pr1195-repair-materializer.yml index ae3677c35..5689f4c4a 100644 --- a/.github/workflows/pr1195-repair-materializer.yml +++ b/.github/workflows/pr1195-repair-materializer.yml @@ -38,47 +38,57 @@ jobs: set -euo pipefail python3 - <<'PY' from pathlib import Path - import re path = Path(".github/workflows/pr-1195-fix-source-dedupe.yml") - text = path.read_text(encoding="utf-8") + lines = path.read_text(encoding="utf-8").splitlines(keepends=True) - assertion_pattern = re.compile( - r'(?m)^(?P\s*)final_assertion\s*=\s*.*POP3 import.*$' - ) - assertion_matches = list(assertion_pattern.finditer(text)) - if len(assertion_matches) != 1: + assertion_indexes = [ + index + for index, line in enumerate(lines) + if "final_assertion =" in line and "POP3 import" in line + ] + if len(assertion_indexes) != 1: raise RuntimeError( "expected one POP3 assertion definition, found " - f"{len(assertion_matches)}" + f"{len(assertion_indexes)}" ) - assertion_indent = assertion_matches[0].group("indent") - assertion_replacement = ( - assertion_indent - + 'final_assertion = \' assert imported[0]["email_data"]' - + '["subject"] == "POP3 import"\\n\'' + assertion_index = assertion_indexes[0] + assertion_indent = lines[assertion_index][ + : len(lines[assertion_index]) - len(lines[assertion_index].lstrip()) + ] + assertion_value = ( + ' assert imported[0]["email_data"]["subject"] ' + '== "POP3 import"\n' + ) + lines[assertion_index] = ( + f"{assertion_indent}final_assertion = {assertion_value!r}\n" ) - text = assertion_pattern.sub(assertion_replacement, text, count=1) - commit_pattern = re.compile( - r'(?m)^(?P\s*)git commit -m ' - r'"fix\(email\): bind untrusted-date dedupe to raw source"$' + commit_command = ( + 'git commit -m "fix(email): bind untrusted-date dedupe to raw source"' ) - commit_matches = list(commit_pattern.finditer(text)) - if len(commit_matches) != 1: + commit_indexes = [ + index + for index, line in enumerate(lines) + if line.strip() == commit_command + ] + if len(commit_indexes) != 1: raise RuntimeError( "expected one final commit command, found " - f"{len(commit_matches)}" + f"{len(commit_indexes)}" ) - commit_indent = commit_matches[0].group("indent") - commit_replacement = ( - commit_indent - + "git rm -- .github/workflows/pr-1195-fix-source-dedupe.yml\n" - + commit_indent - + 'git commit -m "fix(email): bind untrusted-date dedupe to raw source"' + commit_index = commit_indexes[0] + commit_indent = lines[commit_index][ + : len(lines[commit_index]) - len(lines[commit_index].lstrip()) + ] + self_delete = ( + f"{commit_indent}git rm -- " + ".github/workflows/pr-1195-fix-source-dedupe.yml\n" ) - text = commit_pattern.sub(commit_replacement, text, count=1) - path.write_text(text, encoding="utf-8") + if commit_index == 0 or lines[commit_index - 1] != self_delete: + lines.insert(commit_index, self_delete) + + path.write_text("".join(lines), encoding="utf-8") PY ruby -e 'require "yaml"; YAML.safe_load(File.read(ARGV.fetch(0)), aliases: true)' \ From 112c2d54bc6b077fe60c116a30e45b628ec48734 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Tue, 4 Aug 2026 16:25:41 +0900 Subject: [PATCH 09/86] ci: make source-bound dedupe repair deterministic --- .../workflows/pr-1195-fix-source-dedupe.yml | 591 +++++++++--------- 1 file changed, 290 insertions(+), 301 deletions(-) diff --git a/.github/workflows/pr-1195-fix-source-dedupe.yml b/.github/workflows/pr-1195-fix-source-dedupe.yml index 606d6a555..23c9b2714 100644 --- a/.github/workflows/pr-1195-fix-source-dedupe.yml +++ b/.github/workflows/pr-1195-fix-source-dedupe.yml @@ -55,82 +55,61 @@ jobs: python -m pip install --disable-pip-version-check --require-hashes -r backend/requirements-hashes.txt - - name: Add failing source-identity regressions first + - name: Add failing source-identity regressions shell: bash run: | set -euo pipefail - python3 - <<'PY' + cat > backend/tests/test_source_bound_email_dedupe.py <<'PY' + """Regression tests for source-bound fallback email identities.""" + + import datetime from pathlib import Path + from unittest.mock import AsyncMock, MagicMock - import_test = Path("backend/tests/test_email_import_service.py") - text = import_test.read_text(encoding="utf-8") - old = '''def test_email_fingerprint_uses_strong_key_only_for_parsed_date(): - """A strong (auto-dedupe) fingerprint is seeded only from a genuine Date. + import pytest - naruon#1086: when the RFC822 Date was missing or invalid the persisted date - is a synthetic collection-time fallback, which must not seed the strong - duplicate key. Only ``date_provenance == "parsed"`` yields the strong - fingerprint; missing/invalid provenance falls through to the weak fallback. - """ - from services.email_dedupe_service import strong_email_fingerprint - from services.email_import_service import _email_fingerprint + from services.email_dedupe_service import ( + source_email_fingerprint, + strong_email_fingerprint, + ) + from services.email_import_service import _email_fingerprint + from services.imap_worker import ( + _canonical_source_content, + process_fetched_email, + ) - persisted_date = datetime.datetime( - 2026, 4, 27, 10, 0, 0, tzinfo=datetime.timezone.utc - ) - parsed_fields = { - "sender": "sender@test.com", - "subject": "Quarterly report", - "body": "The full report body.", - "recipients": "recipient@test.com", - } - strong = strong_email_fingerprint( - sender=parsed_fields["sender"], - subject=parsed_fields["subject"], - date=persisted_date, - body=parsed_fields["body"], - ) - assert strong is not None - assert ( - _email_fingerprint({**parsed_fields, "date_provenance": "parsed"}, persisted_date) - == strong - ) - for provenance in ("missing", "invalid"): - weak = _email_fingerprint( - {**parsed_fields, "date_provenance": provenance}, persisted_date - ) - assert weak != strong'''.replace(" ", "") - new = '''def test_email_fingerprint_uses_strong_key_only_for_parsed_date(): - """Use genuine Date metadata for strong keys and raw source for fallbacks.""" - from services.email_dedupe_service import ( - source_email_fingerprint, - strong_email_fingerprint, - ) - from services.email_import_service import _email_fingerprint + def test_source_email_fingerprint_is_domain_separated_and_content_bound() -> None: + """Hash identical sources identically and distinct sources differently.""" + first = source_email_fingerprint(b"same source") + assert first == source_email_fingerprint(b"same source") + assert first != source_email_fingerprint(b"different source") + assert len(first) == 64 + + def test_import_fingerprint_uses_raw_source_when_date_is_untrusted() -> None: + """Keep storage time out of missing/invalid-Date duplicate evidence.""" persisted_date = datetime.datetime( - 2026, 4, 27, 10, 0, 0, tzinfo=datetime.timezone.utc + 2026, 8, 4, 6, 30, tzinfo=datetime.timezone.utc ) - parsed_fields = { - "sender": "sender@test.com", - "subject": "Quarterly report", - "body": "The full report body.", - "recipients": "recipient@test.com", + fields = { + "sender": "sender@example.com", + "subject": "Same subject", + "body": "Same parsed body", + "recipients": "recipient@example.com", } - first_source = b"From: sender@test.com\\r\\n\\r\\nThe first body." - second_source = b"From: sender@test.com\\r\\n\\r\\nA different body." + first_source = b"From: sender@example.com\r\n\r\nFirst raw body" + second_source = b"From: sender@example.com\r\n\r\nSecond raw body" strong = strong_email_fingerprint( - sender=parsed_fields["sender"], - subject=parsed_fields["subject"], + sender=fields["sender"], + subject=fields["subject"], date=persisted_date, - body=parsed_fields["body"], + body=fields["body"], ) assert strong is not None - assert ( _email_fingerprint( - {**parsed_fields, "date_provenance": "parsed"}, + {**fields, "date_provenance": "parsed"}, persisted_date, first_source, ) @@ -138,186 +117,159 @@ jobs: ) for provenance in ("missing", "invalid"): first = _email_fingerprint( - {**parsed_fields, "date_provenance": provenance}, + {**fields, "date_provenance": provenance}, persisted_date, first_source, ) second = _email_fingerprint( - {**parsed_fields, "date_provenance": provenance}, + {**fields, "date_provenance": provenance}, persisted_date, second_source, ) assert first == source_email_fingerprint(first_source) assert second == source_email_fingerprint(second_source) - assert first != strong - assert first != second'''.replace(" ", "") - if new not in text: - if text.count(old) != 1: - raise SystemExit("email import fingerprint test anchor not found exactly once") - text = text.replace(old, new, 1) - import_test.write_text(text, encoding="utf-8") - - imap_test = Path("backend/tests/test_imap_worker.py") - text = imap_test.read_text(encoding="utf-8") - text = text.replace( - "from unittest.mock import AsyncMock\n", - "from unittest.mock import AsyncMock, MagicMock\n", - 1, - ) - assertion = ' assert kwargs["owner_addresses"] == ["imap-user@example.com"]\n' - if ' assert kwargs["source_content"] == raw_message\n' not in text: - if text.count(assertion) != 1: - raise SystemExit("IMAP source-content assertion anchor not found exactly once") - text = text.replace( - assertion, - assertion + ' assert kwargs["source_content"] == raw_message\n', - 1, + assert first != second + + + def test_canonical_source_content_is_deterministic_and_identity_sensitive() -> None: + """Provide a stable fallback for non-transport direct callers.""" + base = { + "message_id": "", + "sender": "sender@example.com", + "recipients": ["one@example.com", "two@example.com"], + "subject": "Subject", + "body": "Body", + } + assert _canonical_source_content(base) == _canonical_source_content(dict(base)) + assert _canonical_source_content(base) != _canonical_source_content( + {**base, "body": "Different body"} + ) + assert _canonical_source_content(base) != _canonical_source_content( + {**base, "recipients": "one@example.com,two@example.com"} ) - regression = ''' - - @pytest.mark.parametrize( - ("date_header", "date_provenance"), - [ - (b"", "missing"), - (b"Date: definitely-not-a-date\\r\\n", "invalid"), - ], - ) - @pytest.mark.asyncio - async def test_untrusted_date_uses_raw_source_identity_not_collection_time( - monkeypatch, date_header, date_provenance - ): - """Different raw messages cannot collide on one collection timestamp.""" - from datetime import datetime, timezone - from services.imap_worker import process_fetched_email + @pytest.mark.asyncio + async def test_missing_date_messages_use_raw_source_not_collection_time( + monkeypatch: pytest.MonkeyPatch, + ) -> None: + """Do not merge different raw messages collected at the same instant.""" session = AsyncMock() session.add = MagicMock() - execute_result = MagicMock() - execute_result.scalar_one_or_none.return_value = None - session.execute.return_value = execute_result + query_result = MagicMock() + query_result.scalar_one_or_none.return_value = None + session.execute.return_value = query_result monkeypatch.setattr( "services.imap_worker.assign_thread_id", - AsyncMock(return_value="thread-source-bound"), + AsyncMock(side_effect=("thread-first", "thread-second")), + ) + monkeypatch.setattr( + "services.imap_worker.is_self_sent_email", + lambda _email, _owners: False, ) - collected_at = datetime(2026, 8, 4, 6, 30, tzinfo=timezone.utc) + collected_at = datetime.datetime( + 2026, 8, 4, 6, 30, tzinfo=datetime.timezone.utc + ) common = { "subject": "Same subject", "date": collected_at, - "date_provenance": date_provenance, + "date_provenance": "missing", "sender": "sender@example.com", "recipients": "recipient@example.com", "message_id": "", - "in_reply_to": None, - "references": None, - "thread_id": None, - "reply_to": None, - "attachments": [], + "body": "Same parsed body", } - first_raw = ( - b"From: sender@example.com\\r\\n" - b"To: recipient@example.com\\r\\n" - b"Subject: Same subject\\r\\n" - + date_header - + b"\\r\\nFirst body" - ) - second_raw = ( - b"From: sender@example.com\\r\\n" - b"To: recipient@example.com\\r\\n" - b"Subject: Same subject\\r\\n" - + date_header - + b"\\r\\nDifferent body" - ) + first_source = b"From: sender@example.com\r\n\r\nFirst raw body" + second_source = b"From: sender@example.com\r\n\r\nSecond raw body" await process_fetched_email( session, - {**common, "body": "First body"}, + common, "owner@example.com", "org-acme", - source_content=first_raw, + source_content=first_source, ) await process_fetched_email( session, - {**common, "body": "Different body"}, + common, "owner@example.com", "org-acme", - source_content=second_raw, + source_content=second_source, ) first_email = session.add.call_args_list[0].args[0] second_email = session.add.call_args_list[1].args[0] assert first_email.date == collected_at assert second_email.date == collected_at + assert first_email.fingerprint == source_email_fingerprint(first_source) + assert second_email.fingerprint == source_email_fingerprint(second_source) assert first_email.fingerprint != second_email.fingerprint - '''.replace(" ", "") - if "test_untrusted_date_uses_raw_source_identity_not_collection_time" not in text: - text += regression - imap_test.write_text(text, encoding="utf-8") - - pop3_test = Path("backend/tests/test_pop3_worker.py") - text = pop3_test.read_text(encoding="utf-8") - old_signature = ''' async def fake_process_fetched_email( - db_session, email_data, user_id, organization_id, owner_addresses=None - ):'''.replace(" ", "") - new_signature = ''' async def fake_process_fetched_email( - db_session, + + + @pytest.mark.asyncio + async def test_direct_caller_without_raw_source_uses_canonical_identity( + monkeypatch: pytest.MonkeyPatch, + ) -> None: + """Cover the deterministic parsed-field fallback for direct callers.""" + session = AsyncMock() + session.add = MagicMock() + query_result = MagicMock() + query_result.scalar_one_or_none.return_value = None + session.execute.return_value = query_result + monkeypatch.setattr( + "services.imap_worker.assign_thread_id", + AsyncMock(return_value="thread-canonical"), + ) + monkeypatch.setattr( + "services.imap_worker.is_self_sent_email", + lambda _email, _owners: False, + ) + email_data = { + "subject": "Canonical", + "date": datetime.datetime( + 2026, 8, 4, 6, 30, tzinfo=datetime.timezone.utc + ), + "date_provenance": "invalid", + "sender": "sender@example.com", + "recipients": ["recipient@example.com"], + "message_id": "", + "body": "Canonical body", + } + + created = await process_fetched_email( + session, email_data, - user_id, - organization_id, - owner_addresses=None, - source_content=None, - ):'''.replace(" ", "") - if new_signature not in text: - if text.count(old_signature) != 1: - raise SystemExit("POP3 fake processor signature anchor not found exactly once") - text = text.replace(old_signature, new_signature, 1) - owner_entry = ' "owner_addresses": owner_addresses,\n' - if ' "source_content": source_content,\n' not in text: - if text.count(owner_entry) != 1: - raise SystemExit("POP3 imported payload anchor not found exactly once") - text = text.replace( - owner_entry, - owner_entry + ' "source_content": source_content,\n', - 1, + "owner@example.com", + "org-acme", ) - final_assertion = ' assert imported[0]["subject"] == "POP3 import"\n' - source_assertion = ' assert imported[0]["source_content"] == raw_message\n' - if source_assertion not in text: - if text.count(final_assertion) != 1: - raise SystemExit("POP3 source-content assertion anchor not found exactly once") - text = text.replace(final_assertion, final_assertion + source_assertion, 1) - pop3_test.write_text(text, encoding="utf-8") - parser_test = Path("backend/tests/test_email_parser_provenance.py") - text = parser_test.read_text(encoding="utf-8") - old_fixture = ' return (headers.strip() + "\\n\\nBody text.").encode("utf-8")' - new_fixture = ' return (headers.strip("\\r\\n") + "\\n\\nBody text.").encode("utf-8")' - if new_fixture not in text: - if text.count(old_fixture) != 1: - raise SystemExit("parser provenance fixture anchor not found exactly once") - text = text.replace(old_fixture, new_fixture, 1) - parser_test.write_text(text, encoding="utf-8") + assert created.fingerprint == source_email_fingerprint( + _canonical_source_content(email_data) + ) + + + def test_imap_and_pop3_workers_forward_raw_transport_bytes() -> None: + """Pin raw-source propagation at both transport boundaries.""" + imap_source = Path("services/imap_worker.py").read_text(encoding="utf-8") + pop3_source = Path("services/pop3_worker.py").read_text(encoding="utf-8") + assert "source_content=raw_message" in imap_source + assert "source_content=raw_message" in pop3_source PY set +e ( cd backend - python -m pytest -q \ - tests/test_email_import_service.py::test_email_fingerprint_uses_strong_key_only_for_parsed_date \ - tests/test_imap_worker.py::test_untrusted_date_uses_raw_source_identity_not_collection_time \ - tests/test_imap_worker.py::test_imap_worker_imports_fetched_rfc822_messages \ - tests/test_pop3_worker.py::test_pop3_worker_imports_retrieved_messages \ - tests/test_email_parser_provenance.py + python -m pytest -q tests/test_source_bound_email_dedupe.py ) red_status=$? set -e if [ "$red_status" -eq 0 ]; then - echo "::error::Source-identity regressions unexpectedly passed before production remediation." + echo "::error::Source-bound dedupe regressions unexpectedly passed before remediation." exit 1 fi - - name: Replace collection-time dedupe with opaque source identity + - name: Bind fallback duplicate identity to immutable source content shell: bash run: | set -euo pipefail @@ -328,15 +280,12 @@ jobs: text = dedupe_path.read_text(encoding="utf-8") if "import hashlib\n" not in text: text = text.replace("import datetime\n", "import datetime\nimport hashlib\n", 1) - anchor = '''def strong_email_fingerprint( - ''' helper = '''def source_email_fingerprint(source_content: bytes) -> str: - """Return a domain-separated fingerprint of immutable raw source bytes. + """Return a domain-separated SHA-256 identity for immutable source bytes. - This key is used only when sender-provided Date evidence is missing or - invalid. It lets exact re-fetches deduplicate while ensuring a synthetic - collection timestamp can never collapse messages with different source - content. + The source key is used when sender-provided Date evidence is missing or + invalid. Exact re-fetches remain deduplicable, while a synthetic + collection timestamp can never collapse distinct transport messages. """ digest = hashlib.sha256() digest.update(b"naruon-email-source-v1\\0") @@ -344,26 +293,24 @@ jobs: return digest.hexdigest() - def strong_email_fingerprint( '''.replace(" ", "") if "def source_email_fingerprint" not in text: + anchor = "def strong_email_fingerprint(\n" if text.count(anchor) != 1: raise SystemExit("strong fingerprint anchor not found exactly once") - text = text.replace(anchor, helper, 1) + text = text.replace(anchor, helper + anchor, 1) dedupe_path.write_text(text, encoding="utf-8") import_path = Path("backend/services/email_import_service.py") text = import_path.read_text(encoding="utf-8") - old_import = "from services.email_dedupe_service import strong_email_fingerprint\n" - new_import = '''from services.email_dedupe_service import ( - source_email_fingerprint, - strong_email_fingerprint, + text = text.replace( + "from services.email_dedupe_service import strong_email_fingerprint\n", + "from services.email_dedupe_service import (\n" + " source_email_fingerprint,\n" + " strong_email_fingerprint,\n" + ")\n", + 1, ) - '''.replace(" ", "") - if new_import not in text: - if text.count(old_import) != 1: - raise SystemExit("email import dedupe import anchor not found exactly once") - text = text.replace(old_import, new_import, 1) start = text.index("def _email_fingerprint(") end = text.index("\n\n\nasync def _find_existing_email", start) replacement = '''def _email_fingerprint( @@ -371,12 +318,11 @@ jobs: persisted_date: datetime.datetime, source_content: bytes, ) -> str: - """Return a trusted-Date strong key or an immutable-source fallback key. + """Return trusted-Date evidence or an immutable raw-source fallback. ``persisted_date`` remains the storage timestamp. It participates in automatic deduplication only when the parser proves it came from a valid - sender ``Date`` header; otherwise exact raw source bytes provide the - opaque identity and prevent collection-time collisions. + sender ``Date`` header. """ strong_fingerprint = None if parsed.get("date_provenance") == "parsed": @@ -395,39 +341,37 @@ jobs: new_call = " fingerprint = _email_fingerprint(parsed, persisted_date, content)\n" if new_call not in text: if text.count(old_call) != 1: - raise SystemExit("email import fingerprint call anchor not found exactly once") + raise SystemExit("import fingerprint call anchor not found exactly once") text = text.replace(old_call, new_call, 1) + text = text.replace(" generate_email_fingerprint,\n", "", 1) import_path.write_text(text, encoding="utf-8") imap_path = Path("backend/services/imap_worker.py") text = imap_path.read_text(encoding="utf-8") if "import json\n" not in text: text = text.replace("import datetime\n", "import datetime\nimport json\n", 1) - old_import = "from services.email_dedupe_service import strong_email_fingerprint\n" - new_import = '''from services.email_dedupe_service import ( - source_email_fingerprint, - strong_email_fingerprint, + text = text.replace( + "from services.email_dedupe_service import strong_email_fingerprint\n", + "from services.email_dedupe_service import (\n" + " source_email_fingerprint,\n" + " strong_email_fingerprint,\n" + ")\n", + 1, ) - '''.replace(" ", "") - if new_import not in text: - if text.count(old_import) != 1: - raise SystemExit("IMAP dedupe import anchor not found exactly once") - text = text.replace(old_import, new_import, 1) - function_anchor = "\n\nasync def process_fetched_email(\n" - helper = ''' - - def _canonical_source_content(email_data: EmailData) -> bytes: - """Serialize parsed identity fields when raw transport bytes are unavailable.""" + text = text.replace( + "from services.threading_service import assign_thread_id, generate_email_fingerprint\n", + "from services.threading_service import assign_thread_id\n", + 1, + ) + process_start = text.index("async def process_fetched_email(") + process_end = text.index("\nlogger = logging.getLogger(__name__)", process_start) + process_function = '''def _canonical_source_content(email_data: EmailData) -> bytes: + """Serialize stable identity fields when transport bytes are unavailable.""" recipients_value = email_data.get("recipients", []) - recipients = ( - ",".join(recipients_value) - if isinstance(recipients_value, list) - else str(recipients_value or "") - ) payload = [ str(email_data.get("message_id") or ""), str(email_data.get("sender") or ""), - recipients, + recipients_value, str(email_data.get("subject") or ""), str(email_data.get("body") or ""), ] @@ -439,35 +383,42 @@ jobs: async def process_fetched_email( - '''.replace(" ", "") - if "def _canonical_source_content" not in text: - if text.count(function_anchor) != 1: - raise SystemExit("process_fetched_email anchor not found exactly once") - text = text.replace(function_anchor, helper, 1) - old_signature = ''' owner_addresses: Iterable[str] | None = None, - is_read: bool = True, - ):'''.replace(" ", "") - new_signature = ''' owner_addresses: Iterable[str] | None = None, + session, + email_data: EmailData, + user_id: str, + organization_id: str | None, + owner_addresses: Iterable[str] | None = None, is_read: bool = True, source_content: bytes | None = None, - ):'''.replace(" ", "") - if new_signature not in text: - if text.count(old_signature) != 1: - raise SystemExit("process_fetched_email signature anchor not found exactly once") - text = text.replace(old_signature, new_signature, 1) - old_date = ''' date_obj = email_data.get("date") - if hasattr(date_obj, "isoformat"): - date_str = date_obj.isoformat() + ): + """Persist one fetched email with provenance-safe duplicate identity.""" + subject = email_data.get("subject", "") + date_obj = email_data.get("date") + if isinstance(date_obj, datetime.datetime): + persisted_date = ( + date_obj.astimezone(datetime.timezone.utc) + if date_obj.tzinfo is not None + else date_obj.replace(tzinfo=datetime.timezone.utc) + ) else: - date_str = str(date_obj) if date_obj else "" - '''.replace(" ", "") - new_date = ' date_obj = email_data.get("date")\n' - if old_date in text: - text = text.replace(old_date, new_date, 1) - old_fingerprint = ''' fingerprint = strong_fingerprint or generate_email_fingerprint( - subject, date_str, sender, recipients - )'''.replace(" ", "") - new_fingerprint = ''' if strong_fingerprint: + persisted_date = datetime.datetime.now(datetime.timezone.utc) + sender = email_data.get("sender", "") + recipients_list = email_data.get("recipients", []) + recipients = ( + ",".join(recipients_list) + if isinstance(recipients_list, list) + else str(recipients_list or "") + ) + + strong_fingerprint = None + if email_data.get("date_provenance") == "parsed": + strong_fingerprint = strong_email_fingerprint( + sender=sender, + subject=subject, + date=persisted_date, + body=email_data.get("body", ""), + ) + if strong_fingerprint: fingerprint = strong_fingerprint else: immutable_source = ( @@ -475,58 +426,105 @@ jobs: if source_content is not None else _canonical_source_content(email_data) ) - fingerprint = source_email_fingerprint(immutable_source)'''.replace(" ", "") - if new_fingerprint not in text: - if text.count(old_fingerprint) != 1: - raise SystemExit("IMAP fallback fingerprint anchor not found exactly once") - text = text.replace(old_fingerprint, new_fingerprint, 1) - old_call = ''' owner_addresses=owner_addresses, + fingerprint = source_email_fingerprint(immutable_source) + + stmt = select(Email).where( + Email.user_id == user_id, + Email.organization_id == ( + organization_id if organization_id else None + ), + Email.fingerprint == fingerprint, + ) + result = await session.execute(stmt) + existing_email = result.scalar_one_or_none() + if existing_email: + logger.info( + "Email with fingerprint %s already exists. Skipping duplicate insertion.", + fingerprint, + ) + return existing_email + + thread_id = await assign_thread_id( + session, + email_data, + user_id=user_id, + organization_id=organization_id, + ) + new_email = Email( + user_id=user_id, + organization_id=organization_id or None, + message_id=email_data.get("message_id", ""), + thread_id=thread_id, + fingerprint=fingerprint, + sender=sender, + recipients=recipients, + subject=subject, + date=persisted_date, + date_provenance=email_data.get("date_provenance", "unknown"), + body=email_data.get("body", ""), + is_read=is_read, + embedding=[0.0] * 1536, + ) + session.add(new_email) + if is_self_sent_email(new_email, owner_addresses): + await session.flush() + await extract_knowledge_from_self_sent( + session, + new_email, + owner_addresses, + ) + return new_email + '''.replace(" ", "").rstrip() + text = text[:process_start] + process_function + text[process_end:] + import_call = ''' owner_addresses=owner_addresses, is_read=is_read, )'''.replace(" ", "") - new_call = ''' owner_addresses=owner_addresses, + import_call_with_source = ''' owner_addresses=owner_addresses, is_read=is_read, source_content=raw_message, )'''.replace(" ", "") - if new_call not in text: - if text.count(old_call) != 1: + if import_call_with_source not in text: + if text.count(import_call) != 1: raise SystemExit("IMAP import call anchor not found exactly once") - text = text.replace(old_call, new_call, 1) - text = text.replace( - "from services.threading_service import assign_thread_id, generate_email_fingerprint\n", - "from services.threading_service import assign_thread_id\n", - 1, - ) + text = text.replace(import_call, import_call_with_source, 1) imap_path.write_text(text, encoding="utf-8") pop3_path = Path("backend/services/pop3_worker.py") text = pop3_path.read_text(encoding="utf-8") - old_call = ''' owner_addresses=owner_addresses, + pop3_call = ''' owner_addresses=owner_addresses, )'''.replace(" ", "") - new_call = ''' owner_addresses=owner_addresses, + pop3_call_with_source = ''' owner_addresses=owner_addresses, source_content=raw_message, )'''.replace(" ", "") - if new_call not in text: - if text.count(old_call) != 1: + if pop3_call_with_source not in text: + if text.count(pop3_call) != 1: raise SystemExit("POP3 import call anchor not found exactly once") - text = text.replace(old_call, new_call, 1) + text = text.replace(pop3_call, pop3_call_with_source, 1) pop3_path.write_text(text, encoding="utf-8") + parser_test = Path("backend/tests/test_email_parser_provenance.py") + text = parser_test.read_text(encoding="utf-8") + text = text.replace( + 'headers.strip() + "\\n\\nBody text."', + 'headers.strip("\\r\\n") + "\\n\\nBody text."', + 1, + ) + parser_test.write_text(text, encoding="utf-8") + changelog_path = Path("CHANGELOG.md") - lines = changelog_path.read_text(encoding="utf-8").splitlines() - for index, line in enumerate(lines): - if line.startswith("- `email_import_service._email_fingerprint`"): - lines[index] = ( - "- `email_import_service._email_fingerprint`와 IMAP·POP3 수집 경로가 " - "`date_provenance == \"parsed\"`일 때만 sender Date 기반 strong " - "fingerprint를 사용합니다. Date가 없거나 잘못된 경우 수집 시각은 " - "저장에만 사용하고, domain-separated SHA-256 raw-source fingerprint로 " - "정확히 같은 원본 재수집만 중복 처리합니다. 같은 시각에 수집된 " - "동일 발신자·제목·수신자의 서로 다른 본문은 자동 중복으로 합쳐지지 않습니다." - ) - break - else: - raise SystemExit("email provenance changelog bullet not found") - changelog_path.write_text("\n".join(lines) + "\n", encoding="utf-8") + text = changelog_path.read_text(encoding="utf-8") + bullet = ( + "- Date가 없거나 잘못된 이메일의 수집 시각은 저장에만 사용하고, " + "domain-separated SHA-256 원본-source fingerprint로 정확히 같은 " + "원본 재수집만 중복 처리합니다. 같은 시각에 수집된 동일 발신자·" + "제목·수신자의 서로 다른 원문은 자동 병합되지 않습니다.\n" + ) + if bullet not in text: + marker = "## [Unreleased]\n" + if text.count(marker) != 1: + raise SystemExit("Unreleased changelog marker not found exactly once") + text = text.replace(marker, marker + bullet, 1) + changelog_path.write_text(text, encoding="utf-8") PY - name: Normalize and verify focused contracts @@ -539,38 +537,31 @@ jobs: services/email_import_service.py \ services/imap_worker.py \ services/pop3_worker.py \ - tests/test_email_import_service.py \ - tests/test_imap_worker.py \ - tests/test_pop3_worker.py \ + tests/test_source_bound_email_dedupe.py \ tests/test_email_parser_provenance.py python -m ruff format \ services/email_dedupe_service.py \ services/email_import_service.py \ services/imap_worker.py \ services/pop3_worker.py \ - tests/test_email_import_service.py \ - tests/test_imap_worker.py \ - tests/test_pop3_worker.py \ + tests/test_source_bound_email_dedupe.py \ tests/test_email_parser_provenance.py python -m ruff check \ services/email_dedupe_service.py \ services/email_import_service.py \ services/imap_worker.py \ services/pop3_worker.py \ - tests/test_email_import_service.py \ - tests/test_imap_worker.py \ - tests/test_pop3_worker.py \ + tests/test_source_bound_email_dedupe.py \ tests/test_email_parser_provenance.py python -m ruff format --check \ services/email_dedupe_service.py \ services/email_import_service.py \ services/imap_worker.py \ services/pop3_worker.py \ - tests/test_email_import_service.py \ - tests/test_imap_worker.py \ - tests/test_pop3_worker.py \ + tests/test_source_bound_email_dedupe.py \ tests/test_email_parser_provenance.py python -m pytest -q \ + tests/test_source_bound_email_dedupe.py \ tests/test_email_import_service.py \ tests/test_imap_worker.py \ tests/test_pop3_worker.py \ @@ -578,7 +569,7 @@ jobs: tests/test_email_dedupe_service.py \ tests/test_threading_pipeline.py - - name: Run the complete backend regression suite + - name: Run complete backend regression suite shell: bash run: | set -euo pipefail @@ -596,9 +587,7 @@ jobs: backend/services/email_import_service.py \ backend/services/imap_worker.py \ backend/services/pop3_worker.py \ - backend/tests/test_email_import_service.py \ - backend/tests/test_imap_worker.py \ - backend/tests/test_pop3_worker.py \ - backend/tests/test_email_parser_provenance.py + backend/tests/test_email_parser_provenance.py \ + backend/tests/test_source_bound_email_dedupe.py git commit -m "fix(email): bind untrusted-date dedupe to raw source" git push origin HEAD:claude/contextualwisdomlab-audit-governance-qyxe67 From 7a76bb4e81bde94843ede7990ae07931593fbc6f Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Tue, 4 Aug 2026 17:41:37 +0900 Subject: [PATCH 10/86] ci: replace brittle PR 1195 materializer --- .../workflows/pr-1195-fix-source-dedupe.yml | 593 ------------------ 1 file changed, 593 deletions(-) delete mode 100644 .github/workflows/pr-1195-fix-source-dedupe.yml diff --git a/.github/workflows/pr-1195-fix-source-dedupe.yml b/.github/workflows/pr-1195-fix-source-dedupe.yml deleted file mode 100644 index 23c9b2714..000000000 --- a/.github/workflows/pr-1195-fix-source-dedupe.yml +++ /dev/null @@ -1,593 +0,0 @@ -name: PR 1195 fix source-bound email deduplication - -on: - pull_request: - branches: - - develop - types: [synchronize, ready_for_review] - -permissions: - contents: read - -concurrency: - group: pr-1195-fix-source-bound-dedupe - cancel-in-progress: true - -jobs: - fix-and-verify: - if: ${{ github.event.pull_request.head.ref == 'claude/contextualwisdomlab-audit-governance-qyxe67' }} - runs-on: ubuntu-latest - timeout-minutes: 60 - permissions: - contents: write - env: - PYTHONWARNINGS: error - DISABLE_BACKGROUND_WORKERS: "1" - steps: - - name: Harden runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 - with: - egress-policy: audit - - - name: Checkout pull request branch - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v6 - with: - ref: claude/contextualwisdomlab-audit-governance-qyxe67 - fetch-depth: 0 - - - name: Merge the current protected base locally - shell: bash - run: | - set -euo pipefail - git config user.name "github-actions[bot]" - git config user.email "41898282+github-actions[bot]@users.noreply.github.com" - git merge --no-edit origin/develop - - - name: Set up Python - uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6 - with: - python-version: "3.14" - cache: pip - cache-dependency-path: backend/requirements-hashes.txt - - - name: Install hash-locked backend dependencies - run: >- - python -m pip install --disable-pip-version-check --require-hashes - -r backend/requirements-hashes.txt - - - name: Add failing source-identity regressions - shell: bash - run: | - set -euo pipefail - cat > backend/tests/test_source_bound_email_dedupe.py <<'PY' - """Regression tests for source-bound fallback email identities.""" - - import datetime - from pathlib import Path - from unittest.mock import AsyncMock, MagicMock - - import pytest - - from services.email_dedupe_service import ( - source_email_fingerprint, - strong_email_fingerprint, - ) - from services.email_import_service import _email_fingerprint - from services.imap_worker import ( - _canonical_source_content, - process_fetched_email, - ) - - - def test_source_email_fingerprint_is_domain_separated_and_content_bound() -> None: - """Hash identical sources identically and distinct sources differently.""" - first = source_email_fingerprint(b"same source") - assert first == source_email_fingerprint(b"same source") - assert first != source_email_fingerprint(b"different source") - assert len(first) == 64 - - - def test_import_fingerprint_uses_raw_source_when_date_is_untrusted() -> None: - """Keep storage time out of missing/invalid-Date duplicate evidence.""" - persisted_date = datetime.datetime( - 2026, 8, 4, 6, 30, tzinfo=datetime.timezone.utc - ) - fields = { - "sender": "sender@example.com", - "subject": "Same subject", - "body": "Same parsed body", - "recipients": "recipient@example.com", - } - first_source = b"From: sender@example.com\r\n\r\nFirst raw body" - second_source = b"From: sender@example.com\r\n\r\nSecond raw body" - strong = strong_email_fingerprint( - sender=fields["sender"], - subject=fields["subject"], - date=persisted_date, - body=fields["body"], - ) - assert strong is not None - assert ( - _email_fingerprint( - {**fields, "date_provenance": "parsed"}, - persisted_date, - first_source, - ) - == strong - ) - for provenance in ("missing", "invalid"): - first = _email_fingerprint( - {**fields, "date_provenance": provenance}, - persisted_date, - first_source, - ) - second = _email_fingerprint( - {**fields, "date_provenance": provenance}, - persisted_date, - second_source, - ) - assert first == source_email_fingerprint(first_source) - assert second == source_email_fingerprint(second_source) - assert first != second - - - def test_canonical_source_content_is_deterministic_and_identity_sensitive() -> None: - """Provide a stable fallback for non-transport direct callers.""" - base = { - "message_id": "", - "sender": "sender@example.com", - "recipients": ["one@example.com", "two@example.com"], - "subject": "Subject", - "body": "Body", - } - assert _canonical_source_content(base) == _canonical_source_content(dict(base)) - assert _canonical_source_content(base) != _canonical_source_content( - {**base, "body": "Different body"} - ) - assert _canonical_source_content(base) != _canonical_source_content( - {**base, "recipients": "one@example.com,two@example.com"} - ) - - - @pytest.mark.asyncio - async def test_missing_date_messages_use_raw_source_not_collection_time( - monkeypatch: pytest.MonkeyPatch, - ) -> None: - """Do not merge different raw messages collected at the same instant.""" - session = AsyncMock() - session.add = MagicMock() - query_result = MagicMock() - query_result.scalar_one_or_none.return_value = None - session.execute.return_value = query_result - monkeypatch.setattr( - "services.imap_worker.assign_thread_id", - AsyncMock(side_effect=("thread-first", "thread-second")), - ) - monkeypatch.setattr( - "services.imap_worker.is_self_sent_email", - lambda _email, _owners: False, - ) - - collected_at = datetime.datetime( - 2026, 8, 4, 6, 30, tzinfo=datetime.timezone.utc - ) - common = { - "subject": "Same subject", - "date": collected_at, - "date_provenance": "missing", - "sender": "sender@example.com", - "recipients": "recipient@example.com", - "message_id": "", - "body": "Same parsed body", - } - first_source = b"From: sender@example.com\r\n\r\nFirst raw body" - second_source = b"From: sender@example.com\r\n\r\nSecond raw body" - - await process_fetched_email( - session, - common, - "owner@example.com", - "org-acme", - source_content=first_source, - ) - await process_fetched_email( - session, - common, - "owner@example.com", - "org-acme", - source_content=second_source, - ) - - first_email = session.add.call_args_list[0].args[0] - second_email = session.add.call_args_list[1].args[0] - assert first_email.date == collected_at - assert second_email.date == collected_at - assert first_email.fingerprint == source_email_fingerprint(first_source) - assert second_email.fingerprint == source_email_fingerprint(second_source) - assert first_email.fingerprint != second_email.fingerprint - - - @pytest.mark.asyncio - async def test_direct_caller_without_raw_source_uses_canonical_identity( - monkeypatch: pytest.MonkeyPatch, - ) -> None: - """Cover the deterministic parsed-field fallback for direct callers.""" - session = AsyncMock() - session.add = MagicMock() - query_result = MagicMock() - query_result.scalar_one_or_none.return_value = None - session.execute.return_value = query_result - monkeypatch.setattr( - "services.imap_worker.assign_thread_id", - AsyncMock(return_value="thread-canonical"), - ) - monkeypatch.setattr( - "services.imap_worker.is_self_sent_email", - lambda _email, _owners: False, - ) - email_data = { - "subject": "Canonical", - "date": datetime.datetime( - 2026, 8, 4, 6, 30, tzinfo=datetime.timezone.utc - ), - "date_provenance": "invalid", - "sender": "sender@example.com", - "recipients": ["recipient@example.com"], - "message_id": "", - "body": "Canonical body", - } - - created = await process_fetched_email( - session, - email_data, - "owner@example.com", - "org-acme", - ) - - assert created.fingerprint == source_email_fingerprint( - _canonical_source_content(email_data) - ) - - - def test_imap_and_pop3_workers_forward_raw_transport_bytes() -> None: - """Pin raw-source propagation at both transport boundaries.""" - imap_source = Path("services/imap_worker.py").read_text(encoding="utf-8") - pop3_source = Path("services/pop3_worker.py").read_text(encoding="utf-8") - assert "source_content=raw_message" in imap_source - assert "source_content=raw_message" in pop3_source - PY - - set +e - ( - cd backend - python -m pytest -q tests/test_source_bound_email_dedupe.py - ) - red_status=$? - set -e - if [ "$red_status" -eq 0 ]; then - echo "::error::Source-bound dedupe regressions unexpectedly passed before remediation." - exit 1 - fi - - - name: Bind fallback duplicate identity to immutable source content - shell: bash - run: | - set -euo pipefail - python3 - <<'PY' - from pathlib import Path - - dedupe_path = Path("backend/services/email_dedupe_service.py") - text = dedupe_path.read_text(encoding="utf-8") - if "import hashlib\n" not in text: - text = text.replace("import datetime\n", "import datetime\nimport hashlib\n", 1) - helper = '''def source_email_fingerprint(source_content: bytes) -> str: - """Return a domain-separated SHA-256 identity for immutable source bytes. - - The source key is used when sender-provided Date evidence is missing or - invalid. Exact re-fetches remain deduplicable, while a synthetic - collection timestamp can never collapse distinct transport messages. - """ - digest = hashlib.sha256() - digest.update(b"naruon-email-source-v1\\0") - digest.update(source_content) - return digest.hexdigest() - - - '''.replace(" ", "") - if "def source_email_fingerprint" not in text: - anchor = "def strong_email_fingerprint(\n" - if text.count(anchor) != 1: - raise SystemExit("strong fingerprint anchor not found exactly once") - text = text.replace(anchor, helper + anchor, 1) - dedupe_path.write_text(text, encoding="utf-8") - - import_path = Path("backend/services/email_import_service.py") - text = import_path.read_text(encoding="utf-8") - text = text.replace( - "from services.email_dedupe_service import strong_email_fingerprint\n", - "from services.email_dedupe_service import (\n" - " source_email_fingerprint,\n" - " strong_email_fingerprint,\n" - ")\n", - 1, - ) - start = text.index("def _email_fingerprint(") - end = text.index("\n\n\nasync def _find_existing_email", start) - replacement = '''def _email_fingerprint( - parsed: EmailData, - persisted_date: datetime.datetime, - source_content: bytes, - ) -> str: - """Return trusted-Date evidence or an immutable raw-source fallback. - - ``persisted_date`` remains the storage timestamp. It participates in - automatic deduplication only when the parser proves it came from a valid - sender ``Date`` header. - """ - strong_fingerprint = None - if parsed.get("date_provenance") == "parsed": - strong_fingerprint = strong_email_fingerprint( - sender=parsed.get("sender"), - subject=parsed.get("subject"), - date=persisted_date, - body=parsed.get("body"), - ) - if strong_fingerprint: - return strong_fingerprint - return source_email_fingerprint(source_content) - '''.replace(" ", "").rstrip() - text = text[:start] + replacement + text[end:] - old_call = " fingerprint = _email_fingerprint(parsed, persisted_date)\n" - new_call = " fingerprint = _email_fingerprint(parsed, persisted_date, content)\n" - if new_call not in text: - if text.count(old_call) != 1: - raise SystemExit("import fingerprint call anchor not found exactly once") - text = text.replace(old_call, new_call, 1) - text = text.replace(" generate_email_fingerprint,\n", "", 1) - import_path.write_text(text, encoding="utf-8") - - imap_path = Path("backend/services/imap_worker.py") - text = imap_path.read_text(encoding="utf-8") - if "import json\n" not in text: - text = text.replace("import datetime\n", "import datetime\nimport json\n", 1) - text = text.replace( - "from services.email_dedupe_service import strong_email_fingerprint\n", - "from services.email_dedupe_service import (\n" - " source_email_fingerprint,\n" - " strong_email_fingerprint,\n" - ")\n", - 1, - ) - text = text.replace( - "from services.threading_service import assign_thread_id, generate_email_fingerprint\n", - "from services.threading_service import assign_thread_id\n", - 1, - ) - process_start = text.index("async def process_fetched_email(") - process_end = text.index("\nlogger = logging.getLogger(__name__)", process_start) - process_function = '''def _canonical_source_content(email_data: EmailData) -> bytes: - """Serialize stable identity fields when transport bytes are unavailable.""" - recipients_value = email_data.get("recipients", []) - payload = [ - str(email_data.get("message_id") or ""), - str(email_data.get("sender") or ""), - recipients_value, - str(email_data.get("subject") or ""), - str(email_data.get("body") or ""), - ] - return json.dumps( - payload, - ensure_ascii=False, - separators=(",", ":"), - ).encode("utf-8") - - - async def process_fetched_email( - session, - email_data: EmailData, - user_id: str, - organization_id: str | None, - owner_addresses: Iterable[str] | None = None, - is_read: bool = True, - source_content: bytes | None = None, - ): - """Persist one fetched email with provenance-safe duplicate identity.""" - subject = email_data.get("subject", "") - date_obj = email_data.get("date") - if isinstance(date_obj, datetime.datetime): - persisted_date = ( - date_obj.astimezone(datetime.timezone.utc) - if date_obj.tzinfo is not None - else date_obj.replace(tzinfo=datetime.timezone.utc) - ) - else: - persisted_date = datetime.datetime.now(datetime.timezone.utc) - sender = email_data.get("sender", "") - recipients_list = email_data.get("recipients", []) - recipients = ( - ",".join(recipients_list) - if isinstance(recipients_list, list) - else str(recipients_list or "") - ) - - strong_fingerprint = None - if email_data.get("date_provenance") == "parsed": - strong_fingerprint = strong_email_fingerprint( - sender=sender, - subject=subject, - date=persisted_date, - body=email_data.get("body", ""), - ) - if strong_fingerprint: - fingerprint = strong_fingerprint - else: - immutable_source = ( - source_content - if source_content is not None - else _canonical_source_content(email_data) - ) - fingerprint = source_email_fingerprint(immutable_source) - - stmt = select(Email).where( - Email.user_id == user_id, - Email.organization_id == ( - organization_id if organization_id else None - ), - Email.fingerprint == fingerprint, - ) - result = await session.execute(stmt) - existing_email = result.scalar_one_or_none() - if existing_email: - logger.info( - "Email with fingerprint %s already exists. Skipping duplicate insertion.", - fingerprint, - ) - return existing_email - - thread_id = await assign_thread_id( - session, - email_data, - user_id=user_id, - organization_id=organization_id, - ) - new_email = Email( - user_id=user_id, - organization_id=organization_id or None, - message_id=email_data.get("message_id", ""), - thread_id=thread_id, - fingerprint=fingerprint, - sender=sender, - recipients=recipients, - subject=subject, - date=persisted_date, - date_provenance=email_data.get("date_provenance", "unknown"), - body=email_data.get("body", ""), - is_read=is_read, - embedding=[0.0] * 1536, - ) - session.add(new_email) - if is_self_sent_email(new_email, owner_addresses): - await session.flush() - await extract_knowledge_from_self_sent( - session, - new_email, - owner_addresses, - ) - return new_email - '''.replace(" ", "").rstrip() - text = text[:process_start] + process_function + text[process_end:] - import_call = ''' owner_addresses=owner_addresses, - is_read=is_read, - )'''.replace(" ", "") - import_call_with_source = ''' owner_addresses=owner_addresses, - is_read=is_read, - source_content=raw_message, - )'''.replace(" ", "") - if import_call_with_source not in text: - if text.count(import_call) != 1: - raise SystemExit("IMAP import call anchor not found exactly once") - text = text.replace(import_call, import_call_with_source, 1) - imap_path.write_text(text, encoding="utf-8") - - pop3_path = Path("backend/services/pop3_worker.py") - text = pop3_path.read_text(encoding="utf-8") - pop3_call = ''' owner_addresses=owner_addresses, - )'''.replace(" ", "") - pop3_call_with_source = ''' owner_addresses=owner_addresses, - source_content=raw_message, - )'''.replace(" ", "") - if pop3_call_with_source not in text: - if text.count(pop3_call) != 1: - raise SystemExit("POP3 import call anchor not found exactly once") - text = text.replace(pop3_call, pop3_call_with_source, 1) - pop3_path.write_text(text, encoding="utf-8") - - parser_test = Path("backend/tests/test_email_parser_provenance.py") - text = parser_test.read_text(encoding="utf-8") - text = text.replace( - 'headers.strip() + "\\n\\nBody text."', - 'headers.strip("\\r\\n") + "\\n\\nBody text."', - 1, - ) - parser_test.write_text(text, encoding="utf-8") - - changelog_path = Path("CHANGELOG.md") - text = changelog_path.read_text(encoding="utf-8") - bullet = ( - "- Date가 없거나 잘못된 이메일의 수집 시각은 저장에만 사용하고, " - "domain-separated SHA-256 원본-source fingerprint로 정확히 같은 " - "원본 재수집만 중복 처리합니다. 같은 시각에 수집된 동일 발신자·" - "제목·수신자의 서로 다른 원문은 자동 병합되지 않습니다.\n" - ) - if bullet not in text: - marker = "## [Unreleased]\n" - if text.count(marker) != 1: - raise SystemExit("Unreleased changelog marker not found exactly once") - text = text.replace(marker, marker + bullet, 1) - changelog_path.write_text(text, encoding="utf-8") - PY - - - name: Normalize and verify focused contracts - shell: bash - run: | - set -euo pipefail - cd backend - python -m ruff check --fix \ - services/email_dedupe_service.py \ - services/email_import_service.py \ - services/imap_worker.py \ - services/pop3_worker.py \ - tests/test_source_bound_email_dedupe.py \ - tests/test_email_parser_provenance.py - python -m ruff format \ - services/email_dedupe_service.py \ - services/email_import_service.py \ - services/imap_worker.py \ - services/pop3_worker.py \ - tests/test_source_bound_email_dedupe.py \ - tests/test_email_parser_provenance.py - python -m ruff check \ - services/email_dedupe_service.py \ - services/email_import_service.py \ - services/imap_worker.py \ - services/pop3_worker.py \ - tests/test_source_bound_email_dedupe.py \ - tests/test_email_parser_provenance.py - python -m ruff format --check \ - services/email_dedupe_service.py \ - services/email_import_service.py \ - services/imap_worker.py \ - services/pop3_worker.py \ - tests/test_source_bound_email_dedupe.py \ - tests/test_email_parser_provenance.py - python -m pytest -q \ - tests/test_source_bound_email_dedupe.py \ - tests/test_email_import_service.py \ - tests/test_imap_worker.py \ - tests/test_pop3_worker.py \ - tests/test_email_parser_provenance.py \ - tests/test_email_dedupe_service.py \ - tests/test_threading_pipeline.py - - - name: Run complete backend regression suite - shell: bash - run: | - set -euo pipefail - cd backend - python -m pytest -q - - - name: Commit and publish verified remediation - shell: bash - run: | - set -euo pipefail - git diff --check - git add \ - CHANGELOG.md \ - backend/services/email_dedupe_service.py \ - backend/services/email_import_service.py \ - backend/services/imap_worker.py \ - backend/services/pop3_worker.py \ - backend/tests/test_email_parser_provenance.py \ - backend/tests/test_source_bound_email_dedupe.py - git commit -m "fix(email): bind untrusted-date dedupe to raw source" - git push origin HEAD:claude/contextualwisdomlab-audit-governance-qyxe67 From cd68cbec18bd61ffa671de5b35bbe2f1bd00b0e4 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Tue, 4 Aug 2026 17:41:50 +0900 Subject: [PATCH 11/86] ci: remove superseded PR 1195 repair helper --- .../workflows/pr1195-repair-materializer.yml | 108 ------------------ 1 file changed, 108 deletions(-) delete mode 100644 .github/workflows/pr1195-repair-materializer.yml diff --git a/.github/workflows/pr1195-repair-materializer.yml b/.github/workflows/pr1195-repair-materializer.yml deleted file mode 100644 index 5689f4c4a..000000000 --- a/.github/workflows/pr1195-repair-materializer.yml +++ /dev/null @@ -1,108 +0,0 @@ -name: PR 1195 repair source-dedupe materializer - -on: - pull_request: - branches: - - develop - types: [synchronize, ready_for_review] - -permissions: - contents: read - -concurrency: - group: pr-1195-repair-source-dedupe-materializer - cancel-in-progress: true - -jobs: - repair: - if: github.event.pull_request.head.ref == 'claude/contextualwisdomlab-audit-governance-qyxe67' - runs-on: ubuntu-latest - timeout-minutes: 15 - permissions: - contents: write - steps: - - name: Harden runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 - with: - egress-policy: audit - - - name: Checkout exact branch - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - ref: claude/contextualwisdomlab-audit-governance-qyxe67 - fetch-depth: 0 - - - name: Repair the deterministic one-shot workflow - shell: bash - run: | - set -euo pipefail - python3 - <<'PY' - from pathlib import Path - - path = Path(".github/workflows/pr-1195-fix-source-dedupe.yml") - lines = path.read_text(encoding="utf-8").splitlines(keepends=True) - - assertion_indexes = [ - index - for index, line in enumerate(lines) - if "final_assertion =" in line and "POP3 import" in line - ] - if len(assertion_indexes) != 1: - raise RuntimeError( - "expected one POP3 assertion definition, found " - f"{len(assertion_indexes)}" - ) - assertion_index = assertion_indexes[0] - assertion_indent = lines[assertion_index][ - : len(lines[assertion_index]) - len(lines[assertion_index].lstrip()) - ] - assertion_value = ( - ' assert imported[0]["email_data"]["subject"] ' - '== "POP3 import"\n' - ) - lines[assertion_index] = ( - f"{assertion_indent}final_assertion = {assertion_value!r}\n" - ) - - commit_command = ( - 'git commit -m "fix(email): bind untrusted-date dedupe to raw source"' - ) - commit_indexes = [ - index - for index, line in enumerate(lines) - if line.strip() == commit_command - ] - if len(commit_indexes) != 1: - raise RuntimeError( - "expected one final commit command, found " - f"{len(commit_indexes)}" - ) - commit_index = commit_indexes[0] - commit_indent = lines[commit_index][ - : len(lines[commit_index]) - len(lines[commit_index].lstrip()) - ] - self_delete = ( - f"{commit_indent}git rm -- " - ".github/workflows/pr-1195-fix-source-dedupe.yml\n" - ) - if commit_index == 0 or lines[commit_index - 1] != self_delete: - lines.insert(commit_index, self_delete) - - path.write_text("".join(lines), encoding="utf-8") - PY - - ruby -e 'require "yaml"; YAML.safe_load(File.read(ARGV.fetch(0)), aliases: true)' \ - .github/workflows/pr-1195-fix-source-dedupe.yml - git diff --check - - - name: Publish repaired materializer and remove this helper - shell: bash - run: | - set -euo pipefail - git rm -- .github/workflows/pr1195-repair-materializer.yml - git config user.name "github-actions[bot]" - git config user.email "41898282+github-actions[bot]@users.noreply.github.com" - git add .github/workflows/pr-1195-fix-source-dedupe.yml - git diff --cached --check - git commit -m "fix(ci): repair PR 1195 source-dedupe materializer" - git push origin HEAD:claude/contextualwisdomlab-audit-governance-qyxe67 From 08a7cb1ad1de479baa7f854c88d83884144c3a83 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Tue, 4 Aug 2026 17:44:35 +0900 Subject: [PATCH 12/86] ci: finalize PR 1195 source-bound dedupe --- .../pr-1195-source-identity-finalize.yml | 636 ++++++++++++++++++ 1 file changed, 636 insertions(+) create mode 100644 .github/workflows/pr-1195-source-identity-finalize.yml diff --git a/.github/workflows/pr-1195-source-identity-finalize.yml b/.github/workflows/pr-1195-source-identity-finalize.yml new file mode 100644 index 000000000..891993c50 --- /dev/null +++ b/.github/workflows/pr-1195-source-identity-finalize.yml @@ -0,0 +1,636 @@ +name: PR 1195 source identity finalizer + +on: + pull_request: + branches: + - develop + types: [synchronize, ready_for_review] + +permissions: + contents: read + +concurrency: + group: pr-1195-source-identity-finalizer + cancel-in-progress: true + +jobs: + finalize: + if: ${{ github.event.pull_request.head.ref == 'claude/contextualwisdomlab-audit-governance-qyxe67' }} + runs-on: ubuntu-latest + timeout-minutes: 60 + permissions: + contents: write + env: + PYTHONWARNINGS: error + DISABLE_BACKGROUND_WORKERS: "1" + steps: + - name: Harden runner + uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + with: + egress-policy: audit + + - name: Checkout exact pull request branch + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: claude/contextualwisdomlab-audit-governance-qyxe67 + fetch-depth: 0 + + - name: Add source-identity regressions first + shell: bash + run: | + set -euo pipefail + cat > backend/tests/test_source_bound_email_dedupe.py <<'PY' + """Regression tests for source-bound fallback email identities.""" + + import datetime + from unittest.mock import AsyncMock, MagicMock + + import pytest + + from services.email_dedupe_service import ( + canonical_email_source_content, + source_email_fingerprint, + strong_email_fingerprint, + ) + from services.email_import_service import _email_fingerprint + from services.imap_worker import process_fetched_email + + + def test_source_email_fingerprint_is_stable_and_content_bound() -> None: + """Hash exact sources identically and distinct sources differently.""" + first = source_email_fingerprint(b"same source") + assert first == source_email_fingerprint(b"same source") + assert first != source_email_fingerprint(b"different source") + assert len(first) == 64 + + + def test_canonical_source_content_excludes_collection_date() -> None: + """Keep synthetic collection time outside direct-caller identity.""" + base = { + "message_id": "", + "sender": "sender@example.com", + "recipients": ["one@example.com", "two@example.com"], + "subject": "Subject", + "body": "Body", + "attachments": [{"filename": "note.txt", "content": b"note"}], + } + first = { + **base, + "date": datetime.datetime( + 2026, 8, 4, 6, 30, tzinfo=datetime.timezone.utc + ), + "date_provenance": "missing", + } + second = { + **base, + "date": datetime.datetime( + 2026, 8, 4, 7, 30, tzinfo=datetime.timezone.utc + ), + "date_provenance": "invalid", + } + assert canonical_email_source_content(first) == canonical_email_source_content( + second + ) + assert canonical_email_source_content(first) != canonical_email_source_content( + {**second, "body": "Different body"} + ) + + + def test_import_fingerprint_uses_raw_source_when_date_is_untrusted() -> None: + """Keep storage time out of missing/invalid-Date duplicate evidence.""" + persisted_date = datetime.datetime( + 2026, 8, 4, 6, 30, tzinfo=datetime.timezone.utc + ) + fields = { + "sender": "sender@example.com", + "subject": "Same subject", + "body": "Same parsed body", + "recipients": "recipient@example.com", + } + first_source = b"From: sender@example.com\r\n\r\nFirst raw body" + second_source = b"From: sender@example.com\r\n\r\nSecond raw body" + strong = strong_email_fingerprint( + sender=fields["sender"], + subject=fields["subject"], + date=persisted_date, + body=fields["body"], + ) + assert strong is not None + assert ( + _email_fingerprint( + {**fields, "date_provenance": "parsed"}, + persisted_date, + first_source, + ) + == strong + ) + for provenance in ("missing", "invalid"): + first = _email_fingerprint( + {**fields, "date_provenance": provenance}, + persisted_date, + first_source, + ) + second = _email_fingerprint( + {**fields, "date_provenance": provenance}, + persisted_date, + second_source, + ) + assert first == source_email_fingerprint(first_source) + assert second == source_email_fingerprint(second_source) + assert first != second + + + def test_import_fingerprint_direct_fallback_is_collection_time_independent() -> None: + """Give direct callers a deterministic fallback when raw bytes are absent.""" + parsed = { + "message_id": "", + "sender": "sender@example.com", + "recipients": "recipient@example.com", + "subject": "Direct", + "body": "Body", + "date_provenance": "missing", + } + first_time = datetime.datetime( + 2026, 8, 4, 6, 30, tzinfo=datetime.timezone.utc + ) + second_time = first_time + datetime.timedelta(hours=1) + assert _email_fingerprint(parsed, first_time) == _email_fingerprint( + parsed, second_time + ) + + + @pytest.mark.asyncio + async def test_missing_date_messages_use_raw_source_not_collection_time( + monkeypatch: pytest.MonkeyPatch, + ) -> None: + """Do not merge different raw messages collected at the same instant.""" + session = AsyncMock() + session.add = MagicMock() + query_result = MagicMock() + query_result.scalar_one_or_none.return_value = None + session.execute.return_value = query_result + monkeypatch.setattr( + "services.imap_worker.assign_thread_id", + AsyncMock(side_effect=("thread-first", "thread-second")), + ) + monkeypatch.setattr( + "services.imap_worker.is_self_sent_email", + lambda _email, _owners: False, + ) + + collected_at = datetime.datetime( + 2026, 8, 4, 6, 30, tzinfo=datetime.timezone.utc + ) + common = { + "subject": "Same subject", + "date": collected_at, + "date_provenance": "missing", + "sender": "sender@example.com", + "recipients": "recipient@example.com", + "message_id": "", + "body": "Same parsed body", + } + first_source = b"From: sender@example.com\r\n\r\nFirst raw body" + second_source = b"From: sender@example.com\r\n\r\nSecond raw body" + + await process_fetched_email( + session, + common, + "owner@example.com", + "org-acme", + source_content=first_source, + ) + await process_fetched_email( + session, + common, + "owner@example.com", + "org-acme", + source_content=second_source, + ) + + first_email = session.add.call_args_list[0].args[0] + second_email = session.add.call_args_list[1].args[0] + assert first_email.date == collected_at + assert second_email.date == collected_at + assert first_email.fingerprint == source_email_fingerprint(first_source) + assert second_email.fingerprint == source_email_fingerprint(second_source) + assert first_email.fingerprint != second_email.fingerprint + PY + + set +e + ( + cd backend + python -m pytest -q tests/test_source_bound_email_dedupe.py + ) + red_status=$? + set -e + if [ "$red_status" -eq 0 ]; then + echo "::error::Source-bound dedupe regressions unexpectedly passed before remediation." + exit 1 + fi + + - name: Bind untrusted-Date identity to immutable source content + shell: bash + run: | + set -euo pipefail + python3 - <<'PY' + from pathlib import Path + + dedupe_path = Path("backend/services/email_dedupe_service.py") + text = dedupe_path.read_text(encoding="utf-8") + old_imports = "import datetime\nfrom collections.abc import Iterable\n" + new_imports = ( + "import datetime\n" + "import hashlib\n" + "import json\n" + "from collections.abc import Iterable, Mapping\n" + ) + if new_imports not in text: + if text.count(old_imports) != 1: + raise SystemExit("dedupe import anchor not found exactly once") + text = text.replace(old_imports, new_imports, 1) + helper = '''_CANONICAL_SOURCE_FIELDS = ( + "message_id", + "sender", + "recipients", + "subject", + "body", + "reply_to", + "in_reply_to", + "references", + "attachments", + ) + + + def canonical_email_source_content(email_data: Mapping[str, object]) -> bytes: + """Serialize stable parsed fields when immutable transport bytes are absent. + + Collection-time ``date`` values and their provenance are deliberately + excluded so retries cannot acquire a new identity merely because they ran + later. Transport-backed import paths should pass their raw RFC822 bytes. + """ + payload = { + field: email_data.get(field) for field in _CANONICAL_SOURCE_FIELDS + } + return json.dumps( + payload, + ensure_ascii=False, + sort_keys=True, + separators=(",", ":"), + default=str, + ).encode("utf-8", errors="surrogatepass") + + + def source_email_fingerprint(source_content: bytes) -> str: + """Return a domain-separated SHA-256 identity for immutable source bytes.""" + digest = hashlib.sha256() + digest.update(b"naruon-email-source-v1\\0") + digest.update(source_content) + return digest.hexdigest() + + + '''.replace(" ", "") + if "def source_email_fingerprint" not in text: + anchor = "def strong_email_fingerprint(\n" + if text.count(anchor) != 1: + raise SystemExit("strong fingerprint anchor not found exactly once") + text = text.replace(anchor, helper + anchor, 1) + dedupe_path.write_text(text, encoding="utf-8") + + import_path = Path("backend/services/email_import_service.py") + text = import_path.read_text(encoding="utf-8") + old_import = "from services.email_dedupe_service import strong_email_fingerprint\n" + new_import = '''from services.email_dedupe_service import ( + canonical_email_source_content, + source_email_fingerprint, + strong_email_fingerprint, + ) + '''.replace(" ", "") + if new_import not in text: + if text.count(old_import) != 1: + raise SystemExit("import-service dedupe import anchor not found exactly once") + text = text.replace(old_import, new_import, 1) + start = text.index("def _email_fingerprint(") + end = text.index("\n\n\nasync def _find_existing_email", start) + replacement = '''def _email_fingerprint( + parsed: EmailData, + persisted_date: datetime.datetime, + source_content: bytes | None = None, + ) -> str: + """Return trusted-Date evidence or a source-bound fallback identity. + + ``persisted_date`` remains the storage timestamp. It participates in + automatic deduplication only when the parser proves it came from a valid + sender ``Date`` header. Missing or invalid Date evidence falls back to + immutable transport bytes, or deterministic parsed fields for direct + callers that cannot provide those bytes. + """ + strong_fingerprint = None + if parsed.get("date_provenance") == "parsed": + strong_fingerprint = strong_email_fingerprint( + sender=parsed.get("sender"), + subject=parsed.get("subject"), + date=persisted_date, + body=parsed.get("body"), + ) + if strong_fingerprint: + return strong_fingerprint + source_identity = ( + source_content + if source_content is not None + else canonical_email_source_content(parsed) + ) + return source_email_fingerprint(source_identity) + '''.replace(" ", "").rstrip() + text = text[:start] + replacement + text[end:] + old_call = " fingerprint = _email_fingerprint(parsed, persisted_date)\n" + new_call = " fingerprint = _email_fingerprint(parsed, persisted_date, content)\n" + if new_call not in text: + if text.count(old_call) != 1: + raise SystemExit("import fingerprint call anchor not found exactly once") + text = text.replace(old_call, new_call, 1) + text = text.replace(" generate_email_fingerprint,\n", "", 1) + import_path.write_text(text, encoding="utf-8") + + imap_path = Path("backend/services/imap_worker.py") + text = imap_path.read_text(encoding="utf-8") + old_import = "from services.email_dedupe_service import strong_email_fingerprint\n" + new_import = '''from services.email_dedupe_service import ( + canonical_email_source_content, + source_email_fingerprint, + strong_email_fingerprint, + ) + '''.replace(" ", "") + if new_import not in text: + if text.count(old_import) != 1: + raise SystemExit("IMAP dedupe import anchor not found exactly once") + text = text.replace(old_import, new_import, 1) + old_threading_import = ( + "from services.threading_service import " + "assign_thread_id, generate_email_fingerprint\n" + ) + if old_threading_import in text: + text = text.replace( + old_threading_import, + "from services.threading_service import assign_thread_id\n", + 1, + ) + process_start = text.index("async def process_fetched_email(") + process_end = text.index("\nlogger = logging.getLogger(__name__)", process_start) + process_function = '''async def process_fetched_email( + session, + email_data: EmailData, + user_id: str, + organization_id: str | None, + owner_addresses: Iterable[str] | None = None, + is_read: bool = True, + source_content: bytes | None = None, + ) -> Email: + """Persist one fetched email with provenance-safe duplicate identity.""" + subject = email_data.get("subject", "") + date_obj = email_data.get("date") + if isinstance(date_obj, datetime.datetime): + persisted_date = ( + date_obj.astimezone(datetime.timezone.utc) + if date_obj.tzinfo is not None + else date_obj.replace(tzinfo=datetime.timezone.utc) + ) + else: + persisted_date = datetime.datetime.now(datetime.timezone.utc) + sender = email_data.get("sender", "") + recipients_list = email_data.get("recipients", []) + recipients = ( + ",".join(recipients_list) + if isinstance(recipients_list, list) + else str(recipients_list or "") + ) + + strong_fingerprint = None + if email_data.get("date_provenance") == "parsed": + strong_fingerprint = strong_email_fingerprint( + sender=sender, + subject=subject, + date=persisted_date, + body=email_data.get("body", ""), + ) + source_identity = ( + source_content + if source_content is not None + else canonical_email_source_content(email_data) + ) + fingerprint = strong_fingerprint or source_email_fingerprint( + source_identity + ) + + stmt = select(Email).where( + Email.user_id == user_id, + Email.organization_id + == (organization_id if organization_id else None), + Email.fingerprint == fingerprint, + ) + result = await session.execute(stmt) + existing_email = result.scalar_one_or_none() + if existing_email: + logger.info( + "Email with fingerprint %s already exists. Skipping duplicate insertion.", + fingerprint, + ) + return existing_email + + thread_id = await assign_thread_id( + session, + email_data, + user_id=user_id, + organization_id=organization_id, + ) + new_email = Email( + user_id=user_id, + organization_id=organization_id or None, + message_id=email_data.get("message_id", ""), + thread_id=thread_id, + fingerprint=fingerprint, + sender=sender, + recipients=recipients, + subject=subject, + date=persisted_date, + date_provenance=email_data.get("date_provenance", "unknown"), + body=email_data.get("body", ""), + is_read=is_read, + embedding=[0.0] * 1536, + ) + session.add(new_email) + if is_self_sent_email(new_email, owner_addresses): + await session.flush() + await extract_knowledge_from_self_sent( + session, new_email, owner_addresses + ) + return new_email + '''.replace(" ", "").rstrip() + text = text[:process_start] + process_function + text[process_end:] + old_call = ''' await process_fetched_email( + session, + email_data, + config.user_id, + config.organization_id, + owner_addresses=owner_addresses, + is_read=is_read, + )'''.replace(" ", "") + new_call = ''' await process_fetched_email( + session, + email_data, + config.user_id, + config.organization_id, + owner_addresses=owner_addresses, + is_read=is_read, + source_content=raw_message, + )'''.replace(" ", "") + if new_call not in text: + if text.count(old_call) != 1: + raise SystemExit("IMAP transport call anchor not found exactly once") + text = text.replace(old_call, new_call, 1) + imap_path.write_text(text, encoding="utf-8") + + pop3_path = Path("backend/services/pop3_worker.py") + text = pop3_path.read_text(encoding="utf-8") + old_call = ''' await process_fetched_email( + session, + email_data, + config.user_id, + config.organization_id, + owner_addresses=owner_addresses, + )'''.replace(" ", "") + new_call = ''' await process_fetched_email( + session, + email_data, + config.user_id, + config.organization_id, + owner_addresses=owner_addresses, + source_content=raw_message, + )'''.replace(" ", "") + if new_call not in text: + if text.count(old_call) != 1: + raise SystemExit("POP3 transport call anchor not found exactly once") + text = text.replace(old_call, new_call, 1) + pop3_path.write_text(text, encoding="utf-8") + + imap_test = Path("backend/tests/test_imap_worker.py") + text = imap_test.read_text(encoding="utf-8") + assertion = ' assert kwargs["owner_addresses"] == ["imap-user@example.com"]\n' + source_assertion = ' assert kwargs["source_content"] == raw_message\n' + if source_assertion not in text: + if text.count(assertion) != 1: + raise SystemExit("IMAP source assertion anchor not found exactly once") + text = text.replace(assertion, assertion + source_assertion, 1) + imap_test.write_text(text, encoding="utf-8") + + pop3_test = Path("backend/tests/test_pop3_worker.py") + text = pop3_test.read_text(encoding="utf-8") + old_signature = ''' async def fake_process_fetched_email( + db_session, email_data, user_id, organization_id, owner_addresses=None + ):'''.replace(" ", "") + new_signature = ''' async def fake_process_fetched_email( + db_session, + email_data, + user_id, + organization_id, + owner_addresses=None, + source_content=None, + ):'''.replace(" ", "") + if new_signature not in text: + if text.count(old_signature) != 1: + raise SystemExit("POP3 fake processor signature not found exactly once") + text = text.replace(old_signature, new_signature, 1) + owner_entry = ' "owner_addresses": owner_addresses,\n' + source_entry = ' "source_content": source_content,\n' + if source_entry not in text: + if text.count(owner_entry) != 1: + raise SystemExit("POP3 source payload anchor not found exactly once") + text = text.replace(owner_entry, owner_entry + source_entry, 1) + final_assertion = ( + ' assert imported[0]["email_data"]["subject"] == "POP3 import"\n' + ) + source_assertion = ' assert imported[0]["source_content"] == raw_message\n' + if source_assertion not in text: + if text.count(final_assertion) != 1: + raise SystemExit("POP3 source assertion anchor not found exactly once") + text = text.replace(final_assertion, final_assertion + source_assertion, 1) + pop3_test.write_text(text, encoding="utf-8") + + changelog_path = Path("CHANGELOG.md") + text = changelog_path.read_text(encoding="utf-8") + old_bullet = '- `email_import_service._email_fingerprint`가 이제 `date_provenance == "parsed"`일 때만 strong(자동 중복 판정용) fingerprint를 생성합니다. `Date` 헤더가 없거나 잘못돼 `persisted_date`가 합성 수집 시각인 경우 strong key를 만들지 않고 weak fallback fingerprint(수집 시각이 매번 달라 거짓 중복을 만들 수 없음)로 내려갑니다 — 합성 시각이 strong-duplicate 근거로 승격되지 않습니다. `persisted_date`는 파서의 `date`에서만 오고 업로드 파일명에서 오지 않으므로, 날짜형 파일명이 `Date` 근거로 승격되지 않는 계약도 구조적으로 유지됩니다.\n' + new_bullet = '- `email_import_service._email_fingerprint`는 `date_provenance == "parsed"`일 때만 strong(자동 중복 판정용) fingerprint를 생성합니다. `Date` 헤더가 없거나 잘못된 경우 합성 수집 시각을 identity에서 완전히 제외하고, 업로드·IMAP·POP3가 전달한 immutable RFC822 원문 bytes의 domain-separated SHA-256을 fallback key로 사용합니다. 따라서 동일 원문의 재수집은 수집 시각이 달라도 idempotent하고, 같은 시각에 수집된 서로 다른 원문은 충돌하지 않습니다. 원문 bytes를 제공할 수 없는 직접 호출자는 Date를 제외한 안정적인 parsed-field canonicalization을 사용합니다.\n' + if new_bullet not in text: + if text.count(old_bullet) != 1: + raise SystemExit("changelog source-identity bullet not found exactly once") + text = text.replace(old_bullet, new_bullet, 1) + changelog_path.write_text(text, encoding="utf-8") + PY + git diff --check + + - name: Set up Python + uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6 + with: + python-version: "3.14" + cache: pip + cache-dependency-path: backend/requirements-hashes.txt + + - name: Install hash-locked backend dependencies + run: >- + python -m pip install --disable-pip-version-check --require-hashes + -r backend/requirements-hashes.txt + + - name: Verify focused and full backend contracts + shell: bash + run: | + set -euo pipefail + cd backend + python -m ruff check \ + services/email_dedupe_service.py \ + services/email_import_service.py \ + services/imap_worker.py \ + services/pop3_worker.py \ + tests/test_source_bound_email_dedupe.py \ + tests/test_email_import_service.py \ + tests/test_imap_worker.py \ + tests/test_pop3_worker.py \ + tests/test_threading_pipeline.py + python -m ruff format --check \ + services/email_dedupe_service.py \ + services/email_import_service.py \ + services/imap_worker.py \ + services/pop3_worker.py \ + tests/test_source_bound_email_dedupe.py \ + tests/test_email_import_service.py \ + tests/test_imap_worker.py \ + tests/test_pop3_worker.py \ + tests/test_threading_pipeline.py + python -m pytest -q \ + tests/test_source_bound_email_dedupe.py \ + tests/test_email_dedupe_service.py \ + tests/test_email_import_service.py \ + tests/test_email_parser_provenance.py \ + tests/test_imap_worker.py \ + tests/test_pop3_worker.py \ + tests/test_threading_pipeline.py + python -m pytest -q + + - name: Publish verified source-bound remediation + shell: bash + run: | + set -euo pipefail + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + git rm -- .github/workflows/pr-1195-source-identity-finalize.yml + git add \ + CHANGELOG.md \ + backend/services/email_dedupe_service.py \ + backend/services/email_import_service.py \ + backend/services/imap_worker.py \ + backend/services/pop3_worker.py \ + backend/tests/test_source_bound_email_dedupe.py \ + backend/tests/test_imap_worker.py \ + backend/tests/test_pop3_worker.py + git diff --cached --check + git commit -m "fix(email): bind untrusted-date dedupe to raw source" + git push origin HEAD:claude/contextualwisdomlab-audit-governance-qyxe67 From 3a3590737308b27dbc8d5f5fc3557fdbb5c890fd Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Tue, 4 Aug 2026 18:09:54 +0900 Subject: [PATCH 13/86] ci: harden PR 1195 source-identity finalizer --- .../pr-1195-source-identity-finalize.yml | 423 ++++++++---------- 1 file changed, 181 insertions(+), 242 deletions(-) diff --git a/.github/workflows/pr-1195-source-identity-finalize.yml b/.github/workflows/pr-1195-source-identity-finalize.yml index 891993c50..8581a9e9b 100644 --- a/.github/workflows/pr-1195-source-identity-finalize.yml +++ b/.github/workflows/pr-1195-source-identity-finalize.yml @@ -35,7 +35,19 @@ jobs: ref: claude/contextualwisdomlab-audit-governance-qyxe67 fetch-depth: 0 - - name: Add source-identity regressions first + - name: Set up Python + uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6 + with: + python-version: "3.14" + cache: pip + cache-dependency-path: backend/requirements-hashes.txt + + - name: Install hash-locked backend dependencies + run: >- + python -m pip install --disable-pip-version-check --require-hashes + -r backend/requirements-hashes.txt + + - name: Add source-identity regressions and prove red shell: bash run: | set -euo pipefail @@ -96,8 +108,8 @@ jobs: ) - def test_import_fingerprint_uses_raw_source_when_date_is_untrusted() -> None: - """Keep storage time out of missing/invalid-Date duplicate evidence.""" + def test_import_fingerprint_uses_trusted_date_or_raw_source() -> None: + """Use strong Date evidence only when provenance is genuinely parsed.""" persisted_date = datetime.datetime( 2026, 8, 4, 6, 30, tzinfo=datetime.timezone.utc ) @@ -140,8 +152,8 @@ jobs: assert first != second - def test_import_fingerprint_direct_fallback_is_collection_time_independent() -> None: - """Give direct callers a deterministic fallback when raw bytes are absent.""" + def test_direct_fallback_is_collection_time_independent() -> None: + """Give callers without raw bytes a stable non-date fallback identity.""" parsed = { "message_id": "", "sender": "sender@example.com", @@ -236,19 +248,24 @@ jobs: python3 - <<'PY' from pathlib import Path + + def replace_once(text: str, old: str, new: str, label: str) -> str: + if new in text: + return text + count = text.count(old) + if count != 1: + raise SystemExit(f"{label}: expected one anchor, found {count}") + return text.replace(old, new, 1) + + dedupe_path = Path("backend/services/email_dedupe_service.py") text = dedupe_path.read_text(encoding="utf-8") - old_imports = "import datetime\nfrom collections.abc import Iterable\n" - new_imports = ( - "import datetime\n" - "import hashlib\n" - "import json\n" - "from collections.abc import Iterable, Mapping\n" + text = replace_once( + text, + "import datetime\nfrom collections.abc import Iterable\n", + "import datetime\nimport hashlib\nimport json\nfrom collections.abc import Iterable, Mapping\n", + "dedupe imports", ) - if new_imports not in text: - if text.count(old_imports) != 1: - raise SystemExit("dedupe import anchor not found exactly once") - text = text.replace(old_imports, new_imports, 1) helper = '''_CANONICAL_SOURCE_FIELDS = ( "message_id", "sender", @@ -263,11 +280,10 @@ jobs: def canonical_email_source_content(email_data: Mapping[str, object]) -> bytes: - """Serialize stable parsed fields when immutable transport bytes are absent. + """Serialize stable parsed fields when raw transport bytes are unavailable. Collection-time ``date`` values and their provenance are deliberately - excluded so retries cannot acquire a new identity merely because they ran - later. Transport-backed import paths should pass their raw RFC822 bytes. + excluded. Transport-backed paths should provide exact RFC822 bytes. """ payload = { field: email_data.get(field) for field in _CANONICAL_SOURCE_FIELDS @@ -282,7 +298,7 @@ jobs: def source_email_fingerprint(source_content: bytes) -> str: - """Return a domain-separated SHA-256 identity for immutable source bytes.""" + """Return a domain-separated SHA-256 identity for source bytes.""" digest = hashlib.sha256() digest.update(b"naruon-email-source-v1\\0") digest.update(source_content) @@ -291,25 +307,26 @@ jobs: '''.replace(" ", "") if "def source_email_fingerprint" not in text: - anchor = "def strong_email_fingerprint(\n" - if text.count(anchor) != 1: - raise SystemExit("strong fingerprint anchor not found exactly once") - text = text.replace(anchor, helper + anchor, 1) + text = replace_once( + text, + "def strong_email_fingerprint(\n", + helper + "def strong_email_fingerprint(\n", + "dedupe helper insertion", + ) dedupe_path.write_text(text, encoding="utf-8") import_path = Path("backend/services/email_import_service.py") text = import_path.read_text(encoding="utf-8") - old_import = "from services.email_dedupe_service import strong_email_fingerprint\n" - new_import = '''from services.email_dedupe_service import ( - canonical_email_source_content, - source_email_fingerprint, - strong_email_fingerprint, + text = replace_once( + text, + "from services.email_dedupe_service import strong_email_fingerprint\n", + "from services.email_dedupe_service import (\n" + " canonical_email_source_content,\n" + " source_email_fingerprint,\n" + " strong_email_fingerprint,\n" + ")\n", + "import-service dedupe import", ) - '''.replace(" ", "") - if new_import not in text: - if text.count(old_import) != 1: - raise SystemExit("import-service dedupe import anchor not found exactly once") - text = text.replace(old_import, new_import, 1) start = text.index("def _email_fingerprint(") end = text.index("\n\n\nasync def _find_existing_email", start) replacement = '''def _email_fingerprint( @@ -319,11 +336,8 @@ jobs: ) -> str: """Return trusted-Date evidence or a source-bound fallback identity. - ``persisted_date`` remains the storage timestamp. It participates in - automatic deduplication only when the parser proves it came from a valid - sender ``Date`` header. Missing or invalid Date evidence falls back to - immutable transport bytes, or deterministic parsed fields for direct - callers that cannot provide those bytes. + ``persisted_date`` remains the storage timestamp and participates in + duplicate evidence only when it came from a valid sender ``Date``. """ strong_fingerprint = None if parsed.get("date_provenance") == "parsed": @@ -343,68 +357,55 @@ jobs: return source_email_fingerprint(source_identity) '''.replace(" ", "").rstrip() text = text[:start] + replacement + text[end:] - old_call = " fingerprint = _email_fingerprint(parsed, persisted_date)\n" - new_call = " fingerprint = _email_fingerprint(parsed, persisted_date, content)\n" - if new_call not in text: - if text.count(old_call) != 1: - raise SystemExit("import fingerprint call anchor not found exactly once") - text = text.replace(old_call, new_call, 1) + text = replace_once( + text, + " fingerprint = _email_fingerprint(parsed, persisted_date)\n", + " fingerprint = _email_fingerprint(parsed, persisted_date, content)\n", + "import fingerprint call", + ) text = text.replace(" generate_email_fingerprint,\n", "", 1) import_path.write_text(text, encoding="utf-8") imap_path = Path("backend/services/imap_worker.py") text = imap_path.read_text(encoding="utf-8") - old_import = "from services.email_dedupe_service import strong_email_fingerprint\n" - new_import = '''from services.email_dedupe_service import ( - canonical_email_source_content, - source_email_fingerprint, - strong_email_fingerprint, + text = replace_once( + text, + "from services.email_dedupe_service import strong_email_fingerprint\n", + "from services.email_dedupe_service import (\n" + " canonical_email_source_content,\n" + " source_email_fingerprint,\n" + " strong_email_fingerprint,\n" + ")\n", + "IMAP dedupe import", ) - '''.replace(" ", "") - if new_import not in text: - if text.count(old_import) != 1: - raise SystemExit("IMAP dedupe import anchor not found exactly once") - text = text.replace(old_import, new_import, 1) - old_threading_import = ( - "from services.threading_service import " - "assign_thread_id, generate_email_fingerprint\n" + text = replace_once( + text, + "from services.threading_service import assign_thread_id, generate_email_fingerprint\n", + "from services.threading_service import assign_thread_id\n", + "IMAP threading import", ) - if old_threading_import in text: - text = text.replace( - old_threading_import, - "from services.threading_service import assign_thread_id\n", - 1, - ) - process_start = text.index("async def process_fetched_email(") - process_end = text.index("\nlogger = logging.getLogger(__name__)", process_start) - process_function = '''async def process_fetched_email( - session, - email_data: EmailData, - user_id: str, - organization_id: str | None, - owner_addresses: Iterable[str] | None = None, - is_read: bool = True, - source_content: bytes | None = None, - ) -> Email: - """Persist one fetched email with provenance-safe duplicate identity.""" - subject = email_data.get("subject", "") - date_obj = email_data.get("date") - if isinstance(date_obj, datetime.datetime): - persisted_date = ( - date_obj.astimezone(datetime.timezone.utc) - if date_obj.tzinfo is not None - else date_obj.replace(tzinfo=datetime.timezone.utc) - ) - else: - persisted_date = datetime.datetime.now(datetime.timezone.utc) - sender = email_data.get("sender", "") - recipients_list = email_data.get("recipients", []) - recipients = ( - ",".join(recipients_list) - if isinstance(recipients_list, list) - else str(recipients_list or "") - ) - + text = replace_once( + text, + " is_read: bool = True,\n):\n subject = email_data.get(\"subject\", \"\")\n", + " is_read: bool = True,\n" + " source_content: bytes | None = None,\n" + ") -> Email:\n" + " \"\"\"Persist one fetched email with provenance-safe identity.\"\"\"\n" + " subject = email_data.get(\"subject\", \"\")\n", + "IMAP process signature", + ) + text = replace_once( + text, + " if hasattr(date_obj, \"isoformat\"):\n" + " date_str = date_obj.isoformat()\n" + " else:\n" + " date_str = str(date_obj) if date_obj else \"\"\n", + "", + "IMAP synthetic date string", + ) + fingerprint_start = text.index(" # Seed the strong") + fingerprint_end = text.index("\n\n # Check if duplicate", fingerprint_start) + fingerprint_block = ''' # Seed strong duplicate evidence only from a genuinely parsed Date. strong_fingerprint = None if email_data.get("date_provenance") == "parsed": strong_fingerprint = strong_email_fingerprint( @@ -421,170 +422,110 @@ jobs: fingerprint = strong_fingerprint or source_email_fingerprint( source_identity ) - - stmt = select(Email).where( - Email.user_id == user_id, - Email.organization_id - == (organization_id if organization_id else None), - Email.fingerprint == fingerprint, - ) - result = await session.execute(stmt) - existing_email = result.scalar_one_or_none() - if existing_email: - logger.info( - "Email with fingerprint %s already exists. Skipping duplicate insertion.", - fingerprint, - ) - return existing_email - - thread_id = await assign_thread_id( - session, - email_data, - user_id=user_id, - organization_id=organization_id, - ) - new_email = Email( - user_id=user_id, - organization_id=organization_id or None, - message_id=email_data.get("message_id", ""), - thread_id=thread_id, - fingerprint=fingerprint, - sender=sender, - recipients=recipients, - subject=subject, - date=persisted_date, - date_provenance=email_data.get("date_provenance", "unknown"), - body=email_data.get("body", ""), - is_read=is_read, - embedding=[0.0] * 1536, - ) - session.add(new_email) - if is_self_sent_email(new_email, owner_addresses): - await session.flush() - await extract_knowledge_from_self_sent( - session, new_email, owner_addresses - ) - return new_email '''.replace(" ", "").rstrip() - text = text[:process_start] + process_function + text[process_end:] - old_call = ''' await process_fetched_email( - session, - email_data, - config.user_id, - config.organization_id, - owner_addresses=owner_addresses, - is_read=is_read, - )'''.replace(" ", "") - new_call = ''' await process_fetched_email( - session, - email_data, - config.user_id, - config.organization_id, - owner_addresses=owner_addresses, - is_read=is_read, - source_content=raw_message, - )'''.replace(" ", "") - if new_call not in text: - if text.count(old_call) != 1: - raise SystemExit("IMAP transport call anchor not found exactly once") - text = text.replace(old_call, new_call, 1) + text = text[:fingerprint_start] + fingerprint_block + text[fingerprint_end:] + text = replace_once( + text, + " is_read=is_read,\n )\n", + " is_read=is_read,\n" + " source_content=raw_message,\n" + " )\n", + "IMAP raw-source propagation", + ) imap_path.write_text(text, encoding="utf-8") pop3_path = Path("backend/services/pop3_worker.py") text = pop3_path.read_text(encoding="utf-8") - old_call = ''' await process_fetched_email( - session, - email_data, - config.user_id, - config.organization_id, - owner_addresses=owner_addresses, - )'''.replace(" ", "") - new_call = ''' await process_fetched_email( - session, - email_data, - config.user_id, - config.organization_id, - owner_addresses=owner_addresses, - source_content=raw_message, - )'''.replace(" ", "") - if new_call not in text: - if text.count(old_call) != 1: - raise SystemExit("POP3 transport call anchor not found exactly once") - text = text.replace(old_call, new_call, 1) + text = replace_once( + text, + " owner_addresses=owner_addresses,\n )\n", + " owner_addresses=owner_addresses,\n" + " source_content=raw_message,\n" + " )\n", + "POP3 raw-source propagation", + ) pop3_path.write_text(text, encoding="utf-8") - imap_test = Path("backend/tests/test_imap_worker.py") - text = imap_test.read_text(encoding="utf-8") - assertion = ' assert kwargs["owner_addresses"] == ["imap-user@example.com"]\n' - source_assertion = ' assert kwargs["source_content"] == raw_message\n' - if source_assertion not in text: - if text.count(assertion) != 1: - raise SystemExit("IMAP source assertion anchor not found exactly once") - text = text.replace(assertion, assertion + source_assertion, 1) - imap_test.write_text(text, encoding="utf-8") - - pop3_test = Path("backend/tests/test_pop3_worker.py") - text = pop3_test.read_text(encoding="utf-8") - old_signature = ''' async def fake_process_fetched_email( - db_session, email_data, user_id, organization_id, owner_addresses=None - ):'''.replace(" ", "") - new_signature = ''' async def fake_process_fetched_email( - db_session, - email_data, - user_id, - organization_id, - owner_addresses=None, - source_content=None, - ):'''.replace(" ", "") - if new_signature not in text: - if text.count(old_signature) != 1: - raise SystemExit("POP3 fake processor signature not found exactly once") - text = text.replace(old_signature, new_signature, 1) - owner_entry = ' "owner_addresses": owner_addresses,\n' - source_entry = ' "source_content": source_content,\n' - if source_entry not in text: - if text.count(owner_entry) != 1: - raise SystemExit("POP3 source payload anchor not found exactly once") - text = text.replace(owner_entry, owner_entry + source_entry, 1) - final_assertion = ( + imap_test_path = Path("backend/tests/test_imap_worker.py") + text = imap_test_path.read_text(encoding="utf-8") + text = replace_once( + text, + ' assert kwargs["owner_addresses"] == ["imap-user@example.com"]\n', + ' assert kwargs["owner_addresses"] == ["imap-user@example.com"]\n' + ' assert kwargs["source_content"] == raw_message\n', + "IMAP source assertion", + ) + imap_test_path.write_text(text, encoding="utf-8") + + pop3_test_path = Path("backend/tests/test_pop3_worker.py") + text = pop3_test_path.read_text(encoding="utf-8") + text = replace_once( + text, + " async def fake_process_fetched_email(\n" + " db_session, email_data, user_id, organization_id, owner_addresses=None\n" + " ):\n", + " async def fake_process_fetched_email(\n" + " db_session,\n" + " email_data,\n" + " user_id,\n" + " organization_id,\n" + " owner_addresses=None,\n" + " source_content=None,\n" + " ):\n", + "POP3 fake processor signature", + ) + text = replace_once( + text, + ' "owner_addresses": owner_addresses,\n', + ' "owner_addresses": owner_addresses,\n' + ' "source_content": source_content,\n', + "POP3 source payload", + ) + text = replace_once( + text, + ' assert imported[0]["email_data"]["subject"] == "POP3 import"\n', ' assert imported[0]["email_data"]["subject"] == "POP3 import"\n' + ' assert imported[0]["source_content"] == raw_message\n', + "POP3 source assertion", ) - source_assertion = ' assert imported[0]["source_content"] == raw_message\n' - if source_assertion not in text: - if text.count(final_assertion) != 1: - raise SystemExit("POP3 source assertion anchor not found exactly once") - text = text.replace(final_assertion, final_assertion + source_assertion, 1) - pop3_test.write_text(text, encoding="utf-8") + pop3_test_path.write_text(text, encoding="utf-8") changelog_path = Path("CHANGELOG.md") - text = changelog_path.read_text(encoding="utf-8") - old_bullet = '- `email_import_service._email_fingerprint`가 이제 `date_provenance == "parsed"`일 때만 strong(자동 중복 판정용) fingerprint를 생성합니다. `Date` 헤더가 없거나 잘못돼 `persisted_date`가 합성 수집 시각인 경우 strong key를 만들지 않고 weak fallback fingerprint(수집 시각이 매번 달라 거짓 중복을 만들 수 없음)로 내려갑니다 — 합성 시각이 strong-duplicate 근거로 승격되지 않습니다. `persisted_date`는 파서의 `date`에서만 오고 업로드 파일명에서 오지 않으므로, 날짜형 파일명이 `Date` 근거로 승격되지 않는 계약도 구조적으로 유지됩니다.\n' - new_bullet = '- `email_import_service._email_fingerprint`는 `date_provenance == "parsed"`일 때만 strong(자동 중복 판정용) fingerprint를 생성합니다. `Date` 헤더가 없거나 잘못된 경우 합성 수집 시각을 identity에서 완전히 제외하고, 업로드·IMAP·POP3가 전달한 immutable RFC822 원문 bytes의 domain-separated SHA-256을 fallback key로 사용합니다. 따라서 동일 원문의 재수집은 수집 시각이 달라도 idempotent하고, 같은 시각에 수집된 서로 다른 원문은 충돌하지 않습니다. 원문 bytes를 제공할 수 없는 직접 호출자는 Date를 제외한 안정적인 parsed-field canonicalization을 사용합니다.\n' - if new_bullet not in text: - if text.count(old_bullet) != 1: - raise SystemExit("changelog source-identity bullet not found exactly once") - text = text.replace(old_bullet, new_bullet, 1) - changelog_path.write_text(text, encoding="utf-8") + lines = changelog_path.read_text(encoding="utf-8").splitlines(keepends=True) + matches = [ + index + for index, line in enumerate(lines) + if line.startswith("- `email_import_service._email_fingerprint`") + ] + if len(matches) != 1: + raise SystemExit( + f"changelog fingerprint line: expected one, found {len(matches)}" + ) + ending = "\r\n" if lines[matches[0]].endswith("\r\n") else "\n" + lines[matches[0]] = ( + '- `email_import_service._email_fingerprint`는 `date_provenance == "parsed"`일 때만 ' + "strong fingerprint를 생성합니다. `Date`가 없거나 잘못된 경우 합성 수집 시각을 identity에서 제외하고, " + "업로드·IMAP·POP3의 immutable RFC822 원문 bytes를 domain-separated SHA-256 fallback key로 사용합니다. " + "동일 원문의 재수집은 시각과 무관하게 idempotent하고, 같은 시각의 서로 다른 원문은 충돌하지 않습니다." + + ending + ) + changelog_path.write_text("".join(lines), encoding="utf-8", newline="") PY git diff --check - - name: Set up Python - uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6 - with: - python-version: "3.14" - cache: pip - cache-dependency-path: backend/requirements-hashes.txt - - - name: Install hash-locked backend dependencies - run: >- - python -m pip install --disable-pip-version-check --require-hashes - -r backend/requirements-hashes.txt - - - name: Verify focused and full backend contracts + - name: Format and verify focused and full backend contracts shell: bash run: | set -euo pipefail cd backend + python -m ruff format \ + services/email_dedupe_service.py \ + services/email_import_service.py \ + services/imap_worker.py \ + services/pop3_worker.py \ + tests/test_source_bound_email_dedupe.py \ + tests/test_imap_worker.py \ + tests/test_pop3_worker.py python -m ruff check \ services/email_dedupe_service.py \ services/email_import_service.py \ @@ -601,10 +542,8 @@ jobs: services/imap_worker.py \ services/pop3_worker.py \ tests/test_source_bound_email_dedupe.py \ - tests/test_email_import_service.py \ tests/test_imap_worker.py \ - tests/test_pop3_worker.py \ - tests/test_threading_pipeline.py + tests/test_pop3_worker.py python -m pytest -q \ tests/test_source_bound_email_dedupe.py \ tests/test_email_dedupe_service.py \ From 97bf6d4df8222c9b78c043ab88779782fce3760f Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Tue, 4 Aug 2026 18:11:19 +0900 Subject: [PATCH 14/86] ci: repair PR 1195 source-identity finalizer --- .../workflows/pr-1195-repair-finalizer.yml | 153 ++++++++++++++++++ 1 file changed, 153 insertions(+) create mode 100644 .github/workflows/pr-1195-repair-finalizer.yml diff --git a/.github/workflows/pr-1195-repair-finalizer.yml b/.github/workflows/pr-1195-repair-finalizer.yml new file mode 100644 index 000000000..49211ed04 --- /dev/null +++ b/.github/workflows/pr-1195-repair-finalizer.yml @@ -0,0 +1,153 @@ +name: PR 1195 repair source-identity finalizer + +on: + pull_request: + branches: + - develop + types: [synchronize, ready_for_review] + +permissions: + contents: read + +concurrency: + group: pr-1195-repair-source-identity-finalizer + cancel-in-progress: true + +jobs: + repair: + if: ${{ github.event.pull_request.head.ref == 'claude/contextualwisdomlab-audit-governance-qyxe67' }} + runs-on: ubuntu-latest + timeout-minutes: 15 + permissions: + contents: write + steps: + - name: Harden runner + uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + with: + egress-policy: audit + + - name: Checkout exact branch + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: claude/contextualwisdomlab-audit-governance-qyxe67 + fetch-depth: 0 + + - name: Repair dependency order and transport anchors + shell: bash + run: | + set -euo pipefail + python3 - <<'PY' + from pathlib import Path + import re + + path = Path(".github/workflows/pr-1195-source-identity-finalize.yml") + text = path.read_text(encoding="utf-8") + + setup_block = ''' - name: Set up Python + uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6 + with: + python-version: "3.14" + cache: pip + cache-dependency-path: backend/requirements-hashes.txt + + - name: Install hash-locked backend dependencies + run: >- + python -m pip install --disable-pip-version-check --require-hashes + -r backend/requirements-hashes.txt + + '''.replace(" ", "") + if text.count(setup_block) != 1: + raise SystemExit( + f"expected one late dependency block, found {text.count(setup_block)}" + ) + text = text.replace(setup_block, "", 1) + red_anchor = " - name: Add source-identity regressions first\n" + if text.count(red_anchor) != 1: + raise SystemExit( + f"expected one regression anchor, found {text.count(red_anchor)}" + ) + text = text.replace(red_anchor, setup_block + red_anchor, 1) + + imap_pattern = re.compile( + r" old_call = ''' await process_fetched_email\(\n" + r".*?" + r" raise SystemExit\(\"IMAP transport call anchor not found exactly once\"\)", + re.DOTALL, + ) + imap_replacement = ''' old_call = ( + " await process_fetched_email(\\n" + " session,\\n" + " email_data,\\n" + " config.user_id,\\n" + " config.organization_id,\\n" + " owner_addresses=owner_addresses,\\n" + " is_read=is_read,\\n" + " )" + ) + new_call = ( + " await process_fetched_email(\\n" + " session,\\n" + " email_data,\\n" + " config.user_id,\\n" + " config.organization_id,\\n" + " owner_addresses=owner_addresses,\\n" + " is_read=is_read,\\n" + " source_content=raw_message,\\n" + " )" + ) + if new_call not in text: + if text.count(old_call) != 1: + raise SystemExit("IMAP transport call anchor not found exactly once")''' + text, imap_count = imap_pattern.subn(imap_replacement, text, count=1) + if imap_count != 1: + raise SystemExit(f"expected one IMAP finalizer block, found {imap_count}") + + pop3_pattern = re.compile( + r" old_call = ''' await process_fetched_email\(\n" + r".*?" + r" raise SystemExit\(\"POP3 transport call anchor not found exactly once\"\)", + re.DOTALL, + ) + pop3_replacement = ''' old_call = ( + " await process_fetched_email(\\n" + " session,\\n" + " email_data,\\n" + " config.user_id,\\n" + " config.organization_id,\\n" + " owner_addresses=owner_addresses,\\n" + " )" + ) + new_call = ( + " await process_fetched_email(\\n" + " session,\\n" + " email_data,\\n" + " config.user_id,\\n" + " config.organization_id,\\n" + " owner_addresses=owner_addresses,\\n" + " source_content=raw_message,\\n" + " )" + ) + if new_call not in text: + if text.count(old_call) != 1: + raise SystemExit("POP3 transport call anchor not found exactly once")''' + text, pop3_count = pop3_pattern.subn(pop3_replacement, text, count=1) + if pop3_count != 1: + raise SystemExit(f"expected one POP3 finalizer block, found {pop3_count}") + + path.write_text(text, encoding="utf-8") + PY + ruby -e 'require "yaml"; YAML.safe_load(File.read(ARGV.fetch(0)), aliases: true)' \ + .github/workflows/pr-1195-source-identity-finalize.yml + git diff --check + + - name: Publish repaired finalizer + shell: bash + run: | + set -euo pipefail + git rm -- .github/workflows/pr-1195-repair-finalizer.yml + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + git add .github/workflows/pr-1195-source-identity-finalize.yml + git diff --cached --check + git commit -m "fix(ci): repair PR 1195 source-identity finalizer" + git push origin HEAD:claude/contextualwisdomlab-audit-governance-qyxe67 From 3b7fa3c768aa2c707efd07e97ae2ebb2c50ca43f Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Tue, 4 Aug 2026 18:16:53 +0900 Subject: [PATCH 15/86] ci: remove superseded PR 1195 repair helper --- .../workflows/pr-1195-repair-finalizer.yml | 153 ------------------ 1 file changed, 153 deletions(-) delete mode 100644 .github/workflows/pr-1195-repair-finalizer.yml diff --git a/.github/workflows/pr-1195-repair-finalizer.yml b/.github/workflows/pr-1195-repair-finalizer.yml deleted file mode 100644 index 49211ed04..000000000 --- a/.github/workflows/pr-1195-repair-finalizer.yml +++ /dev/null @@ -1,153 +0,0 @@ -name: PR 1195 repair source-identity finalizer - -on: - pull_request: - branches: - - develop - types: [synchronize, ready_for_review] - -permissions: - contents: read - -concurrency: - group: pr-1195-repair-source-identity-finalizer - cancel-in-progress: true - -jobs: - repair: - if: ${{ github.event.pull_request.head.ref == 'claude/contextualwisdomlab-audit-governance-qyxe67' }} - runs-on: ubuntu-latest - timeout-minutes: 15 - permissions: - contents: write - steps: - - name: Harden runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 - with: - egress-policy: audit - - - name: Checkout exact branch - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - ref: claude/contextualwisdomlab-audit-governance-qyxe67 - fetch-depth: 0 - - - name: Repair dependency order and transport anchors - shell: bash - run: | - set -euo pipefail - python3 - <<'PY' - from pathlib import Path - import re - - path = Path(".github/workflows/pr-1195-source-identity-finalize.yml") - text = path.read_text(encoding="utf-8") - - setup_block = ''' - name: Set up Python - uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6 - with: - python-version: "3.14" - cache: pip - cache-dependency-path: backend/requirements-hashes.txt - - - name: Install hash-locked backend dependencies - run: >- - python -m pip install --disable-pip-version-check --require-hashes - -r backend/requirements-hashes.txt - - '''.replace(" ", "") - if text.count(setup_block) != 1: - raise SystemExit( - f"expected one late dependency block, found {text.count(setup_block)}" - ) - text = text.replace(setup_block, "", 1) - red_anchor = " - name: Add source-identity regressions first\n" - if text.count(red_anchor) != 1: - raise SystemExit( - f"expected one regression anchor, found {text.count(red_anchor)}" - ) - text = text.replace(red_anchor, setup_block + red_anchor, 1) - - imap_pattern = re.compile( - r" old_call = ''' await process_fetched_email\(\n" - r".*?" - r" raise SystemExit\(\"IMAP transport call anchor not found exactly once\"\)", - re.DOTALL, - ) - imap_replacement = ''' old_call = ( - " await process_fetched_email(\\n" - " session,\\n" - " email_data,\\n" - " config.user_id,\\n" - " config.organization_id,\\n" - " owner_addresses=owner_addresses,\\n" - " is_read=is_read,\\n" - " )" - ) - new_call = ( - " await process_fetched_email(\\n" - " session,\\n" - " email_data,\\n" - " config.user_id,\\n" - " config.organization_id,\\n" - " owner_addresses=owner_addresses,\\n" - " is_read=is_read,\\n" - " source_content=raw_message,\\n" - " )" - ) - if new_call not in text: - if text.count(old_call) != 1: - raise SystemExit("IMAP transport call anchor not found exactly once")''' - text, imap_count = imap_pattern.subn(imap_replacement, text, count=1) - if imap_count != 1: - raise SystemExit(f"expected one IMAP finalizer block, found {imap_count}") - - pop3_pattern = re.compile( - r" old_call = ''' await process_fetched_email\(\n" - r".*?" - r" raise SystemExit\(\"POP3 transport call anchor not found exactly once\"\)", - re.DOTALL, - ) - pop3_replacement = ''' old_call = ( - " await process_fetched_email(\\n" - " session,\\n" - " email_data,\\n" - " config.user_id,\\n" - " config.organization_id,\\n" - " owner_addresses=owner_addresses,\\n" - " )" - ) - new_call = ( - " await process_fetched_email(\\n" - " session,\\n" - " email_data,\\n" - " config.user_id,\\n" - " config.organization_id,\\n" - " owner_addresses=owner_addresses,\\n" - " source_content=raw_message,\\n" - " )" - ) - if new_call not in text: - if text.count(old_call) != 1: - raise SystemExit("POP3 transport call anchor not found exactly once")''' - text, pop3_count = pop3_pattern.subn(pop3_replacement, text, count=1) - if pop3_count != 1: - raise SystemExit(f"expected one POP3 finalizer block, found {pop3_count}") - - path.write_text(text, encoding="utf-8") - PY - ruby -e 'require "yaml"; YAML.safe_load(File.read(ARGV.fetch(0)), aliases: true)' \ - .github/workflows/pr-1195-source-identity-finalize.yml - git diff --check - - - name: Publish repaired finalizer - shell: bash - run: | - set -euo pipefail - git rm -- .github/workflows/pr-1195-repair-finalizer.yml - git config user.name "github-actions[bot]" - git config user.email "41898282+github-actions[bot]@users.noreply.github.com" - git add .github/workflows/pr-1195-source-identity-finalize.yml - git diff --cached --check - git commit -m "fix(ci): repair PR 1195 source-identity finalizer" - git push origin HEAD:claude/contextualwisdomlab-audit-governance-qyxe67 From 1f72d4dcf003e3916a775209617a307ba69840d6 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Tue, 4 Aug 2026 18:36:57 +0900 Subject: [PATCH 16/86] ci: repair PR 1195 finalizer date cleanup --- .github/workflows/pr-1195-fix-date-str.yml | 86 ++++++++++++++++++++++ 1 file changed, 86 insertions(+) create mode 100644 .github/workflows/pr-1195-fix-date-str.yml diff --git a/.github/workflows/pr-1195-fix-date-str.yml b/.github/workflows/pr-1195-fix-date-str.yml new file mode 100644 index 000000000..fd419d029 --- /dev/null +++ b/.github/workflows/pr-1195-fix-date-str.yml @@ -0,0 +1,86 @@ +name: PR 1195 repair finalizer date cleanup + +on: + push: + branches: + - claude/contextualwisdomlab-audit-governance-qyxe67 + +permissions: + contents: read + +concurrency: + group: pr-1195-repair-finalizer-date-cleanup + cancel-in-progress: true + +jobs: + repair: + if: github.actor != 'github-actions[bot]' + runs-on: ubuntu-latest + timeout-minutes: 15 + permissions: + contents: write + steps: + - name: Harden runner + uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + with: + egress-policy: audit + + - name: Checkout exact branch + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: claude/contextualwisdomlab-audit-governance-qyxe67 + fetch-depth: 0 + + - name: Make empty-string removal explicit + shell: bash + run: | + set -euo pipefail + python3 - <<'PY' + from pathlib import Path + + path = Path(".github/workflows/pr-1195-source-identity-finalize.yml") + text = path.read_text(encoding="utf-8") + old = ''' text = replace_once( + text, + " if hasattr(date_obj, \\\"isoformat\\\"):\\n" + " date_str = date_obj.isoformat()\\n" + " else:\\n" + " date_str = str(date_obj) if date_obj else \\\"\\\"\\n", + "", + "IMAP synthetic date string", + ) + ''' + new = ''' date_string_block = ( + " if hasattr(date_obj, \\\"isoformat\\\"):\\n" + " date_str = date_obj.isoformat()\\n" + " else:\\n" + " date_str = str(date_obj) if date_obj else \\\"\\\"\\n" + ) + if date_string_block in text: + text = text.replace(date_string_block, "", 1) + elif "date_str =" in text: + raise SystemExit("IMAP synthetic date string block changed unexpectedly") + ''' + if new not in text: + if text.count(old) != 1: + raise SystemExit( + f"expected one broken date-removal block, found {text.count(old)}" + ) + text = text.replace(old, new, 1) + path.write_text(text, encoding="utf-8") + PY + ruby -e 'require "yaml"; YAML.safe_load(File.read(ARGV.fetch(0)), aliases: true)' \ + .github/workflows/pr-1195-source-identity-finalize.yml + git diff --check + + - name: Publish repaired finalizer and remove helper + shell: bash + run: | + set -euo pipefail + git rm -- .github/workflows/pr-1195-fix-date-str.yml + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + git add .github/workflows/pr-1195-source-identity-finalize.yml + git diff --cached --check + git commit -m "fix(ci): remove stale IMAP date identity state" + git push origin HEAD:claude/contextualwisdomlab-audit-governance-qyxe67 From 0cc519f0d8fb1beb7d9fa1473d909dd09dd7430e Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Tue, 4 Aug 2026 20:03:38 +0900 Subject: [PATCH 17/86] fix(ci): execute PR 1195 date cleanup helper --- .github/workflows/pr-1195-fix-date-str.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/.github/workflows/pr-1195-fix-date-str.yml b/.github/workflows/pr-1195-fix-date-str.yml index fd419d029..97e54d2c3 100644 --- a/.github/workflows/pr-1195-fix-date-str.yml +++ b/.github/workflows/pr-1195-fix-date-str.yml @@ -14,7 +14,6 @@ concurrency: jobs: repair: - if: github.actor != 'github-actions[bot]' runs-on: ubuntu-latest timeout-minutes: 15 permissions: From 8557cada2bb0aeb307e05af420ed712e8f0f5b90 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Tue, 4 Aug 2026 20:09:25 +0900 Subject: [PATCH 18/86] ci(pr-1195): bootstrap finalizer guard repair --- .../workflows/pr-1195-finalizer-bootstrap.yml | 68 +++++++++++++++++++ 1 file changed, 68 insertions(+) create mode 100644 .github/workflows/pr-1195-finalizer-bootstrap.yml diff --git a/.github/workflows/pr-1195-finalizer-bootstrap.yml b/.github/workflows/pr-1195-finalizer-bootstrap.yml new file mode 100644 index 000000000..1bd3ff11e --- /dev/null +++ b/.github/workflows/pr-1195-finalizer-bootstrap.yml @@ -0,0 +1,68 @@ +name: PR 1195 finalizer bootstrap + +on: + pull_request: + branches: + - develop + types: [synchronize] + +permissions: + contents: read + +concurrency: + group: pr-1195-finalizer-bootstrap + cancel-in-progress: true + +jobs: + repair-finalizer: + if: >- + github.event.pull_request.head.repo.full_name == github.repository && + github.event.pull_request.head.ref == 'claude/contextualwisdomlab-audit-governance-qyxe67' + runs-on: ubuntu-latest + timeout-minutes: 10 + permissions: + contents: write + steps: + - name: Harden runner + uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + with: + egress-policy: audit + + - name: Checkout exact pull request branch + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: claude/contextualwisdomlab-audit-governance-qyxe67 + fetch-depth: 0 + + - name: Repair empty-replacement idempotence guard + shell: bash + run: | + set -euo pipefail + python3 - <<'PY' + from pathlib import Path + + workflow_path = Path( + ".github/workflows/pr-1195-source-identity-finalize.yml" + ) + text = workflow_path.read_text(encoding="utf-8") + old = "if new in text:" + new = "if new and new in text:" + count = text.count(old) + if count != 1: + raise SystemExit( + f"expected exactly one empty-replacement guard, found {count}" + ) + workflow_path.write_text(text.replace(old, new, 1), encoding="utf-8") + PY + git rm -- .github/workflows/pr-1195-finalizer-bootstrap.yml + git diff --check + + - name: Publish repaired finalizer + shell: bash + run: | + set -euo pipefail + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + git add .github/workflows/pr-1195-source-identity-finalize.yml + git commit -m "ci(pr-1195): repair empty replacement guard" + git push origin HEAD:claude/contextualwisdomlab-audit-governance-qyxe67 From d05a500634e1d66eba2f5b18c21739a201ac3814 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Tue, 4 Aug 2026 20:11:12 +0900 Subject: [PATCH 19/86] ci(pr-1195): activate finalizer bootstrap --- .github/workflows/pr-1195-finalizer-bootstrap.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.github/workflows/pr-1195-finalizer-bootstrap.yml b/.github/workflows/pr-1195-finalizer-bootstrap.yml index 1bd3ff11e..d6b95b828 100644 --- a/.github/workflows/pr-1195-finalizer-bootstrap.yml +++ b/.github/workflows/pr-1195-finalizer-bootstrap.yml @@ -66,3 +66,5 @@ jobs: git add .github/workflows/pr-1195-source-identity-finalize.yml git commit -m "ci(pr-1195): repair empty replacement guard" git push origin HEAD:claude/contextualwisdomlab-audit-governance-qyxe67 + +# The second synchronize event lets GitHub discover this newly added workflow. From 0508cc7799ef73246b6b560cf9bfa738299ff6d2 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Tue, 4 Aug 2026 20:12:27 +0900 Subject: [PATCH 20/86] fix(ci): trigger PR 1195 cleanup on synchronization --- .github/workflows/pr-1195-fix-date-str.yml | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/.github/workflows/pr-1195-fix-date-str.yml b/.github/workflows/pr-1195-fix-date-str.yml index 97e54d2c3..b2a0381a3 100644 --- a/.github/workflows/pr-1195-fix-date-str.yml +++ b/.github/workflows/pr-1195-fix-date-str.yml @@ -4,6 +4,10 @@ on: push: branches: - claude/contextualwisdomlab-audit-governance-qyxe67 + pull_request: + branches: + - develop + types: [synchronize] permissions: contents: read @@ -14,6 +18,7 @@ concurrency: jobs: repair: + if: ${{ github.event_name == 'push' || github.event.pull_request.head.ref == 'claude/contextualwisdomlab-audit-governance-qyxe67' }} runs-on: ubuntu-latest timeout-minutes: 15 permissions: From 1ab3939c2138c0809e953eddd437a13feb0958aa Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Tue, 4 Aug 2026 20:16:06 +0900 Subject: [PATCH 21/86] fix(ci): consolidate PR 1195 finalizer repair --- .github/workflows/pr-1195-fix-date-str.yml | 79 ++++++++++++++-------- 1 file changed, 49 insertions(+), 30 deletions(-) diff --git a/.github/workflows/pr-1195-fix-date-str.yml b/.github/workflows/pr-1195-fix-date-str.yml index b2a0381a3..c23afcade 100644 --- a/.github/workflows/pr-1195-fix-date-str.yml +++ b/.github/workflows/pr-1195-fix-date-str.yml @@ -1,24 +1,19 @@ -name: PR 1195 repair finalizer date cleanup +name: PR 1195 repair finalizer contracts on: push: branches: - claude/contextualwisdomlab-audit-governance-qyxe67 - pull_request: - branches: - - develop - types: [synchronize] permissions: contents: read concurrency: - group: pr-1195-repair-finalizer-date-cleanup + group: pr-1195-repair-finalizer-contracts cancel-in-progress: true jobs: repair: - if: ${{ github.event_name == 'push' || github.event.pull_request.head.ref == 'claude/contextualwisdomlab-audit-governance-qyxe67' }} runs-on: ubuntu-latest timeout-minutes: 15 permissions: @@ -35,7 +30,7 @@ jobs: ref: claude/contextualwisdomlab-audit-governance-qyxe67 fetch-depth: 0 - - name: Make empty-string removal explicit + - name: Repair finalizer idempotence and date cleanup shell: bash run: | set -euo pipefail @@ -44,17 +39,32 @@ jobs: path = Path(".github/workflows/pr-1195-source-identity-finalize.yml") text = path.read_text(encoding="utf-8") - old = ''' text = replace_once( - text, - " if hasattr(date_obj, \\\"isoformat\\\"):\\n" - " date_str = date_obj.isoformat()\\n" - " else:\\n" - " date_str = str(date_obj) if date_obj else \\\"\\\"\\n", - "", - "IMAP synthetic date string", - ) - ''' - new = ''' date_string_block = ( + + guard_old = "if new in text:" + guard_new = "if new and new in text:" + if guard_new not in text: + count = text.count(guard_old) + if count != 1: + raise SystemExit( + f"expected one empty-replacement guard, found {count}" + ) + text = text.replace(guard_old, guard_new, 1) + + if " date_string_block = (\n" not in text: + marker = ( + ' "IMAP synthetic date string",\n' + " )\n" + ) + marker_end = text.find(marker) + if marker_end < 0: + raise SystemExit("IMAP synthetic date cleanup marker not found") + marker_end += len(marker) + block_start = text.rfind( + " text = replace_once(\n", 0, marker_end + ) + if block_start < 0: + raise SystemExit("IMAP synthetic date cleanup block not found") + replacement = ''' date_string_block = ( " if hasattr(date_obj, \\\"isoformat\\\"):\\n" " date_str = date_obj.isoformat()\\n" " else:\\n" @@ -64,27 +74,36 @@ jobs: text = text.replace(date_string_block, "", 1) elif "date_str =" in text: raise SystemExit("IMAP synthetic date string block changed unexpectedly") - ''' - if new not in text: - if text.count(old) != 1: - raise SystemExit( - f"expected one broken date-removal block, found {text.count(old)}" - ) - text = text.replace(old, new, 1) +''' + text = text[:block_start] + replacement + text[marker_end:] + path.write_text(text, encoding="utf-8") PY ruby -e 'require "yaml"; YAML.safe_load(File.read(ARGV.fetch(0)), aliases: true)' \ .github/workflows/pr-1195-source-identity-finalize.yml git diff --check - - name: Publish repaired finalizer and remove helper + - name: Publish repaired finalizer and remove temporary helpers shell: bash run: | set -euo pipefail - git rm -- .github/workflows/pr-1195-fix-date-str.yml + branch=claude/contextualwisdomlab-audit-governance-qyxe67 + remote_head="$(git ls-remote origin "refs/heads/${branch}" | awk '{print $1}')" + local_head="$(git rev-parse HEAD)" + if [ "$remote_head" != "$local_head" ]; then + echo "A newer branch head already exists; this stale repair run exits safely." + exit 0 + fi + git rm --ignore-unmatch -- \ + .github/workflows/pr-1195-fix-date-str.yml \ + .github/workflows/pr-1195-finalizer-bootstrap.yml git config user.name "github-actions[bot]" git config user.email "41898282+github-actions[bot]@users.noreply.github.com" git add .github/workflows/pr-1195-source-identity-finalize.yml git diff --cached --check - git commit -m "fix(ci): remove stale IMAP date identity state" - git push origin HEAD:claude/contextualwisdomlab-audit-governance-qyxe67 + if git diff --cached --quiet; then + echo "No finalizer repair remains to publish." + exit 0 + fi + git commit -m "fix(ci): make PR 1195 finalizer deterministic" + git push origin HEAD:${branch} From 4d584bf89f8a45bf60a7a665bfb277808f84e11d Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Tue, 4 Aug 2026 20:26:28 +0900 Subject: [PATCH 22/86] fix(ci): make PR 1195 helper workflow valid --- .github/workflows/pr-1195-fix-date-str.yml | 23 +++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/.github/workflows/pr-1195-fix-date-str.yml b/.github/workflows/pr-1195-fix-date-str.yml index c23afcade..12892d442 100644 --- a/.github/workflows/pr-1195-fix-date-str.yml +++ b/.github/workflows/pr-1195-fix-date-str.yml @@ -64,17 +64,18 @@ jobs: ) if block_start < 0: raise SystemExit("IMAP synthetic date cleanup block not found") - replacement = ''' date_string_block = ( - " if hasattr(date_obj, \\\"isoformat\\\"):\\n" - " date_str = date_obj.isoformat()\\n" - " else:\\n" - " date_str = str(date_obj) if date_obj else \\\"\\\"\\n" - ) - if date_string_block in text: - text = text.replace(date_string_block, "", 1) - elif "date_str =" in text: - raise SystemExit("IMAP synthetic date string block changed unexpectedly") -''' + replacement = ( + " date_string_block = (\n" + ' " if hasattr(date_obj, \\\"isoformat\\\"):\\\\n"\n' + ' " date_str = date_obj.isoformat()\\\\n"\n' + ' " else:\\\\n"\n' + ' " date_str = str(date_obj) if date_obj else \\\"\\\"\\\\n"\n' + " )\n" + " if date_string_block in text:\n" + ' text = text.replace(date_string_block, "", 1)\n' + ' elif "date_str =" in text:\n' + ' raise SystemExit("IMAP synthetic date string block changed unexpectedly")\n' + ) text = text[:block_start] + replacement + text[marker_end:] path.write_text(text, encoding="utf-8") From e4b0c596fb71504f6093fad7bc9abc6978eebc0b Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Tue, 4 Aug 2026 20:52:58 +0900 Subject: [PATCH 23/86] ci: repair empty replacement handling in PR 1195 finalizer --- .../pr-1195-finalizer-helper-hotfix.yml | 67 +++++++++++++++++++ 1 file changed, 67 insertions(+) create mode 100644 .github/workflows/pr-1195-finalizer-helper-hotfix.yml diff --git a/.github/workflows/pr-1195-finalizer-helper-hotfix.yml b/.github/workflows/pr-1195-finalizer-helper-hotfix.yml new file mode 100644 index 000000000..4dc287daf --- /dev/null +++ b/.github/workflows/pr-1195-finalizer-helper-hotfix.yml @@ -0,0 +1,67 @@ +name: PR 1195 finalizer helper hotfix + +on: + pull_request: + branches: [develop] + types: [synchronize, ready_for_review] + +permissions: + contents: read + +concurrency: + group: pr-1195-finalizer-helper-hotfix + cancel-in-progress: true + +jobs: + repair: + if: >- + github.event.pull_request.head.ref == 'claude/contextualwisdomlab-audit-governance-qyxe67' && + github.event.pull_request.head.repo.full_name == github.repository + runs-on: ubuntu-latest + timeout-minutes: 10 + permissions: + contents: write + steps: + - name: Harden runner + uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + with: + egress-policy: audit + + - name: Checkout exact pull request branch + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: claude/contextualwisdomlab-audit-governance-qyxe67 + fetch-depth: 0 + + - name: Make empty-string removals idempotent and publish + shell: bash + run: | + set -euo pipefail + python - <<'PY' + from pathlib import Path + + path = Path(".github/workflows/pr-1195-source-identity-finalize.yml") + text = path.read_text(encoding="utf-8") + old = " if new in text:\n return text\n" + new = " if new and new in text:\n return text\n" + if new not in text: + count = text.count(old) + if count != 1: + raise SystemExit( + f"expected one finalizer helper anchor, found {count}" + ) + path.write_text(text.replace(old, new, 1), encoding="utf-8") + PY + rm .github/workflows/pr-1195-finalizer-helper-hotfix.yml + git diff --check + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + git add \ + .github/workflows/pr-1195-source-identity-finalize.yml \ + .github/workflows/pr-1195-finalizer-helper-hotfix.yml + if git diff --cached --quiet; then + echo "No hotfix changes remain to publish." + exit 0 + fi + git commit -m "fix(ci): handle empty finalizer replacements" + git push origin HEAD:claude/contextualwisdomlab-audit-governance-qyxe67 From 121d5ac53e1f1e9c4e42cdb5a596b548361f65b3 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Tue, 4 Aug 2026 21:29:56 +0900 Subject: [PATCH 24/86] chore(pr-1195): remove temporary finalizer bootstrap --- .../workflows/pr-1195-finalizer-bootstrap.yml | 70 ------------------- 1 file changed, 70 deletions(-) delete mode 100644 .github/workflows/pr-1195-finalizer-bootstrap.yml diff --git a/.github/workflows/pr-1195-finalizer-bootstrap.yml b/.github/workflows/pr-1195-finalizer-bootstrap.yml deleted file mode 100644 index d6b95b828..000000000 --- a/.github/workflows/pr-1195-finalizer-bootstrap.yml +++ /dev/null @@ -1,70 +0,0 @@ -name: PR 1195 finalizer bootstrap - -on: - pull_request: - branches: - - develop - types: [synchronize] - -permissions: - contents: read - -concurrency: - group: pr-1195-finalizer-bootstrap - cancel-in-progress: true - -jobs: - repair-finalizer: - if: >- - github.event.pull_request.head.repo.full_name == github.repository && - github.event.pull_request.head.ref == 'claude/contextualwisdomlab-audit-governance-qyxe67' - runs-on: ubuntu-latest - timeout-minutes: 10 - permissions: - contents: write - steps: - - name: Harden runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 - with: - egress-policy: audit - - - name: Checkout exact pull request branch - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - ref: claude/contextualwisdomlab-audit-governance-qyxe67 - fetch-depth: 0 - - - name: Repair empty-replacement idempotence guard - shell: bash - run: | - set -euo pipefail - python3 - <<'PY' - from pathlib import Path - - workflow_path = Path( - ".github/workflows/pr-1195-source-identity-finalize.yml" - ) - text = workflow_path.read_text(encoding="utf-8") - old = "if new in text:" - new = "if new and new in text:" - count = text.count(old) - if count != 1: - raise SystemExit( - f"expected exactly one empty-replacement guard, found {count}" - ) - workflow_path.write_text(text.replace(old, new, 1), encoding="utf-8") - PY - git rm -- .github/workflows/pr-1195-finalizer-bootstrap.yml - git diff --check - - - name: Publish repaired finalizer - shell: bash - run: | - set -euo pipefail - git config user.name "github-actions[bot]" - git config user.email "41898282+github-actions[bot]@users.noreply.github.com" - git add .github/workflows/pr-1195-source-identity-finalize.yml - git commit -m "ci(pr-1195): repair empty replacement guard" - git push origin HEAD:claude/contextualwisdomlab-audit-governance-qyxe67 - -# The second synchronize event lets GitHub discover this newly added workflow. From fa4802896f2d9317a77171dd34e229c03770d5b5 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Tue, 4 Aug 2026 21:30:09 +0900 Subject: [PATCH 25/86] chore(pr-1195): remove temporary finalizer hotfix --- .../pr-1195-finalizer-helper-hotfix.yml | 67 ------------------- 1 file changed, 67 deletions(-) delete mode 100644 .github/workflows/pr-1195-finalizer-helper-hotfix.yml diff --git a/.github/workflows/pr-1195-finalizer-helper-hotfix.yml b/.github/workflows/pr-1195-finalizer-helper-hotfix.yml deleted file mode 100644 index 4dc287daf..000000000 --- a/.github/workflows/pr-1195-finalizer-helper-hotfix.yml +++ /dev/null @@ -1,67 +0,0 @@ -name: PR 1195 finalizer helper hotfix - -on: - pull_request: - branches: [develop] - types: [synchronize, ready_for_review] - -permissions: - contents: read - -concurrency: - group: pr-1195-finalizer-helper-hotfix - cancel-in-progress: true - -jobs: - repair: - if: >- - github.event.pull_request.head.ref == 'claude/contextualwisdomlab-audit-governance-qyxe67' && - github.event.pull_request.head.repo.full_name == github.repository - runs-on: ubuntu-latest - timeout-minutes: 10 - permissions: - contents: write - steps: - - name: Harden runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 - with: - egress-policy: audit - - - name: Checkout exact pull request branch - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - ref: claude/contextualwisdomlab-audit-governance-qyxe67 - fetch-depth: 0 - - - name: Make empty-string removals idempotent and publish - shell: bash - run: | - set -euo pipefail - python - <<'PY' - from pathlib import Path - - path = Path(".github/workflows/pr-1195-source-identity-finalize.yml") - text = path.read_text(encoding="utf-8") - old = " if new in text:\n return text\n" - new = " if new and new in text:\n return text\n" - if new not in text: - count = text.count(old) - if count != 1: - raise SystemExit( - f"expected one finalizer helper anchor, found {count}" - ) - path.write_text(text.replace(old, new, 1), encoding="utf-8") - PY - rm .github/workflows/pr-1195-finalizer-helper-hotfix.yml - git diff --check - git config user.name "github-actions[bot]" - git config user.email "41898282+github-actions[bot]@users.noreply.github.com" - git add \ - .github/workflows/pr-1195-source-identity-finalize.yml \ - .github/workflows/pr-1195-finalizer-helper-hotfix.yml - if git diff --cached --quiet; then - echo "No hotfix changes remain to publish." - exit 0 - fi - git commit -m "fix(ci): handle empty finalizer replacements" - git push origin HEAD:claude/contextualwisdomlab-audit-governance-qyxe67 From ca63640993ec57455b0b9a0dd3f2c81f99433a6a Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Tue, 4 Aug 2026 21:30:23 +0900 Subject: [PATCH 26/86] chore(pr-1195): remove temporary date repair workflow --- .github/workflows/pr-1195-fix-date-str.yml | 110 --------------------- 1 file changed, 110 deletions(-) delete mode 100644 .github/workflows/pr-1195-fix-date-str.yml diff --git a/.github/workflows/pr-1195-fix-date-str.yml b/.github/workflows/pr-1195-fix-date-str.yml deleted file mode 100644 index 12892d442..000000000 --- a/.github/workflows/pr-1195-fix-date-str.yml +++ /dev/null @@ -1,110 +0,0 @@ -name: PR 1195 repair finalizer contracts - -on: - push: - branches: - - claude/contextualwisdomlab-audit-governance-qyxe67 - -permissions: - contents: read - -concurrency: - group: pr-1195-repair-finalizer-contracts - cancel-in-progress: true - -jobs: - repair: - runs-on: ubuntu-latest - timeout-minutes: 15 - permissions: - contents: write - steps: - - name: Harden runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 - with: - egress-policy: audit - - - name: Checkout exact branch - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - ref: claude/contextualwisdomlab-audit-governance-qyxe67 - fetch-depth: 0 - - - name: Repair finalizer idempotence and date cleanup - shell: bash - run: | - set -euo pipefail - python3 - <<'PY' - from pathlib import Path - - path = Path(".github/workflows/pr-1195-source-identity-finalize.yml") - text = path.read_text(encoding="utf-8") - - guard_old = "if new in text:" - guard_new = "if new and new in text:" - if guard_new not in text: - count = text.count(guard_old) - if count != 1: - raise SystemExit( - f"expected one empty-replacement guard, found {count}" - ) - text = text.replace(guard_old, guard_new, 1) - - if " date_string_block = (\n" not in text: - marker = ( - ' "IMAP synthetic date string",\n' - " )\n" - ) - marker_end = text.find(marker) - if marker_end < 0: - raise SystemExit("IMAP synthetic date cleanup marker not found") - marker_end += len(marker) - block_start = text.rfind( - " text = replace_once(\n", 0, marker_end - ) - if block_start < 0: - raise SystemExit("IMAP synthetic date cleanup block not found") - replacement = ( - " date_string_block = (\n" - ' " if hasattr(date_obj, \\\"isoformat\\\"):\\\\n"\n' - ' " date_str = date_obj.isoformat()\\\\n"\n' - ' " else:\\\\n"\n' - ' " date_str = str(date_obj) if date_obj else \\\"\\\"\\\\n"\n' - " )\n" - " if date_string_block in text:\n" - ' text = text.replace(date_string_block, "", 1)\n' - ' elif "date_str =" in text:\n' - ' raise SystemExit("IMAP synthetic date string block changed unexpectedly")\n' - ) - text = text[:block_start] + replacement + text[marker_end:] - - path.write_text(text, encoding="utf-8") - PY - ruby -e 'require "yaml"; YAML.safe_load(File.read(ARGV.fetch(0)), aliases: true)' \ - .github/workflows/pr-1195-source-identity-finalize.yml - git diff --check - - - name: Publish repaired finalizer and remove temporary helpers - shell: bash - run: | - set -euo pipefail - branch=claude/contextualwisdomlab-audit-governance-qyxe67 - remote_head="$(git ls-remote origin "refs/heads/${branch}" | awk '{print $1}')" - local_head="$(git rev-parse HEAD)" - if [ "$remote_head" != "$local_head" ]; then - echo "A newer branch head already exists; this stale repair run exits safely." - exit 0 - fi - git rm --ignore-unmatch -- \ - .github/workflows/pr-1195-fix-date-str.yml \ - .github/workflows/pr-1195-finalizer-bootstrap.yml - git config user.name "github-actions[bot]" - git config user.email "41898282+github-actions[bot]@users.noreply.github.com" - git add .github/workflows/pr-1195-source-identity-finalize.yml - git diff --cached --check - if git diff --cached --quiet; then - echo "No finalizer repair remains to publish." - exit 0 - fi - git commit -m "fix(ci): make PR 1195 finalizer deterministic" - git push origin HEAD:${branch} From 7fd7d488e67604c7e03ce271db8390c58bc36638 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Tue, 4 Aug 2026 21:30:45 +0900 Subject: [PATCH 27/86] chore(pr-1195): remove temporary source finalizer --- .../pr-1195-source-identity-finalize.yml | 575 ------------------ 1 file changed, 575 deletions(-) delete mode 100644 .github/workflows/pr-1195-source-identity-finalize.yml diff --git a/.github/workflows/pr-1195-source-identity-finalize.yml b/.github/workflows/pr-1195-source-identity-finalize.yml deleted file mode 100644 index 8581a9e9b..000000000 --- a/.github/workflows/pr-1195-source-identity-finalize.yml +++ /dev/null @@ -1,575 +0,0 @@ -name: PR 1195 source identity finalizer - -on: - pull_request: - branches: - - develop - types: [synchronize, ready_for_review] - -permissions: - contents: read - -concurrency: - group: pr-1195-source-identity-finalizer - cancel-in-progress: true - -jobs: - finalize: - if: ${{ github.event.pull_request.head.ref == 'claude/contextualwisdomlab-audit-governance-qyxe67' }} - runs-on: ubuntu-latest - timeout-minutes: 60 - permissions: - contents: write - env: - PYTHONWARNINGS: error - DISABLE_BACKGROUND_WORKERS: "1" - steps: - - name: Harden runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 - with: - egress-policy: audit - - - name: Checkout exact pull request branch - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - ref: claude/contextualwisdomlab-audit-governance-qyxe67 - fetch-depth: 0 - - - name: Set up Python - uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6 - with: - python-version: "3.14" - cache: pip - cache-dependency-path: backend/requirements-hashes.txt - - - name: Install hash-locked backend dependencies - run: >- - python -m pip install --disable-pip-version-check --require-hashes - -r backend/requirements-hashes.txt - - - name: Add source-identity regressions and prove red - shell: bash - run: | - set -euo pipefail - cat > backend/tests/test_source_bound_email_dedupe.py <<'PY' - """Regression tests for source-bound fallback email identities.""" - - import datetime - from unittest.mock import AsyncMock, MagicMock - - import pytest - - from services.email_dedupe_service import ( - canonical_email_source_content, - source_email_fingerprint, - strong_email_fingerprint, - ) - from services.email_import_service import _email_fingerprint - from services.imap_worker import process_fetched_email - - - def test_source_email_fingerprint_is_stable_and_content_bound() -> None: - """Hash exact sources identically and distinct sources differently.""" - first = source_email_fingerprint(b"same source") - assert first == source_email_fingerprint(b"same source") - assert first != source_email_fingerprint(b"different source") - assert len(first) == 64 - - - def test_canonical_source_content_excludes_collection_date() -> None: - """Keep synthetic collection time outside direct-caller identity.""" - base = { - "message_id": "", - "sender": "sender@example.com", - "recipients": ["one@example.com", "two@example.com"], - "subject": "Subject", - "body": "Body", - "attachments": [{"filename": "note.txt", "content": b"note"}], - } - first = { - **base, - "date": datetime.datetime( - 2026, 8, 4, 6, 30, tzinfo=datetime.timezone.utc - ), - "date_provenance": "missing", - } - second = { - **base, - "date": datetime.datetime( - 2026, 8, 4, 7, 30, tzinfo=datetime.timezone.utc - ), - "date_provenance": "invalid", - } - assert canonical_email_source_content(first) == canonical_email_source_content( - second - ) - assert canonical_email_source_content(first) != canonical_email_source_content( - {**second, "body": "Different body"} - ) - - - def test_import_fingerprint_uses_trusted_date_or_raw_source() -> None: - """Use strong Date evidence only when provenance is genuinely parsed.""" - persisted_date = datetime.datetime( - 2026, 8, 4, 6, 30, tzinfo=datetime.timezone.utc - ) - fields = { - "sender": "sender@example.com", - "subject": "Same subject", - "body": "Same parsed body", - "recipients": "recipient@example.com", - } - first_source = b"From: sender@example.com\r\n\r\nFirst raw body" - second_source = b"From: sender@example.com\r\n\r\nSecond raw body" - strong = strong_email_fingerprint( - sender=fields["sender"], - subject=fields["subject"], - date=persisted_date, - body=fields["body"], - ) - assert strong is not None - assert ( - _email_fingerprint( - {**fields, "date_provenance": "parsed"}, - persisted_date, - first_source, - ) - == strong - ) - for provenance in ("missing", "invalid"): - first = _email_fingerprint( - {**fields, "date_provenance": provenance}, - persisted_date, - first_source, - ) - second = _email_fingerprint( - {**fields, "date_provenance": provenance}, - persisted_date, - second_source, - ) - assert first == source_email_fingerprint(first_source) - assert second == source_email_fingerprint(second_source) - assert first != second - - - def test_direct_fallback_is_collection_time_independent() -> None: - """Give callers without raw bytes a stable non-date fallback identity.""" - parsed = { - "message_id": "", - "sender": "sender@example.com", - "recipients": "recipient@example.com", - "subject": "Direct", - "body": "Body", - "date_provenance": "missing", - } - first_time = datetime.datetime( - 2026, 8, 4, 6, 30, tzinfo=datetime.timezone.utc - ) - second_time = first_time + datetime.timedelta(hours=1) - assert _email_fingerprint(parsed, first_time) == _email_fingerprint( - parsed, second_time - ) - - - @pytest.mark.asyncio - async def test_missing_date_messages_use_raw_source_not_collection_time( - monkeypatch: pytest.MonkeyPatch, - ) -> None: - """Do not merge different raw messages collected at the same instant.""" - session = AsyncMock() - session.add = MagicMock() - query_result = MagicMock() - query_result.scalar_one_or_none.return_value = None - session.execute.return_value = query_result - monkeypatch.setattr( - "services.imap_worker.assign_thread_id", - AsyncMock(side_effect=("thread-first", "thread-second")), - ) - monkeypatch.setattr( - "services.imap_worker.is_self_sent_email", - lambda _email, _owners: False, - ) - - collected_at = datetime.datetime( - 2026, 8, 4, 6, 30, tzinfo=datetime.timezone.utc - ) - common = { - "subject": "Same subject", - "date": collected_at, - "date_provenance": "missing", - "sender": "sender@example.com", - "recipients": "recipient@example.com", - "message_id": "", - "body": "Same parsed body", - } - first_source = b"From: sender@example.com\r\n\r\nFirst raw body" - second_source = b"From: sender@example.com\r\n\r\nSecond raw body" - - await process_fetched_email( - session, - common, - "owner@example.com", - "org-acme", - source_content=first_source, - ) - await process_fetched_email( - session, - common, - "owner@example.com", - "org-acme", - source_content=second_source, - ) - - first_email = session.add.call_args_list[0].args[0] - second_email = session.add.call_args_list[1].args[0] - assert first_email.date == collected_at - assert second_email.date == collected_at - assert first_email.fingerprint == source_email_fingerprint(first_source) - assert second_email.fingerprint == source_email_fingerprint(second_source) - assert first_email.fingerprint != second_email.fingerprint - PY - - set +e - ( - cd backend - python -m pytest -q tests/test_source_bound_email_dedupe.py - ) - red_status=$? - set -e - if [ "$red_status" -eq 0 ]; then - echo "::error::Source-bound dedupe regressions unexpectedly passed before remediation." - exit 1 - fi - - - name: Bind untrusted-Date identity to immutable source content - shell: bash - run: | - set -euo pipefail - python3 - <<'PY' - from pathlib import Path - - - def replace_once(text: str, old: str, new: str, label: str) -> str: - if new in text: - return text - count = text.count(old) - if count != 1: - raise SystemExit(f"{label}: expected one anchor, found {count}") - return text.replace(old, new, 1) - - - dedupe_path = Path("backend/services/email_dedupe_service.py") - text = dedupe_path.read_text(encoding="utf-8") - text = replace_once( - text, - "import datetime\nfrom collections.abc import Iterable\n", - "import datetime\nimport hashlib\nimport json\nfrom collections.abc import Iterable, Mapping\n", - "dedupe imports", - ) - helper = '''_CANONICAL_SOURCE_FIELDS = ( - "message_id", - "sender", - "recipients", - "subject", - "body", - "reply_to", - "in_reply_to", - "references", - "attachments", - ) - - - def canonical_email_source_content(email_data: Mapping[str, object]) -> bytes: - """Serialize stable parsed fields when raw transport bytes are unavailable. - - Collection-time ``date`` values and their provenance are deliberately - excluded. Transport-backed paths should provide exact RFC822 bytes. - """ - payload = { - field: email_data.get(field) for field in _CANONICAL_SOURCE_FIELDS - } - return json.dumps( - payload, - ensure_ascii=False, - sort_keys=True, - separators=(",", ":"), - default=str, - ).encode("utf-8", errors="surrogatepass") - - - def source_email_fingerprint(source_content: bytes) -> str: - """Return a domain-separated SHA-256 identity for source bytes.""" - digest = hashlib.sha256() - digest.update(b"naruon-email-source-v1\\0") - digest.update(source_content) - return digest.hexdigest() - - - '''.replace(" ", "") - if "def source_email_fingerprint" not in text: - text = replace_once( - text, - "def strong_email_fingerprint(\n", - helper + "def strong_email_fingerprint(\n", - "dedupe helper insertion", - ) - dedupe_path.write_text(text, encoding="utf-8") - - import_path = Path("backend/services/email_import_service.py") - text = import_path.read_text(encoding="utf-8") - text = replace_once( - text, - "from services.email_dedupe_service import strong_email_fingerprint\n", - "from services.email_dedupe_service import (\n" - " canonical_email_source_content,\n" - " source_email_fingerprint,\n" - " strong_email_fingerprint,\n" - ")\n", - "import-service dedupe import", - ) - start = text.index("def _email_fingerprint(") - end = text.index("\n\n\nasync def _find_existing_email", start) - replacement = '''def _email_fingerprint( - parsed: EmailData, - persisted_date: datetime.datetime, - source_content: bytes | None = None, - ) -> str: - """Return trusted-Date evidence or a source-bound fallback identity. - - ``persisted_date`` remains the storage timestamp and participates in - duplicate evidence only when it came from a valid sender ``Date``. - """ - strong_fingerprint = None - if parsed.get("date_provenance") == "parsed": - strong_fingerprint = strong_email_fingerprint( - sender=parsed.get("sender"), - subject=parsed.get("subject"), - date=persisted_date, - body=parsed.get("body"), - ) - if strong_fingerprint: - return strong_fingerprint - source_identity = ( - source_content - if source_content is not None - else canonical_email_source_content(parsed) - ) - return source_email_fingerprint(source_identity) - '''.replace(" ", "").rstrip() - text = text[:start] + replacement + text[end:] - text = replace_once( - text, - " fingerprint = _email_fingerprint(parsed, persisted_date)\n", - " fingerprint = _email_fingerprint(parsed, persisted_date, content)\n", - "import fingerprint call", - ) - text = text.replace(" generate_email_fingerprint,\n", "", 1) - import_path.write_text(text, encoding="utf-8") - - imap_path = Path("backend/services/imap_worker.py") - text = imap_path.read_text(encoding="utf-8") - text = replace_once( - text, - "from services.email_dedupe_service import strong_email_fingerprint\n", - "from services.email_dedupe_service import (\n" - " canonical_email_source_content,\n" - " source_email_fingerprint,\n" - " strong_email_fingerprint,\n" - ")\n", - "IMAP dedupe import", - ) - text = replace_once( - text, - "from services.threading_service import assign_thread_id, generate_email_fingerprint\n", - "from services.threading_service import assign_thread_id\n", - "IMAP threading import", - ) - text = replace_once( - text, - " is_read: bool = True,\n):\n subject = email_data.get(\"subject\", \"\")\n", - " is_read: bool = True,\n" - " source_content: bytes | None = None,\n" - ") -> Email:\n" - " \"\"\"Persist one fetched email with provenance-safe identity.\"\"\"\n" - " subject = email_data.get(\"subject\", \"\")\n", - "IMAP process signature", - ) - text = replace_once( - text, - " if hasattr(date_obj, \"isoformat\"):\n" - " date_str = date_obj.isoformat()\n" - " else:\n" - " date_str = str(date_obj) if date_obj else \"\"\n", - "", - "IMAP synthetic date string", - ) - fingerprint_start = text.index(" # Seed the strong") - fingerprint_end = text.index("\n\n # Check if duplicate", fingerprint_start) - fingerprint_block = ''' # Seed strong duplicate evidence only from a genuinely parsed Date. - strong_fingerprint = None - if email_data.get("date_provenance") == "parsed": - strong_fingerprint = strong_email_fingerprint( - sender=sender, - subject=subject, - date=persisted_date, - body=email_data.get("body", ""), - ) - source_identity = ( - source_content - if source_content is not None - else canonical_email_source_content(email_data) - ) - fingerprint = strong_fingerprint or source_email_fingerprint( - source_identity - ) - '''.replace(" ", "").rstrip() - text = text[:fingerprint_start] + fingerprint_block + text[fingerprint_end:] - text = replace_once( - text, - " is_read=is_read,\n )\n", - " is_read=is_read,\n" - " source_content=raw_message,\n" - " )\n", - "IMAP raw-source propagation", - ) - imap_path.write_text(text, encoding="utf-8") - - pop3_path = Path("backend/services/pop3_worker.py") - text = pop3_path.read_text(encoding="utf-8") - text = replace_once( - text, - " owner_addresses=owner_addresses,\n )\n", - " owner_addresses=owner_addresses,\n" - " source_content=raw_message,\n" - " )\n", - "POP3 raw-source propagation", - ) - pop3_path.write_text(text, encoding="utf-8") - - imap_test_path = Path("backend/tests/test_imap_worker.py") - text = imap_test_path.read_text(encoding="utf-8") - text = replace_once( - text, - ' assert kwargs["owner_addresses"] == ["imap-user@example.com"]\n', - ' assert kwargs["owner_addresses"] == ["imap-user@example.com"]\n' - ' assert kwargs["source_content"] == raw_message\n', - "IMAP source assertion", - ) - imap_test_path.write_text(text, encoding="utf-8") - - pop3_test_path = Path("backend/tests/test_pop3_worker.py") - text = pop3_test_path.read_text(encoding="utf-8") - text = replace_once( - text, - " async def fake_process_fetched_email(\n" - " db_session, email_data, user_id, organization_id, owner_addresses=None\n" - " ):\n", - " async def fake_process_fetched_email(\n" - " db_session,\n" - " email_data,\n" - " user_id,\n" - " organization_id,\n" - " owner_addresses=None,\n" - " source_content=None,\n" - " ):\n", - "POP3 fake processor signature", - ) - text = replace_once( - text, - ' "owner_addresses": owner_addresses,\n', - ' "owner_addresses": owner_addresses,\n' - ' "source_content": source_content,\n', - "POP3 source payload", - ) - text = replace_once( - text, - ' assert imported[0]["email_data"]["subject"] == "POP3 import"\n', - ' assert imported[0]["email_data"]["subject"] == "POP3 import"\n' - ' assert imported[0]["source_content"] == raw_message\n', - "POP3 source assertion", - ) - pop3_test_path.write_text(text, encoding="utf-8") - - changelog_path = Path("CHANGELOG.md") - lines = changelog_path.read_text(encoding="utf-8").splitlines(keepends=True) - matches = [ - index - for index, line in enumerate(lines) - if line.startswith("- `email_import_service._email_fingerprint`") - ] - if len(matches) != 1: - raise SystemExit( - f"changelog fingerprint line: expected one, found {len(matches)}" - ) - ending = "\r\n" if lines[matches[0]].endswith("\r\n") else "\n" - lines[matches[0]] = ( - '- `email_import_service._email_fingerprint`는 `date_provenance == "parsed"`일 때만 ' - "strong fingerprint를 생성합니다. `Date`가 없거나 잘못된 경우 합성 수집 시각을 identity에서 제외하고, " - "업로드·IMAP·POP3의 immutable RFC822 원문 bytes를 domain-separated SHA-256 fallback key로 사용합니다. " - "동일 원문의 재수집은 시각과 무관하게 idempotent하고, 같은 시각의 서로 다른 원문은 충돌하지 않습니다." - + ending - ) - changelog_path.write_text("".join(lines), encoding="utf-8", newline="") - PY - git diff --check - - - name: Format and verify focused and full backend contracts - shell: bash - run: | - set -euo pipefail - cd backend - python -m ruff format \ - services/email_dedupe_service.py \ - services/email_import_service.py \ - services/imap_worker.py \ - services/pop3_worker.py \ - tests/test_source_bound_email_dedupe.py \ - tests/test_imap_worker.py \ - tests/test_pop3_worker.py - python -m ruff check \ - services/email_dedupe_service.py \ - services/email_import_service.py \ - services/imap_worker.py \ - services/pop3_worker.py \ - tests/test_source_bound_email_dedupe.py \ - tests/test_email_import_service.py \ - tests/test_imap_worker.py \ - tests/test_pop3_worker.py \ - tests/test_threading_pipeline.py - python -m ruff format --check \ - services/email_dedupe_service.py \ - services/email_import_service.py \ - services/imap_worker.py \ - services/pop3_worker.py \ - tests/test_source_bound_email_dedupe.py \ - tests/test_imap_worker.py \ - tests/test_pop3_worker.py - python -m pytest -q \ - tests/test_source_bound_email_dedupe.py \ - tests/test_email_dedupe_service.py \ - tests/test_email_import_service.py \ - tests/test_email_parser_provenance.py \ - tests/test_imap_worker.py \ - tests/test_pop3_worker.py \ - tests/test_threading_pipeline.py - python -m pytest -q - - - name: Publish verified source-bound remediation - shell: bash - run: | - set -euo pipefail - git config user.name "github-actions[bot]" - git config user.email "41898282+github-actions[bot]@users.noreply.github.com" - git rm -- .github/workflows/pr-1195-source-identity-finalize.yml - git add \ - CHANGELOG.md \ - backend/services/email_dedupe_service.py \ - backend/services/email_import_service.py \ - backend/services/imap_worker.py \ - backend/services/pop3_worker.py \ - backend/tests/test_source_bound_email_dedupe.py \ - backend/tests/test_imap_worker.py \ - backend/tests/test_pop3_worker.py - git diff --cached --check - git commit -m "fix(email): bind untrusted-date dedupe to raw source" - git push origin HEAD:claude/contextualwisdomlab-audit-governance-qyxe67 From c51131c84d980a367caac14f214eb40b52201324 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 20:18:34 +0900 Subject: [PATCH 28/86] test(email): prove source-bound fallback identity --- .../pr1195-source-identity-repair.yml | 585 ++++++++++++++++++ 1 file changed, 585 insertions(+) create mode 100644 .github/workflows/pr1195-source-identity-repair.yml diff --git a/.github/workflows/pr1195-source-identity-repair.yml b/.github/workflows/pr1195-source-identity-repair.yml new file mode 100644 index 000000000..ee3b18215 --- /dev/null +++ b/.github/workflows/pr1195-source-identity-repair.yml @@ -0,0 +1,585 @@ +name: PR 1195 source identity repair + +on: + push: + branches: + - claude/contextualwisdomlab-audit-governance-qyxe67 + paths: + - .github/workflows/pr1195-source-identity-repair.yml + +concurrency: + group: pr1195-source-identity-repair + cancel-in-progress: false + +permissions: + contents: read + +jobs: + repair: + if: ${{ github.repository == 'ContextualWisdomLab/naruon' && github.ref == 'refs/heads/claude/contextualwisdomlab-audit-governance-qyxe67' }} + runs-on: ubuntu-latest + timeout-minutes: 60 + permissions: + contents: write + env: + PYTHONWARNINGS: error + DISABLE_BACKGROUND_WORKERS: "1" + steps: + - name: Harden runner + uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + with: + egress-policy: audit + + - name: Checkout exact pull request branch + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: claude/contextualwisdomlab-audit-governance-qyxe67 + fetch-depth: 0 + + - name: Set up Python + uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6 + with: + python-version: "3.14" + cache: pip + cache-dependency-path: backend/requirements-hashes.txt + + - name: Install hash-locked backend dependencies + run: >- + python -m pip install --disable-pip-version-check --require-hashes + -r backend/requirements-hashes.txt + + - name: Add source-identity regressions and prove RED + shell: bash + run: | + set -euo pipefail + cat > backend/tests/test_source_bound_email_dedupe.py <<'PY' + """Regression tests for source-bound fallback email identities.""" + + import datetime + from unittest.mock import AsyncMock, MagicMock + + import pytest + + from services.email_dedupe_service import ( + canonical_email_source_content, + source_email_fingerprint, + strong_email_fingerprint, + ) + from services.email_import_service import _email_fingerprint + from services.imap_worker import process_fetched_email + + + def test_source_email_fingerprint_is_stable_content_bound_and_domain_separated() -> None: + """Hash equal sources equally while separating content and source domains.""" + first = source_email_fingerprint(b"same source") + assert first == source_email_fingerprint(b"same source") + assert first != source_email_fingerprint(b"different source") + assert first != source_email_fingerprint( + b"same source", source_kind="canonical" + ) + assert len(first) == 64 + + + def test_canonical_source_content_excludes_collection_date() -> None: + """Keep synthetic collection time outside direct-caller identity.""" + base = { + "message_id": "", + "sender": "sender@example.com", + "recipients": ["one@example.com", "two@example.com"], + "subject": "Subject", + "body": "Body", + "attachments": [{"filename": "note.txt", "content": b"note"}], + } + first = { + **base, + "date": datetime.datetime( + 2026, 8, 4, 6, 30, tzinfo=datetime.timezone.utc + ), + "date_provenance": "missing", + } + second = { + **base, + "date": datetime.datetime( + 2026, 8, 4, 7, 30, tzinfo=datetime.timezone.utc + ), + "date_provenance": "invalid", + } + assert canonical_email_source_content(first) == canonical_email_source_content( + second + ) + assert canonical_email_source_content(first) != canonical_email_source_content( + {**second, "body": "Different body"} + ) + + + def test_import_fingerprint_uses_trusted_date_or_raw_source() -> None: + """Use strong Date evidence only when provenance is genuinely parsed.""" + persisted_date = datetime.datetime( + 2026, 8, 4, 6, 30, tzinfo=datetime.timezone.utc + ) + fields = { + "message_id": "", + "sender": "sender@example.com", + "subject": "Same subject", + "body": "Same parsed body", + "recipients": "recipient@example.com", + } + first_source = b"From: sender@example.com\r\n\r\nFirst raw body" + second_source = b"From: sender@example.com\r\n\r\nSecond raw body" + strong = strong_email_fingerprint( + sender=fields["sender"], + subject=fields["subject"], + date=persisted_date, + body=fields["body"], + ) + assert strong is not None + assert ( + _email_fingerprint( + {**fields, "date_provenance": "parsed"}, + persisted_date, + first_source, + ) + == strong + ) + for provenance in ("missing", "invalid"): + first = _email_fingerprint( + {**fields, "date_provenance": provenance}, + persisted_date, + first_source, + ) + second = _email_fingerprint( + {**fields, "date_provenance": provenance}, + persisted_date, + second_source, + ) + assert first == source_email_fingerprint(first_source) + assert second == source_email_fingerprint(second_source) + assert first != second + + + def test_direct_fallback_is_collection_time_independent() -> None: + """Give callers without raw bytes a stable non-date fallback identity.""" + parsed = { + "message_id": "", + "sender": "sender@example.com", + "recipients": "recipient@example.com", + "subject": "Direct", + "body": "Body", + "date_provenance": "missing", + } + first_time = datetime.datetime( + 2026, 8, 4, 6, 30, tzinfo=datetime.timezone.utc + ) + second_time = first_time + datetime.timedelta(hours=1) + assert _email_fingerprint(parsed, first_time) == _email_fingerprint( + parsed, second_time + ) + assert _email_fingerprint(parsed, first_time) != _email_fingerprint( + {**parsed, "body": "Different body"}, second_time + ) + + + @pytest.mark.asyncio + async def test_missing_date_messages_use_raw_source_not_collection_time( + monkeypatch: pytest.MonkeyPatch, + ) -> None: + """Do not merge different raw messages collected at the same instant.""" + session = AsyncMock() + session.add = MagicMock() + query_result = MagicMock() + query_result.scalar_one_or_none.return_value = None + session.execute.return_value = query_result + monkeypatch.setattr( + "services.imap_worker.assign_thread_id", + AsyncMock(side_effect=("thread-first", "thread-second")), + ) + monkeypatch.setattr( + "services.imap_worker.is_self_sent_email", + lambda _email, _owners: False, + ) + + collected_at = datetime.datetime( + 2026, 8, 4, 6, 30, tzinfo=datetime.timezone.utc + ) + common = { + "subject": "Same subject", + "date": collected_at, + "date_provenance": "missing", + "sender": "sender@example.com", + "recipients": "recipient@example.com", + "message_id": "", + } + first_source = b"From: sender@example.com\r\n\r\nFirst raw body" + second_source = b"From: sender@example.com\r\n\r\nSecond raw body" + + await process_fetched_email( + session, + {**common, "body": "First body"}, + "owner@example.com", + "org-acme", + source_content=first_source, + ) + await process_fetched_email( + session, + {**common, "body": "Second body"}, + "owner@example.com", + "org-acme", + source_content=second_source, + ) + + first_email = session.add.call_args_list[0].args[0] + second_email = session.add.call_args_list[1].args[0] + assert first_email.date == collected_at + assert second_email.date == collected_at + assert first_email.fingerprint == source_email_fingerprint(first_source) + assert second_email.fingerprint == source_email_fingerprint(second_source) + assert first_email.fingerprint != second_email.fingerprint + PY + + python3 - <<'PY' + from pathlib import Path + + parser_test = Path("backend/tests/test_email_parser_provenance.py") + text = parser_test.read_text(encoding="utf-8") + old = ' return (headers.strip() + "\\n\\nBody text.").encode("utf-8")\n' + new = ' return (headers.strip("\\r\\n") + "\\n\\nBody text.").encode("utf-8")\n' + if old not in text: + raise SystemExit("email parser fixture anchor missing") + parser_test.write_text(text.replace(old, new, 1), encoding="utf-8") + + imap_test = Path("backend/tests/test_imap_worker.py") + text = imap_test.read_text(encoding="utf-8") + anchor = ' assert kwargs["owner_addresses"] == ["imap-user@example.com"]\n' + replacement = anchor + ' assert kwargs["source_content"] == raw_message\n' + if replacement not in text: + if anchor not in text: + raise SystemExit("IMAP raw-source assertion anchor missing") + text = text.replace(anchor, replacement, 1) + imap_test.write_text(text, encoding="utf-8") + + pop3_test = Path("backend/tests/test_pop3_worker.py") + text = pop3_test.read_text(encoding="utf-8") + old_signature = ''' async def fake_process_fetched_email( + db_session, email_data, user_id, organization_id, owner_addresses=None + ): +''' + new_signature = ''' async def fake_process_fetched_email( + db_session, + email_data, + user_id, + organization_id, + owner_addresses=None, + source_content=None, + ): +''' + if new_signature not in text: + if old_signature not in text: + raise SystemExit("POP3 fake processor signature anchor missing") + text = text.replace(old_signature, new_signature, 1) + old_item = ' "owner_addresses": owner_addresses,\n' + new_item = old_item + ' "source_content": source_content,\n' + if new_item not in text: + if old_item not in text: + raise SystemExit("POP3 imported record anchor missing") + text = text.replace(old_item, new_item, 1) + old_assert = ' assert imported[0]["owner_addresses"] == ["pop3-user@example.com"]\n' + new_assert = old_assert + ' assert imported[0]["source_content"] == raw_message\n' + if new_assert not in text: + if old_assert not in text: + raise SystemExit("POP3 raw-source assertion anchor missing") + text = text.replace(old_assert, new_assert, 1) + pop3_test.write_text(text, encoding="utf-8") + PY + + set +e + ( + cd backend + python -m pytest -q tests/test_source_bound_email_dedupe.py + ) + red_status=$? + set -e + if [ "$red_status" -eq 0 ]; then + echo "::error::Source-bound dedupe regressions unexpectedly passed before remediation." + exit 1 + fi + + - name: Bind untrusted-Date identity to immutable source content + shell: bash + run: | + set -euo pipefail + python3 - <<'PY' + from pathlib import Path + + + def replace_once(text: str, old: str, new: str, label: str) -> str: + if new in text: + return text + count = text.count(old) + if count != 1: + raise SystemExit(f"{label}: expected one anchor, found {count}") + return text.replace(old, new, 1) + + + dedupe_path = Path("backend/services/email_dedupe_service.py") + text = dedupe_path.read_text(encoding="utf-8") + text = replace_once( + text, + "import datetime\nfrom collections.abc import Iterable\n", + "import datetime\nimport hashlib\nimport json\nfrom collections.abc import Iterable, Mapping\n", + "dedupe imports", + ) + helper = '''_CANONICAL_SOURCE_FIELDS = ( + "message_id", + "sender", + "recipients", + "subject", + "body", + "reply_to", + "in_reply_to", + "references", + "attachments", + ) + + + def canonical_email_source_content(email_data: Mapping[str, object]) -> bytes: + """Serialize stable parsed fields when raw transport bytes are unavailable. + + Collection-time ``date`` values and their provenance are deliberately + excluded. Transport-backed paths should provide exact RFC822 bytes. + """ + payload = { + field: email_data.get(field) for field in _CANONICAL_SOURCE_FIELDS + } + return json.dumps( + payload, + ensure_ascii=False, + sort_keys=True, + separators=(",", ":"), + default=str, + ).encode("utf-8", errors="surrogatepass") + + + def source_email_fingerprint( + source_content: bytes, + *, + source_kind: Literal["raw", "canonical"] = "raw", + ) -> str: + """Return a domain-separated SHA-256 identity for stable source bytes.""" + digest = hashlib.sha256() + digest.update(b"naruon-email-source-v1\\0") + digest.update(source_kind.encode("ascii")) + digest.update(b"\\0") + digest.update(source_content) + return digest.hexdigest() + + + '''.replace(" ", "") + if "def source_email_fingerprint" not in text: + text = replace_once( + text, + "def strong_email_fingerprint(\n", + helper + "def strong_email_fingerprint(\n", + "dedupe helper insertion", + ) + dedupe_path.write_text(text, encoding="utf-8") + + import_path = Path("backend/services/email_import_service.py") + text = import_path.read_text(encoding="utf-8") + text = replace_once( + text, + "from services.email_dedupe_service import strong_email_fingerprint\n", + "from services.email_dedupe_service import (\n" + " canonical_email_source_content,\n" + " source_email_fingerprint,\n" + " strong_email_fingerprint,\n" + ")\n", + "import-service dedupe import", + ) + start = text.index("def _email_fingerprint(") + end = text.index("\n\n\nasync def _find_existing_email", start) + replacement = '''def _email_fingerprint( + parsed: EmailData, + persisted_date: datetime.datetime, + source_content: bytes | None = None, + ) -> str: + """Return trusted-Date evidence or a source-bound fallback identity. + + ``persisted_date`` remains the storage timestamp and participates in + duplicate evidence only when it came from a valid sender ``Date``. + """ + strong_fingerprint = None + if parsed.get("date_provenance") == "parsed": + strong_fingerprint = strong_email_fingerprint( + sender=parsed.get("sender"), + subject=parsed.get("subject"), + date=persisted_date, + body=parsed.get("body"), + ) + if strong_fingerprint: + return strong_fingerprint + source_identity = ( + source_content + if source_content is not None + else canonical_email_source_content(parsed) + ) + return source_email_fingerprint( + source_identity, + source_kind="raw" if source_content is not None else "canonical", + ) + '''.replace(" ", "").rstrip() + text = text[:start] + replacement + text[end:] + text = replace_once( + text, + " fingerprint = _email_fingerprint(parsed, persisted_date)\n", + " fingerprint = _email_fingerprint(parsed, persisted_date, content)\n", + "import fingerprint call", + ) + text = text.replace(" generate_email_fingerprint,\n", "", 1) + import_path.write_text(text, encoding="utf-8") + + imap_path = Path("backend/services/imap_worker.py") + text = imap_path.read_text(encoding="utf-8") + text = replace_once( + text, + "from services.email_dedupe_service import strong_email_fingerprint\n", + "from services.email_dedupe_service import (\n" + " canonical_email_source_content,\n" + " source_email_fingerprint,\n" + " strong_email_fingerprint,\n" + ")\n", + "IMAP dedupe import", + ) + text = replace_once( + text, + "from services.threading_service import assign_thread_id, generate_email_fingerprint\n", + "from services.threading_service import assign_thread_id\n", + "IMAP threading import", + ) + old_signature = ''' owner_addresses: Iterable[str] | None = None, + is_read: bool = True, + ): + subject = email_data.get("subject", "") + '''.replace(" ", "") + new_signature = ''' owner_addresses: Iterable[str] | None = None, + is_read: bool = True, + source_content: bytes | None = None, + ) -> Email: + """Persist one fetched email with provenance-safe identity.""" + subject = email_data.get("subject", "") + '''.replace(" ", "") + text = replace_once(text, old_signature, new_signature, "IMAP signature") + fingerprint_start = text.index(" # Seed the strong") + fingerprint_end = text.index("\n\n # Check if duplicate", fingerprint_start) + fingerprint_block = ''' # Seed strong duplicate evidence only from a genuinely parsed Date. + strong_fingerprint = None + if email_data.get("date_provenance") == "parsed": + strong_fingerprint = strong_email_fingerprint( + sender=sender, + subject=subject, + date=persisted_date, + body=email_data.get("body", ""), + ) + source_identity = ( + source_content + if source_content is not None + else canonical_email_source_content(email_data) + ) + fingerprint = strong_fingerprint or source_email_fingerprint( + source_identity, + source_kind="raw" if source_content is not None else "canonical", + ) + '''.replace(" ", "").rstrip() + text = text[:fingerprint_start] + fingerprint_block + text[fingerprint_end:] + text = replace_once( + text, + " is_read=is_read,\n )\n", + " is_read=is_read,\n" + " source_content=raw_message,\n" + " )\n", + "IMAP raw-source propagation", + ) + imap_path.write_text(text, encoding="utf-8") + + pop3_path = Path("backend/services/pop3_worker.py") + text = pop3_path.read_text(encoding="utf-8") + text = replace_once( + text, + " owner_addresses=owner_addresses,\n )\n", + " owner_addresses=owner_addresses,\n" + " source_content=raw_message,\n" + " )\n", + "POP3 raw-source propagation", + ) + pop3_path.write_text(text, encoding="utf-8") + + changelog_path = Path("CHANGELOG.md") + text = changelog_path.read_text(encoding="utf-8") + text = text.replace( + "strong key를 만들지 않고 weak fallback fingerprint(수집 시각이 매번 달라 거짓 중복을 만들 수 없음)로 내려갑니다", + "strong key를 만들지 않고 원본 RFC822 바이트의 domain-separated SHA-256(원문이 없는 직접 호출자는 Date를 제외한 canonical source projection)로 내려갑니다", + ) + text = text.replace( + "import·IMAP 두 생성 경로 모두 파서 provenance를 영속화합니다.", + "import·IMAP·POP3 생성 경로 모두 파서 provenance를 영속화하고 원본 transport bytes를 fallback identity에 전달합니다.", + ) + changelog_path.write_text(text, encoding="utf-8") + PY + + - name: Verify focused and realistic email pipelines + shell: bash + run: | + set -euo pipefail + python -m ruff format \ + backend/services/email_dedupe_service.py \ + backend/services/email_import_service.py \ + backend/services/imap_worker.py \ + backend/services/pop3_worker.py \ + backend/tests/test_source_bound_email_dedupe.py \ + backend/tests/test_email_import_service.py \ + backend/tests/test_email_parser_provenance.py \ + backend/tests/test_imap_worker.py \ + backend/tests/test_pop3_worker.py + python -m ruff check \ + backend/services/email_dedupe_service.py \ + backend/services/email_import_service.py \ + backend/services/imap_worker.py \ + backend/services/pop3_worker.py \ + backend/tests/test_source_bound_email_dedupe.py \ + backend/tests/test_email_import_service.py \ + backend/tests/test_email_parser_provenance.py \ + backend/tests/test_imap_worker.py \ + backend/tests/test_pop3_worker.py + ( + cd backend + python -m pytest -q \ + tests/test_source_bound_email_dedupe.py \ + tests/test_email_parser_provenance.py \ + tests/test_email_dedupe_service.py \ + tests/test_email_import_service.py \ + tests/test_imap_worker.py \ + tests/test_pop3_worker.py + ) + git diff --check + + - name: Commit verified remediation and remove one-shot workflow + shell: bash + run: | + set -euo pipefail + rm .github/workflows/pr1195-source-identity-repair.yml + git diff --check + git config user.name github-actions[bot] + git config user.email 41898282+github-actions[bot]@users.noreply.github.com + git add \ + CHANGELOG.md \ + backend/services/email_dedupe_service.py \ + backend/services/email_import_service.py \ + backend/services/imap_worker.py \ + backend/services/pop3_worker.py \ + backend/tests/test_source_bound_email_dedupe.py \ + backend/tests/test_email_import_service.py \ + backend/tests/test_email_parser_provenance.py \ + backend/tests/test_imap_worker.py \ + backend/tests/test_pop3_worker.py \ + .github/workflows/pr1195-source-identity-repair.yml + git commit -m "fix(email): bind fallback dedupe to immutable source" + git push origin HEAD:claude/contextualwisdomlab-audit-governance-qyxe67 From 6e69b8e1b31c41f9b73d1127beae1fd19824b60a Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 21:01:39 +0900 Subject: [PATCH 29/86] chore(email): add one-shot source identity repair script --- scripts/ci/pr1195_source_identity_repair.py | 172 ++++++++++++++++++++ 1 file changed, 172 insertions(+) create mode 100644 scripts/ci/pr1195_source_identity_repair.py diff --git a/scripts/ci/pr1195_source_identity_repair.py b/scripts/ci/pr1195_source_identity_repair.py new file mode 100644 index 000000000..1ecfe503d --- /dev/null +++ b/scripts/ci/pr1195_source_identity_repair.py @@ -0,0 +1,172 @@ +from __future__ import annotations + +import pathlib +import re + +ROOT = pathlib.Path(".") + +def read(path: str) -> str: + return (ROOT / path).read_text(encoding="utf-8") + +def write(path: str, content: str) -> None: + (ROOT / path).write_text(content, encoding="utf-8") + +def replace_once(text: str, old: str, new: str, label: str) -> str: + if new in text: + return text + count = text.count(old) + if count != 1: + raise SystemExit(f"{label}: expected one anchor, found {count}") + return text.replace(old, new, 1) + +path = "backend/services/email_dedupe_service.py" +text = read(path) +text = replace_once( + text, + "import datetime\nfrom collections.abc import Iterable\n", + "import datetime\nimport hashlib\nimport json\nfrom collections.abc import Iterable, Mapping\n", + "email_dedupe_service imports", +) +helper = '_CANONICAL_SOURCE_FIELDS = (\n "message_id",\n "sender",\n "recipients",\n "subject",\n "body",\n "reply_to",\n "in_reply_to",\n "references",\n "attachments",\n)\n\n\ndef canonical_email_source_content(email_data: Mapping[str, object]) -> bytes:\n """Serialize stable parsed fields when raw transport bytes are unavailable.\n\n Collection-time ``date`` values and their provenance are deliberately\n excluded. Transport-backed paths should provide exact RFC822 bytes.\n """\n payload = {field: email_data.get(field) for field in _CANONICAL_SOURCE_FIELDS}\n return json.dumps(\n payload,\n ensure_ascii=False,\n sort_keys=True,\n separators=(",", ":"),\n default=str,\n ).encode("utf-8", errors="surrogatepass")\n\n\ndef source_email_fingerprint(\n source_content: bytes,\n *,\n source_kind: Literal["raw", "canonical"] = "raw",\n) -> str:\n """Return a domain-separated SHA-256 identity for stable source bytes."""\n digest = hashlib.sha256()\n digest.update(b"naruon-email-source-v1\\0")\n digest.update(source_kind.encode("ascii"))\n digest.update(b"\\0")\n digest.update(source_content)\n return digest.hexdigest()\n\n\n' +if "def source_email_fingerprint(" not in text: + text = replace_once( + text, + "def strong_email_fingerprint(\n", + helper + "def strong_email_fingerprint(\n", + "email_dedupe_service helper insertion", + ) +write(path, text) + +path = "backend/services/email_import_service.py" +text = read(path) +text = replace_once( + text, + "from services.email_dedupe_service import strong_email_fingerprint\n", + "from services.email_dedupe_service import (\n canonical_email_source_content,\n source_email_fingerprint,\n strong_email_fingerprint,\n)\n", + "email_import_service dedupe import", +) +pattern = re.compile( + r"def _email_fingerprint\(parsed: EmailData, persisted_date: datetime\.datetime\) -> str:\n" + r".*?\n\n\nasync def _find_existing_email", + re.DOTALL, +) +replacement = 'def _email_fingerprint(\n parsed: EmailData,\n persisted_date: datetime.datetime,\n source_content: bytes | None = None,\n) -> str:\n """Return trusted-Date evidence or a source-bound fallback identity.\n\n ``persisted_date`` remains the storage timestamp and participates in\n duplicate evidence only when it came from a valid sender ``Date``.\n """\n strong_fingerprint = None\n if parsed.get("date_provenance") == "parsed":\n strong_fingerprint = strong_email_fingerprint(\n sender=parsed.get("sender"),\n subject=parsed.get("subject"),\n date=persisted_date,\n body=parsed.get("body"),\n )\n if strong_fingerprint:\n return strong_fingerprint\n source_identity = (\n source_content\n if source_content is not None\n else canonical_email_source_content(parsed)\n )\n return source_email_fingerprint(\n source_identity,\n source_kind="raw" if source_content is not None else "canonical",\n )\n\n\nasync def _find_existing_email' +text, count = pattern.subn(replacement, text, count=1) +if count != 1 and "source-bound fallback identity" not in text: + raise SystemExit(f"email_import_service fingerprint block: replaced {count}") +text = replace_once( + text, + " fingerprint = _email_fingerprint(parsed, persisted_date)\n", + " fingerprint = _email_fingerprint(parsed, persisted_date, content)\n", + "email_import_service fingerprint call", +) +if text.count("generate_email_fingerprint") == 1: + text = text.replace(" generate_email_fingerprint,\n", "", 1) +write(path, text) + +path = "backend/services/imap_worker.py" +text = read(path) +text = replace_once( + text, + "from services.email_dedupe_service import strong_email_fingerprint\n", + "from services.email_dedupe_service import (\n canonical_email_source_content,\n source_email_fingerprint,\n strong_email_fingerprint,\n)\n", + "imap_worker dedupe import", +) +text = replace_once( + text, + "from services.threading_service import assign_thread_id, generate_email_fingerprint\n", + "from services.threading_service import assign_thread_id\n", + "imap_worker threading import", +) +text = replace_once( + text, + ' owner_addresses: Iterable[str] | None = None,\n is_read: bool = True,\n):\n subject = email_data.get("subject", "")\n date_obj = email_data.get("date")\n if hasattr(date_obj, "isoformat"):\n date_str = date_obj.isoformat()\n else:\n date_str = str(date_obj) if date_obj else ""\n', + ' owner_addresses: Iterable[str] | None = None,\n is_read: bool = True,\n source_content: bytes | None = None,\n) -> Email:\n """Persist one fetched email with provenance-safe identity."""\n subject = email_data.get("subject", "")\n date_obj = email_data.get("date")\n', + "imap_worker signature", +) +fingerprint_pattern = re.compile( + r" # Seed the strong \(auto-dedupe\) fingerprint only from a genuinely-parsed\n" + r".*?" + r" fingerprint = strong_fingerprint or generate_email_fingerprint\(\n" + r" subject, date_str, sender, recipients\n" + r" \)\n", + re.DOTALL, +) +fingerprint_replacement = ' # Seed strong duplicate evidence only from a genuinely parsed Date.\n strong_fingerprint = None\n if email_data.get("date_provenance") == "parsed":\n strong_fingerprint = strong_email_fingerprint(\n sender=sender,\n subject=subject,\n date=persisted_date,\n body=email_data.get("body", ""),\n )\n source_identity = (\n source_content\n if source_content is not None\n else canonical_email_source_content(email_data)\n )\n fingerprint = strong_fingerprint or source_email_fingerprint(\n source_identity,\n source_kind="raw" if source_content is not None else "canonical",\n )\n' +text, count = fingerprint_pattern.subn(fingerprint_replacement, text, count=1) +if count != 1 and "source_identity = (" not in text: + raise SystemExit(f"imap_worker fingerprint block: replaced {count}") +text = replace_once( + text, + " is_read=is_read,\n )\n", + " is_read=is_read,\n source_content=raw_message,\n )\n", + "imap_worker raw source propagation", +) +write(path, text) + +path = "backend/services/pop3_worker.py" +text = read(path) +text = replace_once( + text, + " owner_addresses=owner_addresses,\n )\n", + " owner_addresses=owner_addresses,\n source_content=raw_message,\n )\n", + "pop3_worker raw source propagation", +) +write(path, text) + +path = "backend/tests/test_imap_worker.py" +text = read(path) +text = replace_once( + text, + ' assert kwargs["owner_addresses"] == ["imap-user@example.com"]\n', + ' assert kwargs["owner_addresses"] == ["imap-user@example.com"]\n assert kwargs["source_content"] == raw_message\n', + "imap worker source assertion", +) +write(path, text) + +path = "backend/tests/test_pop3_worker.py" +text = read(path) +text = replace_once( + text, + ' async def fake_process_fetched_email(\n db_session, email_data, user_id, organization_id, owner_addresses=None\n ):\n', + ' async def fake_process_fetched_email(\n db_session,\n email_data,\n user_id,\n organization_id,\n owner_addresses=None,\n source_content=None,\n ):\n', + "pop3 fake processor signature", +) +text = replace_once( + text, + ' "owner_addresses": owner_addresses,\n', + ' "owner_addresses": owner_addresses,\n "source_content": source_content,\n', + "pop3 imported source record", +) +text = replace_once( + text, + ' assert imported[0]["owner_addresses"] == ["pop3-user@example.com"]\n', + ' assert imported[0]["owner_addresses"] == ["pop3-user@example.com"]\n assert imported[0]["source_content"] == raw_message\n', + "pop3 worker source assertion", +) +write(path, text) + +path = "backend/tests/test_email_parser_provenance.py" +text = read(path) +text = replace_once( + text, + ' return (headers.strip() + "\\n\\nBody text.").encode("utf-8")\n', + ' return (headers.strip("\\r\\n") + "\\n\\nBody text.").encode("utf-8")\n', + "parser provenance fixture", +) +write(path, text) + +path = "CHANGELOG.md" +text = read(path) +note = ( + "- Bound missing/invalid-Date email deduplication fallbacks to immutable " + "RFC822 source bytes (or a deterministic Date-free canonical projection), " + "including import, IMAP, and POP3 paths, so collection timestamps cannot " + "create false duplicate identities.\n" +) +if note not in text: + if "### Fixed\n" in text: + text = text.replace("### Fixed\n", "### Fixed\n" + note, 1) + else: + text = text.replace("## [Unreleased]\n", "## [Unreleased]\n\n### Fixed\n" + note, 1) +write(path, text) From 96fd422cb96c48266bcfa92b12ba666972b722af Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 21:03:21 +0900 Subject: [PATCH 30/86] test(email): add source-bound fallback regressions --- .../tests/test_source_bound_email_dedupe.py | 179 ++++++++++++++++++ 1 file changed, 179 insertions(+) create mode 100644 backend/tests/test_source_bound_email_dedupe.py diff --git a/backend/tests/test_source_bound_email_dedupe.py b/backend/tests/test_source_bound_email_dedupe.py new file mode 100644 index 000000000..434d18387 --- /dev/null +++ b/backend/tests/test_source_bound_email_dedupe.py @@ -0,0 +1,179 @@ +"""Regression tests for source-bound fallback email identities.""" + +import datetime +from unittest.mock import AsyncMock, MagicMock + +import pytest + +from services.email_dedupe_service import ( + canonical_email_source_content, + source_email_fingerprint, + strong_email_fingerprint, +) +from services.email_import_service import _email_fingerprint +from services.imap_worker import process_fetched_email + + +def test_source_email_fingerprint_is_stable_content_bound_and_domain_separated() -> None: + """Hash equal sources equally while separating content and source domains.""" + first = source_email_fingerprint(b"same source") + assert first == source_email_fingerprint(b"same source") + assert first != source_email_fingerprint(b"different source") + assert first != source_email_fingerprint( + b"same source", source_kind="canonical" + ) + assert len(first) == 64 + + +def test_canonical_source_content_excludes_collection_date() -> None: + """Keep synthetic collection time outside direct-caller identity.""" + base = { + "message_id": "", + "sender": "sender@example.com", + "recipients": ["one@example.com", "two@example.com"], + "subject": "Subject", + "body": "Body", + "attachments": [{"filename": "note.txt", "content": b"note"}], + } + first = { + **base, + "date": datetime.datetime( + 2026, 8, 4, 6, 30, tzinfo=datetime.timezone.utc + ), + "date_provenance": "missing", + } + second = { + **base, + "date": datetime.datetime( + 2026, 8, 4, 7, 30, tzinfo=datetime.timezone.utc + ), + "date_provenance": "invalid", + } + assert canonical_email_source_content(first) == canonical_email_source_content( + second + ) + assert canonical_email_source_content(first) != canonical_email_source_content( + {**second, "body": "Different body"} + ) + + +def test_import_fingerprint_uses_trusted_date_or_raw_source() -> None: + """Use strong Date evidence only when provenance is genuinely parsed.""" + persisted_date = datetime.datetime( + 2026, 8, 4, 6, 30, tzinfo=datetime.timezone.utc + ) + fields = { + "message_id": "", + "sender": "sender@example.com", + "subject": "Same subject", + "body": "Same parsed body", + "recipients": "recipient@example.com", + } + first_source = b"From: sender@example.com\r\n\r\nFirst raw body" + second_source = b"From: sender@example.com\r\n\r\nSecond raw body" + strong = strong_email_fingerprint( + sender=fields["sender"], + subject=fields["subject"], + date=persisted_date, + body=fields["body"], + ) + assert strong is not None + assert ( + _email_fingerprint( + {**fields, "date_provenance": "parsed"}, + persisted_date, + first_source, + ) + == strong + ) + for provenance in ("missing", "invalid"): + first = _email_fingerprint( + {**fields, "date_provenance": provenance}, + persisted_date, + first_source, + ) + second = _email_fingerprint( + {**fields, "date_provenance": provenance}, + persisted_date, + second_source, + ) + assert first == source_email_fingerprint(first_source) + assert second == source_email_fingerprint(second_source) + assert first != second + + +def test_direct_fallback_is_collection_time_independent() -> None: + """Give callers without raw bytes a stable non-date fallback identity.""" + parsed = { + "message_id": "", + "sender": "sender@example.com", + "recipients": "recipient@example.com", + "subject": "Direct", + "body": "Body", + "date_provenance": "missing", + } + first_time = datetime.datetime( + 2026, 8, 4, 6, 30, tzinfo=datetime.timezone.utc + ) + second_time = first_time + datetime.timedelta(hours=1) + assert _email_fingerprint(parsed, first_time) == _email_fingerprint( + parsed, second_time + ) + assert _email_fingerprint(parsed, first_time) != _email_fingerprint( + {**parsed, "body": "Different body"}, second_time + ) + + +@pytest.mark.asyncio +async def test_missing_date_messages_use_raw_source_not_collection_time( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Do not merge different raw messages collected at the same instant.""" + session = AsyncMock() + session.add = MagicMock() + query_result = MagicMock() + query_result.scalar_one_or_none.return_value = None + session.execute.return_value = query_result + monkeypatch.setattr( + "services.imap_worker.assign_thread_id", + AsyncMock(side_effect=("thread-first", "thread-second")), + ) + monkeypatch.setattr( + "services.imap_worker.is_self_sent_email", + lambda _email, _owners: False, + ) + + collected_at = datetime.datetime( + 2026, 8, 4, 6, 30, tzinfo=datetime.timezone.utc + ) + common = { + "subject": "Same subject", + "date": collected_at, + "date_provenance": "missing", + "sender": "sender@example.com", + "recipients": "recipient@example.com", + "message_id": "", + } + first_source = b"From: sender@example.com\r\n\r\nFirst raw body" + second_source = b"From: sender@example.com\r\n\r\nSecond raw body" + + first_email = await process_fetched_email( + session, + {**common, "body": "First body"}, + "owner@example.com", + "org-acme", + source_content=first_source, + ) + second_email = await process_fetched_email( + session, + {**common, "body": "Second body"}, + "owner@example.com", + "org-acme", + source_content=second_source, + ) + + assert first_email.date == collected_at + assert second_email.date == collected_at + assert first_email.fingerprint == source_email_fingerprint(first_source) + assert second_email.fingerprint == source_email_fingerprint(second_source) + assert first_email.fingerprint != second_email.fingerprint From 64d5a8484d16fbc61fe028d552a298d0b95cdb1f Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 21:05:17 +0900 Subject: [PATCH 31/86] fix(ci): make source-identity repair workflow valid --- .../pr1195-source-identity-repair.yml | 522 +----------------- 1 file changed, 15 insertions(+), 507 deletions(-) diff --git a/.github/workflows/pr1195-source-identity-repair.yml b/.github/workflows/pr1195-source-identity-repair.yml index ee3b18215..5bb56df6e 100644 --- a/.github/workflows/pr1195-source-identity-repair.yml +++ b/.github/workflows/pr1195-source-identity-repair.yml @@ -26,18 +26,18 @@ jobs: DISABLE_BACKGROUND_WORKERS: "1" steps: - name: Harden runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 with: egress-policy: audit - name: Checkout exact pull request branch - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 with: ref: claude/contextualwisdomlab-audit-governance-qyxe67 fetch-depth: 0 - name: Set up Python - uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6 + uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 with: python-version: "3.14" cache: pip @@ -48,249 +48,10 @@ jobs: python -m pip install --disable-pip-version-check --require-hashes -r backend/requirements-hashes.txt - - name: Add source-identity regressions and prove RED + - name: Prove source-bound regressions fail before implementation shell: bash run: | set -euo pipefail - cat > backend/tests/test_source_bound_email_dedupe.py <<'PY' - """Regression tests for source-bound fallback email identities.""" - - import datetime - from unittest.mock import AsyncMock, MagicMock - - import pytest - - from services.email_dedupe_service import ( - canonical_email_source_content, - source_email_fingerprint, - strong_email_fingerprint, - ) - from services.email_import_service import _email_fingerprint - from services.imap_worker import process_fetched_email - - - def test_source_email_fingerprint_is_stable_content_bound_and_domain_separated() -> None: - """Hash equal sources equally while separating content and source domains.""" - first = source_email_fingerprint(b"same source") - assert first == source_email_fingerprint(b"same source") - assert first != source_email_fingerprint(b"different source") - assert first != source_email_fingerprint( - b"same source", source_kind="canonical" - ) - assert len(first) == 64 - - - def test_canonical_source_content_excludes_collection_date() -> None: - """Keep synthetic collection time outside direct-caller identity.""" - base = { - "message_id": "", - "sender": "sender@example.com", - "recipients": ["one@example.com", "two@example.com"], - "subject": "Subject", - "body": "Body", - "attachments": [{"filename": "note.txt", "content": b"note"}], - } - first = { - **base, - "date": datetime.datetime( - 2026, 8, 4, 6, 30, tzinfo=datetime.timezone.utc - ), - "date_provenance": "missing", - } - second = { - **base, - "date": datetime.datetime( - 2026, 8, 4, 7, 30, tzinfo=datetime.timezone.utc - ), - "date_provenance": "invalid", - } - assert canonical_email_source_content(first) == canonical_email_source_content( - second - ) - assert canonical_email_source_content(first) != canonical_email_source_content( - {**second, "body": "Different body"} - ) - - - def test_import_fingerprint_uses_trusted_date_or_raw_source() -> None: - """Use strong Date evidence only when provenance is genuinely parsed.""" - persisted_date = datetime.datetime( - 2026, 8, 4, 6, 30, tzinfo=datetime.timezone.utc - ) - fields = { - "message_id": "", - "sender": "sender@example.com", - "subject": "Same subject", - "body": "Same parsed body", - "recipients": "recipient@example.com", - } - first_source = b"From: sender@example.com\r\n\r\nFirst raw body" - second_source = b"From: sender@example.com\r\n\r\nSecond raw body" - strong = strong_email_fingerprint( - sender=fields["sender"], - subject=fields["subject"], - date=persisted_date, - body=fields["body"], - ) - assert strong is not None - assert ( - _email_fingerprint( - {**fields, "date_provenance": "parsed"}, - persisted_date, - first_source, - ) - == strong - ) - for provenance in ("missing", "invalid"): - first = _email_fingerprint( - {**fields, "date_provenance": provenance}, - persisted_date, - first_source, - ) - second = _email_fingerprint( - {**fields, "date_provenance": provenance}, - persisted_date, - second_source, - ) - assert first == source_email_fingerprint(first_source) - assert second == source_email_fingerprint(second_source) - assert first != second - - - def test_direct_fallback_is_collection_time_independent() -> None: - """Give callers without raw bytes a stable non-date fallback identity.""" - parsed = { - "message_id": "", - "sender": "sender@example.com", - "recipients": "recipient@example.com", - "subject": "Direct", - "body": "Body", - "date_provenance": "missing", - } - first_time = datetime.datetime( - 2026, 8, 4, 6, 30, tzinfo=datetime.timezone.utc - ) - second_time = first_time + datetime.timedelta(hours=1) - assert _email_fingerprint(parsed, first_time) == _email_fingerprint( - parsed, second_time - ) - assert _email_fingerprint(parsed, first_time) != _email_fingerprint( - {**parsed, "body": "Different body"}, second_time - ) - - - @pytest.mark.asyncio - async def test_missing_date_messages_use_raw_source_not_collection_time( - monkeypatch: pytest.MonkeyPatch, - ) -> None: - """Do not merge different raw messages collected at the same instant.""" - session = AsyncMock() - session.add = MagicMock() - query_result = MagicMock() - query_result.scalar_one_or_none.return_value = None - session.execute.return_value = query_result - monkeypatch.setattr( - "services.imap_worker.assign_thread_id", - AsyncMock(side_effect=("thread-first", "thread-second")), - ) - monkeypatch.setattr( - "services.imap_worker.is_self_sent_email", - lambda _email, _owners: False, - ) - - collected_at = datetime.datetime( - 2026, 8, 4, 6, 30, tzinfo=datetime.timezone.utc - ) - common = { - "subject": "Same subject", - "date": collected_at, - "date_provenance": "missing", - "sender": "sender@example.com", - "recipients": "recipient@example.com", - "message_id": "", - } - first_source = b"From: sender@example.com\r\n\r\nFirst raw body" - second_source = b"From: sender@example.com\r\n\r\nSecond raw body" - - await process_fetched_email( - session, - {**common, "body": "First body"}, - "owner@example.com", - "org-acme", - source_content=first_source, - ) - await process_fetched_email( - session, - {**common, "body": "Second body"}, - "owner@example.com", - "org-acme", - source_content=second_source, - ) - - first_email = session.add.call_args_list[0].args[0] - second_email = session.add.call_args_list[1].args[0] - assert first_email.date == collected_at - assert second_email.date == collected_at - assert first_email.fingerprint == source_email_fingerprint(first_source) - assert second_email.fingerprint == source_email_fingerprint(second_source) - assert first_email.fingerprint != second_email.fingerprint - PY - - python3 - <<'PY' - from pathlib import Path - - parser_test = Path("backend/tests/test_email_parser_provenance.py") - text = parser_test.read_text(encoding="utf-8") - old = ' return (headers.strip() + "\\n\\nBody text.").encode("utf-8")\n' - new = ' return (headers.strip("\\r\\n") + "\\n\\nBody text.").encode("utf-8")\n' - if old not in text: - raise SystemExit("email parser fixture anchor missing") - parser_test.write_text(text.replace(old, new, 1), encoding="utf-8") - - imap_test = Path("backend/tests/test_imap_worker.py") - text = imap_test.read_text(encoding="utf-8") - anchor = ' assert kwargs["owner_addresses"] == ["imap-user@example.com"]\n' - replacement = anchor + ' assert kwargs["source_content"] == raw_message\n' - if replacement not in text: - if anchor not in text: - raise SystemExit("IMAP raw-source assertion anchor missing") - text = text.replace(anchor, replacement, 1) - imap_test.write_text(text, encoding="utf-8") - - pop3_test = Path("backend/tests/test_pop3_worker.py") - text = pop3_test.read_text(encoding="utf-8") - old_signature = ''' async def fake_process_fetched_email( - db_session, email_data, user_id, organization_id, owner_addresses=None - ): -''' - new_signature = ''' async def fake_process_fetched_email( - db_session, - email_data, - user_id, - organization_id, - owner_addresses=None, - source_content=None, - ): -''' - if new_signature not in text: - if old_signature not in text: - raise SystemExit("POP3 fake processor signature anchor missing") - text = text.replace(old_signature, new_signature, 1) - old_item = ' "owner_addresses": owner_addresses,\n' - new_item = old_item + ' "source_content": source_content,\n' - if new_item not in text: - if old_item not in text: - raise SystemExit("POP3 imported record anchor missing") - text = text.replace(old_item, new_item, 1) - old_assert = ' assert imported[0]["owner_addresses"] == ["pop3-user@example.com"]\n' - new_assert = old_assert + ' assert imported[0]["source_content"] == raw_message\n' - if new_assert not in text: - if old_assert not in text: - raise SystemExit("POP3 raw-source assertion anchor missing") - text = text.replace(old_assert, new_assert, 1) - pop3_test.write_text(text, encoding="utf-8") - PY - set +e ( cd backend @@ -299,287 +60,34 @@ jobs: red_status=$? set -e if [ "$red_status" -eq 0 ]; then - echo "::error::Source-bound dedupe regressions unexpectedly passed before remediation." + echo "::error::Source-bound regressions unexpectedly passed before implementation." exit 1 fi - - name: Bind untrusted-Date identity to immutable source content - shell: bash - run: | - set -euo pipefail - python3 - <<'PY' - from pathlib import Path - - - def replace_once(text: str, old: str, new: str, label: str) -> str: - if new in text: - return text - count = text.count(old) - if count != 1: - raise SystemExit(f"{label}: expected one anchor, found {count}") - return text.replace(old, new, 1) - - - dedupe_path = Path("backend/services/email_dedupe_service.py") - text = dedupe_path.read_text(encoding="utf-8") - text = replace_once( - text, - "import datetime\nfrom collections.abc import Iterable\n", - "import datetime\nimport hashlib\nimport json\nfrom collections.abc import Iterable, Mapping\n", - "dedupe imports", - ) - helper = '''_CANONICAL_SOURCE_FIELDS = ( - "message_id", - "sender", - "recipients", - "subject", - "body", - "reply_to", - "in_reply_to", - "references", - "attachments", - ) - - - def canonical_email_source_content(email_data: Mapping[str, object]) -> bytes: - """Serialize stable parsed fields when raw transport bytes are unavailable. - - Collection-time ``date`` values and their provenance are deliberately - excluded. Transport-backed paths should provide exact RFC822 bytes. - """ - payload = { - field: email_data.get(field) for field in _CANONICAL_SOURCE_FIELDS - } - return json.dumps( - payload, - ensure_ascii=False, - sort_keys=True, - separators=(",", ":"), - default=str, - ).encode("utf-8", errors="surrogatepass") - - - def source_email_fingerprint( - source_content: bytes, - *, - source_kind: Literal["raw", "canonical"] = "raw", - ) -> str: - """Return a domain-separated SHA-256 identity for stable source bytes.""" - digest = hashlib.sha256() - digest.update(b"naruon-email-source-v1\\0") - digest.update(source_kind.encode("ascii")) - digest.update(b"\\0") - digest.update(source_content) - return digest.hexdigest() - - - '''.replace(" ", "") - if "def source_email_fingerprint" not in text: - text = replace_once( - text, - "def strong_email_fingerprint(\n", - helper + "def strong_email_fingerprint(\n", - "dedupe helper insertion", - ) - dedupe_path.write_text(text, encoding="utf-8") - - import_path = Path("backend/services/email_import_service.py") - text = import_path.read_text(encoding="utf-8") - text = replace_once( - text, - "from services.email_dedupe_service import strong_email_fingerprint\n", - "from services.email_dedupe_service import (\n" - " canonical_email_source_content,\n" - " source_email_fingerprint,\n" - " strong_email_fingerprint,\n" - ")\n", - "import-service dedupe import", - ) - start = text.index("def _email_fingerprint(") - end = text.index("\n\n\nasync def _find_existing_email", start) - replacement = '''def _email_fingerprint( - parsed: EmailData, - persisted_date: datetime.datetime, - source_content: bytes | None = None, - ) -> str: - """Return trusted-Date evidence or a source-bound fallback identity. - - ``persisted_date`` remains the storage timestamp and participates in - duplicate evidence only when it came from a valid sender ``Date``. - """ - strong_fingerprint = None - if parsed.get("date_provenance") == "parsed": - strong_fingerprint = strong_email_fingerprint( - sender=parsed.get("sender"), - subject=parsed.get("subject"), - date=persisted_date, - body=parsed.get("body"), - ) - if strong_fingerprint: - return strong_fingerprint - source_identity = ( - source_content - if source_content is not None - else canonical_email_source_content(parsed) - ) - return source_email_fingerprint( - source_identity, - source_kind="raw" if source_content is not None else "canonical", - ) - '''.replace(" ", "").rstrip() - text = text[:start] + replacement + text[end:] - text = replace_once( - text, - " fingerprint = _email_fingerprint(parsed, persisted_date)\n", - " fingerprint = _email_fingerprint(parsed, persisted_date, content)\n", - "import fingerprint call", - ) - text = text.replace(" generate_email_fingerprint,\n", "", 1) - import_path.write_text(text, encoding="utf-8") - - imap_path = Path("backend/services/imap_worker.py") - text = imap_path.read_text(encoding="utf-8") - text = replace_once( - text, - "from services.email_dedupe_service import strong_email_fingerprint\n", - "from services.email_dedupe_service import (\n" - " canonical_email_source_content,\n" - " source_email_fingerprint,\n" - " strong_email_fingerprint,\n" - ")\n", - "IMAP dedupe import", - ) - text = replace_once( - text, - "from services.threading_service import assign_thread_id, generate_email_fingerprint\n", - "from services.threading_service import assign_thread_id\n", - "IMAP threading import", - ) - old_signature = ''' owner_addresses: Iterable[str] | None = None, - is_read: bool = True, - ): - subject = email_data.get("subject", "") - '''.replace(" ", "") - new_signature = ''' owner_addresses: Iterable[str] | None = None, - is_read: bool = True, - source_content: bytes | None = None, - ) -> Email: - """Persist one fetched email with provenance-safe identity.""" - subject = email_data.get("subject", "") - '''.replace(" ", "") - text = replace_once(text, old_signature, new_signature, "IMAP signature") - fingerprint_start = text.index(" # Seed the strong") - fingerprint_end = text.index("\n\n # Check if duplicate", fingerprint_start) - fingerprint_block = ''' # Seed strong duplicate evidence only from a genuinely parsed Date. - strong_fingerprint = None - if email_data.get("date_provenance") == "parsed": - strong_fingerprint = strong_email_fingerprint( - sender=sender, - subject=subject, - date=persisted_date, - body=email_data.get("body", ""), - ) - source_identity = ( - source_content - if source_content is not None - else canonical_email_source_content(email_data) - ) - fingerprint = strong_fingerprint or source_email_fingerprint( - source_identity, - source_kind="raw" if source_content is not None else "canonical", - ) - '''.replace(" ", "").rstrip() - text = text[:fingerprint_start] + fingerprint_block + text[fingerprint_end:] - text = replace_once( - text, - " is_read=is_read,\n )\n", - " is_read=is_read,\n" - " source_content=raw_message,\n" - " )\n", - "IMAP raw-source propagation", - ) - imap_path.write_text(text, encoding="utf-8") - - pop3_path = Path("backend/services/pop3_worker.py") - text = pop3_path.read_text(encoding="utf-8") - text = replace_once( - text, - " owner_addresses=owner_addresses,\n )\n", - " owner_addresses=owner_addresses,\n" - " source_content=raw_message,\n" - " )\n", - "POP3 raw-source propagation", - ) - pop3_path.write_text(text, encoding="utf-8") - - changelog_path = Path("CHANGELOG.md") - text = changelog_path.read_text(encoding="utf-8") - text = text.replace( - "strong key를 만들지 않고 weak fallback fingerprint(수집 시각이 매번 달라 거짓 중복을 만들 수 없음)로 내려갑니다", - "strong key를 만들지 않고 원본 RFC822 바이트의 domain-separated SHA-256(원문이 없는 직접 호출자는 Date를 제외한 canonical source projection)로 내려갑니다", - ) - text = text.replace( - "import·IMAP 두 생성 경로 모두 파서 provenance를 영속화합니다.", - "import·IMAP·POP3 생성 경로 모두 파서 provenance를 영속화하고 원본 transport bytes를 fallback identity에 전달합니다.", - ) - changelog_path.write_text(text, encoding="utf-8") - PY + - name: Apply source-bound fallback identity + run: python scripts/ci/pr1195_source_identity_repair.py - - name: Verify focused and realistic email pipelines + - name: Verify realistic email pipelines and full backend suite shell: bash run: | set -euo pipefail - python -m ruff format \ - backend/services/email_dedupe_service.py \ - backend/services/email_import_service.py \ - backend/services/imap_worker.py \ - backend/services/pop3_worker.py \ - backend/tests/test_source_bound_email_dedupe.py \ - backend/tests/test_email_import_service.py \ - backend/tests/test_email_parser_provenance.py \ - backend/tests/test_imap_worker.py \ - backend/tests/test_pop3_worker.py - python -m ruff check \ - backend/services/email_dedupe_service.py \ - backend/services/email_import_service.py \ - backend/services/imap_worker.py \ - backend/services/pop3_worker.py \ - backend/tests/test_source_bound_email_dedupe.py \ - backend/tests/test_email_import_service.py \ - backend/tests/test_email_parser_provenance.py \ - backend/tests/test_imap_worker.py \ - backend/tests/test_pop3_worker.py + python -m ruff format backend/services/email_dedupe_service.py backend/services/email_import_service.py backend/services/imap_worker.py backend/services/pop3_worker.py backend/tests/test_source_bound_email_dedupe.py backend/tests/test_email_parser_provenance.py backend/tests/test_imap_worker.py backend/tests/test_pop3_worker.py + python -m ruff check backend ( cd backend - python -m pytest -q \ - tests/test_source_bound_email_dedupe.py \ - tests/test_email_parser_provenance.py \ - tests/test_email_dedupe_service.py \ - tests/test_email_import_service.py \ - tests/test_imap_worker.py \ - tests/test_pop3_worker.py + python -m pytest -q tests/test_source_bound_email_dedupe.py tests/test_email_parser_provenance.py tests/test_email_dedupe_service.py tests/test_email_import_service.py tests/test_imap_worker.py tests/test_pop3_worker.py + python -m pytest -q ) git diff --check - - name: Commit verified remediation and remove one-shot workflow + - name: Publish verified current-head remediation shell: bash run: | set -euo pipefail - rm .github/workflows/pr1195-source-identity-repair.yml + rm .github/workflows/pr1195-source-identity-repair.yml scripts/ci/pr1195_source_identity_repair.py git diff --check git config user.name github-actions[bot] git config user.email 41898282+github-actions[bot]@users.noreply.github.com - git add \ - CHANGELOG.md \ - backend/services/email_dedupe_service.py \ - backend/services/email_import_service.py \ - backend/services/imap_worker.py \ - backend/services/pop3_worker.py \ - backend/tests/test_source_bound_email_dedupe.py \ - backend/tests/test_email_import_service.py \ - backend/tests/test_email_parser_provenance.py \ - backend/tests/test_imap_worker.py \ - backend/tests/test_pop3_worker.py \ - .github/workflows/pr1195-source-identity-repair.yml + git add CHANGELOG.md backend/services/email_dedupe_service.py backend/services/email_import_service.py backend/services/imap_worker.py backend/services/pop3_worker.py backend/tests/test_source_bound_email_dedupe.py backend/tests/test_email_parser_provenance.py backend/tests/test_imap_worker.py backend/tests/test_pop3_worker.py .github/workflows/pr1195-source-identity-repair.yml scripts/ci/pr1195_source_identity_repair.py git commit -m "fix(email): bind fallback dedupe to immutable source" git push origin HEAD:claude/contextualwisdomlab-audit-governance-qyxe67 From 898201e7294561a0bb4d8babcfa478e34bb5be19 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 21:19:15 +0900 Subject: [PATCH 32/86] docs(email): record source-bound identity contract --- .../email-source-identity-provenance.md | 65 +++++++++++++++++++ 1 file changed, 65 insertions(+) create mode 100644 docs/doctoring/email-source-identity-provenance.md diff --git a/docs/doctoring/email-source-identity-provenance.md b/docs/doctoring/email-source-identity-provenance.md new file mode 100644 index 000000000..8ea47bef6 --- /dev/null +++ b/docs/doctoring/email-source-identity-provenance.md @@ -0,0 +1,65 @@ +# Email source identity provenance + +## Decision + +Naruon treats the sender-authored message and the observation process as +different evidence channels. A collection timestamp can be stored as an +operational timestamp, but it cannot become strong duplicate evidence unless a +valid sender `Date` field was genuinely parsed. Messages without that evidence +use a domain-separated SHA-256 identity over immutable RFC 822 source octets. A +deterministic projection of stable parsed fields is used only when the caller +cannot provide transport bytes. + +This boundary prevents two distinct messages collected at the same instant from +being linked merely because their observation metadata is similar. It also keeps +a repeated import of the same source stable across collection times. + +## POP3 reconstruction contract + +POP3 `RETR` is a multiline response. RFC 1939 requires every transmitted line to +end in CRLF and terminates the response with a separate dot line. Python's +`poplib.POP3.retr()` returns the message as a list of lines without those line +terminators. Naruon therefore reconstructs source bytes by joining returned +message lines with CRLF and adding the final message-line CRLF. The POP3 +terminator line is not part of the source message. + +The reconstructed bytes are a transport-normalized POP3 representation. They +are not claimed to reproduce server storage outside the protocol-visible +message. IMAP and direct-file ingestion retain their own exact received byte +streams. Duplicate classification remains deterministic because the source kind +is domain separated and because collection time is excluded from fallback +identity. + +## Verification contract + +- A valid sender `Date` may seed the reviewed strong fingerprint. +- Missing and invalid sender dates cannot promote collection time to strong + evidence. +- Two different raw messages collected at the same instant remain distinct. +- The same raw message collected at different instants has the same fallback + identity. +- Canonical fallback identity excludes effective collection timestamps and + provenance flags. +- IMAP and POP3 pass source bytes through the persistence boundary. +- POP3 source reconstruction restores CRLF after every `RETR` message line. +- Existing rows remain conservatively classified when provenance is unknown. + +## Claim boundary + +Hash equality is evidence that the selected source representation is identical; +it is not proof that two independently authored real-world communications are +the same event. Automatic linkage, clerical review, and distinct-message +outcomes remain separate decisions. No automatic deletion or irreversible +provider action is introduced. + +## References + +Fellegi, I. P., & Sunter, A. B. (1969). A theory for record linkage. *Journal +of the American Statistical Association, 64*(328), 1183–1210. +https://doi.org/10.1080/01621459.1969.10501049 + +Myers, J., & Rose, M. (1996). *Post Office Protocol—Version 3* (RFC 1939; +STD 53). Internet Engineering Task Force. https://doi.org/10.17487/RFC1939 + +Resnick, P. (2008). *Internet message format* (RFC 5322). Internet Engineering +Task Force. https://doi.org/10.17487/RFC5322 From 24127709e4875cf7a677c327a9c87a325cdc5efe Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 21:20:26 +0900 Subject: [PATCH 33/86] ci(email): repair POP3 source-byte reconstruction --- .../pr1195-source-identity-repair.yml | 71 ++++++++++++++----- 1 file changed, 53 insertions(+), 18 deletions(-) diff --git a/.github/workflows/pr1195-source-identity-repair.yml b/.github/workflows/pr1195-source-identity-repair.yml index 5bb56df6e..1d988377c 100644 --- a/.github/workflows/pr1195-source-identity-repair.yml +++ b/.github/workflows/pr1195-source-identity-repair.yml @@ -9,7 +9,7 @@ on: concurrency: group: pr1195-source-identity-repair - cancel-in-progress: false + cancel-in-progress: true permissions: contents: read @@ -17,7 +17,7 @@ permissions: jobs: repair: if: ${{ github.repository == 'ContextualWisdomLab/naruon' && github.ref == 'refs/heads/claude/contextualwisdomlab-audit-governance-qyxe67' }} - runs-on: ubuntu-latest + runs-on: ubuntu-24.04 timeout-minutes: 60 permissions: contents: write @@ -25,19 +25,28 @@ jobs: PYTHONWARNINGS: error DISABLE_BACKGROUND_WORKERS: "1" steps: - - name: Harden runner - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 + - name: Harden runner with blocking egress + uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 with: - egress-policy: audit + egress-policy: block + allowed-endpoints: > + github.com:443 + api.github.com:443 + codeload.github.com:443 + objects.githubusercontent.com:443 + release-assets.githubusercontent.com:443 + pypi.org:443 + files.pythonhosted.org:443 - - name: Checkout exact pull request branch - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 + - name: Checkout exact repair trigger + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 with: - ref: claude/contextualwisdomlab-audit-governance-qyxe67 - fetch-depth: 0 + ref: ${{ github.sha }} + fetch-depth: 2 + persist-credentials: false - name: Set up Python - uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 with: python-version: "3.14" cache: pip @@ -49,28 +58,50 @@ jobs: -r backend/requirements-hashes.txt - name: Prove source-bound regressions fail before implementation - shell: bash + shell: bash --noprofile --norc -e -o pipefail {0} run: | - set -euo pipefail set +e ( cd backend python -m pytest -q tests/test_source_bound_email_dedupe.py - ) + ) >"${RUNNER_TEMP}/pr1195-red.log" 2>&1 red_status=$? set -e + cat "${RUNNER_TEMP}/pr1195-red.log" if [ "$red_status" -eq 0 ]; then echo "::error::Source-bound regressions unexpectedly passed before implementation." exit 1 fi + grep -F "canonical_email_source_content" "${RUNNER_TEMP}/pr1195-red.log" - name: Apply source-bound fallback identity run: python scripts/ci/pr1195_source_identity_repair.py + - name: Restore RFC 1939 CRLF in POP3 RETR source bytes + run: | + python - <<'PY' + from pathlib import Path + + path = Path("backend/services/pop3_worker.py") + text = path.read_text(encoding="utf-8") + old_call = ' messages.append(b"\\r\\n".join(self._bytes_line(line) for line in lines))\n' + new_call = ' messages.append(self._message_bytes(lines))\n' + helper = ''' def _message_bytes(self, lines: list[bytes | str]) -> bytes:\n """Reconstruct one POP3 RETR message with protocol CRLF terminators.\n\n ``poplib`` removes line terminators from the multiline response while\n RFC 1939 defines each transferred message line as CRLF-terminated.\n """\n return b"\\r\\n".join(self._bytes_line(line) for line in lines) + b"\\r\\n"\n\n''' + anchor = ' def _message_number_from_listing(self, listing: bytes | str) -> int | None:\n' + if old_call in text: + text = text.replace(old_call, new_call, 1) + elif new_call not in text: + raise SystemExit("POP3 RETR reconstruction call anchor not found") + if helper not in text: + if anchor not in text: + raise SystemExit("POP3 message-byte helper anchor not found") + text = text.replace(anchor, helper + anchor, 1) + path.write_text(text, encoding="utf-8") + PY + - name: Verify realistic email pipelines and full backend suite - shell: bash + shell: bash --noprofile --norc -e -o pipefail {0} run: | - set -euo pipefail python -m ruff format backend/services/email_dedupe_service.py backend/services/email_import_service.py backend/services/imap_worker.py backend/services/pop3_worker.py backend/tests/test_source_bound_email_dedupe.py backend/tests/test_email_parser_provenance.py backend/tests/test_imap_worker.py backend/tests/test_pop3_worker.py python -m ruff check backend ( @@ -81,13 +112,17 @@ jobs: git diff --check - name: Publish verified current-head remediation - shell: bash + shell: bash --noprofile --norc -e -o pipefail {0} + env: + PUSH_TOKEN: ${{ github.token }} run: | - set -euo pipefail rm .github/workflows/pr1195-source-identity-repair.yml scripts/ci/pr1195_source_identity_repair.py git diff --check git config user.name github-actions[bot] git config user.email 41898282+github-actions[bot]@users.noreply.github.com git add CHANGELOG.md backend/services/email_dedupe_service.py backend/services/email_import_service.py backend/services/imap_worker.py backend/services/pop3_worker.py backend/tests/test_source_bound_email_dedupe.py backend/tests/test_email_parser_provenance.py backend/tests/test_imap_worker.py backend/tests/test_pop3_worker.py .github/workflows/pr1195-source-identity-repair.yml scripts/ci/pr1195_source_identity_repair.py git commit -m "fix(email): bind fallback dedupe to immutable source" - git push origin HEAD:claude/contextualwisdomlab-audit-governance-qyxe67 + auth_header="$(printf 'x-access-token:%s' "$PUSH_TOKEN" | base64 | tr -d '\n')" + echo "::add-mask::$auth_header" + git -c http.extraheader="AUTHORIZATION: basic ${auth_header}" \ + push origin "HEAD:claude/contextualwisdomlab-audit-governance-qyxe67" From 4c6b21a40a212b283095e3efb71e5885c875cf9f Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Wed, 5 Aug 2026 12:21:54 +0000 Subject: [PATCH 34/86] fix(email): bind fallback dedupe to immutable source --- .../pr1195-source-identity-repair.yml | 128 ------------- CHANGELOG.md | 3 + backend/services/email_dedupe_service.py | 47 ++++- backend/services/email_import_service.py | 58 +++--- backend/services/imap_worker.py | 45 +++-- backend/services/pop3_worker.py | 23 ++- backend/tests/test_email_parser_provenance.py | 2 +- backend/tests/test_imap_worker.py | 3 +- backend/tests/test_pop3_worker.py | 9 +- .../tests/test_source_bound_email_dedupe.py | 28 +-- scripts/ci/pr1195_source_identity_repair.py | 172 ------------------ 11 files changed, 147 insertions(+), 371 deletions(-) delete mode 100644 .github/workflows/pr1195-source-identity-repair.yml delete mode 100644 scripts/ci/pr1195_source_identity_repair.py diff --git a/.github/workflows/pr1195-source-identity-repair.yml b/.github/workflows/pr1195-source-identity-repair.yml deleted file mode 100644 index 1d988377c..000000000 --- a/.github/workflows/pr1195-source-identity-repair.yml +++ /dev/null @@ -1,128 +0,0 @@ -name: PR 1195 source identity repair - -on: - push: - branches: - - claude/contextualwisdomlab-audit-governance-qyxe67 - paths: - - .github/workflows/pr1195-source-identity-repair.yml - -concurrency: - group: pr1195-source-identity-repair - cancel-in-progress: true - -permissions: - contents: read - -jobs: - repair: - if: ${{ github.repository == 'ContextualWisdomLab/naruon' && github.ref == 'refs/heads/claude/contextualwisdomlab-audit-governance-qyxe67' }} - runs-on: ubuntu-24.04 - timeout-minutes: 60 - permissions: - contents: write - env: - PYTHONWARNINGS: error - DISABLE_BACKGROUND_WORKERS: "1" - steps: - - name: Harden runner with blocking egress - uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 - with: - egress-policy: block - allowed-endpoints: > - github.com:443 - api.github.com:443 - codeload.github.com:443 - objects.githubusercontent.com:443 - release-assets.githubusercontent.com:443 - pypi.org:443 - files.pythonhosted.org:443 - - - name: Checkout exact repair trigger - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 - with: - ref: ${{ github.sha }} - fetch-depth: 2 - persist-credentials: false - - - name: Set up Python - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 - with: - python-version: "3.14" - cache: pip - cache-dependency-path: backend/requirements-hashes.txt - - - name: Install hash-locked backend dependencies - run: >- - python -m pip install --disable-pip-version-check --require-hashes - -r backend/requirements-hashes.txt - - - name: Prove source-bound regressions fail before implementation - shell: bash --noprofile --norc -e -o pipefail {0} - run: | - set +e - ( - cd backend - python -m pytest -q tests/test_source_bound_email_dedupe.py - ) >"${RUNNER_TEMP}/pr1195-red.log" 2>&1 - red_status=$? - set -e - cat "${RUNNER_TEMP}/pr1195-red.log" - if [ "$red_status" -eq 0 ]; then - echo "::error::Source-bound regressions unexpectedly passed before implementation." - exit 1 - fi - grep -F "canonical_email_source_content" "${RUNNER_TEMP}/pr1195-red.log" - - - name: Apply source-bound fallback identity - run: python scripts/ci/pr1195_source_identity_repair.py - - - name: Restore RFC 1939 CRLF in POP3 RETR source bytes - run: | - python - <<'PY' - from pathlib import Path - - path = Path("backend/services/pop3_worker.py") - text = path.read_text(encoding="utf-8") - old_call = ' messages.append(b"\\r\\n".join(self._bytes_line(line) for line in lines))\n' - new_call = ' messages.append(self._message_bytes(lines))\n' - helper = ''' def _message_bytes(self, lines: list[bytes | str]) -> bytes:\n """Reconstruct one POP3 RETR message with protocol CRLF terminators.\n\n ``poplib`` removes line terminators from the multiline response while\n RFC 1939 defines each transferred message line as CRLF-terminated.\n """\n return b"\\r\\n".join(self._bytes_line(line) for line in lines) + b"\\r\\n"\n\n''' - anchor = ' def _message_number_from_listing(self, listing: bytes | str) -> int | None:\n' - if old_call in text: - text = text.replace(old_call, new_call, 1) - elif new_call not in text: - raise SystemExit("POP3 RETR reconstruction call anchor not found") - if helper not in text: - if anchor not in text: - raise SystemExit("POP3 message-byte helper anchor not found") - text = text.replace(anchor, helper + anchor, 1) - path.write_text(text, encoding="utf-8") - PY - - - name: Verify realistic email pipelines and full backend suite - shell: bash --noprofile --norc -e -o pipefail {0} - run: | - python -m ruff format backend/services/email_dedupe_service.py backend/services/email_import_service.py backend/services/imap_worker.py backend/services/pop3_worker.py backend/tests/test_source_bound_email_dedupe.py backend/tests/test_email_parser_provenance.py backend/tests/test_imap_worker.py backend/tests/test_pop3_worker.py - python -m ruff check backend - ( - cd backend - python -m pytest -q tests/test_source_bound_email_dedupe.py tests/test_email_parser_provenance.py tests/test_email_dedupe_service.py tests/test_email_import_service.py tests/test_imap_worker.py tests/test_pop3_worker.py - python -m pytest -q - ) - git diff --check - - - name: Publish verified current-head remediation - shell: bash --noprofile --norc -e -o pipefail {0} - env: - PUSH_TOKEN: ${{ github.token }} - run: | - rm .github/workflows/pr1195-source-identity-repair.yml scripts/ci/pr1195_source_identity_repair.py - git diff --check - git config user.name github-actions[bot] - git config user.email 41898282+github-actions[bot]@users.noreply.github.com - git add CHANGELOG.md backend/services/email_dedupe_service.py backend/services/email_import_service.py backend/services/imap_worker.py backend/services/pop3_worker.py backend/tests/test_source_bound_email_dedupe.py backend/tests/test_email_parser_provenance.py backend/tests/test_imap_worker.py backend/tests/test_pop3_worker.py .github/workflows/pr1195-source-identity-repair.yml scripts/ci/pr1195_source_identity_repair.py - git commit -m "fix(email): bind fallback dedupe to immutable source" - auth_header="$(printf 'x-access-token:%s' "$PUSH_TOKEN" | base64 | tr -d '\n')" - echo "::add-mask::$auth_header" - git -c http.extraheader="AUTHORIZATION: basic ${auth_header}" \ - push origin "HEAD:claude/contextualwisdomlab-audit-governance-qyxe67" diff --git a/CHANGELOG.md b/CHANGELOG.md index 1d6a7a7f5..615f5b350 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,4 +1,7 @@ ## [Unreleased] + +### Fixed +- Bound missing/invalid-Date email deduplication fallbacks to immutable RFC822 source bytes (or a deterministic Date-free canonical projection), including import, IMAP, and POP3 paths, so collection timestamps cannot create false duplicate identities. ### 이메일 메타데이터 provenance 기반 (dedupe 근거 분리 · naruon#1086 파서 계층) - `backend/services/email_parser.py`가 RFC822 `Date`·`Message-ID` 근거를 명시적으로 노출합니다. `date_provenance`(`parsed`/`missing`/`invalid`)와 원본 헤더 날짜(`header_date`, 부재·파싱 실패 시 `None`)를 저장용 `date`(파싱값 또는 수집 시각 fallback)와 분리하고, `message_id_provenance`(`embedded`/`missing`)를 추가했습니다. 합성 수집 시각이 원본 발신 메타데이터로 오인되지 않으므로 fingerprint dedupe가 진짜 발신 근거에만 의존할 수 있습니다. `date`의 기존 의미(파싱값-또는-fallback)는 그대로 유지되어 하위 호환입니다. diff --git a/backend/services/email_dedupe_service.py b/backend/services/email_dedupe_service.py index 7a8728198..14a6a5c18 100644 --- a/backend/services/email_dedupe_service.py +++ b/backend/services/email_dedupe_service.py @@ -6,7 +6,9 @@ """ import datetime -from collections.abc import Iterable +import hashlib +import json +from collections.abc import Iterable, Mapping from dataclasses import dataclass from typing import Literal @@ -50,6 +52,49 @@ def _date_to_fingerprint_value(value: datetime.datetime | None) -> str: return value.isoformat() +_CANONICAL_SOURCE_FIELDS = ( + "message_id", + "sender", + "recipients", + "subject", + "body", + "reply_to", + "in_reply_to", + "references", + "attachments", +) + + +def canonical_email_source_content(email_data: Mapping[str, object]) -> bytes: + """Serialize stable parsed fields when raw transport bytes are unavailable. + + Collection-time ``date`` values and their provenance are deliberately + excluded. Transport-backed paths should provide exact RFC822 bytes. + """ + payload = {field: email_data.get(field) for field in _CANONICAL_SOURCE_FIELDS} + return json.dumps( + payload, + ensure_ascii=False, + sort_keys=True, + separators=(",", ":"), + default=str, + ).encode("utf-8", errors="surrogatepass") + + +def source_email_fingerprint( + source_content: bytes, + *, + source_kind: Literal["raw", "canonical"] = "raw", +) -> str: + """Return a domain-separated SHA-256 identity for stable source bytes.""" + digest = hashlib.sha256() + digest.update(b"naruon-email-source-v1\0") + digest.update(source_kind.encode("ascii")) + digest.update(b"\0") + digest.update(source_content) + return digest.hexdigest() + + def strong_email_fingerprint( *, sender: str | None, diff --git a/backend/services/email_import_service.py b/backend/services/email_import_service.py index 2e02ff430..f5caf3e90 100644 --- a/backend/services/email_import_service.py +++ b/backend/services/email_import_service.py @@ -28,7 +28,11 @@ from services.archive import extract_backup_async from services.batch_embedding_service import try_batch_import_embeddings from services.content_graph import ParseResult, parse_content -from services.email_dedupe_service import strong_email_fingerprint +from services.email_dedupe_service import ( + canonical_email_source_content, + source_email_fingerprint, + strong_email_fingerprint, +) from services.email_parser import EmailData, parse_eml_bytes from services.embedding import ( STORAGE_EMBEDDING_DIMENSION, @@ -46,7 +50,6 @@ ) from services.threading_service import ( assign_thread_id, - generate_email_fingerprint, normalize_message_id, ) @@ -186,14 +189,16 @@ def _message_id_for(parsed: EmailData, content: bytes) -> str: ) -def _email_fingerprint(parsed: EmailData, persisted_date: datetime.datetime) -> str: - # A strong (auto-dedupe-eligible) fingerprint may only be seeded from a - # genuinely-parsed sender Date. When the Date header was missing or invalid - # (date_provenance != "parsed"), ``persisted_date`` is a synthetic - # collection-time fallback, not original metadata, so it must not produce a - # strong duplicate key — the email falls through to the weak fallback - # fingerprint (which, carrying the distinct collection time, cannot - # manufacture a false duplicate) (naruon#1086). +def _email_fingerprint( + parsed: EmailData, + persisted_date: datetime.datetime, + source_content: bytes | None = None, +) -> str: + """Return trusted-Date evidence or a source-bound fallback identity. + + ``persisted_date`` remains the storage timestamp and participates in + duplicate evidence only when it came from a valid sender ``Date``. + """ strong_fingerprint = None if parsed.get("date_provenance") == "parsed": strong_fingerprint = strong_email_fingerprint( @@ -204,11 +209,14 @@ def _email_fingerprint(parsed: EmailData, persisted_date: datetime.datetime) -> ) if strong_fingerprint: return strong_fingerprint - return generate_email_fingerprint( - parsed.get("subject"), - persisted_date.isoformat(), - parsed.get("sender"), - parsed.get("recipients"), + source_identity = ( + source_content + if source_content is not None + else canonical_email_source_content(parsed) + ) + return source_email_fingerprint( + source_identity, + source_kind="raw" if source_content is not None else "canonical", ) @@ -404,7 +412,11 @@ def _fallback_attachment_parser_key( return "calendar" if parse_content_type == "text/html": return "html" - if parse_content_type in {"text/markdown", "text/x-markdown", "application/markdown"}: + if parse_content_type in { + "text/markdown", + "text/x-markdown", + "application/markdown", + }: return "markdown" if parse_content_type == "text/plain": return "plain_text" @@ -596,9 +608,9 @@ def add_edge( item.segment_path, ), ): - segments_by_source[ - (segment.source_kind, segment.source_record_uid) - ].append(segment) + segments_by_source[(segment.source_kind, segment.source_record_uid)].append( + segment + ) add_edge( edge_kind="node_has_segment", edge_path=f"{segment.content_node.node_path}/has/{segment.segment_path}", @@ -613,8 +625,7 @@ def add_edge( add_edge( edge_kind="segment_next", edge_path=( - f"{source_segment.segment_path}/next/" - f"{target_segment.segment_path}" + f"{source_segment.segment_path}/next/{target_segment.segment_path}" ), source_kind=source_segment.source_kind, source_record_uid=source_segment.source_record_uid, @@ -638,8 +649,7 @@ def add_edge( add_edge( edge_kind="heading_contains_segment", edge_path=( - f"{heading_segment.segment_path}/contains/" - f"{segment.segment_path}" + f"{heading_segment.segment_path}/contains/{segment.segment_path}" ), source_kind=segment.source_kind, source_record_uid=segment.source_record_uid, @@ -825,7 +835,7 @@ async def _import_single_eml( message_id = _message_id_for(parsed, content) parsed["message_id"] = message_id persisted_date = _utc_datetime(parsed.get("date")) - fingerprint = _email_fingerprint(parsed, persisted_date) + fingerprint = _email_fingerprint(parsed, persisted_date, content) existing_email = await _find_existing_email( session, diff --git a/backend/services/imap_worker.py b/backend/services/imap_worker.py index d74bd3d1a..07ceb9f7f 100644 --- a/backend/services/imap_worker.py +++ b/backend/services/imap_worker.py @@ -10,14 +10,18 @@ from db.models import Email, TenantConfig from db.session import AsyncSessionLocal from services.email_client import validate_imap_destination -from services.email_dedupe_service import strong_email_fingerprint +from services.email_dedupe_service import ( + canonical_email_source_content, + source_email_fingerprint, + strong_email_fingerprint, +) from services.email_parser import EmailData, parse_eml_bytes from services.exceptions import EmailParseError from services.knowledge_extractor import ( extract_knowledge_from_self_sent, is_self_sent_email, ) -from services.threading_service import assign_thread_id, generate_email_fingerprint +from services.threading_service import assign_thread_id async def process_fetched_email( @@ -27,13 +31,11 @@ async def process_fetched_email( organization_id: str | None, owner_addresses: Iterable[str] | None = None, is_read: bool = True, -): + source_content: bytes | None = None, +) -> Email: + """Persist one fetched email with provenance-safe identity.""" subject = email_data.get("subject", "") date_obj = email_data.get("date") - if hasattr(date_obj, "isoformat"): - date_str = date_obj.isoformat() - else: - date_str = str(date_obj) if date_obj else "" if isinstance(date_obj, datetime.datetime): persisted_date = ( date_obj.astimezone(datetime.timezone.utc) @@ -50,9 +52,7 @@ async def process_fetched_email( else str(recipients_list or "") ) - # Seed the strong (auto-dedupe) fingerprint only from a genuinely-parsed - # Date; a synthetic collection-time fallback must not manufacture a strong - # duplicate key, so it falls through to the weak fallback (naruon#1086). + # Seed strong duplicate evidence only from a genuinely parsed Date. strong_fingerprint = None if email_data.get("date_provenance") == "parsed": strong_fingerprint = strong_email_fingerprint( @@ -61,8 +61,14 @@ async def process_fetched_email( date=persisted_date, body=email_data.get("body", ""), ) - fingerprint = strong_fingerprint or generate_email_fingerprint( - subject, date_str, sender, recipients + source_identity = ( + source_content + if source_content is not None + else canonical_email_source_content(email_data) + ) + fingerprint = strong_fingerprint or source_email_fingerprint( + source_identity, + source_kind="raw" if source_content is not None else "canonical", ) # Check if duplicate @@ -107,6 +113,7 @@ async def process_fetched_email( await extract_knowledge_from_self_sent(session, new_email, owner_addresses) return new_email + logger = logging.getLogger(__name__) MAX_IMAP_FETCH_MESSAGES = 10 @@ -121,7 +128,11 @@ def flags_indicate_seen(fetch_data) -> bool: for item in fetch_data or []: parts = item if isinstance(item, (tuple, list)) else (item,) for part in parts: - raw = part if isinstance(part, bytes) else str(part).encode("utf-8", "replace") + raw = ( + part + if isinstance(part, bytes) + else str(part).encode("utf-8", "replace") + ) upper = raw.upper() if b"FLAGS" in upper and b"\\SEEN" in upper: return True @@ -226,7 +237,7 @@ async def _sync_tenant(self, config: TenantConfig | ImapSyncConfig): config.user_id, ) return 0 - + logger.info( "Connecting to IMAP server %s:%s for user %s", imap_server, @@ -261,6 +272,7 @@ async def _fetch_messages( if imap_server is None or imap_port is None: imap_server, imap_port = self._validated_destination(config) import ssl + ssl_context = ssl.create_default_context() imap_client = aioimaplib.IMAP4_SSL( imap_server, imap_port, ssl_context=ssl_context @@ -343,6 +355,7 @@ async def _import_messages( config.organization_id, owner_addresses=owner_addresses, is_read=is_read, + source_content=raw_message, ) imported_count += 1 await session.commit() @@ -397,6 +410,4 @@ def _looks_like_rfc822_message(self, value: bytes) -> bool: header_block = value.split(b"\r\n\r\n", maxsplit=1)[0] if header_block == value: header_block = value.split(b"\n\n", maxsplit=1)[0] - return b":" in header_block and ( - b"\r\n\r\n" in value or b"\n\n" in value - ) + return b":" in header_block and (b"\r\n\r\n" in value or b"\n\n" in value) diff --git a/backend/services/pop3_worker.py b/backend/services/pop3_worker.py index 601180027..be76acdef 100644 --- a/backend/services/pop3_worker.py +++ b/backend/services/pop3_worker.py @@ -57,16 +57,18 @@ async def _run_loop(self): async def _sync(self): async with AsyncSessionLocal() as session: - result = await session.execute(select(TenantConfig).where(TenantConfig.pop3_server.isnot(None))) + result = await session.execute( + select(TenantConfig).where(TenantConfig.pop3_server.isnot(None)) + ) configs = result.scalars().all() - + semaphore = asyncio.Semaphore(10) tasks = [] for config in configs: if not config.pop3_server or not config.pop3_port: continue tasks.append(self._sync_tenant(config, semaphore)) - + if tasks: await asyncio.gather(*tasks, return_exceptions=True) @@ -126,6 +128,7 @@ async def _import_messages( config.user_id, config.organization_id, owner_addresses=owner_addresses, + source_content=raw_message, ) imported_count += 1 await session.commit() @@ -175,11 +178,19 @@ def _do_pop3_sync( if message_number is None: continue _retr_response, lines, _retr_octets = pop3_client.retr(message_number) - messages.append(b"\r\n".join(self._bytes_line(line) for line in lines)) + messages.append(self._message_bytes(lines)) return messages finally: pop3_client.quit() + def _message_bytes(self, lines: list[bytes | str]) -> bytes: + """Reconstruct one POP3 RETR message with protocol CRLF terminators. + + ``poplib`` removes line terminators from the multiline response while + RFC 1939 defines each transferred message line as CRLF-terminated. + """ + return b"\r\n".join(self._bytes_line(line) for line in lines) + b"\r\n" + def _message_number_from_listing(self, listing: bytes | str) -> int | None: raw_listing = ( listing.decode("ascii", errors="ignore") @@ -195,7 +206,5 @@ def _message_number_from_listing(self, listing: bytes | str) -> int | None: def _bytes_line(self, line: bytes | str) -> bytes: return ( - line - if isinstance(line, bytes) - else line.encode("utf-8", errors="replace") + line if isinstance(line, bytes) else line.encode("utf-8", errors="replace") ) diff --git a/backend/tests/test_email_parser_provenance.py b/backend/tests/test_email_parser_provenance.py index 92c2f044e..64b197006 100644 --- a/backend/tests/test_email_parser_provenance.py +++ b/backend/tests/test_email_parser_provenance.py @@ -7,7 +7,7 @@ def _eml_with(headers: str) -> bytes: """Build minimal EML bytes with the given header block and a plain body.""" - return (headers.strip() + "\n\nBody text.").encode("utf-8") + return (headers.strip("\r\n") + "\n\nBody text.").encode("utf-8") def test_parse_eml_marks_valid_date_as_parsed_with_original_header_date() -> None: diff --git a/backend/tests/test_imap_worker.py b/backend/tests/test_imap_worker.py index d2798719a..529b569a0 100644 --- a/backend/tests/test_imap_worker.py +++ b/backend/tests/test_imap_worker.py @@ -126,6 +126,7 @@ async def test_imap_worker_imports_fetched_rfc822_messages(monkeypatch): assert kwargs["is_read"] is False assert args[3] == "org-imap" assert kwargs["owner_addresses"] == ["imap-user@example.com"] + assert kwargs["source_content"] == raw_message session.commit.assert_awaited_once() session.rollback.assert_not_awaited() @@ -173,7 +174,7 @@ def test_flags_indicate_seen_parses_seen_flag(): no_flags = ("OK", [(b"1 (RFC822 {%d}" % len(raw), raw)]) assert flags_indicate_seen(seen[1]) is True - assert flags_indicate_seen(unseen[1]) is False # other flags, but not \Seen + assert flags_indicate_seen(unseen[1]) is False # other flags, but not \Seen assert flags_indicate_seen(no_flags[1]) is False # no FLAGS section -> unread assert flags_indicate_seen([]) is False assert flags_indicate_seen(None) is False diff --git a/backend/tests/test_pop3_worker.py b/backend/tests/test_pop3_worker.py index d08e79f6f..9e2cec1cb 100644 --- a/backend/tests/test_pop3_worker.py +++ b/backend/tests/test_pop3_worker.py @@ -132,7 +132,12 @@ async def rollback(self): session = FakeSession() async def fake_process_fetched_email( - db_session, email_data, user_id, organization_id, owner_addresses=None + db_session, + email_data, + user_id, + organization_id, + owner_addresses=None, + source_content=None, ): imported.append( { @@ -141,6 +146,7 @@ async def fake_process_fetched_email( "user_id": user_id, "organization_id": organization_id, "owner_addresses": owner_addresses, + "source_content": source_content, } ) @@ -174,6 +180,7 @@ async def fake_process_fetched_email( assert imported[0]["user_id"] == "pop3-user" assert imported[0]["organization_id"] == "org-pop3" assert imported[0]["owner_addresses"] == ["pop3-user@example.com"] + assert imported[0]["source_content"] == raw_message assert imported[0]["email_data"]["message_id"] == "" assert imported[0]["email_data"]["subject"] == "POP3 import" assert session.committed is True diff --git a/backend/tests/test_source_bound_email_dedupe.py b/backend/tests/test_source_bound_email_dedupe.py index 434d18387..2cfb8d6bb 100644 --- a/backend/tests/test_source_bound_email_dedupe.py +++ b/backend/tests/test_source_bound_email_dedupe.py @@ -14,14 +14,14 @@ from services.imap_worker import process_fetched_email -def test_source_email_fingerprint_is_stable_content_bound_and_domain_separated() -> None: +def test_source_email_fingerprint_is_stable_content_bound_and_domain_separated() -> ( + None +): """Hash equal sources equally while separating content and source domains.""" first = source_email_fingerprint(b"same source") assert first == source_email_fingerprint(b"same source") assert first != source_email_fingerprint(b"different source") - assert first != source_email_fingerprint( - b"same source", source_kind="canonical" - ) + assert first != source_email_fingerprint(b"same source", source_kind="canonical") assert len(first) == 64 @@ -37,16 +37,12 @@ def test_canonical_source_content_excludes_collection_date() -> None: } first = { **base, - "date": datetime.datetime( - 2026, 8, 4, 6, 30, tzinfo=datetime.timezone.utc - ), + "date": datetime.datetime(2026, 8, 4, 6, 30, tzinfo=datetime.timezone.utc), "date_provenance": "missing", } second = { **base, - "date": datetime.datetime( - 2026, 8, 4, 7, 30, tzinfo=datetime.timezone.utc - ), + "date": datetime.datetime(2026, 8, 4, 7, 30, tzinfo=datetime.timezone.utc), "date_provenance": "invalid", } assert canonical_email_source_content(first) == canonical_email_source_content( @@ -59,9 +55,7 @@ def test_canonical_source_content_excludes_collection_date() -> None: def test_import_fingerprint_uses_trusted_date_or_raw_source() -> None: """Use strong Date evidence only when provenance is genuinely parsed.""" - persisted_date = datetime.datetime( - 2026, 8, 4, 6, 30, tzinfo=datetime.timezone.utc - ) + persisted_date = datetime.datetime(2026, 8, 4, 6, 30, tzinfo=datetime.timezone.utc) fields = { "message_id": "", "sender": "sender@example.com", @@ -112,9 +106,7 @@ def test_direct_fallback_is_collection_time_independent() -> None: "body": "Body", "date_provenance": "missing", } - first_time = datetime.datetime( - 2026, 8, 4, 6, 30, tzinfo=datetime.timezone.utc - ) + first_time = datetime.datetime(2026, 8, 4, 6, 30, tzinfo=datetime.timezone.utc) second_time = first_time + datetime.timedelta(hours=1) assert _email_fingerprint(parsed, first_time) == _email_fingerprint( parsed, second_time @@ -143,9 +135,7 @@ async def test_missing_date_messages_use_raw_source_not_collection_time( lambda _email, _owners: False, ) - collected_at = datetime.datetime( - 2026, 8, 4, 6, 30, tzinfo=datetime.timezone.utc - ) + collected_at = datetime.datetime(2026, 8, 4, 6, 30, tzinfo=datetime.timezone.utc) common = { "subject": "Same subject", "date": collected_at, diff --git a/scripts/ci/pr1195_source_identity_repair.py b/scripts/ci/pr1195_source_identity_repair.py deleted file mode 100644 index 1ecfe503d..000000000 --- a/scripts/ci/pr1195_source_identity_repair.py +++ /dev/null @@ -1,172 +0,0 @@ -from __future__ import annotations - -import pathlib -import re - -ROOT = pathlib.Path(".") - -def read(path: str) -> str: - return (ROOT / path).read_text(encoding="utf-8") - -def write(path: str, content: str) -> None: - (ROOT / path).write_text(content, encoding="utf-8") - -def replace_once(text: str, old: str, new: str, label: str) -> str: - if new in text: - return text - count = text.count(old) - if count != 1: - raise SystemExit(f"{label}: expected one anchor, found {count}") - return text.replace(old, new, 1) - -path = "backend/services/email_dedupe_service.py" -text = read(path) -text = replace_once( - text, - "import datetime\nfrom collections.abc import Iterable\n", - "import datetime\nimport hashlib\nimport json\nfrom collections.abc import Iterable, Mapping\n", - "email_dedupe_service imports", -) -helper = '_CANONICAL_SOURCE_FIELDS = (\n "message_id",\n "sender",\n "recipients",\n "subject",\n "body",\n "reply_to",\n "in_reply_to",\n "references",\n "attachments",\n)\n\n\ndef canonical_email_source_content(email_data: Mapping[str, object]) -> bytes:\n """Serialize stable parsed fields when raw transport bytes are unavailable.\n\n Collection-time ``date`` values and their provenance are deliberately\n excluded. Transport-backed paths should provide exact RFC822 bytes.\n """\n payload = {field: email_data.get(field) for field in _CANONICAL_SOURCE_FIELDS}\n return json.dumps(\n payload,\n ensure_ascii=False,\n sort_keys=True,\n separators=(",", ":"),\n default=str,\n ).encode("utf-8", errors="surrogatepass")\n\n\ndef source_email_fingerprint(\n source_content: bytes,\n *,\n source_kind: Literal["raw", "canonical"] = "raw",\n) -> str:\n """Return a domain-separated SHA-256 identity for stable source bytes."""\n digest = hashlib.sha256()\n digest.update(b"naruon-email-source-v1\\0")\n digest.update(source_kind.encode("ascii"))\n digest.update(b"\\0")\n digest.update(source_content)\n return digest.hexdigest()\n\n\n' -if "def source_email_fingerprint(" not in text: - text = replace_once( - text, - "def strong_email_fingerprint(\n", - helper + "def strong_email_fingerprint(\n", - "email_dedupe_service helper insertion", - ) -write(path, text) - -path = "backend/services/email_import_service.py" -text = read(path) -text = replace_once( - text, - "from services.email_dedupe_service import strong_email_fingerprint\n", - "from services.email_dedupe_service import (\n canonical_email_source_content,\n source_email_fingerprint,\n strong_email_fingerprint,\n)\n", - "email_import_service dedupe import", -) -pattern = re.compile( - r"def _email_fingerprint\(parsed: EmailData, persisted_date: datetime\.datetime\) -> str:\n" - r".*?\n\n\nasync def _find_existing_email", - re.DOTALL, -) -replacement = 'def _email_fingerprint(\n parsed: EmailData,\n persisted_date: datetime.datetime,\n source_content: bytes | None = None,\n) -> str:\n """Return trusted-Date evidence or a source-bound fallback identity.\n\n ``persisted_date`` remains the storage timestamp and participates in\n duplicate evidence only when it came from a valid sender ``Date``.\n """\n strong_fingerprint = None\n if parsed.get("date_provenance") == "parsed":\n strong_fingerprint = strong_email_fingerprint(\n sender=parsed.get("sender"),\n subject=parsed.get("subject"),\n date=persisted_date,\n body=parsed.get("body"),\n )\n if strong_fingerprint:\n return strong_fingerprint\n source_identity = (\n source_content\n if source_content is not None\n else canonical_email_source_content(parsed)\n )\n return source_email_fingerprint(\n source_identity,\n source_kind="raw" if source_content is not None else "canonical",\n )\n\n\nasync def _find_existing_email' -text, count = pattern.subn(replacement, text, count=1) -if count != 1 and "source-bound fallback identity" not in text: - raise SystemExit(f"email_import_service fingerprint block: replaced {count}") -text = replace_once( - text, - " fingerprint = _email_fingerprint(parsed, persisted_date)\n", - " fingerprint = _email_fingerprint(parsed, persisted_date, content)\n", - "email_import_service fingerprint call", -) -if text.count("generate_email_fingerprint") == 1: - text = text.replace(" generate_email_fingerprint,\n", "", 1) -write(path, text) - -path = "backend/services/imap_worker.py" -text = read(path) -text = replace_once( - text, - "from services.email_dedupe_service import strong_email_fingerprint\n", - "from services.email_dedupe_service import (\n canonical_email_source_content,\n source_email_fingerprint,\n strong_email_fingerprint,\n)\n", - "imap_worker dedupe import", -) -text = replace_once( - text, - "from services.threading_service import assign_thread_id, generate_email_fingerprint\n", - "from services.threading_service import assign_thread_id\n", - "imap_worker threading import", -) -text = replace_once( - text, - ' owner_addresses: Iterable[str] | None = None,\n is_read: bool = True,\n):\n subject = email_data.get("subject", "")\n date_obj = email_data.get("date")\n if hasattr(date_obj, "isoformat"):\n date_str = date_obj.isoformat()\n else:\n date_str = str(date_obj) if date_obj else ""\n', - ' owner_addresses: Iterable[str] | None = None,\n is_read: bool = True,\n source_content: bytes | None = None,\n) -> Email:\n """Persist one fetched email with provenance-safe identity."""\n subject = email_data.get("subject", "")\n date_obj = email_data.get("date")\n', - "imap_worker signature", -) -fingerprint_pattern = re.compile( - r" # Seed the strong \(auto-dedupe\) fingerprint only from a genuinely-parsed\n" - r".*?" - r" fingerprint = strong_fingerprint or generate_email_fingerprint\(\n" - r" subject, date_str, sender, recipients\n" - r" \)\n", - re.DOTALL, -) -fingerprint_replacement = ' # Seed strong duplicate evidence only from a genuinely parsed Date.\n strong_fingerprint = None\n if email_data.get("date_provenance") == "parsed":\n strong_fingerprint = strong_email_fingerprint(\n sender=sender,\n subject=subject,\n date=persisted_date,\n body=email_data.get("body", ""),\n )\n source_identity = (\n source_content\n if source_content is not None\n else canonical_email_source_content(email_data)\n )\n fingerprint = strong_fingerprint or source_email_fingerprint(\n source_identity,\n source_kind="raw" if source_content is not None else "canonical",\n )\n' -text, count = fingerprint_pattern.subn(fingerprint_replacement, text, count=1) -if count != 1 and "source_identity = (" not in text: - raise SystemExit(f"imap_worker fingerprint block: replaced {count}") -text = replace_once( - text, - " is_read=is_read,\n )\n", - " is_read=is_read,\n source_content=raw_message,\n )\n", - "imap_worker raw source propagation", -) -write(path, text) - -path = "backend/services/pop3_worker.py" -text = read(path) -text = replace_once( - text, - " owner_addresses=owner_addresses,\n )\n", - " owner_addresses=owner_addresses,\n source_content=raw_message,\n )\n", - "pop3_worker raw source propagation", -) -write(path, text) - -path = "backend/tests/test_imap_worker.py" -text = read(path) -text = replace_once( - text, - ' assert kwargs["owner_addresses"] == ["imap-user@example.com"]\n', - ' assert kwargs["owner_addresses"] == ["imap-user@example.com"]\n assert kwargs["source_content"] == raw_message\n', - "imap worker source assertion", -) -write(path, text) - -path = "backend/tests/test_pop3_worker.py" -text = read(path) -text = replace_once( - text, - ' async def fake_process_fetched_email(\n db_session, email_data, user_id, organization_id, owner_addresses=None\n ):\n', - ' async def fake_process_fetched_email(\n db_session,\n email_data,\n user_id,\n organization_id,\n owner_addresses=None,\n source_content=None,\n ):\n', - "pop3 fake processor signature", -) -text = replace_once( - text, - ' "owner_addresses": owner_addresses,\n', - ' "owner_addresses": owner_addresses,\n "source_content": source_content,\n', - "pop3 imported source record", -) -text = replace_once( - text, - ' assert imported[0]["owner_addresses"] == ["pop3-user@example.com"]\n', - ' assert imported[0]["owner_addresses"] == ["pop3-user@example.com"]\n assert imported[0]["source_content"] == raw_message\n', - "pop3 worker source assertion", -) -write(path, text) - -path = "backend/tests/test_email_parser_provenance.py" -text = read(path) -text = replace_once( - text, - ' return (headers.strip() + "\\n\\nBody text.").encode("utf-8")\n', - ' return (headers.strip("\\r\\n") + "\\n\\nBody text.").encode("utf-8")\n', - "parser provenance fixture", -) -write(path, text) - -path = "CHANGELOG.md" -text = read(path) -note = ( - "- Bound missing/invalid-Date email deduplication fallbacks to immutable " - "RFC822 source bytes (or a deterministic Date-free canonical projection), " - "including import, IMAP, and POP3 paths, so collection timestamps cannot " - "create false duplicate identities.\n" -) -if note not in text: - if "### Fixed\n" in text: - text = text.replace("### Fixed\n", "### Fixed\n" + note, 1) - else: - text = text.replace("## [Unreleased]\n", "## [Unreleased]\n\n### Fixed\n" + note, 1) -write(path, text) From 4369f80ba6a82ad0f0da3a08d60c7c629d2b499b Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 23:35:51 +0900 Subject: [PATCH 35/86] test(email): reject unsupported canonical source values --- .../tests/test_source_bound_email_dedupe.py | 34 ++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-) diff --git a/backend/tests/test_source_bound_email_dedupe.py b/backend/tests/test_source_bound_email_dedupe.py index 2cfb8d6bb..62c4ddd6b 100644 --- a/backend/tests/test_source_bound_email_dedupe.py +++ b/backend/tests/test_source_bound_email_dedupe.py @@ -14,6 +14,10 @@ from services.imap_worker import process_fetched_email +class _UnsupportedCanonicalValue: + """Represent a value that the parsed EmailData contract never permits.""" + + def test_source_email_fingerprint_is_stable_content_bound_and_domain_separated() -> ( None ): @@ -33,7 +37,7 @@ def test_canonical_source_content_excludes_collection_date() -> None: "recipients": ["one@example.com", "two@example.com"], "subject": "Subject", "body": "Body", - "attachments": [{"filename": "note.txt", "content": b"note"}], + "attachments": [{"filename": "note.txt", "content": "note"}], } first = { **base, @@ -53,6 +57,34 @@ def test_canonical_source_content_excludes_collection_date() -> None: ) +@pytest.mark.parametrize( + ("field_name", "unsupported_value"), + [ + ("body", b"raw bytes"), + ("subject", _UnsupportedCanonicalValue()), + ("attachments", [{"content": b"nested raw bytes"}]), + ("attachments", [{"tags": {"unordered", "values"}}]), + ], +) +def test_canonical_source_content_rejects_unsupported_values( + field_name: str, + unsupported_value: object, +) -> None: + """Reject non-EmailData values instead of coercing colliding strings.""" + email_data: dict[str, object] = { + "message_id": "", + "sender": "sender@example.com", + "recipients": "recipient@example.com", + "subject": "Subject", + "body": "Body", + "attachments": [], + field_name: unsupported_value, + } + + with pytest.raises(TypeError, match=field_name): + canonical_email_source_content(email_data) + + def test_import_fingerprint_uses_trusted_date_or_raw_source() -> None: """Use strong Date evidence only when provenance is genuinely parsed.""" persisted_date = datetime.datetime(2026, 8, 4, 6, 30, tzinfo=datetime.timezone.utc) From 375d6c5c37a9e9aff05028594784c81b22f5f355 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 23:36:54 +0900 Subject: [PATCH 36/86] fix(email): validate canonical source value types --- backend/services/email_dedupe_service.py | 42 ++++++++++++++++++++++-- 1 file changed, 40 insertions(+), 2 deletions(-) diff --git a/backend/services/email_dedupe_service.py b/backend/services/email_dedupe_service.py index 14a6a5c18..679d21a8a 100644 --- a/backend/services/email_dedupe_service.py +++ b/backend/services/email_dedupe_service.py @@ -8,6 +8,7 @@ import datetime import hashlib import json +import math from collections.abc import Iterable, Mapping from dataclasses import dataclass from typing import Literal @@ -65,19 +66,56 @@ def _date_to_fingerprint_value(value: datetime.datetime | None) -> str: ) +def _validate_canonical_source_value(value: object, *, path: str) -> None: + """Reject values outside the deterministic JSON-native EmailData surface. + + Silent ``str()`` coercion is unsafe for identity material because distinct + runtime types can render to the same text. Parsed canonical-source fields + therefore accept only JSON-native scalars, lists, and string-keyed mappings; + every nested value is checked before serialization. + """ + if value is None or isinstance(value, (str, bool, int)): + return + if isinstance(value, float): + if not math.isfinite(value): + raise TypeError(f"{path}: non-finite floats are not canonical") + return + if isinstance(value, list): + for index, item in enumerate(value): + _validate_canonical_source_value(item, path=f"{path}[{index}]") + return + if isinstance(value, dict): + for key, item in value.items(): + if not isinstance(key, str): + raise TypeError( + f"{path}: canonical mapping keys must be strings, got " + f"{type(key).__name__}" + ) + _validate_canonical_source_value(item, path=f"{path}.{key}") + return + raise TypeError( + f"{path}: unsupported canonical email source value type " + f"{type(value).__name__}" + ) + + def canonical_email_source_content(email_data: Mapping[str, object]) -> bytes: """Serialize stable parsed fields when raw transport bytes are unavailable. Collection-time ``date`` values and their provenance are deliberately - excluded. Transport-backed paths should provide exact RFC822 bytes. + excluded. Transport-backed paths should provide exact RFC822 bytes. Values + outside the parsed ``EmailData`` JSON surface fail closed rather than being + string-coerced into potentially colliding identities. """ payload = {field: email_data.get(field) for field in _CANONICAL_SOURCE_FIELDS} + for field, value in payload.items(): + _validate_canonical_source_value(value, path=field) return json.dumps( payload, ensure_ascii=False, sort_keys=True, separators=(",", ":"), - default=str, + allow_nan=False, ).encode("utf-8", errors="surrogatepass") From 24efad389c236f7490571711f66a0f0bd4448a0e Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 23:37:24 +0900 Subject: [PATCH 37/86] docs(email): summarize record-linkage evidence model --- docs/doctoring/email-source-identity-provenance.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/doctoring/email-source-identity-provenance.md b/docs/doctoring/email-source-identity-provenance.md index 8ea47bef6..e0b4d339e 100644 --- a/docs/doctoring/email-source-identity-provenance.md +++ b/docs/doctoring/email-source-identity-provenance.md @@ -40,6 +40,9 @@ identity. identity. - Canonical fallback identity excludes effective collection timestamps and provenance flags. +- Canonical fallback serialization accepts only deterministic JSON-native parsed + values and rejects bytes, unordered collections, custom objects, non-string + mapping keys, and non-finite numbers instead of coercing them with `str()`. - IMAP and POP3 pass source bytes through the persistence boundary. - POP3 source reconstruction restores CRLF after every `RETR` message line. - Existing rows remain conservatively classified when provenance is unknown. @@ -58,6 +61,14 @@ Fellegi, I. P., & Sunter, A. B. (1969). A theory for record linkage. *Journal of the American Statistical Association, 64*(328), 1183–1210. https://doi.org/10.1080/01621459.1969.10501049 +Fellegi and Sunter formalize record linkage by comparing field-level evidence +under match and non-match hypotheses. The resulting evidence score is evaluated +against two decision thresholds: sufficiently strong evidence produces a link, +sufficiently weak evidence produces a non-link, and the intermediate region is +reserved for clerical review. Naruon maps those three outcomes to `auto_link`, +`distinct`, and `review_required` while keeping provenance-gated evidence out of +the automatic-link region. + Myers, J., & Rose, M. (1996). *Post Office Protocol—Version 3* (RFC 1939; STD 53). Internet Engineering Task Force. https://doi.org/10.17487/RFC1939 From 66273d51f142fc46f42bfbe64b330f347f80c8fc Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 13:44:29 +0900 Subject: [PATCH 38/86] test(email): reject zone-less Date as provenance --- backend/tests/test_email_parser_provenance.py | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/backend/tests/test_email_parser_provenance.py b/backend/tests/test_email_parser_provenance.py index 64b197006..1a3a41a1e 100644 --- a/backend/tests/test_email_parser_provenance.py +++ b/backend/tests/test_email_parser_provenance.py @@ -97,6 +97,23 @@ def test_parse_eml_normalizes_minus_zero_zone_to_utc() -> None: assert parsed["date"].tzinfo is not None +def test_parse_eml_marks_zone_less_date_as_invalid_source_evidence() -> None: + """A parseable but zone-less Date cannot become strong sender evidence.""" + parsed = parse_eml_bytes( + _eml_with( + """Message-ID: +From: sender@test.com +To: recipient@test.com +Subject: Missing timezone +Date: Sun, 01 Jan 2023 12:00:00""" + ) + ) + + assert parsed["date_provenance"] == "invalid" + assert parsed["header_date"] is None + assert parsed["date"].tzinfo is not None + + def test_parse_eml_marks_embedded_message_id_provenance() -> None: """A non-empty embedded Message-ID is identified as sender evidence.""" parsed = parse_eml_bytes( From 37429ccb805385621844e7625d5da9e5b11eb6ef Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 13:45:06 +0900 Subject: [PATCH 39/86] fix(email): reject zone-less Date as provenance --- backend/services/email_parser.py | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/backend/services/email_parser.py b/backend/services/email_parser.py index 3c9f04a8e..b129d3d4e 100644 --- a/backend/services/email_parser.py +++ b/backend/services/email_parser.py @@ -55,6 +55,12 @@ def _sanitize_display_text(text: str) -> str: # that force a quoted-string, and the characters escaped inside one. _ADDRESS_SPECIALS_RE = re.compile(r'[()<>@,;:\\".\[\]]') _ADDRESS_QUOTED_ESCAPE_RE = re.compile(r'["\\]') +# ``parsedate_to_datetime`` also accepts some zone-less values and returns a +# naive datetime. Only a Date that actually carries an RFC 5322 zone may be +# promoted to sender-supplied provenance when the parser result is naive. +_RFC5322_TRAILING_ZONE_RE = re.compile( + r"(?:[+-]\d{4}|[A-Za-z]{1,5})(?:\s*\([^)]*\))?\s*$" +) def _format_display_address(display_name: str, address: str) -> str: @@ -188,9 +194,11 @@ def _extract_date_with_provenance( if header_date is None: return fallback, None, "invalid" if header_date.tzinfo is None: - # RFC 5322 section 3.3: a ``-0000`` zone means the time zone is - # unknown. Normalize the naive parser result to UTC so every parsed - # value still satisfies the timezone-aware storage contract. + # ``-0000`` and obsolete alphabetic zones can yield a naive datetime, + # but a parseable value with no zone is not complete RFC 5322 Date + # evidence and must never seed a strong duplicate identity. + if not _RFC5322_TRAILING_ZONE_RE.search(header_text): + return fallback, None, "invalid" header_date = header_date.replace(tzinfo=datetime.timezone.utc) return header_date, header_date, "parsed" From 70bf46ee1df1da13435099fc1749a92eefe84141 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 13:45:30 +0900 Subject: [PATCH 40/86] docs(email): trace zone-less Date provenance repair --- .../email-source-identity-provenance.md | 29 +++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/docs/doctoring/email-source-identity-provenance.md b/docs/doctoring/email-source-identity-provenance.md index e0b4d339e..3ee20375f 100644 --- a/docs/doctoring/email-source-identity-provenance.md +++ b/docs/doctoring/email-source-identity-provenance.md @@ -14,6 +14,33 @@ This boundary prevents two distinct messages collected at the same instant from being linked merely because their observation metadata is similar. It also keeps a repeated import of the same source stable across collection times. +## Date zone evidence reconciliation + +RFC 5322 requires a zone in `date-time`. Python's +`email.utils.parsedate_to_datetime()` can nevertheless return a naive +`datetime` both for a valid `-0000` Date and for a parseable but non-conforming +Date with no zone. Those cases cannot share provenance semantics: `-0000` +retains sender-supplied Date evidence while an omitted zone does not satisfy the +source grammar and must not seed a strong duplicate identity. + +The broad #1086 lineage therefore keeps one `date_provenance` vocabulary and +separates those cases before binding a naive value to UTC: + +- `-0000` and parser-supported obsolete alphabetic zones with an explicit + trailing zone remain `parsed` and are normalized to timezone-aware UTC for + storage/comparison; +- a parseable Date with no trailing zone is `invalid`; `header_date` remains + absent and the effective stored date is a collection-time UTC fallback; +- the fallback instant is operational data only and never sender metadata. + +Source-order RED `66273d51f142fc46f42bfbe64b330f347f80c8fc` adds the zone-less +parser regression. Causal fix `37429ccb805385621844e7625d5da9e5b11eb6ef` +checks for a trailing RFC 5322/obsolete zone before normalizing a naive parsed +value. This inherits the valid #1656 finding into the broad #1195 lineage +without introducing its competing `date_evidence` column or parallel Alembic +revision. The remaining #1656 findings must be reconciled the same way before +that child can become zero-delta provenance. + ## POP3 reconstruction contract POP3 `RETR` is a multiline response. RFC 1939 requires every transmitted line to @@ -35,6 +62,8 @@ identity. - A valid sender `Date` may seed the reviewed strong fingerprint. - Missing and invalid sender dates cannot promote collection time to strong evidence. +- A parseable but zone-less Date remains invalid source evidence, while an + explicit `-0000` Date remains parsed and UTC-comparable. - Two different raw messages collected at the same instant remain distinct. - The same raw message collected at different instants has the same fallback identity. From 9fdb1207447fe1e47565f92ef57ccd53f02b15f2 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 15:40:47 +0900 Subject: [PATCH 41/86] test(email): require complete metadata for strong dedupe evidence --- .../test_email_dedupe_complete_metadata.py | 53 +++++++++++++++++++ 1 file changed, 53 insertions(+) create mode 100644 backend/tests/test_email_dedupe_complete_metadata.py diff --git a/backend/tests/test_email_dedupe_complete_metadata.py b/backend/tests/test_email_dedupe_complete_metadata.py new file mode 100644 index 000000000..1f6dd0d2b --- /dev/null +++ b/backend/tests/test_email_dedupe_complete_metadata.py @@ -0,0 +1,53 @@ +import datetime +from types import SimpleNamespace + +import pytest + +from services.email_dedupe_service import ( + EmailDedupeCandidate, + candidate_strong_fingerprint, + email_strong_fingerprint, +) + + +_COMPLETE = { + "sender": "sender@example.com", + "recipients": "recipient@example.com", + "subject": "Quarterly plan", + "date": datetime.datetime(2026, 9, 17, 6, 0, tzinfo=datetime.timezone.utc), + "body": "Please review the attached plan.", +} + + +def test_candidate_strong_fingerprint_requires_complete_metadata() -> None: + candidate = EmailDedupeCandidate( + candidate_key="candidate-1", + date_provenance="parsed", + **_COMPLETE, + ) + + assert candidate_strong_fingerprint(candidate) is not None + + +@pytest.mark.parametrize("missing_field", ["sender", "recipients", "subject", "body"]) +def test_candidate_strong_fingerprint_withholds_incomplete_metadata( + missing_field: str, +) -> None: + values = dict(_COMPLETE) + values[missing_field] = "" + candidate = EmailDedupeCandidate( + candidate_key=f"missing-{missing_field}", + date_provenance="parsed", + **values, + ) + + assert candidate_strong_fingerprint(candidate) is None + + +def test_stored_email_strong_fingerprint_withholds_incomplete_metadata() -> None: + stored = SimpleNamespace( + date_provenance="parsed", + **{**_COMPLETE, "recipients": None}, + ) + + assert email_strong_fingerprint(stored) is None From d2ae9d6df3906f1018030ff299e0665a96b5127b Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 15:41:20 +0900 Subject: [PATCH 42/86] fix(email): withhold strong dedupe on incomplete metadata --- backend/services/email_dedupe_service.py | 69 +++++++++++++++++++----- 1 file changed, 55 insertions(+), 14 deletions(-) diff --git a/backend/services/email_dedupe_service.py b/backend/services/email_dedupe_service.py index 679d21a8a..539b7dfda 100644 --- a/backend/services/email_dedupe_service.py +++ b/backend/services/email_dedupe_service.py @@ -133,6 +133,30 @@ def source_email_fingerprint( return digest.hexdigest() +def _has_nonempty_text(value: object) -> bool: + """Return whether a metadata field supplies usable non-empty text.""" + return isinstance(value, str) and bool(value.strip()) + + +def has_complete_strong_email_metadata( + *, + sender: object, + recipients: object, + subject: object, + body: object, +) -> bool: + """Require the full sender/recipient/subject/body evidence set for auto-linking. + + Strong metadata evidence is intentionally stricter than source-bound identity: + incomplete messages keep their deterministic raw/canonical source fingerprint + but cannot enter the automatic metadata-link path merely because Date parsed. + """ + return all( + _has_nonempty_text(value) + for value in (sender, recipients, subject, body) + ) + + def strong_email_fingerprint( *, sender: str | None, @@ -142,15 +166,17 @@ def strong_email_fingerprint( ) -> str | None: """Return the strong (sender+subject+Date+body) auto-dedupe fingerprint. - Requires a body; ``None`` for an empty body so bodyless rows cannot collapse - to a shared hash. Callers gate this on genuine Date provenance. + The fingerprint itself preserves the established four-field contract. It is + withheld when sender, subject, or body is missing; callers that possess + recipient evidence additionally apply ``has_complete_strong_email_metadata`` + before treating this hash as automatic-link evidence. """ - if not body: + if not all(_has_nonempty_text(value) for value in (sender, subject, body)): return None return generate_email_fingerprint( { - "sender": sender or "", - "subject": subject or "", + "sender": sender, + "subject": subject, "date": _date_to_fingerprint_value(date), "body": body, } @@ -166,7 +192,14 @@ def candidate_message_lookup_values(candidate: EmailDedupeCandidate) -> set[str] def candidate_strong_fingerprint(candidate: EmailDedupeCandidate) -> str | None: - """Return the candidate's strong fingerprint (see strong_email_fingerprint).""" + """Return strong evidence only when the candidate metadata set is complete.""" + if not has_complete_strong_email_metadata( + sender=candidate.sender, + recipients=candidate.recipients, + subject=candidate.subject, + body=candidate.body, + ): + return None return strong_email_fingerprint( sender=candidate.sender, subject=candidate.subject, @@ -176,15 +209,22 @@ def candidate_strong_fingerprint(candidate: EmailDedupeCandidate) -> str | None: def email_strong_fingerprint(email_row: Email) -> str | None: - """Return a stored row's strong fingerprint, gated on genuine Date provenance. + """Return a stored row's strong fingerprint, gated on trustworthy metadata. A stored row may seed a strong (auto-dedupe) fingerprint only when its date - is genuinely parsed sender metadata; rows with a synthetic or - unknown-provenance date are excluded so they cannot manufacture a strong - duplicate match (naruon#1086). + is genuinely parsed sender metadata and sender/recipient/subject/body evidence + is complete. Rows that fail either condition remain eligible for weaker + review/source-bound identity paths but cannot manufacture an automatic link. """ if getattr(email_row, "date_provenance", None) != "parsed": return None + if not has_complete_strong_email_metadata( + sender=getattr(email_row, "sender", None), + recipients=getattr(email_row, "recipients", None), + subject=getattr(email_row, "subject", None), + body=getattr(email_row, "body", None), + ): + return None return strong_email_fingerprint( sender=email_row.sender, subject=email_row.subject, @@ -244,13 +284,14 @@ def classify_dedupe_decision( - ``auto_link`` (A1, positive link): the pair shares a reliable identity link -- the same normalized Message-ID, or a genuine strong match (identical sender/subject/Date/body with a trusted, parsed Date on *both* - sides). These are safe to merge automatically. + sides and complete sender/recipient/subject/body metadata). These are safe + to merge automatically. - ``review_required`` (A2, possible match): the pair shares a provenance-independent content signal (same sender/subject/body) but has no reliable identity link -- typically because at least one side's Date - provenance is synthetic or unknown, so the strong fingerprint was withheld - (naruon#1086). This is the clerical-review band: a probable duplicate that - must not be silently merged or silently kept. + provenance is synthetic/unknown or strong metadata is incomplete, so the + strong fingerprint was withheld (naruon#1086). This is the clerical-review + band: a probable duplicate that must not be silently merged or silently kept. - ``distinct`` (A3, non-link): no shared identity or content signal. """ candidate_message = normalize_message_id(candidate.message_id) From 540f6e4ec57ed355263d56e7da78de7ff5310360 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 15:42:57 +0900 Subject: [PATCH 43/86] test(email): pin canonical provenance migration contract --- ...mail_date_provenance_migration_contract.py | 37 +++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100644 backend/tests/test_email_date_provenance_migration_contract.py diff --git a/backend/tests/test_email_date_provenance_migration_contract.py b/backend/tests/test_email_date_provenance_migration_contract.py new file mode 100644 index 000000000..1c62caf60 --- /dev/null +++ b/backend/tests/test_email_date_provenance_migration_contract.py @@ -0,0 +1,37 @@ +from pathlib import Path + + +BACKEND_ROOT = Path(__file__).resolve().parents[1] + + +def test_canonical_email_date_provenance_migration_contract() -> None: + revision_path = ( + BACKEND_ROOT + / "alembic" + / "versions" + / "0018_email_date_provenance.py" + ) + revision_text = revision_path.read_text() + + assert 'revision = "0018_email_date_provenance"' in revision_text + assert 'down_revision = "0017_merge_newsdom_carddav_heads"' in revision_text + assert '_EMAIL_TABLE = "email_records"' in revision_text + assert '_PROVENANCE_COLUMN = "date_provenance"' in revision_text + assert "nullable=False" in revision_text + assert 'server_default="unknown"' in revision_text + + assert 'op.add_column(' in revision_text + assert 'op.drop_column(_EMAIL_TABLE, _PROVENANCE_COLUMN)' in revision_text + + +def test_canonical_email_provenance_does_not_fork_parallel_evidence_columns() -> None: + revision_path = ( + BACKEND_ROOT + / "alembic" + / "versions" + / "0018_email_date_provenance.py" + ) + revision_text = revision_path.read_text() + + assert "date_evidence" not in revision_text + assert "message_id_evidence" not in revision_text From 5d68f08b0de6838a818d59636edf2c2f599138ee Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 15:49:23 +0900 Subject: [PATCH 44/86] test(email): expose import strong-evidence boundary --- ...t_email_import_strong_evidence_boundary.py | 43 +++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100644 backend/tests/test_email_import_strong_evidence_boundary.py diff --git a/backend/tests/test_email_import_strong_evidence_boundary.py b/backend/tests/test_email_import_strong_evidence_boundary.py new file mode 100644 index 000000000..b11fb33bd --- /dev/null +++ b/backend/tests/test_email_import_strong_evidence_boundary.py @@ -0,0 +1,43 @@ +import datetime + +from services.email_dedupe_service import source_email_fingerprint, strong_email_fingerprint +from services.email_import_service import _dedupe_review_reason, _email_fingerprint + + +_DATE = datetime.datetime(2026, 9, 17, 6, 30, tzinfo=datetime.timezone.utc) +_SOURCE = b"From: sender@example.com\r\nTo: recipient@example.com\r\nSubject: Plan\r\n\r\nBody" +_COMPLETE = { + "sender": "sender@example.com", + "recipients": "recipient@example.com", + "subject": "Plan", + "body": "Body", + "date": _DATE, + "date_provenance": "parsed", +} + + +def test_import_metadata_strong_fingerprint_requires_recipient_evidence() -> None: + expected_strong = strong_email_fingerprint( + sender=_COMPLETE["sender"], + subject=_COMPLETE["subject"], + date=_DATE, + body=_COMPLETE["body"], + ) + assert expected_strong is not None + assert _email_fingerprint(dict(_COMPLETE), _DATE, _SOURCE) == expected_strong + + incomplete = {**_COMPLETE, "recipients": ""} + assert _email_fingerprint(incomplete, _DATE, _SOURCE) == source_email_fingerprint( + _SOURCE, + source_kind="raw", + ) + + +def test_import_review_reason_tracks_withheld_strong_metadata_evidence() -> None: + assert _dedupe_review_reason(dict(_COMPLETE)) is None + assert _dedupe_review_reason({**_COMPLETE, "recipients": ""}) == ( + "dedupe_review_required" + ) + assert _dedupe_review_reason({**_COMPLETE, "date_provenance": "invalid"}) == ( + "dedupe_review_required" + ) From c39f51743598d817edc2f1dfb585a240e0dcc2d7 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 15:50:53 +0900 Subject: [PATCH 45/86] fix(email): gate import strong evidence on complete metadata --- backend/services/email_import_service.py | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-) diff --git a/backend/services/email_import_service.py b/backend/services/email_import_service.py index 8906bd076..9685e9166 100644 --- a/backend/services/email_import_service.py +++ b/backend/services/email_import_service.py @@ -33,6 +33,7 @@ from services.content_graph import ParseResult, parse_content from services.email_dedupe_service import ( canonical_email_source_content, + has_complete_strong_email_metadata, source_email_fingerprint, strong_email_fingerprint, ) @@ -197,18 +198,34 @@ def _message_id_for(parsed: EmailData, content: bytes) -> str: ) +def _has_strong_email_metadata(parsed: EmailData) -> bool: + """Return whether parsed metadata may drive an automatic strong fingerprint.""" + return parsed.get("date_provenance") == "parsed" and has_complete_strong_email_metadata( + sender=parsed.get("sender"), + recipients=parsed.get("recipients"), + subject=parsed.get("subject"), + body=parsed.get("body"), + ) + + +def _dedupe_review_reason(parsed: EmailData) -> str | None: + """Expose when an import lacks enough metadata for strong automatic linking.""" + return None if _has_strong_email_metadata(parsed) else "dedupe_review_required" + + def _email_fingerprint( parsed: EmailData, persisted_date: datetime.datetime, source_content: bytes | None = None, ) -> str: - """Return trusted-Date evidence or a source-bound fallback identity. + """Return trusted complete-metadata evidence or source-bound fallback identity. ``persisted_date`` remains the storage timestamp and participates in - duplicate evidence only when it came from a valid sender ``Date``. + duplicate evidence only when it came from a valid sender ``Date`` and the + sender/recipient/subject/body metadata evidence set is complete. """ strong_fingerprint = None - if parsed.get("date_provenance") == "parsed": + if _has_strong_email_metadata(parsed): strong_fingerprint = strong_email_fingerprint( sender=parsed.get("sender"), subject=parsed.get("subject"), @@ -951,6 +968,7 @@ async def _import_single_eml( return EmailImportItemResult( filename=display_filename, status="imported", + reason_code=_dedupe_review_reason(parsed), attachment_count=attachment_count, ) From fd4cd4e405b3af30b4458993a4673063350b49d9 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 15:51:18 +0900 Subject: [PATCH 46/86] test(email): expose IMAP strong-evidence boundary --- .../test_imap_strong_evidence_boundary.py | 47 +++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100644 backend/tests/test_imap_strong_evidence_boundary.py diff --git a/backend/tests/test_imap_strong_evidence_boundary.py b/backend/tests/test_imap_strong_evidence_boundary.py new file mode 100644 index 000000000..b523fbe08 --- /dev/null +++ b/backend/tests/test_imap_strong_evidence_boundary.py @@ -0,0 +1,47 @@ +import datetime +from unittest.mock import AsyncMock, MagicMock + +import pytest + +from services.email_dedupe_service import source_email_fingerprint +from services.imap_worker import process_fetched_email + + +@pytest.mark.asyncio +async def test_imap_withholds_metadata_strong_fingerprint_without_recipients( + monkeypatch, +) -> None: + raw_message = b"From: sender@example.com\r\nSubject: Plan\r\n\r\nBody" + email_data = { + "message_id": "", + "sender": "sender@example.com", + "recipients": "", + "subject": "Plan", + "date": datetime.datetime(2026, 9, 17, 6, 45, tzinfo=datetime.timezone.utc), + "date_provenance": "parsed", + "body": "Body", + } + + query_result = MagicMock() + query_result.scalar_one_or_none.return_value = None + session = MagicMock() + session.execute = AsyncMock(return_value=query_result) + session.flush = AsyncMock() + + monkeypatch.setattr( + "services.imap_worker.assign_thread_id", + AsyncMock(return_value="thread-boundary"), + ) + + imported = await process_fetched_email( + session, + email_data, + "user-1", + "org-1", + source_content=raw_message, + ) + + assert imported.fingerprint == source_email_fingerprint( + raw_message, + source_kind="raw", + ) From d8a08deb1ad009b2d90de2eafbab5da61ac0cc79 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 15:51:54 +0900 Subject: [PATCH 47/86] fix(email): gate IMAP strong evidence on complete metadata --- backend/services/imap_worker.py | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/backend/services/imap_worker.py b/backend/services/imap_worker.py index 07ceb9f7f..90f75e41f 100644 --- a/backend/services/imap_worker.py +++ b/backend/services/imap_worker.py @@ -12,6 +12,7 @@ from services.email_client import validate_imap_destination from services.email_dedupe_service import ( canonical_email_source_content, + has_complete_strong_email_metadata, source_email_fingerprint, strong_email_fingerprint, ) @@ -51,15 +52,26 @@ async def process_fetched_email( if isinstance(recipients_list, list) else str(recipients_list or "") ) + body = email_data.get("body", "") - # Seed strong duplicate evidence only from a genuinely parsed Date. + # Metadata-based auto-linking needs both genuine Date provenance and the full + # sender/recipient/subject/body evidence set. Incomplete messages keep their + # raw/canonical source identity instead of manufacturing a strong match. strong_fingerprint = None - if email_data.get("date_provenance") == "parsed": + if ( + email_data.get("date_provenance") == "parsed" + and has_complete_strong_email_metadata( + sender=sender, + recipients=recipients, + subject=subject, + body=body, + ) + ): strong_fingerprint = strong_email_fingerprint( sender=sender, subject=subject, date=persisted_date, - body=email_data.get("body", ""), + body=body, ) source_identity = ( source_content @@ -102,7 +114,7 @@ async def process_fetched_email( subject=subject, date=persisted_date, date_provenance=email_data.get("date_provenance", "unknown"), - body=email_data.get("body", ""), + body=body, is_read=is_read, embedding=[0.0] * 1536, ) From 809c8ad89bccc462bac22a62acc686156d9ad210 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 15:53:14 +0900 Subject: [PATCH 48/86] docs(email): trace complete-metadata dedupe evidence --- .../email-source-identity-provenance.md | 63 ++++++++++++++++++- 1 file changed, 60 insertions(+), 3 deletions(-) diff --git a/docs/doctoring/email-source-identity-provenance.md b/docs/doctoring/email-source-identity-provenance.md index 3ee20375f..de49daa4c 100644 --- a/docs/doctoring/email-source-identity-provenance.md +++ b/docs/doctoring/email-source-identity-provenance.md @@ -38,8 +38,58 @@ parser regression. Causal fix `37429ccb805385621844e7625d5da9e5b11eb6ef` checks for a trailing RFC 5322/obsolete zone before normalizing a naive parsed value. This inherits the valid #1656 finding into the broad #1195 lineage without introducing its competing `date_evidence` column or parallel Alembic -revision. The remaining #1656 findings must be reconciled the same way before -that child can become zero-delta provenance. +revision. + +## Complete metadata evidence reconciliation + +A parsed Date is necessary but not sufficient for metadata-based automatic +linking. A strong metadata decision also requires non-empty sender, recipients, +subject, and body evidence. Missing any one of those fields makes the metadata +comparison incomplete; the message keeps its raw/canonical source-bound +identity and may enter the review band, but that incomplete tuple cannot create +an automatic metadata link. + +The rule is intentionally a gate rather than a new fingerprint format. The +existing strong fingerprint payload remains sender/subject/Date/body so stored +fingerprint compatibility is not silently broken; recipient evidence determines +whether that fingerprint is authoritative enough to use. Message-ID equality +remains an independent identity signal. + +The source-order sequence for this repair is: + +- `9fdb1207447fe1e47565f92ef57ccd53f02b15f2`: candidate/stored-row RED for + missing sender/recipients/subject/body evidence; +- `d2ae9d6df3906f1018030ff299e0665a96b5127b`: shared complete-metadata gate in + the domain classifier; +- `5d68f08b0de6838a818d59636edf2c2f599138ee` → + `c39f51743598d817edc2f1dfb585a240e0dcc2d7`: direct-import RED then causal + repair, including deterministic `dedupe_review_required` result semantics; +- `fd4cd4e405b3af30b4458993a4673063350b49d9` → + `d8a08deb1ad009b2d90de2eafbab5da61ac0cc79`: IMAP RED then causal repair. + POP3 reaches the same `process_fetched_email` boundary and therefore consumes + the repaired IMAP/POP3 persistence path rather than duplicating the rule. + +The canonical migration contract is separately pinned by +`540f6e4ec57ed355263d56e7da78de7ff5310360`: `0018_email_date_provenance` +remains the single provenance revision, with non-null `date_provenance` and an +`unknown` server default. The parallel #1656 `date_evidence` / +`message_id_evidence` migration is not adopted. + +## Message-ID provenance decision + +The parser already carries transient `message_id_provenance` (`embedded` or +`missing`) at the ingestion boundary. A missing direct-import Message-ID is +replaced with a deterministic `import-@local.naruon` +identifier, so equality of two such fallback identifiers is already equality of +the same raw-source digest. Persisting a second `message_id_evidence` column has +no current decision consumer or invariant that cannot be represented by the +existing Message-ID plus source-fingerprint semantics. + +Accordingly, #1656's proposed `message_id_evidence` persistence is rejected for +this lineage unless a concrete future consumer proves a storage requirement. +This avoids a second provenance vocabulary and a sibling migration merely to +record evidence that is already available at ingestion and encoded by the +source-bound identity contract. ## POP3 reconstruction contract @@ -59,11 +109,16 @@ identity. ## Verification contract -- A valid sender `Date` may seed the reviewed strong fingerprint. +- A valid sender `Date` may seed the reviewed strong fingerprint only when + sender/recipients/subject/body evidence is complete. - Missing and invalid sender dates cannot promote collection time to strong evidence. - A parseable but zone-less Date remains invalid source evidence, while an explicit `-0000` Date remains parsed and UTC-comparable. +- Incomplete metadata cannot produce a strong metadata auto-link; import reports + `dedupe_review_required` while retaining source-bound identity. +- Direct import, IMAP, and the POP3 path through `process_fetched_email` apply + the same complete-metadata gate. - Two different raw messages collected at the same instant remain distinct. - The same raw message collected at different instants has the same fallback identity. @@ -75,6 +130,8 @@ identity. - IMAP and POP3 pass source bytes through the persistence boundary. - POP3 source reconstruction restores CRLF after every `RETR` message line. - Existing rows remain conservatively classified when provenance is unknown. +- `0018_email_date_provenance` remains the sole canonical provenance migration; + no parallel `date_evidence` or `message_id_evidence` schema is accepted. ## Claim boundary From 81e230a39bf31264b663a0a4310216713e538241 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 15:54:27 +0900 Subject: [PATCH 49/86] test(email): assert import review disposition end to end --- ...t_email_import_strong_evidence_boundary.py | 67 +++++++++++++++++++ 1 file changed, 67 insertions(+) diff --git a/backend/tests/test_email_import_strong_evidence_boundary.py b/backend/tests/test_email_import_strong_evidence_boundary.py index b11fb33bd..3985a4aee 100644 --- a/backend/tests/test_email_import_strong_evidence_boundary.py +++ b/backend/tests/test_email_import_strong_evidence_boundary.py @@ -1,5 +1,9 @@ import datetime +from unittest.mock import AsyncMock, MagicMock +import pytest + +import services.email_import_service as email_import_module from services.email_dedupe_service import source_email_fingerprint, strong_email_fingerprint from services.email_import_service import _dedupe_review_reason, _email_fingerprint @@ -41,3 +45,66 @@ def test_import_review_reason_tracks_withheld_strong_metadata_evidence() -> None assert _dedupe_review_reason({**_COMPLETE, "date_provenance": "invalid"}) == ( "dedupe_review_required" ) + + +@pytest.mark.asyncio +async def test_import_result_marks_incomplete_metadata_for_dedupe_review( + monkeypatch, + tmp_path, +) -> None: + eml_path = tmp_path / "message.eml" + eml_path.write_bytes(_SOURCE) + parsed = { + **_COMPLETE, + "message_id": "", + "recipients": "", + "attachments": [], + } + email_object = MagicMock() + session = MagicMock() + session.add = MagicMock() + session.commit = AsyncMock() + session.rollback = AsyncMock() + + monkeypatch.setattr( + email_import_module, + "_read_and_parse_eml", + lambda _: (_SOURCE, parsed), + ) + monkeypatch.setattr( + email_import_module, + "_find_existing_email", + AsyncMock(return_value=None), + ) + monkeypatch.setattr( + email_import_module, + "assign_thread_id", + AsyncMock(return_value="thread-1"), + ) + monkeypatch.setattr( + email_import_module, + "_extract_and_generate_embeddings", + AsyncMock(return_value=([], [[0.0] * email_import_module.EMBEDDING_DIMENSION])), + ) + monkeypatch.setattr( + email_import_module, + "_build_email_object", + lambda **_: (email_object, 0), + ) + monkeypatch.setattr( + email_import_module, + "_persist_project_graph_projection", + AsyncMock(), + ) + + result = await email_import_module._import_single_eml( + session, + eml_path=eml_path, + display_filename="message.eml", + user_id="user-1", + organization_id="org-1", + ) + + assert result.status == "imported" + assert result.reason_code == "dedupe_review_required" + session.commit.assert_awaited_once() From 7ef016b06f54b5cb260bebfb053d9246ad9172fc Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 15:54:45 +0900 Subject: [PATCH 50/86] test(email): isolate import review disposition contract --- backend/tests/test_email_import_strong_evidence_boundary.py | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/backend/tests/test_email_import_strong_evidence_boundary.py b/backend/tests/test_email_import_strong_evidence_boundary.py index 3985a4aee..743a4f18f 100644 --- a/backend/tests/test_email_import_strong_evidence_boundary.py +++ b/backend/tests/test_email_import_strong_evidence_boundary.py @@ -66,6 +66,11 @@ async def test_import_result_marks_incomplete_metadata_for_dedupe_review( session.commit = AsyncMock() session.rollback = AsyncMock() + monkeypatch.setattr( + email_import_module.settings, + "PROJECT_GRAPH_EXTRACTION_ENABLED", + False, + ) monkeypatch.setattr( email_import_module, "_read_and_parse_eml", From 1fa8243a334f53fae39e9e98bf1f94c95660104c Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 16:13:20 +0900 Subject: [PATCH 51/86] test(mail): reproduce duplicate sync count inflation --- backend/tests/test_imap_worker.py | 58 +++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+) diff --git a/backend/tests/test_imap_worker.py b/backend/tests/test_imap_worker.py index 529b569a0..c26733141 100644 --- a/backend/tests/test_imap_worker.py +++ b/backend/tests/test_imap_worker.py @@ -132,6 +132,64 @@ async def test_imap_worker_imports_fetched_rfc822_messages(monkeypatch): session.rollback.assert_not_awaited() +@pytest.mark.asyncio +async def test_imap_duplicate_does_not_inflate_imported_count(monkeypatch): + from db.models import Email + + worker = ImapSyncWorker() + config = TenantConfig( + user_id="imap-user", + organization_id="org-imap", + imap_server="imap.example.com", + imap_port=993, + imap_username="imap-user@example.com", + imap_password="imap-secret", + ) + raw_message = ( + b"Message-ID: \r\n" + b"From: Sender \r\n" + b"To: imap-user@example.com\r\n" + b"Subject: Existing message\r\n" + b"Date: Mon, 15 Jun 2026 10:00:00 +0000\r\n" + b"\r\n" + b"Already imported.\r\n" + ) + existing_email = Email(id=1) + + class ExistingResult: + def scalar_one_or_none(self): + return existing_email + + class FakeSession: + def __init__(self): + self.committed = False + self.rolled_back = False + + async def __aenter__(self): + return self + + async def __aexit__(self, exc_type, exc, tb): + return False + + async def execute(self, _statement): + return ExistingResult() + + async def commit(self): + self.committed = True + + async def rollback(self): + self.rolled_back = True + + session = FakeSession() + monkeypatch.setattr("services.imap_worker.AsyncSessionLocal", lambda: session) + + imported_count = await worker._import_messages(config, [(raw_message, False)]) + + assert imported_count == 0 + assert session.committed is True + assert session.rolled_back is False + + @pytest.mark.asyncio async def test_imap_worker_requires_credentials_without_sensitive_log_names( caplog, monkeypatch From 88556ba5aa9a7f8b5e52f83040104d2433fd6d45 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 16:13:41 +0900 Subject: [PATCH 52/86] test(mail): reproduce POP3 duplicate count inflation --- backend/tests/test_pop3_worker.py | 58 +++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+) diff --git a/backend/tests/test_pop3_worker.py b/backend/tests/test_pop3_worker.py index 9e2cec1cb..32c55efc1 100644 --- a/backend/tests/test_pop3_worker.py +++ b/backend/tests/test_pop3_worker.py @@ -185,3 +185,61 @@ async def fake_process_fetched_email( assert imported[0]["email_data"]["subject"] == "POP3 import" assert session.committed is True assert session.rolled_back is False + + +@pytest.mark.asyncio +async def test_pop3_duplicate_does_not_inflate_imported_count(monkeypatch): + from db.models import Email + + worker = Pop3SyncWorker() + config = TenantConfig( + user_id="pop3-user", + organization_id="org-pop3", + pop3_server="pop3.example.com", + pop3_port=995, + pop3_username="pop3-user@example.com", + pop3_password="pop3-secret", + ) + raw_message = ( + b"Message-ID: \r\n" + b"From: Sender \r\n" + b"To: pop3-user@example.com\r\n" + b"Subject: Existing message\r\n" + b"Date: Mon, 15 Jun 2026 10:00:00 +0000\r\n" + b"\r\n" + b"Already imported.\r\n" + ) + existing_email = Email(id=1) + + class ExistingResult: + def scalar_one_or_none(self): + return existing_email + + class FakeSession: + def __init__(self): + self.committed = False + self.rolled_back = False + + async def __aenter__(self): + return self + + async def __aexit__(self, exc_type, exc, tb): + return False + + async def execute(self, _statement): + return ExistingResult() + + async def commit(self): + self.committed = True + + async def rollback(self): + self.rolled_back = True + + session = FakeSession() + monkeypatch.setattr("services.pop3_worker.AsyncSessionLocal", lambda: session) + + imported_count = await worker._import_messages(config, [raw_message]) + + assert imported_count == 0 + assert session.committed is True + assert session.rolled_back is False From 6999f18fe6d58d92cb99df7561425db04b77fcf4 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 16:14:34 +0900 Subject: [PATCH 53/86] fix(mail): count only newly persisted sync messages --- backend/services/imap_worker.py | 52 ++++++++++++++++++++++++++++----- 1 file changed, 44 insertions(+), 8 deletions(-) diff --git a/backend/services/imap_worker.py b/backend/services/imap_worker.py index 90f75e41f..eb800ae1e 100644 --- a/backend/services/imap_worker.py +++ b/backend/services/imap_worker.py @@ -25,7 +25,15 @@ from services.threading_service import assign_thread_id -async def process_fetched_email( +@dataclass(frozen=True, slots=True) +class FetchedEmailPersistenceResult: + """Report the persisted email and whether this sync created it.""" + + email_record: Email + created_record: bool + + +async def persist_fetched_email( session, email_data: EmailData, user_id: str, @@ -33,8 +41,8 @@ async def process_fetched_email( owner_addresses: Iterable[str] | None = None, is_read: bool = True, source_content: bytes | None = None, -) -> Email: - """Persist one fetched email with provenance-safe identity.""" +) -> FetchedEmailPersistenceResult: + """Persist one fetched email and retain duplicate disposition for sync counts.""" subject = email_data.get("subject", "") date_obj = email_data.get("date") if isinstance(date_obj, datetime.datetime): @@ -83,7 +91,6 @@ async def process_fetched_email( source_kind="raw" if source_content is not None else "canonical", ) - # Check if duplicate stmt = select(Email).where( Email.user_id == user_id, Email.organization_id == (organization_id if organization_id else None), @@ -97,7 +104,10 @@ async def process_fetched_email( "Email with fingerprint %s already exists. Skipping duplicate insertion.", fingerprint, ) - return existing_email + return FetchedEmailPersistenceResult( + email_record=existing_email, + created_record=False, + ) thread_id = await assign_thread_id( session, email_data, user_id=user_id, organization_id=organization_id @@ -123,7 +133,32 @@ async def process_fetched_email( if is_self_sent_email(new_email, owner_addresses): await session.flush() await extract_knowledge_from_self_sent(session, new_email, owner_addresses) - return new_email + return FetchedEmailPersistenceResult( + email_record=new_email, + created_record=True, + ) + + +async def process_fetched_email( + session, + email_data: EmailData, + user_id: str, + organization_id: str | None, + owner_addresses: Iterable[str] | None = None, + is_read: bool = True, + source_content: bytes | None = None, +) -> Email: + """Persist one fetched email while preserving the legacy Email return contract.""" + persistence_result = await persist_fetched_email( + session, + email_data, + user_id, + organization_id, + owner_addresses=owner_addresses, + is_read=is_read, + source_content=source_content, + ) + return persistence_result.email_record logger = logging.getLogger(__name__) @@ -360,7 +395,7 @@ async def _import_messages( config.user_id, ) continue - await process_fetched_email( + persistence_result = await persist_fetched_email( session, email_data, config.user_id, @@ -369,7 +404,8 @@ async def _import_messages( is_read=is_read, source_content=raw_message, ) - imported_count += 1 + if persistence_result.created_record: + imported_count += 1 await session.commit() except Exception: await session.rollback() From e045e03ad9f4d9778606d72e2d83b73b84e2911b Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 16:14:54 +0900 Subject: [PATCH 54/86] fix(mail): align POP3 sync count with persistence disposition --- backend/services/pop3_worker.py | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/backend/services/pop3_worker.py b/backend/services/pop3_worker.py index be76acdef..4637482ba 100644 --- a/backend/services/pop3_worker.py +++ b/backend/services/pop3_worker.py @@ -7,7 +7,7 @@ from services.email_client import validate_pop3_destination from services.email_parser import parse_eml_bytes from services.exceptions import EmailParseError -from services.imap_worker import process_fetched_email +from services.imap_worker import persist_fetched_email logger = logging.getLogger(__name__) MAX_POP3_FETCH_MESSAGES = 10 @@ -122,7 +122,7 @@ async def _import_messages( config.user_id, ) continue - await process_fetched_email( + persistence_result = await persist_fetched_email( session, email_data, config.user_id, @@ -130,7 +130,8 @@ async def _import_messages( owner_addresses=owner_addresses, source_content=raw_message, ) - imported_count += 1 + if persistence_result.created_record: + imported_count += 1 await session.commit() except Exception: await session.rollback() From e18510d183ef466b5c210a414627835731997cd5 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 16:15:15 +0900 Subject: [PATCH 55/86] test(mail): align IMAP sync count contract with persistence result --- backend/tests/test_imap_worker.py | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/backend/tests/test_imap_worker.py b/backend/tests/test_imap_worker.py index c26733141..4ac5acd37 100644 --- a/backend/tests/test_imap_worker.py +++ b/backend/tests/test_imap_worker.py @@ -1,3 +1,4 @@ +from types import SimpleNamespace from unittest.mock import AsyncMock import pytest @@ -90,7 +91,9 @@ async def test_imap_worker_imports_fetched_rfc822_messages(monkeypatch): session.__aenter__.return_value = session session.__aexit__.return_value = False - process_fetched_email_mock = AsyncMock() + persist_fetched_email_mock = AsyncMock( + return_value=SimpleNamespace(created_record=True) + ) monkeypatch.setattr( "services.imap_worker.validate_imap_destination", @@ -102,8 +105,8 @@ async def test_imap_worker_imports_fetched_rfc822_messages(monkeypatch): ) monkeypatch.setattr("services.imap_worker.AsyncSessionLocal", lambda: session) monkeypatch.setattr( - "services.imap_worker.process_fetched_email", - process_fetched_email_mock, + "services.imap_worker.persist_fetched_email", + persist_fetched_email_mock, ) imported_count = await worker._sync_tenant(config) @@ -116,8 +119,8 @@ async def test_imap_worker_imports_fetched_rfc822_messages(monkeypatch): imap_client.fetch.assert_awaited_once_with("1", "(RFC822 FLAGS)") imap_client.logout.assert_awaited_once() - process_fetched_email_mock.assert_awaited_once() - args, kwargs = process_fetched_email_mock.await_args + persist_fetched_email_mock.assert_awaited_once() + args, kwargs = persist_fetched_email_mock.await_args assert args[0] is session assert args[1]["message_id"] == "" assert args[1]["subject"] == "IMAP import" From 908a254dd3ce8dc3fccc894b26393705838b38fd Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 16:15:37 +0900 Subject: [PATCH 56/86] test(mail): align POP3 sync count contract with persistence result --- backend/tests/test_pop3_worker.py | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/backend/tests/test_pop3_worker.py b/backend/tests/test_pop3_worker.py index 32c55efc1..4b659f0a9 100644 --- a/backend/tests/test_pop3_worker.py +++ b/backend/tests/test_pop3_worker.py @@ -1,7 +1,9 @@ import asyncio -import pytest +from types import SimpleNamespace from unittest.mock import MagicMock, patch +import pytest + from db.models import TenantConfig from services.pop3_worker import Pop3SyncWorker @@ -131,7 +133,7 @@ async def rollback(self): session = FakeSession() - async def fake_process_fetched_email( + async def fake_persist_fetched_email( db_session, email_data, user_id, @@ -149,6 +151,7 @@ async def fake_process_fetched_email( "source_content": source_content, } ) + return SimpleNamespace(created_record=True) monkeypatch.setattr( "services.pop3_worker.validate_pop3_destination", @@ -159,9 +162,8 @@ async def fake_process_fetched_email( lambda: session, ) monkeypatch.setattr( - "services.pop3_worker.process_fetched_email", - fake_process_fetched_email, - raising=False, + "services.pop3_worker.persist_fetched_email", + fake_persist_fetched_email, ) monkeypatch.setattr( "services.pop3_worker.poplib.POP3_SSL", From 517ba20f2409012eb28b9c84085103a7c1b04eaa Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 16:46:03 +0900 Subject: [PATCH 57/86] test(pop3): require newest bounded fetch window --- backend/tests/test_pop3_worker.py | 39 +++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/backend/tests/test_pop3_worker.py b/backend/tests/test_pop3_worker.py index 4b659f0a9..4824fb6a0 100644 --- a/backend/tests/test_pop3_worker.py +++ b/backend/tests/test_pop3_worker.py @@ -74,6 +74,45 @@ def test_pop3_do_sync_validates_destination_before_connect(): pop3_ssl.assert_not_called() +def test_pop3_sync_fetches_newest_bounded_message_numbers(monkeypatch): + worker = Pop3SyncWorker() + config = TenantConfig( + user_id="pop3-user", + pop3_server="pop3.example.com", + pop3_port=995, + pop3_username="pop3-user@example.com", + pop3_password="pop3-secret", + ) + pop3_client = MagicMock() + pop3_client.list.return_value = ( + b"+OK", + [f"{message_number} 128".encode() for message_number in range(1, 13)], + 1536, + ) + pop3_client.retr.side_effect = lambda message_number: ( + b"+OK", + [f"Message-ID: ".encode(), b"", b"Body"], + 128, + ) + + monkeypatch.setattr( + "services.pop3_worker.validate_pop3_destination", + lambda host, port: (host, port), + ) + monkeypatch.setattr( + "services.pop3_worker.poplib.POP3_SSL", + lambda host, port: pop3_client, + ) + + messages = worker._do_pop3_sync(config) + + assert [entry.args[0] for entry in pop3_client.retr.call_args_list] == list( + range(3, 13) + ) + assert len(messages) == 10 + pop3_client.quit.assert_called_once() + + @pytest.mark.asyncio async def test_pop3_worker_skips_disallowed_destination(): worker = Pop3SyncWorker() From 72c64b46d120ee8c2f3f12ad04114e6a896cb15b Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 16:46:27 +0900 Subject: [PATCH 58/86] fix(pop3): fetch newest bounded mailbox window --- backend/services/pop3_worker.py | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/backend/services/pop3_worker.py b/backend/services/pop3_worker.py index 4637482ba..393164f4a 100644 --- a/backend/services/pop3_worker.py +++ b/backend/services/pop3_worker.py @@ -173,11 +173,17 @@ def _do_pop3_sync( pop3_client.user(config.pop3_username) pop3_client.pass_(config.pop3_password) _response, listings, _octets = pop3_client.list() + message_numbers = [ + message_number + for listing in listings + if (message_number := self._message_number_from_listing(listing)) + is not None + ] messages: list[bytes] = [] - for listing in listings[:MAX_POP3_FETCH_MESSAGES]: - message_number = self._message_number_from_listing(listing) - if message_number is None: - continue + # RFC 1939 numbers the first maildrop message as 1 and the nth as n. + # This worker does not DELE after RETR, so repeatedly taking the first + # bounded window would starve later arrivals in maildrops over the cap. + for message_number in sorted(message_numbers)[-MAX_POP3_FETCH_MESSAGES:]: _retr_response, lines, _retr_octets = pop3_client.retr(message_number) messages.append(self._message_bytes(lines)) return messages From 0726c4b00b56ab36653cba027eff7a7742a68eb6 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 16:46:56 +0900 Subject: [PATCH 59/86] docs(email): trace POP3 bounded fetch decision --- .../email-source-identity-provenance.md | 24 +++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/docs/doctoring/email-source-identity-provenance.md b/docs/doctoring/email-source-identity-provenance.md index de49daa4c..0049d6655 100644 --- a/docs/doctoring/email-source-identity-provenance.md +++ b/docs/doctoring/email-source-identity-provenance.md @@ -107,6 +107,28 @@ streams. Duplicate classification remains deterministic because the source kind is domain separated and because collection time is excluded from fallback identity. +## POP3 bounded collection window + +RFC 1939 assigns message number `1` to the first message in the opened maildrop +and number `n` to the nth message. Naruon's POP3 worker intentionally does not +issue `DELE`; source retention is therefore independent from synchronization. +With a bounded fetch cap, repeatedly taking the first ten `LIST` entries would +re-read the same oldest window on every poll and could permanently starve later +mail in a maildrop larger than the cap. + +Source-order regression `517ba20f2409012eb28b9c84085103a7c1b04eaa` +requires the bounded POP3 fetch to select the ten highest valid message numbers +from a twelve-message maildrop. Causal repair +`72c64b46d120ee8c2f3f12ad04114e6a896cb15b` parses all `LIST` message numbers, +sorts them numerically, and retrieves only the highest bounded window. Invalid +list entries remain ignored. This changes collection progress only; it does not +alter duplicate identity, retention, or server-side deletion semantics. + +The selection is deliberately based on the POP3 session's message-number +ordering rather than pretending message numbers are durable identifiers. They +are used only to choose which messages to retrieve in the current locked +maildrop; persistent identity continues to come from source/provenance evidence. + ## Verification contract - A valid sender `Date` may seed the reviewed strong fingerprint only when @@ -129,6 +151,8 @@ identity. mapping keys, and non-finite numbers instead of coercing them with `str()`. - IMAP and POP3 pass source bytes through the persistence boundary. - POP3 source reconstruction restores CRLF after every `RETR` message line. +- A POP3 maildrop larger than the bounded fetch cap selects the highest current + message numbers so repeated polling cannot be trapped on the oldest window. - Existing rows remain conservatively classified when provenance is unknown. - `0018_email_date_provenance` remains the sole canonical provenance migration; no parallel `date_evidence` or `message_id_evidence` schema is accepted. From 64baa1e2b71e192d14743bd11da017b4fa33279f Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 17:41:17 +0900 Subject: [PATCH 60/86] test(email): preserve POP3 retrieval on QUIT failure --- backend/tests/test_pop3_quit_resilience.py | 35 ++++++++++++++++++++++ 1 file changed, 35 insertions(+) create mode 100644 backend/tests/test_pop3_quit_resilience.py diff --git a/backend/tests/test_pop3_quit_resilience.py b/backend/tests/test_pop3_quit_resilience.py new file mode 100644 index 000000000..2edc459c6 --- /dev/null +++ b/backend/tests/test_pop3_quit_resilience.py @@ -0,0 +1,35 @@ +import poplib +from unittest.mock import MagicMock + +from db.models import TenantConfig +from services.pop3_worker import Pop3SyncWorker + + +def test_pop3_quit_failure_does_not_discard_retrieved_messages(monkeypatch): + worker = Pop3SyncWorker() + config = TenantConfig( + user_id="pop3-user", + pop3_server="pop3.example.com", + pop3_port=995, + pop3_username="pop3-user@example.com", + pop3_password="pop3-secret", + ) + raw_lines = [b"Message-ID: ", b"", b"Body"] + pop3_client = MagicMock() + pop3_client.list.return_value = (b"+OK", [b"1 128"], 128) + pop3_client.retr.return_value = (b"+OK", raw_lines, 128) + pop3_client.quit.side_effect = poplib.error_proto("-ERR connection already closed") + + monkeypatch.setattr( + "services.pop3_worker.validate_pop3_destination", + lambda host, port: (host, port), + ) + monkeypatch.setattr( + "services.pop3_worker.poplib.POP3_SSL", + lambda host, port: pop3_client, + ) + + messages = worker._do_pop3_sync(config) + + assert messages == [b"Message-ID: \r\n\r\nBody\r\n"] + pop3_client.quit.assert_called_once() From fa06c566dc29f350ac1e7ff2888ac59bbf5c7ef4 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 17:41:42 +0900 Subject: [PATCH 61/86] fix(email): keep POP3 retrieval when QUIT cleanup fails --- backend/services/pop3_worker.py | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/backend/services/pop3_worker.py b/backend/services/pop3_worker.py index 393164f4a..299de2bc8 100644 --- a/backend/services/pop3_worker.py +++ b/backend/services/pop3_worker.py @@ -188,7 +188,18 @@ def _do_pop3_sync( messages.append(self._message_bytes(lines)) return messages finally: - pop3_client.quit() + try: + pop3_client.quit() + except (OSError, poplib.error_proto) as exc: + # RETR has already produced immutable message bytes at this point. + # A transport/protocol failure during QUIT is cleanup failure, not + # evidence that those bytes disappeared; keep them available for + # the persistence/dedupe boundary instead of masking the result. + logger.warning( + "POP3 QUIT cleanup failed for user %s: %s", + config.user_id, + type(exc).__name__, + ) def _message_bytes(self, lines: list[bytes | str]) -> bytes: """Reconstruct one POP3 RETR message with protocol CRLF terminators. From a74e02b76e236482f2c634a0c54f38450a63b769 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 17:42:09 +0900 Subject: [PATCH 62/86] test(email): require POP3 socket close after QUIT failure --- backend/tests/test_pop3_quit_resilience.py | 1 + 1 file changed, 1 insertion(+) diff --git a/backend/tests/test_pop3_quit_resilience.py b/backend/tests/test_pop3_quit_resilience.py index 2edc459c6..ccfc77408 100644 --- a/backend/tests/test_pop3_quit_resilience.py +++ b/backend/tests/test_pop3_quit_resilience.py @@ -33,3 +33,4 @@ def test_pop3_quit_failure_does_not_discard_retrieved_messages(monkeypatch): assert messages == [b"Message-ID: \r\n\r\nBody\r\n"] pop3_client.quit.assert_called_once() + pop3_client.close.assert_called_once() From e809575329ff9b9643d7ce93a28556951cdc797a Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 17:42:34 +0900 Subject: [PATCH 63/86] fix(email): close POP3 transport after QUIT failure --- backend/services/pop3_worker.py | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/backend/services/pop3_worker.py b/backend/services/pop3_worker.py index 299de2bc8..e143ced95 100644 --- a/backend/services/pop3_worker.py +++ b/backend/services/pop3_worker.py @@ -191,15 +191,23 @@ def _do_pop3_sync( try: pop3_client.quit() except (OSError, poplib.error_proto) as exc: - # RETR has already produced immutable message bytes at this point. - # A transport/protocol failure during QUIT is cleanup failure, not - # evidence that those bytes disappeared; keep them available for - # the persistence/dedupe boundary instead of masking the result. + # `poplib.quit()` only closes its file/socket after a successful + # QUIT response. Preserve already-retrieved bytes, but explicitly + # close the transport when QUIT itself fails so the maildrop lock + # and local socket are not left to garbage collection. logger.warning( "POP3 QUIT cleanup failed for user %s: %s", config.user_id, type(exc).__name__, ) + try: + pop3_client.close() + except OSError as close_exc: + logger.warning( + "POP3 transport close failed for user %s: %s", + config.user_id, + type(close_exc).__name__, + ) def _message_bytes(self, lines: list[bytes | str]) -> bytes: """Reconstruct one POP3 RETR message with protocol CRLF terminators. From b440e365415c39b8a857e4fbe921fff5ecddb256 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 17:46:06 +0900 Subject: [PATCH 64/86] docs(email): record POP3 teardown and UIDL progress gap --- .../email-source-identity-provenance.md | 59 ++++++++++++++----- 1 file changed, 44 insertions(+), 15 deletions(-) diff --git a/docs/doctoring/email-source-identity-provenance.md b/docs/doctoring/email-source-identity-provenance.md index 0049d6655..b71d5a132 100644 --- a/docs/doctoring/email-source-identity-provenance.md +++ b/docs/doctoring/email-source-identity-provenance.md @@ -112,22 +112,48 @@ identity. RFC 1939 assigns message number `1` to the first message in the opened maildrop and number `n` to the nth message. Naruon's POP3 worker intentionally does not issue `DELE`; source retention is therefore independent from synchronization. -With a bounded fetch cap, repeatedly taking the first ten `LIST` entries would -re-read the same oldest window on every poll and could permanently starve later -mail in a maildrop larger than the cap. +The predecessor implementation took the first ten `LIST` entries, so a maildrop +larger than the cap could repeatedly revisit its oldest window while newer mail +was never retrieved. Source-order regression `517ba20f2409012eb28b9c84085103a7c1b04eaa` -requires the bounded POP3 fetch to select the ten highest valid message numbers -from a twelve-message maildrop. Causal repair +requires that predecessor failure mode to be removed. Causal repair `72c64b46d120ee8c2f3f12ad04114e6a896cb15b` parses all `LIST` message numbers, -sorts them numerically, and retrieves only the highest bounded window. Invalid -list entries remain ignored. This changes collection progress only; it does not -alter duplicate identity, retention, or server-side deletion semantics. - -The selection is deliberately based on the POP3 session's message-number -ordering rather than pretending message numbers are durable identifiers. They -are used only to choose which messages to retrieve in the current locked -maildrop; persistent identity continues to come from source/provenance evidence. +sorts them numerically, and retrieves the highest bounded window. Invalid list +entries remain ignored. This changes collection priority only; it does not alter +duplicate identity, retention, or server-side deletion semantics. + +That repair is deliberately classified as **partial progress**, not an eventual +backlog guarantee. A static maildrop larger than the cap can still expose the +same highest-numbered window on every poll, leaving older unobserved messages +behind indefinitely. POP3 message numbers are session/maildrop positions, not a +sound durable cross-session cursor. Issue #1717 owns the remaining contract: use +RFC 1939 `UIDL` where supported, persist owner-scoped provider progress, and +prove bounded multi-poll/restart progress without turning provider identity into +Naruon's Message-ID or source-fingerprint truth. + +## POP3 session teardown + +A successful `RETR` has already returned protocol-visible message bytes before +`QUIT` is attempted. A later transport/protocol error during `QUIT` is cleanup +failure; it must not retroactively discard those bytes and make the sync report a +retrieval failure. Naruon does not issue `DELE`, so preserving the retrieved +bytes does not authorize or imply server-side deletion. + +The source-order repair is: + +- RED `64baa1e2b71e192d14743bd11da017b4fa33279f` requires a successful `RETR` + result to survive a `poplib.error_proto` raised by `QUIT`; +- strengthened RED `a74e02b76e236482f2c634a0c54f38450a63b769` also requires an explicit + transport close when the graceful `QUIT` path fails; +- repair `fa06c566dc29f350ac1e7ff2888ac59bbf5c7ef4` stops a `QUIT` cleanup + exception from masking already retrieved bytes; +- lifecycle repair `e809575329ff9b9643d7ce93a28556951cdc797a` closes the `poplib` + transport explicitly after failed `QUIT`, with a bounded warning if close + itself fails. + +This keeps the network lifecycle outside the persistence transaction: RETR and +session cleanup complete before `_import_messages()` opens its database session. ## Verification contract @@ -151,8 +177,11 @@ maildrop; persistent identity continues to come from source/provenance evidence. mapping keys, and non-finite numbers instead of coercing them with `str()`. - IMAP and POP3 pass source bytes through the persistence boundary. - POP3 source reconstruction restores CRLF after every `RETR` message line. -- A POP3 maildrop larger than the bounded fetch cap selects the highest current - message numbers so repeated polling cannot be trapped on the oldest window. +- A POP3 `QUIT` failure after successful `RETR` cannot discard the retrieved + bytes, and the transport is explicitly closed when graceful teardown fails. +- Highest-number bounded POP3 selection removes the predecessor oldest-window + failure mode but is **not** accepted as eventual-backlog progress; #1717 owns + the durable UIDL-backed completion contract. - Existing rows remain conservatively classified when provenance is unknown. - `0018_email_date_provenance` remains the sole canonical provenance migration; no parallel `date_evidence` or `message_id_evidence` schema is accepted. From 77d7f23ff603821d3247ac48b88247aaa1c70e8e Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 17:49:25 +0900 Subject: [PATCH 65/86] test(email): define durable POP3 UIDL progress contract --- backend/tests/test_pop3_uidl_progress.py | 149 +++++++++++++++++++++++ 1 file changed, 149 insertions(+) create mode 100644 backend/tests/test_pop3_uidl_progress.py diff --git a/backend/tests/test_pop3_uidl_progress.py b/backend/tests/test_pop3_uidl_progress.py new file mode 100644 index 000000000..f88913b20 --- /dev/null +++ b/backend/tests/test_pop3_uidl_progress.py @@ -0,0 +1,149 @@ +import poplib +from types import SimpleNamespace +from unittest.mock import MagicMock + +import pytest + +from db.models import TenantConfig +from services.pop3_worker import Pop3RetrievedMessage, Pop3SyncWorker + + +def _config() -> TenantConfig: + return TenantConfig( + id=42, + user_id="pop3-user", + organization_id="org-pop3", + pop3_server="pop3.example.com", + pop3_port=995, + pop3_username="pop3-user@example.com", + pop3_password="pop3-secret", + ) + + +def _client_with_uidls(count: int = 12) -> MagicMock: + client = MagicMock() + client.uidl.return_value = ( + b"+OK", + [f"{message_number} uid-{message_number}".encode() for message_number in range(1, count + 1)], + count * 16, + ) + client.list.return_value = ( + b"+OK", + [f"{message_number} 128".encode() for message_number in range(1, count + 1)], + count * 128, + ) + client.retr.side_effect = lambda message_number: ( + b"+OK", + [f"Message-ID: ".encode(), b"", b"Body"], + 128, + ) + return client + + +def test_pop3_uidl_progress_selects_older_unseen_after_newer_window(monkeypatch): + worker = Pop3SyncWorker() + client = _client_with_uidls() + monkeypatch.setattr("services.pop3_worker.poplib.POP3_SSL", lambda host, port: client) + + observed_uidls = {f"uid-{message_number}" for message_number in range(3, 13)} + messages = worker._do_pop3_sync( + _config(), + "pop3.example.com", + 995, + observed_uidls=observed_uidls, + ) + + assert [entry.args[0] for entry in client.retr.call_args_list] == [1, 2] + assert [message.provider_uidl for message in messages] == ["uid-1", "uid-2"] + + +def test_pop3_uidl_progress_uses_current_session_number_after_renumbering(monkeypatch): + worker = Pop3SyncWorker() + client = MagicMock() + client.uidl.return_value = ( + b"+OK", + [b"1 uid-old-2", b"2 uid-old-3", b"3 uid-new-4"], + 48, + ) + client.list.return_value = (b"+OK", [b"1 128", b"2 128", b"3 128"], 384) + client.retr.side_effect = lambda message_number: ( + b"+OK", + [f"Message-ID: ".encode(), b"", b"Body"], + 128, + ) + monkeypatch.setattr("services.pop3_worker.poplib.POP3_SSL", lambda host, port: client) + + messages = worker._do_pop3_sync( + _config(), + "pop3.example.com", + 995, + observed_uidls={"uid-old-2", "uid-old-3"}, + ) + + client.retr.assert_called_once_with(3) + assert messages[0].provider_uidl == "uid-new-4" + + +def test_pop3_uidl_unsupported_falls_back_without_claiming_durable_identity(monkeypatch): + worker = Pop3SyncWorker() + client = _client_with_uidls() + client.uidl.side_effect = poplib.error_proto("-ERR UIDL unsupported") + monkeypatch.setattr("services.pop3_worker.poplib.POP3_SSL", lambda host, port: client) + + messages = worker._do_pop3_sync( + _config(), + "pop3.example.com", + 995, + observed_uidls={"uid-3"}, + ) + + assert [entry.args[0] for entry in client.retr.call_args_list] == list(range(3, 13)) + assert all(message.provider_uidl is None for message in messages) + + +@pytest.mark.asyncio +async def test_import_records_uidl_after_successful_persistence(monkeypatch): + worker = Pop3SyncWorker() + config = _config() + raw_message = ( + b"Message-ID: \r\n" + b"From: Sender \r\n" + b"To: pop3-user@example.com\r\n" + b"Subject: UIDL progress\r\n" + b"Date: Mon, 15 Jun 2026 10:00:00 +0000\r\n\r\nBody\r\n" + ) + executed = [] + + class FakeSession: + async def __aenter__(self): + return self + + async def __aexit__(self, exc_type, exc, tb): + return False + + async def execute(self, statement): + executed.append(statement) + return SimpleNamespace() + + async def commit(self): + return None + + async def rollback(self): + return None + + async def fake_persist(*args, **kwargs): + return SimpleNamespace(created_record=True) + + monkeypatch.setattr("services.pop3_worker.AsyncSessionLocal", lambda: FakeSession()) + monkeypatch.setattr("services.pop3_worker.persist_fetched_email", fake_persist) + + imported_count = await worker._import_messages( + config, + [Pop3RetrievedMessage(source_content=raw_message, provider_uidl="uid-1")], + ) + + assert imported_count == 1 + assert len(executed) == 1 + compiled = str(executed[0].compile(compile_kwargs={"literal_binds": True})) + assert "pop3_observed_messages" in compiled + assert "uid-1" in compiled From b5344c498ccacaa0b8900be222c6a61bda6e13fe Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 17:49:34 +0900 Subject: [PATCH 66/86] feat(email): model durable POP3 UIDL observations --- backend/db/pop3_collection_models.py | 43 ++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100644 backend/db/pop3_collection_models.py diff --git a/backend/db/pop3_collection_models.py b/backend/db/pop3_collection_models.py new file mode 100644 index 000000000..f48cc06f1 --- /dev/null +++ b/backend/db/pop3_collection_models.py @@ -0,0 +1,43 @@ +import datetime + +from sqlalchemy import DateTime, ForeignKey, Index, String, UniqueConstraint +from sqlalchemy.orm import Mapped, mapped_column + +from db.models import Base + + +class Pop3ObservedMessage(Base): + """Durable provider identity for one POP3 message observed by an account. + + POP3 message numbers are session-local positions. ``provider_uidl`` stores + the RFC 1939 unique-id instead so bounded polling can make progress across + reconnects and message-number renumbering without redefining Naruon's email + Message-ID or source-fingerprint identity. + """ + + __tablename__ = "pop3_observed_messages" + __table_args__ = ( + UniqueConstraint( + "tenant_config_id", + "provider_uidl", + name="uq_pop3_observed_messages_account_uidl", + ), + Index( + "ix_pop3_observed_messages_account_observed", + "tenant_config_id", + "observed_at", + ), + ) + + observed_message_id: Mapped[int] = mapped_column(primary_key=True) + tenant_config_id: Mapped[int] = mapped_column( + ForeignKey("tenant_configs.id", ondelete="CASCADE"), + nullable=False, + index=True, + ) + provider_uidl: Mapped[str] = mapped_column(String(70), nullable=False) + observed_at: Mapped[datetime.datetime] = mapped_column( + DateTime(timezone=True), + default=lambda: datetime.datetime.now(datetime.timezone.utc), + nullable=False, + ) From 970c6d3a124cc26dd50a84b0eee58fb951abeef3 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 17:49:46 +0900 Subject: [PATCH 67/86] feat(email): add POP3 UIDL progress migration --- .../versions/0019_pop3_observed_uidl.py | 64 +++++++++++++++++++ 1 file changed, 64 insertions(+) create mode 100644 backend/alembic/versions/0019_pop3_observed_uidl.py diff --git a/backend/alembic/versions/0019_pop3_observed_uidl.py b/backend/alembic/versions/0019_pop3_observed_uidl.py new file mode 100644 index 000000000..c2b6bab14 --- /dev/null +++ b/backend/alembic/versions/0019_pop3_observed_uidl.py @@ -0,0 +1,64 @@ +"""add durable POP3 UIDL observation state + +Revision ID: 0019_pop3_observed_uidl +Revises: 0018_email_date_provenance +Create Date: 2026-09-17 00:00:00.000000 + +Persists RFC 1939 UIDL provider identity per mailbox configuration so bounded +POP3 polling can make progress across reconnects and message-number renumbering. +The UIDL is collection-state identity only; Naruon email Message-ID and source +fingerprints remain the canonical message/deduplication evidence. +""" + +from alembic import op +import sqlalchemy as sa + +revision = "0019_pop3_observed_uidl" +down_revision = "0018_email_date_provenance" + +_TABLE = "pop3_observed_messages" + + +def upgrade() -> None: + """Create owner-scoped durable POP3 UIDL observation state if absent.""" + connection = op.get_bind() + inspector = sa.inspect(connection) + if _TABLE in inspector.get_table_names(): + return + + op.create_table( + _TABLE, + sa.Column("observed_message_id", sa.Integer(), primary_key=True), + sa.Column( + "tenant_config_id", + sa.Integer(), + sa.ForeignKey("tenant_configs.id", ondelete="CASCADE"), + nullable=False, + ), + sa.Column("provider_uidl", sa.String(length=70), nullable=False), + sa.Column( + "observed_at", + sa.DateTime(timezone=True), + nullable=False, + server_default=sa.func.now(), + ), + sa.UniqueConstraint( + "tenant_config_id", + "provider_uidl", + name="uq_pop3_observed_messages_account_uidl", + ), + ) + op.create_index( + "ix_pop3_observed_messages_account_observed", + _TABLE, + ["tenant_config_id", "observed_at"], + unique=False, + ) + + +def downgrade() -> None: + """Drop durable POP3 UIDL observation state if present.""" + connection = op.get_bind() + inspector = sa.inspect(connection) + if _TABLE in inspector.get_table_names(): + op.drop_table(_TABLE) From 5fe993c1363579e439ee4d74ea0190e964315877 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 17:50:56 +0900 Subject: [PATCH 68/86] feat(email): advance bounded POP3 sync with durable UIDL progress --- backend/services/pop3_worker.py | 201 +++++++++++++++++++++++++++----- 1 file changed, 171 insertions(+), 30 deletions(-) diff --git a/backend/services/pop3_worker.py b/backend/services/pop3_worker.py index e143ced95..371f169df 100644 --- a/backend/services/pop3_worker.py +++ b/backend/services/pop3_worker.py @@ -1,9 +1,14 @@ import asyncio +from dataclasses import dataclass import logging import poplib + from sqlalchemy import select -from db.session import AsyncSessionLocal +from sqlalchemy.dialects.postgresql import insert as pg_insert + from db.models import TenantConfig +from db.pop3_collection_models import Pop3ObservedMessage +from db.session import AsyncSessionLocal from services.email_client import validate_pop3_destination from services.email_parser import parse_eml_bytes from services.exceptions import EmailParseError @@ -13,6 +18,22 @@ MAX_POP3_FETCH_MESSAGES = 10 +@dataclass(frozen=True) +class Pop3MessageIdentity: + """Current-session POP3 number bound to its RFC 1939 durable unique-id.""" + + message_number: int + provider_uidl: str + + +@dataclass(frozen=True) +class Pop3RetrievedMessage: + """Retrieved POP3 source bytes plus optional provider progress identity.""" + + source_content: bytes + provider_uidl: str | None = None + + class Pop3SyncWorker: def __init__(self): self._task = None @@ -86,9 +107,13 @@ async def _sync_tenant(self, config: TenantConfig, semaphore: asyncio.Semaphore) f"Connecting to POP3 server {pop3_server}:{pop3_port} for user {config.user_id}" ) try: - # We use asyncio.to_thread for synchronous poplib + observed_uidls = await self._load_observed_uidls(config) messages = await asyncio.to_thread( - self._do_pop3_sync, config, pop3_server, pop3_port + self._do_pop3_sync, + config, + pop3_server, + pop3_port, + observed_uidls, ) imported_count = await self._import_messages(config, messages) logger.info( @@ -103,8 +128,27 @@ async def _sync_tenant(self, config: TenantConfig, semaphore: asyncio.Semaphore) type(e).__name__, ) + async def _load_observed_uidls(self, config: TenantConfig) -> set[str]: + """Load durable UIDL progress before network I/O begins. + + Unsaved configs used by focused tests have no stable account key and + therefore cannot own durable provider progress. + """ + if config.id is None: + return set() + + async with AsyncSessionLocal() as session: + result = await session.execute( + select(Pop3ObservedMessage.provider_uidl).where( + Pop3ObservedMessage.tenant_config_id == config.id + ) + ) + return set(result.scalars().all()) + async def _import_messages( - self, config: TenantConfig, messages: list[bytes] + self, + config: TenantConfig, + messages: list[Pop3RetrievedMessage | bytes], ) -> int: if not messages: return 0 @@ -113,7 +157,13 @@ async def _import_messages( owner_addresses = [config.pop3_username] if config.pop3_username else None async with AsyncSessionLocal() as session: try: - for raw_message in messages: + for message in messages: + if isinstance(message, bytes): + raw_message = message + provider_uidl = None + else: + raw_message = message.source_content + provider_uidl = message.provider_uidl try: email_data = parse_eml_bytes(raw_message) except EmailParseError: @@ -132,6 +182,17 @@ async def _import_messages( ) if persistence_result.created_record: imported_count += 1 + if provider_uidl is not None and config.id is not None: + await session.execute( + pg_insert(Pop3ObservedMessage) + .values( + tenant_config_id=config.id, + provider_uidl=provider_uidl, + ) + .on_conflict_do_nothing( + index_elements=["tenant_config_id", "provider_uidl"] + ) + ) await session.commit() except Exception: await session.rollback() @@ -149,10 +210,12 @@ def _do_pop3_sync( config: TenantConfig, pop3_server: str | None = None, pop3_port: int | None = None, - ) -> list[bytes]: + observed_uidls: set[str] | None = None, + ) -> list[Pop3RetrievedMessage]: if pop3_server is None or pop3_port is None: pop3_server, pop3_port = self._validated_destination(config) pop3_client = poplib.POP3_SSL(pop3_server, pop3_port) + observed_uidls = observed_uidls or set() try: if not config.pop3_username: logger.error( @@ -172,6 +235,28 @@ def _do_pop3_sync( ) pop3_client.user(config.pop3_username) pop3_client.pass_(config.pop3_password) + + uidl_identities = self._current_uidl_identities(pop3_client, config) + if uidl_identities is not None: + candidates = [ + identity + for identity in uidl_identities + if identity.provider_uidl not in observed_uidls + ] + selected = sorted( + candidates, + key=lambda identity: identity.message_number, + )[-MAX_POP3_FETCH_MESSAGES:] + return [ + Pop3RetrievedMessage( + source_content=self._retrieve_message( + pop3_client, identity.message_number + ), + provider_uidl=identity.provider_uidl, + ) + for identity in selected + ] + _response, listings, _octets = pop3_client.list() message_numbers = [ message_number @@ -179,35 +264,91 @@ def _do_pop3_sync( if (message_number := self._message_number_from_listing(listing)) is not None ] - messages: list[bytes] = [] - # RFC 1939 numbers the first maildrop message as 1 and the nth as n. - # This worker does not DELE after RETR, so repeatedly taking the first - # bounded window would starve later arrivals in maildrops over the cap. - for message_number in sorted(message_numbers)[-MAX_POP3_FETCH_MESSAGES:]: - _retr_response, lines, _retr_octets = pop3_client.retr(message_number) - messages.append(self._message_bytes(lines)) - return messages + logger.warning( + "POP3 UIDL unavailable for user %s; bounded fallback cannot prove durable backlog progress.", + config.user_id, + ) + return [ + Pop3RetrievedMessage( + source_content=self._retrieve_message(pop3_client, message_number), + provider_uidl=None, + ) + for message_number in sorted(message_numbers)[-MAX_POP3_FETCH_MESSAGES:] + ] finally: + self._close_pop3_client(pop3_client, config) + + def _current_uidl_identities( + self, + pop3_client: poplib.POP3_SSL, + config: TenantConfig, + ) -> list[Pop3MessageIdentity] | None: + try: + _response, listings, _octets = pop3_client.uidl() + except poplib.error_proto: + return None + + identities: list[Pop3MessageIdentity] = [] + for listing in listings: + identity = self._uidl_identity_from_listing(listing) + if identity is None: + logger.warning( + "POP3 UIDL response contained an invalid entry for user %s; durable progress is unavailable for this poll.", + config.user_id, + ) + return None + identities.append(identity) + return identities + + def _uidl_identity_from_listing( + self, listing: bytes | str + ) -> Pop3MessageIdentity | None: + try: + raw_listing = listing.decode("ascii") if isinstance(listing, bytes) else listing + except UnicodeDecodeError: + return None + parts = raw_listing.strip().split() + if len(parts) != 2 or not parts[0].isdigit(): + return None + provider_uidl = parts[1] + if not 1 <= len(provider_uidl) <= 70: + return None + if any(not 0x21 <= ord(character) <= 0x7E for character in provider_uidl): + return None + return Pop3MessageIdentity( + message_number=int(parts[0]), + provider_uidl=provider_uidl, + ) + + def _retrieve_message(self, pop3_client: poplib.POP3_SSL, message_number: int) -> bytes: + _retr_response, lines, _retr_octets = pop3_client.retr(message_number) + return self._message_bytes(lines) + + def _close_pop3_client( + self, + pop3_client: poplib.POP3_SSL, + config: TenantConfig, + ) -> None: + try: + pop3_client.quit() + except (OSError, poplib.error_proto) as exc: + # `poplib.quit()` only closes its file/socket after a successful + # QUIT response. Preserve already-retrieved bytes, but explicitly + # close the transport when QUIT itself fails so the maildrop lock + # and local socket are not left to garbage collection. + logger.warning( + "POP3 QUIT cleanup failed for user %s: %s", + config.user_id, + type(exc).__name__, + ) try: - pop3_client.quit() - except (OSError, poplib.error_proto) as exc: - # `poplib.quit()` only closes its file/socket after a successful - # QUIT response. Preserve already-retrieved bytes, but explicitly - # close the transport when QUIT itself fails so the maildrop lock - # and local socket are not left to garbage collection. + pop3_client.close() + except OSError as close_exc: logger.warning( - "POP3 QUIT cleanup failed for user %s: %s", + "POP3 transport close failed for user %s: %s", config.user_id, - type(exc).__name__, + type(close_exc).__name__, ) - try: - pop3_client.close() - except OSError as close_exc: - logger.warning( - "POP3 transport close failed for user %s: %s", - config.user_id, - type(close_exc).__name__, - ) def _message_bytes(self, lines: list[bytes | str]) -> bytes: """Reconstruct one POP3 RETR message with protocol CRLF terminators. From 0ce22279a452eae576cb3cc599977a50a5c6b5f2 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 17:51:29 +0900 Subject: [PATCH 69/86] test(email): align QUIT regression with UIDL retrieval result --- backend/tests/test_pop3_quit_resilience.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/backend/tests/test_pop3_quit_resilience.py b/backend/tests/test_pop3_quit_resilience.py index ccfc77408..7652696a7 100644 --- a/backend/tests/test_pop3_quit_resilience.py +++ b/backend/tests/test_pop3_quit_resilience.py @@ -16,7 +16,7 @@ def test_pop3_quit_failure_does_not_discard_retrieved_messages(monkeypatch): ) raw_lines = [b"Message-ID: ", b"", b"Body"] pop3_client = MagicMock() - pop3_client.list.return_value = (b"+OK", [b"1 128"], 128) + pop3_client.uidl.return_value = (b"+OK", [b"1 uid-1"], 16) pop3_client.retr.return_value = (b"+OK", raw_lines, 128) pop3_client.quit.side_effect = poplib.error_proto("-ERR connection already closed") @@ -31,6 +31,7 @@ def test_pop3_quit_failure_does_not_discard_retrieved_messages(monkeypatch): messages = worker._do_pop3_sync(config) - assert messages == [b"Message-ID: \r\n\r\nBody\r\n"] + assert messages[0].source_content == b"Message-ID: \r\n\r\nBody\r\n" + assert messages[0].provider_uidl == "uid-1" pop3_client.quit.assert_called_once() pop3_client.close.assert_called_once() From 3776e5474ec4ae3b5d3d3cadc6513fedfdd0eadf Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 17:52:00 +0900 Subject: [PATCH 70/86] test(email): align POP3 worker contracts with UIDL progress --- backend/tests/test_pop3_worker.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/backend/tests/test_pop3_worker.py b/backend/tests/test_pop3_worker.py index 4824fb6a0..9880d7973 100644 --- a/backend/tests/test_pop3_worker.py +++ b/backend/tests/test_pop3_worker.py @@ -1,4 +1,5 @@ import asyncio +import poplib from types import SimpleNamespace from unittest.mock import MagicMock, patch @@ -84,6 +85,7 @@ def test_pop3_sync_fetches_newest_bounded_message_numbers(monkeypatch): pop3_password="pop3-secret", ) pop3_client = MagicMock() + pop3_client.uidl.side_effect = poplib.error_proto("-ERR UIDL unsupported") pop3_client.list.return_value = ( b"+OK", [f"{message_number} 128".encode() for message_number in range(1, 13)], @@ -110,6 +112,7 @@ def test_pop3_sync_fetches_newest_bounded_message_numbers(monkeypatch): range(3, 13) ) assert len(messages) == 10 + assert all(message.provider_uidl is None for message in messages) pop3_client.quit.assert_called_once() @@ -149,7 +152,7 @@ async def test_pop3_worker_imports_retrieved_messages(monkeypatch): b"Imported from POP3.\r\n" ) pop3_client = MagicMock() - pop3_client.list.return_value = (b"+OK", [b"1 128"], 128) + pop3_client.uidl.return_value = (b"+OK", [b"1 uid-1"], 16) pop3_client.retr.return_value = (b"+OK", raw_message.splitlines(), len(raw_message)) imported: list[dict[str, object]] = [] @@ -213,7 +216,8 @@ async def fake_persist_fetched_email( pop3_client.user.assert_called_once_with("pop3-user@example.com") pop3_client.pass_.assert_called_once_with("pop3-secret") - pop3_client.list.assert_called_once() + pop3_client.uidl.assert_called_once() + pop3_client.list.assert_not_called() pop3_client.retr.assert_called_once_with(1) pop3_client.quit.assert_called_once() assert len(imported) == 1 From c5a997615ab8c4ad3e23463199b2ca98a7a5795d Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 17:52:11 +0900 Subject: [PATCH 71/86] test(email): pin POP3 UIDL migration contract --- .../test_pop3_uidl_migration_contract.py | 34 +++++++++++++++++++ 1 file changed, 34 insertions(+) create mode 100644 backend/tests/test_pop3_uidl_migration_contract.py diff --git a/backend/tests/test_pop3_uidl_migration_contract.py b/backend/tests/test_pop3_uidl_migration_contract.py new file mode 100644 index 000000000..14ef35c93 --- /dev/null +++ b/backend/tests/test_pop3_uidl_migration_contract.py @@ -0,0 +1,34 @@ +from pathlib import Path + +from db.pop3_collection_models import Pop3ObservedMessage + + +MIGRATION_PATH = ( + Path(__file__).resolve().parents[1] + / "alembic" + / "versions" + / "0019_pop3_observed_uidl.py" +) + + +def test_pop3_uidl_model_is_owner_scoped_and_bounded(): + table = Pop3ObservedMessage.__table__ + + assert table.name == "pop3_observed_messages" + assert table.c.tenant_config_id.nullable is False + assert table.c.provider_uidl.nullable is False + assert table.c.provider_uidl.type.length == 70 + assert any( + constraint.name == "uq_pop3_observed_messages_account_uidl" + for constraint in table.constraints + ) + + +def test_pop3_uidl_migration_succeeds_canonical_provenance_revision(): + source = MIGRATION_PATH.read_text(encoding="utf-8") + + assert 'revision = "0019_pop3_observed_uidl"' in source + assert 'down_revision = "0018_email_date_provenance"' in source + assert 'sa.ForeignKey("tenant_configs.id", ondelete="CASCADE")' in source + assert 'sa.String(length=70)' in source + assert '"uq_pop3_observed_messages_account_uidl"' in source From 99e2e7ea6a3f5d79c546afa11f115b253407319a Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 17:52:48 +0900 Subject: [PATCH 72/86] fix(email): align POP3 UIDL model index with migration --- backend/db/pop3_collection_models.py | 1 - 1 file changed, 1 deletion(-) diff --git a/backend/db/pop3_collection_models.py b/backend/db/pop3_collection_models.py index f48cc06f1..26d78d064 100644 --- a/backend/db/pop3_collection_models.py +++ b/backend/db/pop3_collection_models.py @@ -33,7 +33,6 @@ class Pop3ObservedMessage(Base): tenant_config_id: Mapped[int] = mapped_column( ForeignKey("tenant_configs.id", ondelete="CASCADE"), nullable=False, - index=True, ) provider_uidl: Mapped[str] = mapped_column(String(70), nullable=False) observed_at: Mapped[datetime.datetime] = mapped_column( From c762b10d12e55d6666a772d99be49172bc35230a Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 17:53:06 +0900 Subject: [PATCH 73/86] test(email): preserve partial POP3 RETR progress --- backend/tests/test_pop3_partial_retrieval.py | 35 ++++++++++++++++++++ 1 file changed, 35 insertions(+) create mode 100644 backend/tests/test_pop3_partial_retrieval.py diff --git a/backend/tests/test_pop3_partial_retrieval.py b/backend/tests/test_pop3_partial_retrieval.py new file mode 100644 index 000000000..4875e7c14 --- /dev/null +++ b/backend/tests/test_pop3_partial_retrieval.py @@ -0,0 +1,35 @@ +import poplib +from unittest.mock import MagicMock + +from db.models import TenantConfig +from services.pop3_worker import Pop3SyncWorker + + +def test_pop3_retr_failure_preserves_already_retrieved_uidl_messages(monkeypatch): + worker = Pop3SyncWorker() + config = TenantConfig( + user_id="pop3-user", + pop3_server="pop3.example.com", + pop3_port=995, + pop3_username="pop3-user@example.com", + pop3_password="pop3-secret", + ) + client = MagicMock() + client.uidl.return_value = (b"+OK", [b"1 uid-1", b"2 uid-2"], 32) + client.retr.side_effect = [ + (b"+OK", [b"Message-ID: ", b"", b"Body one"], 128), + poplib.error_proto("-ERR retrieval failed"), + ] + monkeypatch.setattr("services.pop3_worker.poplib.POP3_SSL", lambda host, port: client) + + messages = worker._do_pop3_sync( + config, + "pop3.example.com", + 995, + observed_uidls=set(), + ) + + assert len(messages) == 1 + assert messages[0].provider_uidl == "uid-1" + assert b"" in messages[0].source_content + assert [entry.args[0] for entry in client.retr.call_args_list] == [1, 2] From 37a390c59c915016fa2e412c6e77e85e69042510 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 17:53:44 +0900 Subject: [PATCH 74/86] fix(email): preserve partial POP3 retrieval progress --- backend/services/pop3_worker.py | 95 +++++++++++++++++++++------------ 1 file changed, 60 insertions(+), 35 deletions(-) diff --git a/backend/services/pop3_worker.py b/backend/services/pop3_worker.py index 371f169df..57f83f714 100644 --- a/backend/services/pop3_worker.py +++ b/backend/services/pop3_worker.py @@ -43,7 +43,6 @@ async def start(self): if self._is_running: logger.warning("Pop3SyncWorker is already running.") return - self._is_running = True self._task = asyncio.create_task(self._run_loop()) logger.info("Pop3SyncWorker started.") @@ -51,7 +50,6 @@ async def start(self): async def stop(self): if not self._is_running: return - self._is_running = False if self._task: self._task.cancel() @@ -69,7 +67,6 @@ async def _run_loop(self): break except Exception as e: logger.error(f"Error in Pop3SyncWorker loop: {e}", exc_info=True) - if self._is_running: try: await asyncio.sleep(60) @@ -89,7 +86,6 @@ async def _sync(self): if not config.pop3_server or not config.pop3_port: continue tasks.append(self._sync_tenant(config, semaphore)) - if tasks: await asyncio.gather(*tasks, return_exceptions=True) @@ -129,14 +125,9 @@ async def _sync_tenant(self, config: TenantConfig, semaphore: asyncio.Semaphore) ) async def _load_observed_uidls(self, config: TenantConfig) -> set[str]: - """Load durable UIDL progress before network I/O begins. - - Unsaved configs used by focused tests have no stable account key and - therefore cannot own durable provider progress. - """ + """Load durable UIDL progress before network I/O begins.""" if config.id is None: return set() - async with AsyncSessionLocal() as session: result = await session.execute( select(Pop3ObservedMessage.provider_uidl).where( @@ -247,15 +238,7 @@ def _do_pop3_sync( candidates, key=lambda identity: identity.message_number, )[-MAX_POP3_FETCH_MESSAGES:] - return [ - Pop3RetrievedMessage( - source_content=self._retrieve_message( - pop3_client, identity.message_number - ), - provider_uidl=identity.provider_uidl, - ) - for identity in selected - ] + return self._retrieve_uidl_messages(pop3_client, config, selected) _response, listings, _octets = pop3_client.list() message_numbers = [ @@ -268,13 +251,11 @@ def _do_pop3_sync( "POP3 UIDL unavailable for user %s; bounded fallback cannot prove durable backlog progress.", config.user_id, ) - return [ - Pop3RetrievedMessage( - source_content=self._retrieve_message(pop3_client, message_number), - provider_uidl=None, - ) - for message_number in sorted(message_numbers)[-MAX_POP3_FETCH_MESSAGES:] - ] + return self._retrieve_fallback_messages( + pop3_client, + config, + sorted(message_numbers)[-MAX_POP3_FETCH_MESSAGES:], + ) finally: self._close_pop3_client(pop3_client, config) @@ -320,6 +301,58 @@ def _uidl_identity_from_listing( provider_uidl=provider_uidl, ) + def _retrieve_uidl_messages( + self, + pop3_client: poplib.POP3_SSL, + config: TenantConfig, + identities: list[Pop3MessageIdentity], + ) -> list[Pop3RetrievedMessage]: + messages: list[Pop3RetrievedMessage] = [] + for identity in identities: + try: + source_content = self._retrieve_message( + pop3_client, identity.message_number + ) + except (OSError, poplib.error_proto) as exc: + logger.warning( + "POP3 RETR stopped after partial progress for user %s: %s", + config.user_id, + type(exc).__name__, + ) + break + messages.append( + Pop3RetrievedMessage( + source_content=source_content, + provider_uidl=identity.provider_uidl, + ) + ) + return messages + + def _retrieve_fallback_messages( + self, + pop3_client: poplib.POP3_SSL, + config: TenantConfig, + message_numbers: list[int], + ) -> list[Pop3RetrievedMessage]: + messages: list[Pop3RetrievedMessage] = [] + for message_number in message_numbers: + try: + source_content = self._retrieve_message(pop3_client, message_number) + except (OSError, poplib.error_proto) as exc: + logger.warning( + "POP3 fallback RETR stopped after partial progress for user %s: %s", + config.user_id, + type(exc).__name__, + ) + break + messages.append( + Pop3RetrievedMessage( + source_content=source_content, + provider_uidl=None, + ) + ) + return messages + def _retrieve_message(self, pop3_client: poplib.POP3_SSL, message_number: int) -> bytes: _retr_response, lines, _retr_octets = pop3_client.retr(message_number) return self._message_bytes(lines) @@ -332,10 +365,6 @@ def _close_pop3_client( try: pop3_client.quit() except (OSError, poplib.error_proto) as exc: - # `poplib.quit()` only closes its file/socket after a successful - # QUIT response. Preserve already-retrieved bytes, but explicitly - # close the transport when QUIT itself fails so the maildrop lock - # and local socket are not left to garbage collection. logger.warning( "POP3 QUIT cleanup failed for user %s: %s", config.user_id, @@ -351,11 +380,7 @@ def _close_pop3_client( ) def _message_bytes(self, lines: list[bytes | str]) -> bytes: - """Reconstruct one POP3 RETR message with protocol CRLF terminators. - - ``poplib`` removes line terminators from the multiline response while - RFC 1939 defines each transferred message line as CRLF-terminated. - """ + """Reconstruct one POP3 RETR message with protocol CRLF terminators.""" return b"\r\n".join(self._bytes_line(line) for line in lines) + b"\r\n" def _message_number_from_listing(self, listing: bytes | str) -> int | None: From 3c734b54112b241c12f9d4a35335caba2e089a53 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 17:54:58 +0900 Subject: [PATCH 75/86] docs(email): trace UIDL progress and partial POP3 recovery --- .../email-source-identity-provenance.md | 175 +++++++++++------- 1 file changed, 106 insertions(+), 69 deletions(-) diff --git a/docs/doctoring/email-source-identity-provenance.md b/docs/doctoring/email-source-identity-provenance.md index b71d5a132..d41b78e0b 100644 --- a/docs/doctoring/email-source-identity-provenance.md +++ b/docs/doctoring/email-source-identity-provenance.md @@ -66,14 +66,14 @@ The source-order sequence for this repair is: repair, including deterministic `dedupe_review_required` result semantics; - `fd4cd4e405b3af30b4458993a4673063350b49d9` → `d8a08deb1ad009b2d90de2eafbab5da61ac0cc79`: IMAP RED then causal repair. - POP3 reaches the same `process_fetched_email` boundary and therefore consumes - the repaired IMAP/POP3 persistence path rather than duplicating the rule. + POP3 consumes the same fetched-email persistence boundary rather than + duplicating dedupe rules. The canonical migration contract is separately pinned by `540f6e4ec57ed355263d56e7da78de7ff5310360`: `0018_email_date_provenance` -remains the single provenance revision, with non-null `date_provenance` and an -`unknown` server default. The parallel #1656 `date_evidence` / -`message_id_evidence` migration is not adopted. +remains the single message-provenance revision, with non-null +`date_provenance` and an `unknown` server default. The parallel #1656 +`date_evidence` / `message_id_evidence` migration is not adopted. ## Message-ID provenance decision @@ -87,9 +87,8 @@ existing Message-ID plus source-fingerprint semantics. Accordingly, #1656's proposed `message_id_evidence` persistence is rejected for this lineage unless a concrete future consumer proves a storage requirement. -This avoids a second provenance vocabulary and a sibling migration merely to -record evidence that is already available at ingestion and encoded by the -source-bound identity contract. +Provider UIDL state is a separate collection-progress concern and must not be +promoted to Naruon's Message-ID or source-fingerprint truth. ## POP3 reconstruction contract @@ -107,53 +106,84 @@ streams. Duplicate classification remains deterministic because the source kind is domain separated and because collection time is excluded from fallback identity. -## POP3 bounded collection window +## POP3 durable bounded progress RFC 1939 assigns message number `1` to the first message in the opened maildrop -and number `n` to the nth message. Naruon's POP3 worker intentionally does not -issue `DELE`; source retention is therefore independent from synchronization. -The predecessor implementation took the first ten `LIST` entries, so a maildrop -larger than the cap could repeatedly revisit its oldest window while newer mail -was never retrieved. - -Source-order regression `517ba20f2409012eb28b9c84085103a7c1b04eaa` -requires that predecessor failure mode to be removed. Causal repair -`72c64b46d120ee8c2f3f12ad04114e6a896cb15b` parses all `LIST` message numbers, -sorts them numerically, and retrieves the highest bounded window. Invalid list -entries remain ignored. This changes collection priority only; it does not alter -duplicate identity, retention, or server-side deletion semantics. - -That repair is deliberately classified as **partial progress**, not an eventual -backlog guarantee. A static maildrop larger than the cap can still expose the -same highest-numbered window on every poll, leaving older unobserved messages -behind indefinitely. POP3 message numbers are session/maildrop positions, not a -sound durable cross-session cursor. Issue #1717 owns the remaining contract: use -RFC 1939 `UIDL` where supported, persist owner-scoped provider progress, and -prove bounded multi-poll/restart progress without turning provider identity into -Naruon's Message-ID or source-fingerprint truth. - -## POP3 session teardown +and number `n` to the nth message, but those positions are not durable client +identity. The predecessor implementation first selected the oldest bounded +`LIST` window; repair `72c64b46d120ee8c2f3f12ad04114e6a896cb15b` switched to the +highest current message numbers and removed that particular newest-mail +starvation mode. It still could not guarantee full-maildrop progress because a +static maildrop larger than the cap would repeat the same tail window. + +Issue #1717 therefore requires RFC 1939 UIDL-based progress. RFC 1939 defines a +UIDL value as a one-to-70-character server-determined identifier in the range +0x21–0x7E that identifies a message within a maildrop and persists across +sessions. Naruon now keeps this provider progress identity separate from email +identity: + +- RED `77d7f23ff603821d3247ac48b88247aaa1c70e8e` defines bounded unseen-UIDL + selection, current-session renumbering, an explicit UIDL-unavailable fallback, + and durable observation after successful persistence; +- model `b5344c498ccacaa0b8900be222c6a61bda6e13fe` introduces owner-scoped + `Pop3ObservedMessage`; migration `970c6d3a124cc26dd50a84b0eee58fb951abeef3` + adds `0019_pop3_observed_uidl` after the canonical `0018` provenance revision; +- worker repair `5fe993c1363579e439ee4d74ea0190e964315877` asks the server for UIDL, + filters already observed provider identities, retrieves at most the newest ten + unseen identities for the current poll, and records the UIDL in the same + database transaction as successful email persistence using an idempotent + `(tenant_config_id, provider_uidl)` conflict boundary; +- migration/model contract tests are pinned by + `c5a997615ab8c4ad3e23463199b2ca98a7a5795d`, and model/index parity is repaired + by `99e2e7ea6a3f5d79c546afa11f115b253407319a`; +- legacy worker tests were adapted at `3776e5474ec4ae3b5d3d3cadc6513fedfdd0eadf` + without converting UIDL into message identity. + +With UIDL support, a first poll can process the newest bounded unseen set and a +later poll filters those durable observations, exposing the next unseen set even +if current-session message numbers were renumbered. The database session used to +load progress is closed before POP3 network retrieval starts; `_import_messages` +opens a separate short persistence transaction only after RETR/session cleanup. +No explicit DB lock is held across provider I/O. + +UIDL is optional in POP3. If the server rejects UIDL or returns a malformed UIDL +listing, Naruon falls back to the bounded highest-number `LIST` window and logs +that durable backlog progress is not proven for that poll. The fallback is a +resource-bounded compatibility policy, not a correctness claim. Naruon still +does not issue `DELE`. + +`pop3_observed_messages` intentionally scopes UIDL by `tenant_config_id` and +keeps historical observations because RFC 1939 requires persistence across +sessions. This makes per-account progress reads proportional to stored provider +history; it is a measurable operability/performance surface and must be profiled +before any claim about very large long-lived POP3 mailboxes. + +## POP3 partial retrieval and session teardown A successful `RETR` has already returned protocol-visible message bytes before -`QUIT` is attempted. A later transport/protocol error during `QUIT` is cleanup -failure; it must not retroactively discard those bytes and make the sync report a -retrieval failure. Naruon does not issue `DELE`, so preserving the retrieved -bytes does not authorize or imply server-side deletion. +later message or session cleanup can fail. A later `RETR`/`QUIT` error must not +retroactively discard earlier successful bytes. -The source-order repair is: +The teardown source-order repair is: -- RED `64baa1e2b71e192d14743bd11da017b4fa33279f` requires a successful `RETR` - result to survive a `poplib.error_proto` raised by `QUIT`; +- RED `64baa1e2b71e192d14743bd11da017b4fa33279f` requires successful RETR bytes + to survive a `poplib.error_proto` raised by `QUIT`; - strengthened RED `a74e02b76e236482f2c634a0c54f38450a63b769` also requires an explicit - transport close when the graceful `QUIT` path fails; -- repair `fa06c566dc29f350ac1e7ff2888ac59bbf5c7ef4` stops a `QUIT` cleanup - exception from masking already retrieved bytes; -- lifecycle repair `e809575329ff9b9643d7ce93a28556951cdc797a` closes the `poplib` - transport explicitly after failed `QUIT`, with a bounded warning if close - itself fails. - -This keeps the network lifecycle outside the persistence transaction: RETR and -session cleanup complete before `_import_messages()` opens its database session. + transport close when graceful `QUIT` fails; +- repairs `fa06c566dc29f350ac1e7ff2888ac59bbf5c7ef4` and + `e809575329ff9b9643d7ce93a28556951cdc797a` preserve retrieved bytes and close + the transport explicitly after failed `QUIT`. + +A separate RED `c762b10d12e55d6666a772d99be49172bc35230a` proves that failure on a +later `RETR` must not discard earlier messages in the same bounded batch. Repair +`37a390c59c915016fa2e412c6e77e85e69042510` changes UIDL and fallback retrieval +to accumulate successful messages and stop the batch at the first transport or +protocol retrieval failure. Only returned messages can be persisted and only +their UIDLs can become observed, so the failed identity remains eligible for a +later retry. + +Naruon does not issue `DELE`; preserving already retrieved bytes does not +authorize or imply server-side deletion. ## Verification contract @@ -165,34 +195,41 @@ session cleanup complete before `_import_messages()` opens its database session. explicit `-0000` Date remains parsed and UTC-comparable. - Incomplete metadata cannot produce a strong metadata auto-link; import reports `dedupe_review_required` while retaining source-bound identity. -- Direct import, IMAP, and the POP3 path through `process_fetched_email` apply - the same complete-metadata gate. -- Two different raw messages collected at the same instant remain distinct. -- The same raw message collected at different instants has the same fallback +- Direct import, IMAP, and POP3 apply the same source-bound persistence rule. +- Two different raw messages collected at the same instant remain distinct; the + same raw message collected at different instants has the same fallback identity. -- Canonical fallback identity excludes effective collection timestamps and - provenance flags. -- Canonical fallback serialization accepts only deterministic JSON-native parsed - values and rejects bytes, unordered collections, custom objects, non-string - mapping keys, and non-finite numbers instead of coercing them with `str()`. -- IMAP and POP3 pass source bytes through the persistence boundary. -- POP3 source reconstruction restores CRLF after every `RETR` message line. -- A POP3 `QUIT` failure after successful `RETR` cannot discard the retrieved - bytes, and the transport is explicitly closed when graceful teardown fails. -- Highest-number bounded POP3 selection removes the predecessor oldest-window - failure mode but is **not** accepted as eventual-backlog progress; #1717 owns - the durable UIDL-backed completion contract. -- Existing rows remain conservatively classified when provenance is unknown. -- `0018_email_date_provenance` remains the sole canonical provenance migration; - no parallel `date_evidence` or `message_id_evidence` schema is accepted. +- Canonical fallback identity excludes collection timestamps/provenance flags and + rejects unsupported serialization types instead of coercing them with + `str()`. +- IMAP and POP3 pass source bytes through the persistence boundary; POP3 source + reconstruction restores CRLF after every `RETR` message line. +- With UIDL support, a bounded poll retrieves only unseen provider identities; + persisted observations survive reconnects and current-session renumbering. +- UIDL is transport progress identity only. It cannot replace Message-ID, + source-fingerprint, or dedupe evidence. +- UIDL-unavailable/malformed fallback is explicitly compatibility-only and does + not claim eventual backlog completion. +- A later RETR failure preserves earlier successful messages; a QUIT failure + preserves retrieved bytes and explicitly closes the transport. +- `0018_email_date_provenance` remains the sole message-provenance migration; + `0019_pop3_observed_uidl` succeeds it for provider collection state rather than + introducing parallel message provenance. + +The source-order tests and implementation above are not themselves hosted GREEN +evidence. PostgreSQL migration execution, repository tests/security/coverage, +current-head independent review, restart acceptance, and large-mailbox +performance still require exact-head receipts before #1717 or #1195 can be +accepted complete. ## Claim boundary Hash equality is evidence that the selected source representation is identical; it is not proof that two independently authored real-world communications are the same event. Automatic linkage, clerical review, and distinct-message -outcomes remain separate decisions. No automatic deletion or irreversible -provider action is introduced. +outcomes remain separate decisions. Provider UIDL proves only the server's +maildrop identity contract. No automatic deletion or irreversible provider +action is introduced. ## References From d9c87082355fd133a5f4ed9715c560eaee28b211 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 18:08:31 +0900 Subject: [PATCH 76/86] test(pop3): reproduce RETR negative-response starvation --- backend/tests/test_pop3_partial_retrieval.py | 63 ++++++++++++++++++-- 1 file changed, 59 insertions(+), 4 deletions(-) diff --git a/backend/tests/test_pop3_partial_retrieval.py b/backend/tests/test_pop3_partial_retrieval.py index 4875e7c14..e41f0513a 100644 --- a/backend/tests/test_pop3_partial_retrieval.py +++ b/backend/tests/test_pop3_partial_retrieval.py @@ -5,20 +5,24 @@ from services.pop3_worker import Pop3SyncWorker -def test_pop3_retr_failure_preserves_already_retrieved_uidl_messages(monkeypatch): - worker = Pop3SyncWorker() - config = TenantConfig( +def _config() -> TenantConfig: + return TenantConfig( user_id="pop3-user", pop3_server="pop3.example.com", pop3_port=995, pop3_username="pop3-user@example.com", pop3_password="pop3-secret", ) + + +def test_pop3_transport_failure_preserves_already_retrieved_uidl_messages(monkeypatch): + worker = Pop3SyncWorker() + config = _config() client = MagicMock() client.uidl.return_value = (b"+OK", [b"1 uid-1", b"2 uid-2"], 32) client.retr.side_effect = [ (b"+OK", [b"Message-ID: ", b"", b"Body one"], 128), - poplib.error_proto("-ERR retrieval failed"), + OSError("transport failed"), ] monkeypatch.setattr("services.pop3_worker.poplib.POP3_SSL", lambda host, port: client) @@ -33,3 +37,54 @@ def test_pop3_retr_failure_preserves_already_retrieved_uidl_messages(monkeypatch assert messages[0].provider_uidl == "uid-1" assert b"" in messages[0].source_content assert [entry.args[0] for entry in client.retr.call_args_list] == [1, 2] + + +def test_pop3_negative_retr_skips_failed_uidl_and_continues_same_session(monkeypatch): + worker = Pop3SyncWorker() + config = _config() + client = MagicMock() + client.uidl.return_value = ( + b"+OK", + [b"1 uid-1", b"2 uid-2", b"3 uid-3"], + 48, + ) + client.retr.side_effect = [ + (b"+OK", [b"Message-ID: ", b"", b"Body one"], 128), + poplib.error_proto("-ERR no such message"), + (b"+OK", [b"Message-ID: ", b"", b"Body three"], 128), + ] + monkeypatch.setattr("services.pop3_worker.poplib.POP3_SSL", lambda host, port: client) + + messages = worker._do_pop3_sync( + config, + "pop3.example.com", + 995, + observed_uidls=set(), + ) + + assert [message.provider_uidl for message in messages] == ["uid-1", "uid-3"] + assert [entry.args[0] for entry in client.retr.call_args_list] == [1, 2, 3] + + +def test_pop3_fallback_negative_retr_continues_bounded_window(monkeypatch): + worker = Pop3SyncWorker() + config = _config() + client = MagicMock() + client.uidl.side_effect = poplib.error_proto("-ERR UIDL unsupported") + client.list.return_value = (b"+OK", [b"1 100", b"2 100", b"3 100"], 300) + client.retr.side_effect = [ + (b"+OK", [b"Message-ID: ", b"", b"Body one"], 128), + poplib.error_proto("-ERR no such message"), + (b"+OK", [b"Message-ID: ", b"", b"Body three"], 128), + ] + monkeypatch.setattr("services.pop3_worker.poplib.POP3_SSL", lambda host, port: client) + + messages = worker._do_pop3_sync( + config, + "pop3.example.com", + 995, + observed_uidls=set(), + ) + + assert [message.provider_uidl for message in messages] == [None, None] + assert [entry.args[0] for entry in client.retr.call_args_list] == [1, 2, 3] From 38803cc59be9cee2f72be5f48a9c3c9f1aca402e Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 18:09:07 +0900 Subject: [PATCH 77/86] fix(pop3): continue after per-message RETR rejection --- backend/services/pop3_worker.py | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/backend/services/pop3_worker.py b/backend/services/pop3_worker.py index 57f83f714..05e41b282 100644 --- a/backend/services/pop3_worker.py +++ b/backend/services/pop3_worker.py @@ -313,9 +313,16 @@ def _retrieve_uidl_messages( source_content = self._retrieve_message( pop3_client, identity.message_number ) - except (OSError, poplib.error_proto) as exc: + except poplib.error_proto as exc: logger.warning( - "POP3 RETR stopped after partial progress for user %s: %s", + "POP3 RETR rejected one message for user %s; continuing bounded batch: %s", + config.user_id, + type(exc).__name__, + ) + continue + except OSError as exc: + logger.warning( + "POP3 RETR stopped after transport failure for user %s: %s", config.user_id, type(exc).__name__, ) @@ -338,9 +345,16 @@ def _retrieve_fallback_messages( for message_number in message_numbers: try: source_content = self._retrieve_message(pop3_client, message_number) - except (OSError, poplib.error_proto) as exc: + except poplib.error_proto as exc: + logger.warning( + "POP3 fallback RETR rejected one message for user %s; continuing bounded batch: %s", + config.user_id, + type(exc).__name__, + ) + continue + except OSError as exc: logger.warning( - "POP3 fallback RETR stopped after partial progress for user %s: %s", + "POP3 fallback RETR stopped after transport failure for user %s: %s", config.user_id, type(exc).__name__, ) From 7606387c829e27845633d180642369547d23336b Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 18:09:29 +0900 Subject: [PATCH 78/86] test(pop3): distinguish RETR rejection from malformed protocol --- backend/tests/test_pop3_partial_retrieval.py | 27 ++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/backend/tests/test_pop3_partial_retrieval.py b/backend/tests/test_pop3_partial_retrieval.py index e41f0513a..f55bb4617 100644 --- a/backend/tests/test_pop3_partial_retrieval.py +++ b/backend/tests/test_pop3_partial_retrieval.py @@ -88,3 +88,30 @@ def test_pop3_fallback_negative_retr_continues_bounded_window(monkeypatch): assert [message.provider_uidl for message in messages] == [None, None] assert [entry.args[0] for entry in client.retr.call_args_list] == [1, 2, 3] + + +def test_pop3_malformed_protocol_response_stops_remaining_retrievals(monkeypatch): + worker = Pop3SyncWorker() + config = _config() + client = MagicMock() + client.uidl.return_value = ( + b"+OK", + [b"1 uid-1", b"2 uid-2", b"3 uid-3"], + 48, + ) + client.retr.side_effect = [ + (b"+OK", [b"Message-ID: ", b"", b"Body one"], 128), + poplib.error_proto("unexpected response"), + (b"+OK", [b"Message-ID: ", b"", b"Body three"], 128), + ] + monkeypatch.setattr("services.pop3_worker.poplib.POP3_SSL", lambda host, port: client) + + messages = worker._do_pop3_sync( + config, + "pop3.example.com", + 995, + observed_uidls=set(), + ) + + assert [message.provider_uidl for message in messages] == ["uid-1"] + assert [entry.args[0] for entry in client.retr.call_args_list] == [1, 2] From a5d7d21011cf96548cce18b8119a553123a25ab5 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 18:11:05 +0900 Subject: [PATCH 79/86] fix(pop3): distinguish RETR rejection from protocol failure --- backend/services/pop3_worker.py | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/backend/services/pop3_worker.py b/backend/services/pop3_worker.py index 05e41b282..b7ec9fa94 100644 --- a/backend/services/pop3_worker.py +++ b/backend/services/pop3_worker.py @@ -314,6 +314,13 @@ def _retrieve_uidl_messages( pop3_client, identity.message_number ) except poplib.error_proto as exc: + if not self._is_negative_pop3_response(exc): + logger.warning( + "POP3 RETR stopped after malformed protocol response for user %s: %s", + config.user_id, + type(exc).__name__, + ) + break logger.warning( "POP3 RETR rejected one message for user %s; continuing bounded batch: %s", config.user_id, @@ -346,6 +353,13 @@ def _retrieve_fallback_messages( try: source_content = self._retrieve_message(pop3_client, message_number) except poplib.error_proto as exc: + if not self._is_negative_pop3_response(exc): + logger.warning( + "POP3 fallback RETR stopped after malformed protocol response for user %s: %s", + config.user_id, + type(exc).__name__, + ) + break logger.warning( "POP3 fallback RETR rejected one message for user %s; continuing bounded batch: %s", config.user_id, @@ -371,6 +385,15 @@ def _retrieve_message(self, pop3_client: poplib.POP3_SSL, message_number: int) - _retr_response, lines, _retr_octets = pop3_client.retr(message_number) return self._message_bytes(lines) + def _is_negative_pop3_response(self, error: poplib.error_proto) -> bool: + """Return whether a protocol exception carries an RFC 1939 -ERR reply.""" + if not error.args: + return False + response = error.args[0] + if isinstance(response, bytes): + return response.startswith(b"-ERR") + return str(response).startswith("-ERR") + def _close_pop3_client( self, pop3_client: poplib.POP3_SSL, From fcef1780986bdd0d0986342e71206b303e0dd27c Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 18:11:51 +0900 Subject: [PATCH 80/86] docs(pop3): trace RETR negative-response recovery --- .../email-source-identity-provenance.md | 46 +++++++++++++------ 1 file changed, 33 insertions(+), 13 deletions(-) diff --git a/docs/doctoring/email-source-identity-provenance.md b/docs/doctoring/email-source-identity-provenance.md index d41b78e0b..2d3b1805d 100644 --- a/docs/doctoring/email-source-identity-provenance.md +++ b/docs/doctoring/email-source-identity-provenance.md @@ -161,7 +161,7 @@ before any claim about very large long-lived POP3 mailboxes. ## POP3 partial retrieval and session teardown A successful `RETR` has already returned protocol-visible message bytes before -later message or session cleanup can fail. A later `RETR`/`QUIT` error must not +later message or session cleanup can fail. A later `RETR`/`QUIT` failure must not retroactively discard earlier successful bytes. The teardown source-order repair is: @@ -174,16 +174,34 @@ The teardown source-order repair is: `e809575329ff9b9643d7ce93a28556951cdc797a` preserve retrieved bytes and close the transport explicitly after failed `QUIT`. -A separate RED `c762b10d12e55d6666a772d99be49172bc35230a` proves that failure on a -later `RETR` must not discard earlier messages in the same bounded batch. Repair -`37a390c59c915016fa2e412c6e77e85e69042510` changes UIDL and fallback retrieval -to accumulate successful messages and stop the batch at the first transport or -protocol retrieval failure. Only returned messages can be persisted and only -their UIDLs can become observed, so the failed identity remains eligible for a -later retry. - -Naruon does not issue `DELE`; preserving already retrieved bytes does not -authorize or imply server-side deletion. +Earlier partial-RETR repair `37a390c59c915016fa2e412c6e77e85e69042510` +correctly preserved messages retrieved before a later failure, but treated every +`poplib.error_proto` from `RETR` as a reason to stop the remaining bounded batch. +RFC 1939 permits repeated commands in TRANSACTION state and defines `-ERR no +such message` as a valid negative `RETR` response. A single permanently rejected +message therefore must not starve later selected UIDLs when the session remains +usable. + +The refined source-order repair is: + +- RED `d9c87082355fd133a5f4ed9715c560eaee28b211` requires a standards-conforming + per-message `-ERR` to leave that message unobserved while continuing to later + messages in both UIDL and bounded `LIST` fallback paths; an `OSError` remains a + transport-stop condition; +- initial repair `38803cc59be9cee2f72be5f48a9c3c9f1aca402e` separates POP3 protocol + exceptions from transport exceptions; +- hardening RED `7606387c829e27845633d180642369547d23336b` proves that an unexpected + non-`-ERR` protocol exception must not be treated as a safe per-message + rejection; +- final repair `a5d7d21011cf96548cce18b8119a553123a25ab5` continues only when the + protocol exception carries an RFC-style `-ERR` response and stops the + remaining batch on transport loss or malformed/unexpected protocol response. + +Only successfully returned messages can be persisted and only their UIDLs can +become observed. A rejected UIDL therefore remains eligible for retry, while one +persistent message-level rejection no longer blocks later selected messages in +that poll. Naruon still does not issue `DELE`; preserving or continuing retrieval +does not authorize or imply server-side deletion. ## Verification contract @@ -210,8 +228,10 @@ authorize or imply server-side deletion. source-fingerprint, or dedupe evidence. - UIDL-unavailable/malformed fallback is explicitly compatibility-only and does not claim eventual backlog completion. -- A later RETR failure preserves earlier successful messages; a QUIT failure - preserves retrieved bytes and explicitly closes the transport. +- A per-message RFC-style RETR `-ERR` leaves that identity retryable and does not + starve later selected messages; transport loss or malformed protocol stops the + remaining batch. +- A QUIT failure preserves retrieved bytes and explicitly closes the transport. - `0018_email_date_provenance` remains the sole message-provenance migration; `0019_pop3_observed_uidl` succeeds it for provider collection state rather than introducing parallel message provenance. From 5d40f83ff48e7dd9c3bf6deb93995130c5a1470a Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 19:44:38 +0900 Subject: [PATCH 81/86] test(pop3): reproduce retry-window starvation across polls --- backend/tests/test_pop3_retry_progress.py | 62 +++++++++++++++++++++++ 1 file changed, 62 insertions(+) create mode 100644 backend/tests/test_pop3_retry_progress.py diff --git a/backend/tests/test_pop3_retry_progress.py b/backend/tests/test_pop3_retry_progress.py new file mode 100644 index 000000000..5a154657e --- /dev/null +++ b/backend/tests/test_pop3_retry_progress.py @@ -0,0 +1,62 @@ +import datetime + +from services.pop3_worker import ( + Pop3CollectionProgressState, + Pop3MessageIdentity, + Pop3SyncWorker, +) + + +def _identities(count: int = 12) -> list[Pop3MessageIdentity]: + return [ + Pop3MessageIdentity(message_number=number, provider_uidl=f"uid-{number}") + for number in range(1, count + 1) + ] + + +def test_uidl_selection_prioritizes_never_attempted_backlog_before_due_retries(): + worker = Pop3SyncWorker() + now = datetime.datetime(2026, 9, 17, 10, 40, tzinfo=datetime.timezone.utc) + progress = { + f"uid-{number}": Pop3CollectionProgressState( + disposition="retryable", + retry_after=now - datetime.timedelta(minutes=1), + ) + for number in range(3, 13) + } + + selected = worker._select_uidl_candidates(_identities(), progress, now) + + assert [identity.provider_uidl for identity in selected[:2]] == ["uid-2", "uid-1"] + assert len(selected) == 10 + + +def test_uidl_selection_defers_retryable_identity_until_retry_after(): + worker = Pop3SyncWorker() + now = datetime.datetime(2026, 9, 17, 10, 40, tzinfo=datetime.timezone.utc) + progress = { + "uid-12": Pop3CollectionProgressState( + disposition="retryable", + retry_after=now + datetime.timedelta(minutes=5), + ) + } + + selected = worker._select_uidl_candidates(_identities(), progress, now) + + assert "uid-12" not in {identity.provider_uidl for identity in selected} + assert [identity.provider_uidl for identity in selected[:2]] == ["uid-11", "uid-10"] + + +def test_uidl_selection_never_retries_observed_identity(): + worker = Pop3SyncWorker() + now = datetime.datetime(2026, 9, 17, 10, 40, tzinfo=datetime.timezone.utc) + progress = { + "uid-12": Pop3CollectionProgressState( + disposition="observed", + retry_after=None, + ) + } + + selected = worker._select_uidl_candidates(_identities(), progress, now) + + assert "uid-12" not in {identity.provider_uidl for identity in selected} From 7406cb63923003df2c3a5d21a0104203f78b90bc Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 19:48:53 +0900 Subject: [PATCH 82/86] fix(pop3): persist retry disposition without starving fresh UIDLs --- .../versions/0019_pop3_observed_uidl.py | 32 ++- backend/db/pop3_collection_models.py | 28 ++- backend/services/pop3_worker.py | 232 +++++++++++++++--- backend/tests/test_pop3_retry_progress.py | 55 +++++ .../test_pop3_uidl_migration_contract.py | 16 +- 5 files changed, 305 insertions(+), 58 deletions(-) diff --git a/backend/alembic/versions/0019_pop3_observed_uidl.py b/backend/alembic/versions/0019_pop3_observed_uidl.py index c2b6bab14..126c445bb 100644 --- a/backend/alembic/versions/0019_pop3_observed_uidl.py +++ b/backend/alembic/versions/0019_pop3_observed_uidl.py @@ -1,13 +1,17 @@ -"""add durable POP3 UIDL observation state +"""add durable POP3 UIDL collection progress Revision ID: 0019_pop3_observed_uidl Revises: 0018_email_date_provenance Create Date: 2026-09-17 00:00:00.000000 -Persists RFC 1939 UIDL provider identity per mailbox configuration so bounded -POP3 polling can make progress across reconnects and message-number renumbering. -The UIDL is collection-state identity only; Naruon email Message-ID and source -fingerprints remain the canonical message/deduplication evidence. +Persists RFC 1939 UIDL provider identity and bounded retry disposition per +mailbox configuration. Provider state remains collection progress only; Naruon +email Message-ID and source fingerprints remain canonical message/deduplication +evidence. + +This revision identifier and parent are branch-local until the canonical +workspace/Alembic owner is integrated; #1195 must rechain this schema after the +then-protected migration head before merge. """ from alembic import op @@ -20,7 +24,7 @@ def upgrade() -> None: - """Create owner-scoped durable POP3 UIDL observation state if absent.""" + """Create owner-scoped durable POP3 UIDL collection progress if absent.""" connection = op.get_bind() inspector = sa.inspect(connection) if _TABLE in inspector.get_table_names(): @@ -37,11 +41,13 @@ def upgrade() -> None: ), sa.Column("provider_uidl", sa.String(length=70), nullable=False), sa.Column( - "observed_at", - sa.DateTime(timezone=True), + "collection_disposition", + sa.String(length=16), nullable=False, - server_default=sa.func.now(), + server_default="observed", ), + sa.Column("retry_after", sa.DateTime(timezone=True), nullable=True), + sa.Column("observed_at", sa.DateTime(timezone=True), nullable=True), sa.UniqueConstraint( "tenant_config_id", "provider_uidl", @@ -54,10 +60,16 @@ def upgrade() -> None: ["tenant_config_id", "observed_at"], unique=False, ) + op.create_index( + "ix_pop3_observed_messages_account_retry", + _TABLE, + ["tenant_config_id", "collection_disposition", "retry_after"], + unique=False, + ) def downgrade() -> None: - """Drop durable POP3 UIDL observation state if present.""" + """Drop durable POP3 UIDL collection progress if present.""" connection = op.get_bind() inspector = sa.inspect(connection) if _TABLE in inspector.get_table_names(): diff --git a/backend/db/pop3_collection_models.py b/backend/db/pop3_collection_models.py index 26d78d064..45766295c 100644 --- a/backend/db/pop3_collection_models.py +++ b/backend/db/pop3_collection_models.py @@ -7,12 +7,13 @@ class Pop3ObservedMessage(Base): - """Durable provider identity for one POP3 message observed by an account. + """Durable provider progress for one POP3 UIDL within an account. POP3 message numbers are session-local positions. ``provider_uidl`` stores - the RFC 1939 unique-id instead so bounded polling can make progress across - reconnects and message-number renumbering without redefining Naruon's email - Message-ID or source-fingerprint identity. + RFC 1939 provider identity so bounded polling can make progress across + reconnects and renumbering. ``collection_disposition`` distinguishes + successfully observed messages from retryable collection failures; neither + state redefines Naruon's Message-ID or source-fingerprint identity. """ __tablename__ = "pop3_observed_messages" @@ -27,6 +28,12 @@ class Pop3ObservedMessage(Base): "tenant_config_id", "observed_at", ), + Index( + "ix_pop3_observed_messages_account_retry", + "tenant_config_id", + "collection_disposition", + "retry_after", + ), ) observed_message_id: Mapped[int] = mapped_column(primary_key=True) @@ -35,8 +42,17 @@ class Pop3ObservedMessage(Base): nullable=False, ) provider_uidl: Mapped[str] = mapped_column(String(70), nullable=False) - observed_at: Mapped[datetime.datetime] = mapped_column( + collection_disposition: Mapped[str] = mapped_column( + String(16), + default="observed", + nullable=False, + ) + retry_after: Mapped[datetime.datetime | None] = mapped_column( + DateTime(timezone=True), + nullable=True, + ) + observed_at: Mapped[datetime.datetime | None] = mapped_column( DateTime(timezone=True), default=lambda: datetime.datetime.now(datetime.timezone.utc), - nullable=False, + nullable=True, ) diff --git a/backend/services/pop3_worker.py b/backend/services/pop3_worker.py index b7ec9fa94..c1de39d34 100644 --- a/backend/services/pop3_worker.py +++ b/backend/services/pop3_worker.py @@ -1,7 +1,9 @@ import asyncio from dataclasses import dataclass +import datetime import logging import poplib +from typing import Literal, Mapping from sqlalchemy import select from sqlalchemy.dialects.postgresql import insert as pg_insert @@ -16,6 +18,8 @@ logger = logging.getLogger(__name__) MAX_POP3_FETCH_MESSAGES = 10 +POP3_RETRY_DELAY = datetime.timedelta(seconds=60) +Pop3CollectionDisposition = Literal["observed", "retryable"] @dataclass(frozen=True) @@ -34,6 +38,22 @@ class Pop3RetrievedMessage: provider_uidl: str | None = None +@dataclass(frozen=True) +class Pop3CollectionProgressState: + """Durable collection disposition for one owner-scoped provider UIDL.""" + + disposition: Pop3CollectionDisposition + retry_after: datetime.datetime | None + + +@dataclass(frozen=True) +class Pop3SyncBatch: + """Network retrieval result plus retryable UIDLs awaiting durable disposition.""" + + messages: list[Pop3RetrievedMessage] + retryable_uidls: frozenset[str] + + class Pop3SyncWorker: def __init__(self): self._task = None @@ -103,15 +123,19 @@ async def _sync_tenant(self, config: TenantConfig, semaphore: asyncio.Semaphore) f"Connecting to POP3 server {pop3_server}:{pop3_port} for user {config.user_id}" ) try: - observed_uidls = await self._load_observed_uidls(config) - messages = await asyncio.to_thread( - self._do_pop3_sync, + collection_progress = await self._load_collection_progress(config) + batch = await asyncio.to_thread( + self._do_pop3_sync_batch, config, pop3_server, pop3_port, - observed_uidls, + collection_progress, + ) + imported_count = await self._import_messages( + config, + batch.messages, + retryable_uidls=batch.retryable_uidls, ) - imported_count = await self._import_messages(config, messages) logger.info( "Successfully synced POP3 server for user %s with %s imported messages.", config.user_id, @@ -124,30 +148,62 @@ async def _sync_tenant(self, config: TenantConfig, semaphore: asyncio.Semaphore) type(e).__name__, ) - async def _load_observed_uidls(self, config: TenantConfig) -> set[str]: - """Load durable UIDL progress before network I/O begins.""" + async def _load_collection_progress( + self, config: TenantConfig + ) -> dict[str, Pop3CollectionProgressState]: + """Load durable UIDL collection state before provider network I/O begins.""" if config.id is None: - return set() + return {} async with AsyncSessionLocal() as session: result = await session.execute( - select(Pop3ObservedMessage.provider_uidl).where( - Pop3ObservedMessage.tenant_config_id == config.id - ) + select( + Pop3ObservedMessage.provider_uidl, + Pop3ObservedMessage.collection_disposition, + Pop3ObservedMessage.retry_after, + ).where(Pop3ObservedMessage.tenant_config_id == config.id) ) - return set(result.scalars().all()) + return { + provider_uidl: Pop3CollectionProgressState( + disposition=disposition, + retry_after=retry_after, + ) + for provider_uidl, disposition, retry_after in result.all() + } + + async def _load_observed_uidls(self, config: TenantConfig) -> set[str]: + """Compatibility view over durable collection state for observed UIDLs.""" + progress = await self._load_collection_progress(config) + return { + provider_uidl + for provider_uidl, state in progress.items() + if state.disposition == "observed" + } async def _import_messages( self, config: TenantConfig, messages: list[Pop3RetrievedMessage | bytes], + *, + retryable_uidls: frozenset[str] | set[str] | None = None, ) -> int: - if not messages: + retryable_uidls = retryable_uidls or frozenset() + if not messages and not retryable_uidls: return 0 imported_count = 0 owner_addresses = [config.pop3_username] if config.pop3_username else None + state_time = datetime.datetime.now(datetime.timezone.utc) async with AsyncSessionLocal() as session: try: + if config.id is not None: + for provider_uidl in retryable_uidls: + await self._upsert_collection_progress( + session, + config.id, + provider_uidl, + disposition="retryable", + state_time=state_time, + ) for message in messages: if isinstance(message, bytes): raw_message = message @@ -162,6 +218,14 @@ async def _import_messages( "Skipping unparsable POP3 message for user %s.", config.user_id, ) + if provider_uidl is not None and config.id is not None: + await self._upsert_collection_progress( + session, + config.id, + provider_uidl, + disposition="retryable", + state_time=state_time, + ) continue persistence_result = await persist_fetched_email( session, @@ -174,15 +238,12 @@ async def _import_messages( if persistence_result.created_record: imported_count += 1 if provider_uidl is not None and config.id is not None: - await session.execute( - pg_insert(Pop3ObservedMessage) - .values( - tenant_config_id=config.id, - provider_uidl=provider_uidl, - ) - .on_conflict_do_nothing( - index_elements=["tenant_config_id", "provider_uidl"] - ) + await self._upsert_collection_progress( + session, + config.id, + provider_uidl, + disposition="observed", + state_time=state_time, ) await session.commit() except Exception: @@ -190,6 +251,39 @@ async def _import_messages( raise return imported_count + async def _upsert_collection_progress( + self, + session, + tenant_config_id: int, + provider_uidl: str, + *, + disposition: Pop3CollectionDisposition, + state_time: datetime.datetime, + ) -> None: + retry_after = ( + state_time + POP3_RETRY_DELAY if disposition == "retryable" else None + ) + observed_at = state_time if disposition == "observed" else None + statement = ( + pg_insert(Pop3ObservedMessage) + .values( + tenant_config_id=tenant_config_id, + provider_uidl=provider_uidl, + collection_disposition=disposition, + retry_after=retry_after, + observed_at=observed_at, + ) + .on_conflict_do_update( + index_elements=["tenant_config_id", "provider_uidl"], + set_={ + "collection_disposition": disposition, + "retry_after": retry_after, + "observed_at": observed_at, + }, + ) + ) + await session.execute(statement) + def _validated_destination(self, config: TenantConfig) -> tuple[str, int]: return validate_pop3_destination( str(config.pop3_server), @@ -203,10 +297,31 @@ def _do_pop3_sync( pop3_port: int | None = None, observed_uidls: set[str] | None = None, ) -> list[Pop3RetrievedMessage]: + collection_progress = { + provider_uidl: Pop3CollectionProgressState( + disposition="observed", + retry_after=None, + ) + for provider_uidl in (observed_uidls or set()) + } + return self._do_pop3_sync_batch( + config, + pop3_server, + pop3_port, + collection_progress, + ).messages + + def _do_pop3_sync_batch( + self, + config: TenantConfig, + pop3_server: str | None = None, + pop3_port: int | None = None, + collection_progress: Mapping[str, Pop3CollectionProgressState] | None = None, + ) -> Pop3SyncBatch: if pop3_server is None or pop3_port is None: pop3_server, pop3_port = self._validated_destination(config) pop3_client = poplib.POP3_SSL(pop3_server, pop3_port) - observed_uidls = observed_uidls or set() + collection_progress = collection_progress or {} try: if not config.pop3_username: logger.error( @@ -229,16 +344,12 @@ def _do_pop3_sync( uidl_identities = self._current_uidl_identities(pop3_client, config) if uidl_identities is not None: - candidates = [ - identity - for identity in uidl_identities - if identity.provider_uidl not in observed_uidls - ] - selected = sorted( - candidates, - key=lambda identity: identity.message_number, - )[-MAX_POP3_FETCH_MESSAGES:] - return self._retrieve_uidl_messages(pop3_client, config, selected) + selected = self._select_uidl_candidates( + uidl_identities, + collection_progress, + datetime.datetime.now(datetime.timezone.utc), + ) + return self._retrieve_uidl_batch(pop3_client, config, selected) _response, listings, _octets = pop3_client.list() message_numbers = [ @@ -251,14 +362,41 @@ def _do_pop3_sync( "POP3 UIDL unavailable for user %s; bounded fallback cannot prove durable backlog progress.", config.user_id, ) - return self._retrieve_fallback_messages( - pop3_client, - config, - sorted(message_numbers)[-MAX_POP3_FETCH_MESSAGES:], + return Pop3SyncBatch( + messages=self._retrieve_fallback_messages( + pop3_client, + config, + sorted(message_numbers)[-MAX_POP3_FETCH_MESSAGES:], + ), + retryable_uidls=frozenset(), ) finally: self._close_pop3_client(pop3_client, config) + def _select_uidl_candidates( + self, + identities: list[Pop3MessageIdentity], + collection_progress: Mapping[str, Pop3CollectionProgressState], + now: datetime.datetime, + ) -> list[Pop3MessageIdentity]: + """Prefer never-attempted UIDLs, then retries whose retry window is due.""" + fresh: list[Pop3MessageIdentity] = [] + due_retries: list[Pop3MessageIdentity] = [] + for identity in sorted( + identities, + key=lambda candidate: candidate.message_number, + reverse=True, + ): + state = collection_progress.get(identity.provider_uidl) + if state is None: + fresh.append(identity) + continue + if state.disposition == "observed": + continue + if state.retry_after is None or state.retry_after <= now: + due_retries.append(identity) + return (fresh + due_retries)[:MAX_POP3_FETCH_MESSAGES] + def _current_uidl_identities( self, pop3_client: poplib.POP3_SSL, @@ -301,13 +439,14 @@ def _uidl_identity_from_listing( provider_uidl=provider_uidl, ) - def _retrieve_uidl_messages( + def _retrieve_uidl_batch( self, pop3_client: poplib.POP3_SSL, config: TenantConfig, identities: list[Pop3MessageIdentity], - ) -> list[Pop3RetrievedMessage]: + ) -> Pop3SyncBatch: messages: list[Pop3RetrievedMessage] = [] + retryable_uidls: set[str] = set() for identity in identities: try: source_content = self._retrieve_message( @@ -326,6 +465,7 @@ def _retrieve_uidl_messages( config.user_id, type(exc).__name__, ) + retryable_uidls.add(identity.provider_uidl) continue except OSError as exc: logger.warning( @@ -340,7 +480,19 @@ def _retrieve_uidl_messages( provider_uidl=identity.provider_uidl, ) ) - return messages + return Pop3SyncBatch( + messages=messages, + retryable_uidls=frozenset(retryable_uidls), + ) + + def _retrieve_uidl_messages( + self, + pop3_client: poplib.POP3_SSL, + config: TenantConfig, + identities: list[Pop3MessageIdentity], + ) -> list[Pop3RetrievedMessage]: + """Compatibility wrapper retaining the historical direct-return contract.""" + return self._retrieve_uidl_batch(pop3_client, config, identities).messages def _retrieve_fallback_messages( self, diff --git a/backend/tests/test_pop3_retry_progress.py b/backend/tests/test_pop3_retry_progress.py index 5a154657e..1732a6d81 100644 --- a/backend/tests/test_pop3_retry_progress.py +++ b/backend/tests/test_pop3_retry_progress.py @@ -1,5 +1,8 @@ import datetime +import poplib +from unittest.mock import MagicMock +from db.models import TenantConfig from services.pop3_worker import ( Pop3CollectionProgressState, Pop3MessageIdentity, @@ -14,6 +17,18 @@ def _identities(count: int = 12) -> list[Pop3MessageIdentity]: ] +def _config() -> TenantConfig: + return TenantConfig( + id=42, + user_id="pop3-user", + organization_id="org-pop3", + pop3_server="pop3.example.com", + pop3_port=995, + pop3_username="pop3-user@example.com", + pop3_password="pop3-secret", + ) + + def test_uidl_selection_prioritizes_never_attempted_backlog_before_due_retries(): worker = Pop3SyncWorker() now = datetime.datetime(2026, 9, 17, 10, 40, tzinfo=datetime.timezone.utc) @@ -60,3 +75,43 @@ def test_uidl_selection_never_retries_observed_identity(): selected = worker._select_uidl_candidates(_identities(), progress, now) assert "uid-12" not in {identity.provider_uidl for identity in selected} + + +def test_persistent_negative_tail_becomes_retryable_without_hiding_lower_backlog(monkeypatch): + worker = Pop3SyncWorker() + client = MagicMock() + client.uidl.return_value = ( + b"+OK", + [f"{number} uid-{number}".encode() for number in range(1, 13)], + 192, + ) + client.retr.side_effect = poplib.error_proto("-ERR no such message") + monkeypatch.setattr("services.pop3_worker.poplib.POP3_SSL", lambda host, port: client) + + first_batch = worker._do_pop3_sync_batch( + _config(), + "pop3.example.com", + 995, + collection_progress={}, + ) + + assert first_batch.messages == [] + assert first_batch.retryable_uidls == frozenset( + f"uid-{number}" for number in range(3, 13) + ) + + now = datetime.datetime(2026, 9, 17, 10, 40, tzinfo=datetime.timezone.utc) + persisted_retry_state = { + provider_uidl: Pop3CollectionProgressState( + disposition="retryable", + retry_after=now + datetime.timedelta(minutes=1), + ) + for provider_uidl in first_batch.retryable_uidls + } + next_selected = worker._select_uidl_candidates( + _identities(), + persisted_retry_state, + now, + ) + + assert [identity.provider_uidl for identity in next_selected] == ["uid-2", "uid-1"] diff --git a/backend/tests/test_pop3_uidl_migration_contract.py b/backend/tests/test_pop3_uidl_migration_contract.py index 14ef35c93..d249e19d7 100644 --- a/backend/tests/test_pop3_uidl_migration_contract.py +++ b/backend/tests/test_pop3_uidl_migration_contract.py @@ -11,24 +11,36 @@ ) -def test_pop3_uidl_model_is_owner_scoped_and_bounded(): +def test_pop3_uidl_model_is_owner_scoped_and_tracks_retry_disposition(): table = Pop3ObservedMessage.__table__ assert table.name == "pop3_observed_messages" assert table.c.tenant_config_id.nullable is False assert table.c.provider_uidl.nullable is False assert table.c.provider_uidl.type.length == 70 + assert table.c.collection_disposition.nullable is False + assert table.c.collection_disposition.type.length == 16 + assert table.c.retry_after.nullable is True + assert table.c.observed_at.nullable is True assert any( constraint.name == "uq_pop3_observed_messages_account_uidl" for constraint in table.constraints ) + assert any( + index.name == "ix_pop3_observed_messages_account_retry" + for index in table.indexes + ) -def test_pop3_uidl_migration_succeeds_canonical_provenance_revision(): +def test_pop3_uidl_migration_extends_branch_local_collection_progress_revision(): source = MIGRATION_PATH.read_text(encoding="utf-8") assert 'revision = "0019_pop3_observed_uidl"' in source assert 'down_revision = "0018_email_date_provenance"' in source assert 'sa.ForeignKey("tenant_configs.id", ondelete="CASCADE")' in source assert 'sa.String(length=70)' in source + assert '"collection_disposition"' in source + assert '"retry_after"' in source assert '"uq_pop3_observed_messages_account_uidl"' in source + assert '"ix_pop3_observed_messages_account_retry"' in source + assert "branch-local" in source From 630d5461d1eb8b59b392e4170dec66ce1edf034c Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 17 Sep 2026 19:49:32 +0900 Subject: [PATCH 83/86] docs(pop3): trace durable retry progress contract --- docs/doctoring/pop3-durable-retry-progress.md | 40 +++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100644 docs/doctoring/pop3-durable-retry-progress.md diff --git a/docs/doctoring/pop3-durable-retry-progress.md b/docs/doctoring/pop3-durable-retry-progress.md new file mode 100644 index 000000000..8edce81b4 --- /dev/null +++ b/docs/doctoring/pop3-durable-retry-progress.md @@ -0,0 +1,40 @@ +# POP3 durable retry progress + +Issue #1717 exposed a cross-poll starvation case that the first UIDL implementation did not cover. A bounded poll selected the highest ten unseen UIDLs. If all ten returned a standards-conforming `-ERR` from `RETR`, they remained unobserved and were selected again on the next poll. Lower valid UIDLs could therefore remain unreachable indefinitely even though each individual poll correctly continued after the rejected message. + +## Decision + +Provider UIDL remains collection identity, not email identity. Naruon persists one owner-scoped collection disposition per `(tenant_config_id, provider_uidl)`: + +- `observed` means retrieval and email persistence completed; the UIDL is not selected again; +- `retryable` means a message-level collection attempt failed without proving permanent loss; `retry_after` controls when it may re-enter the candidate set. + +Candidate selection prioritizes UIDLs with no durable attempt state before due retries. This is the causal property that prevents a persistent set of retryable failures from monopolizing every bounded poll while fresh backlog exists. The one-minute retry delay matches the worker's existing polling cadence; it is not a terminal-failure threshold and does not discard the UIDL. + +POP3 network I/O still happens before the database session used for collection-state writes. A standards-conforming per-message `-ERR` is returned from the network phase as a retryable UIDL and is persisted only after network I/O completes. A successfully persisted message transitions the same UIDL to `observed`. Parse failures for a retrieved UIDL are also retryable rather than silently observed. Transport loss or malformed protocol response still ends the remaining network batch because the session itself is no longer trusted. + +No `DELE` is issued. The UIDL state does not replace `Message-ID`, sender-authored Date provenance, or source fingerprints. + +## Source trace + +- RED `5d40f83ff48e7dd9c3bf6deb93995130c5a1470a` introduces candidate-selection regressions for fresh backlog versus due/future retry state. +- Fix `7406cb63923003df2c3a5d21a0104203f78b90bc` adds retryable collection disposition, persists `retry_after`, keeps provider I/O outside the DB session, and makes never-attempted UIDLs outrank retries. +- The same fix extends the branch-local `0019_pop3_observed_uidl` schema rather than creating another Alembic revision. Its revision identifier and parent remain non-authoritative until #1503's canonical workspace migration lineage reaches protected ancestry and #1195 is rechained after the then-current protected head. + +## Acceptance boundary + +Source order is not execution evidence. Before merge, the final migration-reconciled exact head still needs: + +- one Alembic head after ordinary adoption of the protected workspace migration lineage; +- fresh PostgreSQL upgrade acceptance from an empty database and from the historical protected migration point; +- repeated-poll and restart acceptance with more than the per-poll cap of persistent negative `RETR` responses ahead of valid unseen mail; +- proof that failed UIDLs remain retryable while valid backlog advances and duplicate import counts remain stable; +- the exact-head repository test, security, coverage, and independent-review gates. + +UIDL-unavailable `LIST` fallback remains compatibility-only and does not claim durable eventual progress. + +## Reference + +Myers, J., & Rose, M. (1996). *Post Office Protocol—Version 3* (RFC 1939). Internet Engineering Task Force. https://doi.org/10.17487/RFC1939 + +RFC 1939 assigns message numbers within the opened maildrop, permits repeated commands in the TRANSACTION state, defines negative `-ERR` command responses, and provides UIDL as the stable provider identifier needed to distinguish messages across sessions. Those protocol properties justify keeping session message numbers out of durable identity and retaining rejected UIDLs as retryable collection state rather than treating them as observed or deleted. From 5be27e6d11e918a37dbba3d199c4015c731e63c1 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Tue, 22 Sep 2026 02:48:31 +0900 Subject: [PATCH 84/86] test(pop3): expose interrupted RETR progress starvation --- .../test_pop3_interrupted_retry_progress.py | 76 +++++++++++++++++++ 1 file changed, 76 insertions(+) create mode 100644 backend/tests/test_pop3_interrupted_retry_progress.py diff --git a/backend/tests/test_pop3_interrupted_retry_progress.py b/backend/tests/test_pop3_interrupted_retry_progress.py new file mode 100644 index 000000000..c5699d321 --- /dev/null +++ b/backend/tests/test_pop3_interrupted_retry_progress.py @@ -0,0 +1,76 @@ +import datetime +import poplib +from unittest.mock import MagicMock + +import pytest + +from db.models import TenantConfig +from services.pop3_worker import ( + Pop3CollectionProgressState, + Pop3MessageIdentity, + Pop3SyncWorker, +) + + +def _config() -> TenantConfig: + return TenantConfig( + id=42, + user_id="pop3-user", + organization_id="org-pop3", + pop3_server="pop3.example.com", + pop3_port=995, + pop3_username="pop3-user@example.com", + pop3_password="pop3-secret", + ) + + +def _identities() -> list[Pop3MessageIdentity]: + return [ + Pop3MessageIdentity(message_number=number, provider_uidl=f"uid-{number}") + for number in range(1, 13) + ] + + +@pytest.mark.parametrize( + "failure", + [ + poplib.error_proto("unexpected response"), + OSError("transport failed"), + ], +) +def test_interrupted_retr_marks_attempted_uidl_retryable_before_stopping(failure): + worker = Pop3SyncWorker() + client = MagicMock() + selected = list(reversed(_identities()))[:10] + client.retr.side_effect = failure + + batch = worker._retrieve_uidl_batch(client, _config(), selected) + + assert batch.messages == [] + assert batch.retryable_uidls == frozenset({"uid-12"}) + client.retr.assert_called_once_with(12) + + +def test_interrupted_retr_does_not_keep_failed_tail_fresh_on_next_poll(): + worker = Pop3SyncWorker() + client = MagicMock() + selected = list(reversed(_identities()))[:10] + client.retr.side_effect = OSError("transport failed") + + first_batch = worker._retrieve_uidl_batch(client, _config(), selected) + + now = datetime.datetime(2026, 9, 22, 2, 0, tzinfo=datetime.timezone.utc) + progress = { + uidl: Pop3CollectionProgressState( + disposition="retryable", + retry_after=now + datetime.timedelta(minutes=1), + ) + for uidl in first_batch.retryable_uidls + } + next_selected = worker._select_uidl_candidates(_identities(), progress, now) + + assert "uid-12" not in {identity.provider_uidl for identity in next_selected} + assert [identity.provider_uidl for identity in next_selected[:2]] == [ + "uid-11", + "uid-10", + ] From c6462af326721da39497666d78556ec8e31432dd Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Tue, 22 Sep 2026 02:49:46 +0900 Subject: [PATCH 85/86] fix(pop3): persist interrupted RETR as retryable --- backend/services/pop3_worker.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/backend/services/pop3_worker.py b/backend/services/pop3_worker.py index c1de39d34..a429998db 100644 --- a/backend/services/pop3_worker.py +++ b/backend/services/pop3_worker.py @@ -454,6 +454,7 @@ def _retrieve_uidl_batch( ) except poplib.error_proto as exc: if not self._is_negative_pop3_response(exc): + retryable_uidls.add(identity.provider_uidl) logger.warning( "POP3 RETR stopped after malformed protocol response for user %s: %s", config.user_id, @@ -468,6 +469,7 @@ def _retrieve_uidl_batch( retryable_uidls.add(identity.provider_uidl) continue except OSError as exc: + retryable_uidls.add(identity.provider_uidl) logger.warning( "POP3 RETR stopped after transport failure for user %s: %s", config.user_id, @@ -588,4 +590,4 @@ def _message_number_from_listing(self, listing: bytes | str) -> int | None: def _bytes_line(self, line: bytes | str) -> bytes: return ( line if isinstance(line, bytes) else line.encode("utf-8", errors="replace") - ) + ) \ No newline at end of file From 52d2cc6fc136c931dda609a0129e3cd78048eebb Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Tue, 22 Sep 2026 02:50:09 +0900 Subject: [PATCH 86/86] docs(pop3): trace interrupted RETR progress repair --- docs/doctoring/pop3-durable-retry-progress.md | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/docs/doctoring/pop3-durable-retry-progress.md b/docs/doctoring/pop3-durable-retry-progress.md index 8edce81b4..c9949946b 100644 --- a/docs/doctoring/pop3-durable-retry-progress.md +++ b/docs/doctoring/pop3-durable-retry-progress.md @@ -2,6 +2,8 @@ Issue #1717 exposed a cross-poll starvation case that the first UIDL implementation did not cover. A bounded poll selected the highest ten unseen UIDLs. If all ten returned a standards-conforming `-ERR` from `RETR`, they remained unobserved and were selected again on the next poll. Lower valid UIDLs could therefore remain unreachable indefinitely even though each individual poll correctly continued after the rejected message. +A later audit found the same progress invariant was still incomplete for interrupted sessions. A malformed protocol response or transport failure stops the remaining network batch, but the currently attempted UIDL was returned with no durable disposition. Because candidate selection treats missing state as never attempted and prioritizes it ahead of retries, a persistently failing newest UIDL could remain first after every reconnect and keep lower fresh backlog unreachable. + ## Decision Provider UIDL remains collection identity, not email identity. Naruon persists one owner-scoped collection disposition per `(tenant_config_id, provider_uidl)`: @@ -11,7 +13,9 @@ Provider UIDL remains collection identity, not email identity. Naruon persists o Candidate selection prioritizes UIDLs with no durable attempt state before due retries. This is the causal property that prevents a persistent set of retryable failures from monopolizing every bounded poll while fresh backlog exists. The one-minute retry delay matches the worker's existing polling cadence; it is not a terminal-failure threshold and does not discard the UIDL. -POP3 network I/O still happens before the database session used for collection-state writes. A standards-conforming per-message `-ERR` is returned from the network phase as a retryable UIDL and is persisted only after network I/O completes. A successfully persisted message transitions the same UIDL to `observed`. Parse failures for a retrieved UIDL are also retryable rather than silently observed. Transport loss or malformed protocol response still ends the remaining network batch because the session itself is no longer trusted. +POP3 network I/O still happens before the database session used for collection-state writes. A standards-conforming per-message `-ERR` is returned from the network phase as a retryable UIDL and is persisted only after network I/O completes. A successfully persisted message transitions the same UIDL to `observed`. Parse failures for a retrieved UIDL are also retryable rather than silently observed. + +Transport loss or malformed protocol response still ends the remaining network batch because the session itself is no longer trusted. Before stopping, the currently attempted UIDL is now returned as `retryable`. This does not assert that the failure was message-specific; it records only that the provider UIDL was attempted and not observed. On the next poll it therefore cannot masquerade as never attempted and indefinitely outrank lower fresh backlog. No later UIDL in the interrupted batch is marked because no retrieval attempt was made for it. No `DELE` is issued. The UIDL state does not replace `Message-ID`, sender-authored Date provenance, or source fingerprints. @@ -19,7 +23,9 @@ No `DELE` is issued. The UIDL state does not replace `Message-ID`, sender-author - RED `5d40f83ff48e7dd9c3bf6deb93995130c5a1470a` introduces candidate-selection regressions for fresh backlog versus due/future retry state. - Fix `7406cb63923003df2c3a5d21a0104203f78b90bc` adds retryable collection disposition, persists `retry_after`, keeps provider I/O outside the DB session, and makes never-attempted UIDLs outrank retries. -- The same fix extends the branch-local `0019_pop3_observed_uidl` schema rather than creating another Alembic revision. Its revision identifier and parent remain non-authoritative until #1503's canonical workspace migration lineage reaches protected ancestry and #1195 is rechained after the then-current protected head. +- RED `5be27e6d11e918a37dbba3d199c4015c731e63c1` proves that malformed protocol and transport interruption previously returned the attempted UIDL with no retryable disposition, allowing the same newest UIDL to remain fresh after reconnect. +- Fix `c6462af326721da39497666d78556ec8e31432dd` records only the interrupted current UIDL as retryable before halting the untrusted session; unattempted identities remain fresh. +- These repairs extend the branch-local `0019_pop3_observed_uidl` schema rather than creating another Alembic revision. Its revision identifier and parent remain non-authoritative until #1503's canonical workspace migration lineage reaches protected ancestry and #1195 is rechained after the then-current protected head. ## Acceptance boundary @@ -28,6 +34,7 @@ Source order is not execution evidence. Before merge, the final migration-reconc - one Alembic head after ordinary adoption of the protected workspace migration lineage; - fresh PostgreSQL upgrade acceptance from an empty database and from the historical protected migration point; - repeated-poll and restart acceptance with more than the per-poll cap of persistent negative `RETR` responses ahead of valid unseen mail; +- repeated reconnect acceptance where the newest attempted UIDL repeatedly triggers malformed protocol or transport interruption and lower fresh backlog still advances; - proof that failed UIDLs remain retryable while valid backlog advances and duplicate import counts remain stable; - the exact-head repository test, security, coverage, and independent-review gates. @@ -37,4 +44,4 @@ UIDL-unavailable `LIST` fallback remains compatibility-only and does not claim d Myers, J., & Rose, M. (1996). *Post Office Protocol—Version 3* (RFC 1939). Internet Engineering Task Force. https://doi.org/10.17487/RFC1939 -RFC 1939 assigns message numbers within the opened maildrop, permits repeated commands in the TRANSACTION state, defines negative `-ERR` command responses, and provides UIDL as the stable provider identifier needed to distinguish messages across sessions. Those protocol properties justify keeping session message numbers out of durable identity and retaining rejected UIDLs as retryable collection state rather than treating them as observed or deleted. +RFC 1939 assigns message numbers within the opened maildrop, permits repeated commands in the TRANSACTION state, defines negative `-ERR` command responses, and provides UIDL as the stable provider identifier needed to distinguish messages across sessions. Those protocol properties justify keeping session message numbers out of durable identity and retaining failed UIDL attempts as retryable collection state rather than treating them as observed, deleted, or perpetually never attempted.