fix(attachments): align deferred parse budget with import transport

[seonghobae]

Physical lease, interruption, and exact-head verification — 2026-09-05

실제 PostgreSQL에서 항목별 commit·rollback 뒤 다른 조회가 작업자의 연결을 재사용하고 잠금이 남는 문제를 재현했습니다. 한 연결을 작업 전체에 고정하고, 취소·연결 종료·해제 실패 때의 복구 순서와 미처리 항목 재개를 보완했습니다.

• Head a312ea5e516fe1d696cfb35ffff7d3a731ad9cd2, tree 83bf3605c961dc7be1c304585762a58de0124ba7; direct fix(data): align PDF DOM upload budget with sidecar #1427 base stays cb08b1c3ea2aba8844fc29ef703c34368cc55e47. Normal commits 47b2347 and a312ea5 preserve the full prior 1b757d5 delta, including HTTP 413 classification, original bytes, migration/search prerequisites, and Proposed ADR lineage.
• Actual PG RED on prior runtime: 2 failed in16.31s, after both commit and rollback. The worker and unrelated reader had the same backend PID; an independent replica could not acquire after completion. JUnit SHA-256: ce3f52eebe36323639ed80ad7e5517428f2f58efa527300b14b5e94af2ef927b.
• Minimal runtime repair: externally hold one connection and bind the existing session to it; preserve item transactions; require explicit true unlock; invalidate before session-close rollback on acquisition/work/release failure or cancellation; abort a disconnected cycle without silently reconnecting; advance cursors only after completed work or healthy item rollback.
• Independent read-only review found two additional issues in the initial repair (prefetched-tail cursor skip and close-before-invalidation). Both were fixed after 5 RED→5 GREEN cases. Re-review found no remaining actionable finding in that scope. Reviewed worker SHA-256 751c99f2d3963dcf79e6011fce8e94185a2a14b58bd389aed49ceb8739c459f2 equals final source; this is not GitHub approval.
• Final exact-head17-file run: 305 passed, 0 failures/errors/skips,225.30s, with clean-lock uv sync --locked, fresh/repeated migration to 0020_search_trigram_storage, explicit -W error, and before/after head/clean-tree guards. Ruff passed; terminal output had no Timeout/Fatal/Warn/Denied matches. Test-owned hardened PG/network resources were removed; existing services were untouched.
• Worker coverage:258/258 statements,58/58 branches,0 exclusions (100%); docstrings22/22 function/class definitions. This is scoped to services/newsdom_worker.py, not every product module or every imaginable edge case.
• Full unchanged40,758,835-byte NASA PDF and active migrated indexes are retained. Real-PG cases exercise same-pool contention, independent-replica acquisition, one-slot completion, acquisition/processing cancellation, backend termination, and failed release. Actual Task.cancel() plus a gated real AsyncSession.close verifies cleanup ordering; the gate is a controlled simulation, not a real network-black-hole test. Unit cases cover both kinds' cursor resume and missing/no-longer-pending reloads.
• Final JUnit SHA-256: a7bf4bab7352c8c1fd9e720b48f59d63057de29a5fdf049ebe2faef8ff3d6ea2; worker coverage JSON: 025ff2ea21aa66efc15bdbc9afc8ab3499591d085219b327d278b240a60b7363. Local evidence directory: /private/tmp/naruon-sweep-lease.QAQAFc. The earlier303-pass checkpoint is recorded separately and was not transferred to the final head.
• Doctoring, alternatives, sequence, and APA references and ADR-0023 remain Proposed. Existing feat: harden live macOS runtime and governance #1317 import and feat(noema-agent): add calendar conflict-check tool #1486 scheduler paths require their own root-cause reproduction/repair; no sibling source was overwritten or claimed fixed. Reusable procedure is in AGENTS owner docs(agents): 증거 기반 작업 절차 정리 #1566@162c0df, with source-only checks explicitly distinguished.

Reproduce from backend against isolated PostgreSQL/pgvector with fresh test credentials using the 17 files listed in doctoring:

uv sync --locked
uv run --locked python scripts/migrate_db.py
uv run --locked python scripts/migrate_db.py
uv run --locked coverage run --branch --source=services.newsdom_worker -m pytest -q -W error -ra --tb=short tests/test_attachment_source_postgres.py tests/test_attachment_parser.py tests/test_newsdom_client.py tests/test_newsdom_worker.py tests/test_email_import_service.py tests/test_email_parser.py tests/test_email_read_state_migration_postgres.py tests/test_alembic_migrations.py tests/test_bootstrap_db.py tests/test_data_api.py tests/test_legacy_document_scope_postgres.py tests/test_workspace_document_migration.py tests/test_container_dependency_pin_contract.py tests/test_search.py tests/test_search_postgres.py tests/test_search_answer.py tests/test_hybrid_retrieval_fusion.py
uv run --locked coverage report -m

Keep Draft. Current-head required hosted checks and qualifying approval, protected foundation integration, immutable NewsDOM release/pin, real64MiB recognition, signed browser recovery, representative capacity/p95 and security acceptance remain open. A database lease is not exactly-once external provider execution. No dependency addition, provider/model timeout, force push, approval, closure, release or protection bypass occurred.

Historical HTTP413 and retained-source receipts (earlier heads)

Current HTTP 413 extension and exact-head revalidation — 2026-09-05

Current head 1b757d5aa25c469157f8f03301964eb3061ed0fe, tree 6b520c3cbe824d4017b97c39ef61fa702434e04a, is a normal descendant of the full retained-source/rollback repair db083c9. Direct parent #1427 stays cb08b1c3ea2aba8844fc29ef703c34368cc55e47; keep Draft.

• Adopted the concurrent test/fix pair 220c7d0751f56244e568513f149669a3856099af / 1b757d5aa25c469157f8f03301964eb3061ed0fe without rewriting history. Provider HTTP 413 now maps to the same size-limit error category as the local guard, so a smaller provider limit is not misclassified as a generic recognition failure.
• The new transport test uses HTTPX MockTransport and a tiny synthetic unit payload; it proves status classification, not a real provider's capacity. The actual retained-source PostgreSQL cases still use the full unchanged NASA PDF.
• Main-agent exact-head verification reran uv sync --locked, fresh/repeat migrations to 0020_search_trigram_storage, and the same full 17-file suite with -W error: 277 passed, 0 failures/errors/skips, 73.71 s. Before/after head and clean-tree guards passed. Changed-file Ruff passed; isolated PostgreSQL/network cleanup completed.
• JUnit SHA-256: 5ec09d7152e0f3551fa670ac15567282ae02ac71170cd51eb5e3e51433abab10. The previous 276-pass result remains bound to db083c9; it was not reused as evidence for this later head.
• Required hosted Checks, post-last-push approval, protected owner integration, released provider pin, actual 64 MiB recognition, and realistic capacity/security acceptance remain open. The default-branch advisories require existing owner repairs; no approval, alert dismissal, bypass or protected merge was performed.

Historical retained-source integration receipt

Current exact authority — 2026-09-05

• direct base: fix(data): align PDF DOM upload budget with sidecar #1427 codex/pdf-dom-upload-64m@cb08b1c3ea2aba8844fc29ef703c34368cc55e47
• branch: feat/naruon-attachment-parse-64m
• exact head: 1b757d5aa25c469157f8f03301964eb3061ed0fe
• lifecycle: Draft / open / mergeable / exact-head hosted evidence absent
• effective child delta: 15 files. The child owns Naruon attachment-ingestion/source retention and NewsDOM consumer behavior; fix(data): align PDF DOM upload budget with sidecar #1427 remains the direct PDF-DOM prerequisite.

Current repair lineage

A fresh read of request_pdf_dom() found a contract hole after the earlier 20 MiB local preflight repair. A payload admitted by Naruon's local guard could still be rejected by the actually deployed NewsDOM provider with HTTP 413, but the client converted every >=400 response into generic NewsdomRequestError. The worker therefore could persist recognition_failed instead of the already-defined provider_payload_size_exceeded evidence.

• RED 220c7d0751f56244e568513f149669a3856099af: adds an executable httpx.MockTransport regression requiring a remote HTTP 413 to raise NewsdomPayloadTooLargeError even when the local payload is below the 20 MiB client guard.
• minimal fix 1b757d5aa25c469157f8f03301964eb3061ed0fe: translates only HTTP 413 to NewsdomPayloadTooLargeError before the generic >=400 branch and updates that exception's docstring to cover both local and provider-side contract rejection. Other provider errors remain generic request failures.

This preserves the domain invariant already consumed by newsdom_worker: a provider-side size rejection is durable, specific evidence and is not silently collapsed into a generic recognition failure. It does not claim that a deployed NewsDOM version accepts 64 MiB.

Owner-stack / ADR boundary

The earlier non-force restack onto #1427 is preserved. ADR-0023 remains Proposed, depends on ADR-0021 for the direct PDF path, and requires an immutable released NewsDOM transport contract before provider-size compatibility can be promoted. ContextualWisdomLab/newsdom-api#665 remains the owner path for the candidate bounded provider transport. No mutable owner head is treated as a released dependency.

The current branch retains the repaired CHANGELOG/proposal-history/database evidence from predecessor db083c9b30dc0156a46baa2cc0ffc3aec5444ab7. That predecessor recorded a clean-lock focused run of 276 tests and real PostgreSQL/corpus evidence, but those results are now predecessor development evidence only because the source and test tree changed at 220c7d0/1b757d5. They are not merge authority for this exact head.

Verification boundary

A fresh exact-head workflow lookup for 1b757d5aa25c469157f8f03301964eb3061ed0fe currently returns no pull-request workflow runs. No qualifying independent post-last-push approval exists. Do not manufacture queue activity with a no-op commit and do not transfer db083c9... checks or local results to the new head.

Keep Draft until the unchanged exact head has the then-live required repository/organization checks terminal-success, zero actionable current-head findings/threads, and a qualifying independent post-last-push approval; #1427 is protected-integrated; and the bounded NewsDOM owner contract is immutably released and pinned where the product claims provider compatibility. Actual 64 MiB provider recognition, representative memory/storage/index/latency/tenant-quota evidence, signed browser upload, and actionable document error/retry UX remain buyer-visible acceptance gaps. No self-approval, force push, destructive rebase, gate weakening, dummy requeue, mutable dependency, or predecessor-evidence transfer.

[coderabbitai]

📝 Walkthrough

Walkthrough

The attachment parser source limit is increased from 20 MiB to 64 MiB. The NewsDOM client rejects PDF payloads above 20 MiB before network activity. The worker records provider payload-size failures. ADRs and related documentation define the bounded, fail-closed contract.

Changes

Attachment Parse Contract

Layer / File(s)	Summary
Parser limit and admission contract backend/services/attachment_parser.py, backend/tests/test_attachment_parser.py, docs/doctoring/bounded-attachment-parse-source-contract.md, docs/adr/0006-bounded-attachment-parse-source-contract.md	The parser limit is set to 64 MiB. Tests verify payloads above 20 MiB. Documentation defines bounded PDF retention, size-limit results, and metadata-only handling for unsupported binaries.
NewsDOM upload guard and failure handling backend/services/newsdom_client.py, backend/services/newsdom_worker.py, backend/api/data.py, backend/tests/test_newsdom_client.py, backend/tests/test_newsdom_worker.py	The NewsDOM client rejects PDF payloads above 20 MiB before network activity. The worker records provider_payload_size_exceeded and marks the attachment as failed. Shared validation uses the NewsDOM upload limit.
Contract decisions and release records docs/adr/0005-bounded-pdf-dom-upload-contract.md, docs/adr/0006-bounded-attachment-parse-source-contract.md, docs/adr/README.md, CHANGELOG.md	The ADRs, ADR index, and changelog record the separate 20 MiB PDF upload and 64 MiB attachment parse-source contracts.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: 🔵 Low · up to 19305

The PR raises the deferred attachment parse limit to match the 64 MiB import transport and preserves explicit unsupported-format statuses. It is mergeable with owner awareness or follow-up for remaining changelog wording corrections and the warning-level output behavior.

Suggested reviewers: cursoragent

Sequence Diagram(s)

sequenceDiagram
  participant AttachmentParser
  participant NewsdomWorker
  participant NewsdomClient
  AttachmentParser->>NewsdomWorker: provide deferred PDF payload
  NewsdomWorker->>NewsdomClient: request_pdf_dom(payload)
  NewsdomClient-->>NewsdomWorker: raise NewsdomPayloadTooLargeError above 20 MiB
  NewsdomWorker->>NewsdomWorker: record provider_payload_size_exceeded and failed status

Loading

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: aligning the deferred attachment parse budget with the authenticated import transport limit.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/naruon-attachment-parse-64m

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[seonghobae]

Current-head verification note (dd1fb33c6ca9fba7d94eed2a13eb3933c60149d3): focused parser tests passed (18 passed), Ruff and git diff --check passed. The full backend suite reached 1832 passed, 2 skipped, 2 failed; both failures are the same pre-existing PostgreSQL smoke-seed root cause (email_records.is_read omitted from raw INSERTs), isolated in dependent PR #1468. This PR keeps that unrelated fixture repair out of the attachment contract. Hosted exact-head Checks remain authoritative.

[seonghobae]

Current head 8bd9c0c60ae05b6161c410240aca7a2d5b05a9d2 only corrects ADR wording so the PR is self-contained before #1427 integrates the separate PDF DOM ADR. Parser Ruff, git diff --check, and backend/tests/test_attachment_parser.py pass (18 passed) on this exact head. The prior full-suite result remains code-equivalent because this commit changes documentation only; hosted exact-head Checks remain authoritative.

[github-actions]

PR governance metadata gate update for 6d6e499d763dcbde6fb9896d9e866b9691b5206b: no current blocking failures remain.

PR governance metadata gate is waiting on current-head requirements; see the latest check for pending reasons.

[seonghobae]

@coderabbitai review

[coderabbitai]

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[seonghobae]

Current-head disposition for de6d712: the four Devin threads are informational. The parser bound now matches the authenticated 64 MiB import ceiling; retained base64 expansion and the external NewsDOM capacity are recorded as the object-storage/sidecar integration gaps. The worker maps sidecar request/response errors to visible recognition failure rather than leaving an unusable payload falsely parsed; only an unconfigured provider remains pending for retry. The oversized test is an intentional boundary test. No additional source change is required in this bounded slice.

[seonghobae]

@coderabbitai review

[coderabbitai]

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[seonghobae]

Maintainer exact-head validation

• Repository: ContextualWisdomLab/naruon
• Pull request: fix(attachments): align deferred parse budget with import transport #1469
• Exact head SHA: 575b0c24fd9cb98106989eb101de74c5ce383db3
• Exact base SHA: e5e99b4e3bb081b92c602358878856536030e2ca
• Changed-surface validation: 109 attachment/import/NewsDOM tests passed; Ruff, compile checks, and git diff --check passed.
• Full selected validation exposed one pre-existing dependency failure in test_data_quality_surface_real_postgres_smoke_uses_signed_scope: the base branch fixture omits required email_records.is_read. That root fix is isolated in PR test(db): align PostgreSQL smoke fixtures with current schema #1468 (exact head 1da167de26b442be6961622f15bb36ae9374e6c4) and is already on the protected normal merge path; no unrelated fixture duplication is added here.
• Current unresolved review threads: 0.
• Decision: keep fix(attachments): align deferred parse budget with import transport #1469 stacked behind the test(db): align PostgreSQL smoke fixtures with current schema #1468 schema-fixture repair; after the base advances, rerun the complete selected suite against this exact head before merge.

[seonghobae]

Exact-head disposition for the prior CodeRabbit changelog observation: current head 575b0c2 already states the bounded range explicitly in CHANGELOG.md: attachments larger than 20 MiB and no larger than 64 MiB, with unsupported binaries remaining metadata-only. ADR-0006 and the doctoring contract state the same boundary. No further source or documentation change is warranted for that observation. The remaining merge blockers are current-head hosted review/security evidence and the protected approval requirement.

[seonghobae]

/coderabbit review

[seonghobae]

@coderabbitai review

[coderabbitai]

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Superseded by current head 258ac41: the requested change is implemented verbatim — the NewsdomPayloadTooLargeError handler now records via logger.info with the operational-information rationale, preserving parse status, error code provider_payload_size_exceeded, and RESULT_FAILED return. All threads resolved; required checks green.

[seonghobae]

Addressed the CodeRabbit round-2 finding in 6d6e499: CHANGELOG.md now names both contract statuses (unsupported_content_type and unsupported_binary) for unsupported binaries, preserving the size-limit descriptions and fail-closed parse_size_limit_exceeded behavior.

Superseded by head 6d6e499: the single requested change is implemented verbatim — CHANGELOG.md now names both contract statuses (unsupported_content_type and unsupported_binary) for unsupported binaries, preserving size-limit descriptions and fail-closed parse_size_limit_exceeded behavior. Thread resolved with the diff reply.

[seonghobae]

현재 head ed4bebeddf05ce1da0c76aca77448deef6254fbb에 최신 develop@042b0c70531b229af3acbd0421a2f23098d848b3을 비강제 merge했습니다. 충돌한 첨부 파서 상수는 PR의 인증된 64 MiB 전송 계약과 base의 제한된 filename decode 계약을 모두 보존해 통합했습니다. uv run --frozen pytest -q -W error tests/test_attachment_parser.py 22개와 Ruff, diff 검사가 통과했습니다. 이전 head의 31개 성공 Check는 현 head에 승계하지 않으며 새 보호 증거를 기다립니다.