feat(export): emit schema-validated NewsDOM JSONL - #820
Conversation
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
NewsDOM JSON을 article/body-block 단위 JSONL로 내보내는 CLI를 추가합니다. 이 exporter의 입력 권위는 별도 ad-hoc dict 규칙이 아니라 기존
newsdom_api.schemas.ParseResponse입니다.RED → causal repair
초기 head
00e9b784badedd398d176667bb1da40bd8029c50은 page/article가 dict가 아니면 조용히 건너뛰고body_blocks가 list가 아니어도 iterable이면 그대로 순회했습니다. 따라서 canonicalParseResponse에 맞지 않는 입력을 성공으로 처리하면서 row를 누락하거나 문자열을 문자 단위 body block으로 내보낼 수 있었습니다.RED
22631eef6e918b3ec53031a6f412899e016a92ff는 세 실제 schema 위반을 고정합니다: non-object page, non-object article, string-valuedbody_blocks. 세 경우 모두 export 전에ValidationError가 발생하고 output file이 생성되지 않아야 합니다. 기존 구현은 이 계약을 만족할 수 없습니다.Repair
165b0a0b1678fbb29732ccc15bc77bf9dd5c7e07은 JSON syntax parsing 뒤ParseResponse.model_validate(data)를 호출하고 typedPageNode/ArticleNode만 순회합니다. 검증은 output file open보다 먼저 끝나므로 schema-invalid DOM은 부분/빈 JSONL을 성공 산출물처럼 남기지 않습니다.document_id,page_number,article_id,headline, orderedbody_blocks의 의미도 canonical schema에서 직접 소비하며 중복 UL/validation contract를 만들지 않습니다.Current exact state — 2026-09-07
develop@e06b1f3fb10903569124af011da213951e6e2473165b0a0b1678fbb29732ccc15bc77bf9dd5c7e07tools/export_jsonl.py,tests/test_tools_export_jsonl.py,CHANGELOG.mdValid articles preserve body-block order and emit one row per block; an article with no body block emits one metadata row with
body_block_index=nulland emptybody_block_text, matching the feature's current documented row contract. Invalid JSON syntax retains the existing ValueError boundary; schema-invalid but syntactically valid JSON now fails closed through Pydantic validation.Do not treat the original
100% coveragesentence as current-head evidence. The branch advanced after the generated claim. Before Ready/merge, the unchanged exact head must obtain applicable test/coverage, Security, SAST, CodeQL and governance GREEN, zero valid unresolved current-head findings, and then-live independent review. Exporting very large NewsDOM files still uses whole-file JSON parsing; if buyer workloads make that a material memory/latency gap, profile representative/right-cleared inputs before introducing a streaming parser or Rust hot path.No self-approval, gate weakening, scanner suppression, force push, destructive rebase, mutable sibling contract, or source-neutral CI retrigger.