feat(export): preserve canonical NewsDOM JSONL artifacts atomically - #791
feat(export): preserve canonical NewsDOM JSONL artifacts atomically#791seonghobae wants to merge 12 commits into
Conversation
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
🔍 연구 근거 자료 누락
새 내보내기 기능에 관련 논문 PDF가 없다. 재배포할 수 없다면 인용, 링크, 요약을 대신 추가해야 한다.
Was this helpful? React with 👍 or 👎 to provide feedback.
|
|
||
| from pydantic import ValidationError | ||
|
|
||
| from newsdom_api.schemas import ParseResponse |
There was a problem hiding this comment.
🟡 직접 실행 시 내보내기 시작 실패
소스 체크아웃에서 python tools/export_jsonl.py ...를 실행하면 newsdom_api.schemas를 찾지 못한다. 도구는 도움말과 내보내기를 시작하기 전에 종료된다.
Prompt for agents
tools/export_jsonl.py가 설치되지 않은 저장소 체크아웃에서도 다른 스키마 기반 CLI처럼 직접 실행되도록 수정한다. tools/validate_dom.py, tools/split_dom.py, tools/anonymize_dom.py는 저장소의 src 디렉터리를 sys.path에 추가한 뒤 newsdom_api.schemas를 import한다. export_jsonl.py에도 같은 실행 경로 지원을 적용하고, src가 sys.path에 없는 상태에서 스크립트의 --help 또는 실제 변환을 runpy/subprocess로 실행하는 회귀 테스트를 추가한다.
Was this helpful? React with 👍 or 👎 to provide feedback.
| except ValidationError as exc: | ||
| raise ValueError("Input is not canonical NewsDOM JSON") from exc | ||
|
|
||
| with output_path.open("w", encoding="utf-8") as jsonlfile: |
There was a problem hiding this comment.
🔴 동일 경로 내보내기가 원본 삭제
입력과 출력에 같은 경로를 지정하면 export_jsonl이 검증된 원본을 JSONL로 덮어쓴다. 성공한 변환 뒤 원본 문서는 복구할 수 없다.
Prompt for agents
tools/export_jsonl.py의 export_jsonl에서 입력 파일과 출력 파일이 같은 실제 파일을 가리키는 경우 쓰기 전에 거부해 원본 손실을 막는다. 상대/절대 경로, 정규화된 동일 경로, 가능한 경우 심볼릭 링크나 하드 링크 별칭도 고려한다. 같은 경로를 전달했을 때 예외가 발생하고 입력 내용이 그대로 유지되는 회귀 테스트를 추가한다.
Was this helpful? React with 👍 or 👎 to provide feedback.
| try: | ||
| document = ParseResponse.model_validate(data) | ||
| except ValidationError as exc: | ||
| raise ValueError("Input is not canonical NewsDOM JSON") from exc | ||
|
|
||
| with output_path.open("w", encoding="utf-8") as jsonlfile: |
There was a problem hiding this comment.
📝 Info: 검증 실패는 쓰기 전에 종료
ParseResponse.model_validate()가 출력 파일을 열기 전에 전체 문서를 검증한다. 잘못된 페이지나 기사는 부분 출력 없이 변환을 중단한다.
Was this helpful? React with 👍 or 👎 to provide feedback.
| article_data = { | ||
| "document_id": document.document_id, | ||
| "page_number": page.page_number, | ||
| **article.model_dump(mode="json"), | ||
| } | ||
| jsonlfile.write(json.dumps(article_data, ensure_ascii=False) + "\n") |
There was a problem hiding this comment.
📝 Info: 기사 출처 정보 보존
기사 전체를 JSON 모드로 직렬화한다. 상위 식별자와 충돌하는 기사 필드가 없어 위치, 이미지, 캡션, 각주가 유지된다.
Was this helpful? React with 👍 or 👎 to provide feedback.
seonghobae
changed the title
1 participant
범위
tools/export_jsonl.py는 NewsDOM의 canonical response model을 그대로 소비해 article 단위 JSONL artifact를 만듭니다. exporter가 별도 schema truth를 만들거나 malformed node를 silent-skip하지 않습니다.develop@e06b1f3fb10903569124af011da213951e6e247310b85cce7b12b6c11d4a9817cba200fb1918e841계약
ParseResponse로 검증합니다.document_id,page_number와 canonicalArticleNode.model_dump(mode="json")전체를 보존합니다.bbox,images,captions,footnotes,body_blocksprovenance를 축소하지 않습니다.os.replace()로 publish하며 실패 시 temp를 제거합니다.RED → GREEN lineage
e1f6e0c85ab886e4d403acb8b452c5222cbcc061→ GREEN6894960e6b67d25a32dfce7d98cf66a0c6497a469fe476719b73befd6342e126c62571d4bf5bcd60, hard-link REDf341bb1f45625662506a1e718eebf94a11d356a6→ GREEN934da87ae774a46a80ff681555015370f63388d34ee1fe616bfd9986d12c12c980cba23546b8a36b→ atomic publish GREEN50fdd8170a3bcf4c24e6e9a3d0f8115013327615docs/product-technical-gap-baseline.md.Successor consolidation
Generated PR #810 adds another
tools/export_jsonl.py, but its valid product intent is fully represented here: JSON input checks,.jsonextension rejection, invalid-JSON error mapping, article-per-line JSONL output, CLI success/error behavior, UTF-8/ensure_ascii=False, tests, and CHANGELOG coverage.#810 additionally silent-skips malformed pages/articles, invents
Unknownidentifiers, and replaces canonicalbody_blocks/provenance with a lossybody_textprojection. Those are alternate schema semantics, not valid canonical NewsDOM contracts, and are intentionally not inherited. Its generated "100% coverage" statement is not exact-head hosted evidence.Exact-head verification state
The latest known runs on unchanged exact head
10b85cce...are terminal but non-passing as a set: tests33908836843, codeql33908836910, ClusterFuzzLite33908836854, container-image33908836836, Security Scan33908836822, SAST33908837011, CodeQL PR33908836983, and scorecards33908836920all concludedcancelled. They are not GREEN and predecessor results do not transfer.Remaining gap / promotion boundary
The implementation still uses
read_text()+json.loads()and therefore does not claim large-dataset streaming or bounded-memory behavior. Follow-up acceptance requires canonical fidelity and atomic publish to survive an incremental path plus real/right-cleared NewsDOM corpus peak-RSS/throughput evidence.Keep Draft until unchanged exact-head applicable tests/security/static-analysis/review gates are terminal-valid. No source-neutral retrigger, self-approval, protection bypass, force update, or predecessor-evidence substitution.