diff --git a/ainiee-translate/SKILL.md b/ainiee-translate/SKILL.md index df02978..d42a148 100644 --- a/ainiee-translate/SKILL.md +++ b/ainiee-translate/SKILL.md @@ -1,13 +1,14 @@ --- name: ainiee-translate -description: Translate books, documents, AiNiee projects, or Unity localization caches end-to-end on the coding agent itself without the AiNiee app. Use for requests such as "用 agent 翻译这本书", "把 epub 翻译了", "翻译 Unity CSV/cache", or "AiNiee-style translation" that need a locked glossary, resumable batches, terminology consistency, validated cache writeback, and export. +description: >- + Translate books, documents, AiNiee projects, Unity localization caches, or flat MTool JSON dictionaries end to end on the coding agent itself without the AiNiee app. Use for requests such as "用 agent 翻译这本书", "把 epub 翻译了", "翻译 Unity CSV/cache", "把这个 MTool JSON 翻译了", or "AiNiee-style translation" that need a locked glossary, resumable batches or bounded subagent groups, terminology consistency, source-aware QA, validated cache writeback, and export. For MTool, preserve the left JSON key as immutable source text and write only the right value as translation. --- # ainiee-translate 技能指南 ## 总览 -本技能让 **编码 agent 本身(Claude Code / Codex 等)** 充当翻译引擎,配合一组确定性 Python 管道脚本,把一本 epub/txt 小说**端到端翻译**: +本技能让 **编码 agent 本身(Claude Code / Codex 等)** 充当翻译引擎,配合一组确定性 Python 管道脚本,把小说、文档或 MTool 本地化字典**端到端翻译**: ``` parse → 构建锁定词汇表 → 逐章翻译(agent 按规则) → 写回缓存 → 导出成品 @@ -22,6 +23,11 @@ parse → 构建锁定词汇表 → 逐章翻译(agent 按规则) → 写回 本技能自带管道脚本(`scripts/ainiee_translate/`),无需单独 `pip install` 本包;解析/导出模块也随技能打包。跨设备复制后,只需要用本目录的 `requirements.txt` 为运行脚本的 Python 安装依赖。 +## 选择工作流 + +- **书籍/普通文档**:继续使用本文件步骤 1–7;按章节或语义边界翻译。 +- **MTool 扁平 JSON**:在采取任何解析、分组、派发或写回动作前,完整阅读 [`references/mtool_translation.md`](references/mtool_translation.md)。MTool 的左侧 key 是运行时匹配原文,绝不改动;右侧 value 才是译文。其完整性交付、技术项直通、子代理并发和进度口径与书籍流程不同。 + --- ## 前置依赖与安装(重要) @@ -147,6 +153,9 @@ mkdir -p "$WORK/work" "$WORK/out" - 找不到时,在 AiNiee 应用的设置/数据目录里定位 `config.json`。 - `--analysis` 可省略(无项目分析时跳过),形如 `.../ProjectCache//AinieeCacheData.json`。 - 生成后,**必须人工 review 并锁定**:检查人名分类(是否保留原文)、地名/种族译法、音译唯一性。复核时特别注意:自动清洗按姓氏末词归并同一角色,可能把**同姓的不同角色**误并为一条——检查每条的 aliases 里没有混入另一个人。 +- 当前项目没有可靠分析缓存,或要补齐本书/本游戏专名时,先运行 `extract_terms` 从 `cache.json` 生成只读候选。默认只将 `confidence: high` 且合并后至少出现 2 行的候选交给人工审校;`resource_hints` 只能辅助比对,绝不进入词典。 +- 将确认译名写入 `term_translations.json` 后运行 `curate_terms` 导出 AiNiee `{src,dst,info}` 表。任一原文或别名对应多个译名时必须阻断导出,不得猜测或静默覆盖。 +- 候选提取、审校和导出不修改 `cache.json`;最终词条仍需并入并锁定 `glossary.locked.json` 后才能作为翻译权威。 - 锁定表格式: ```json @@ -274,6 +283,8 @@ mkdir -p "$WORK/work" "$WORK/out" 完整流程(拆分、抽取、派发模板、风格漂移归一化、串行写回、收尾 verify)见 **`references/parallel_translation.md`**。 +若输入是 MTool JSON,不套用章节拆分模板;改用 **`references/mtool_translation.md`** 的候选选择、试跑估时、原子交付、动态补位和全键验证流程。 + --- ## 步骤 6:导出成品 @@ -346,6 +357,31 @@ verify 是**词汇表执行器**,不是**发现器**。它只能发现「锁 > 经验:写自查脚本做名字比对时,先归一化撇号(弯/直撇号 `'`/`'` 统一),否则 `Mak'ala`、`Quark's`、`O'Brien` 会因撇号不同被误判为「消失」。verify/scan 内部已统一处理。 +### 步骤 7.5:源感知结构审计与审慎修复 + +对 MTool 分组或 AiNiee `cache.json` 转出的译文行,使用源感知审计补足普通 `verify` 的盲区: + +```bash + -m ainiee_translate.audit_translation \ + --source "$WORK/work/source_rows.json" \ + --translation "$WORK/work/translation_rows.json" \ + --glossary "$WORK/work/glossary.locked.json" \ + --repair-manifest "$WORK/work/repair_manifest.json" +``` + +它逐 `text_index` 检查控制符/标签、多类占位符、换行序列、日文对话引号及已锁定术语。`source_rows.json` 与 `translation_rows.json` 可以是批次数组,也可以由脚本接受 AiNiee cache 对象;MTool 左侧 key 仍不可修改。 + +修复必须人工确认:在 manifest 对应项加入 `replacement` 后写入独立文件,再重新审计: + +```bash + -m ainiee_translate.apply_repairs \ + --translation "$WORK/work/translation_rows.json" \ + --manifest "$WORK/work/repair_manifest.json" \ + --output "$WORK/work/translation_rows_repaired.json" +``` + +脚本要求 `expected_before` 精确匹配,拒绝过期修复;不得直接覆盖原始翻译或共享缓存。详细边界见 [`references/source-aware-qa.md`](references/source-aware-qa.md)。 + --- ## 附录 A:命令速查 @@ -366,6 +402,12 @@ $PFX -m ainiee_translate.parse --input book.epub --type AutoType --out "$WORK/wo # 词汇表 $PFX -m ainiee_translate.glossary --config "" --out "$WORK/work/glossary.locked.json" +# 术语候选(只读、待人工确认) +$PFX -m ainiee_translate.extract_terms --input "$WORK/work/cache.json" --output "$WORK/work/term_candidates.json" + +# 审校后导出 AiNiee {src,dst,info} 词典;同源或别名冲突会阻断 +$PFX -m ainiee_translate.curate_terms --candidates "$WORK/work/term_candidates.json" --translations "$WORK/work/term_translations.json" --output "$WORK/work/terms_ainiee.json" + # 用户自定义提示词(汇总 AiNiee 配置里的自定义提示词) $PFX -m ainiee_translate.prompt --config "~/Library/Application Support/AiNiee/config.json" --out work/user_prompt.md @@ -383,6 +425,9 @@ $PFX -m ainiee_translate.verify "$WORK/work/cache.json" "$WORK/work/glossary.loc # 发现(补 verify 盲区:表外被音译/丢失的专名 + 术语漏译 + 幻觉错名 + OCR 粘连词) $PFX -m ainiee_translate.scan "$WORK/work/cache.json" --locked "$WORK/work/glossary.locked.json" --mode all + +# MTool 专项辅助(审计 / 拆组 / 验收 / 进度 / 合并 / 最终键验证) +$PFX -m ainiee_translate.mtool --help ``` --- diff --git a/ainiee-translate/agents/openai.yaml b/ainiee-translate/agents/openai.yaml index 823a8cb..3e84d9c 100644 --- a/ainiee-translate/agents/openai.yaml +++ b/ainiee-translate/agents/openai.yaml @@ -1,4 +1,4 @@ interface: display_name: "AiNiee Agent Translation" - short_description: "Translate books and Unity text caches safely." - default_prompt: "Use $ainiee-translate to translate this book or Unity localization cache with a locked glossary, resumable batches, validation, and safe export." + short_description: "Translate books and MTool dictionaries with safe QA." + default_prompt: "Use $ainiee-translate to translate this book, Unity localization cache, or flat MTool dictionary with a locked glossary, resumable batches, source-aware QA, and safe export." diff --git a/ainiee-translate/references/mtool_translation.md b/ainiee-translate/references/mtool_translation.md new file mode 100644 index 0000000..f2c7a43 --- /dev/null +++ b/ainiee-translate/references/mtool_translation.md @@ -0,0 +1,238 @@ +# MTool 扁平 JSON 翻译工作流 + +用于 `{"原文":"译文"}` 形式的 MTool 游戏本地化字典。先执行审计和小批试跑,再决定是否并行全量翻译。 + +## 目录 + +- [数据契约](#数据契约) +- [阶段 1:审计与可行性门](#阶段-1审计与可行性门) +- [阶段 2:解析、候选选择与直通项](#阶段-2解析候选选择与直通项) +- [阶段 3:小批试跑与估时](#阶段-3小批试跑与估时) +- [阶段 4:拆组与子代理调度](#阶段-4拆组与子代理调度) +- [阶段 5:验收、写回与术语收口](#阶段-5验收写回与术语收口) +- [阶段 6:导出与最终验证](#阶段-6导出与最终验证) +- [取消与异常恢复](#取消与异常恢复) +- [子代理提示词模板](#子代理提示词模板) + +## 数据契约 + +把每个 MTool 条目视为不可交换的映射: + +```json +{ + "左侧运行时原文 key": "右侧显示译文 value" +} +``` + +必须同时满足: + +1. 左侧 key 永远保持逐字符不变;禁止翻译、纠错、规范化、去重或重排 key。 +2. 只修改右侧 value。已有 `key != value` 的条目视为已有译文,默认保留,除非用户明确要求重译。 +3. 最终输出必须与输入拥有相同条目数、相同 key 集合和相同 key 顺序。 +4. 文件名、路径、哈希、脚本命令、资源标识、字符映射表、控制码、占位符和纯技术字符串必须原样直通。 +5. 半句、句首、句尾和解析残片只翻译现有内容,不补全上下文。 +6. MTool writer 只导出 `TRANSLATED`/`POLISHED` 条目;因此不需要翻译的直通项也必须以“译文=原 value(空值需人工判定)”写成已翻译状态,否则导出会漏 key。 + +## 阶段 1:审计与可行性门 + +先确认输入是扁平字符串字典,不要直接按“含日文字符”全量派发:技术映射表也可能含日文或汉字。 + +```bash + -m ainiee_translate.mtool inspect /path/to/game.json +``` + +记录并向用户报告: + +- 总条目数; +- `key == value` 的待判定条目数; +- `key != value` 的已有译文数; +- 空 key/value 数; +- 技术项、资源项和疑似映射表的范围或结构; +- 预计需要人工翻译的候选条目数。 + +在候选量很大时必须先过可行性门:不要用“创建了多少代理”代替完成时间估算。先完成阶段 3 的试跑,用真实吞吐量计算墙钟时间;把估算和替代方案(更小范围、外部 API、本地翻译模型等)告诉用户,再继续大规模投放。 + +## 阶段 2:解析、候选选择与直通项 + +显式指定 MTool 类型,避免 `.json` 内容检测歧义: + +```bash + -m ainiee_translate.parse \ + --input /path/to/game.json \ + --type Mtool \ + --out "$WORK/work/cache.json" +``` + +由主代理审计并生成 `candidate_indices.json`,内容可以是整数数组,也可以是含 `text_index` 的记录数组。选择时遵循: + +- 默认只重译 `key == value` 且确属玩家可见源语言文本的条目; +- 保留 `key != value` 的已有译文; +- 排除路径、文件名、哈希、命令、资源键、纯数字/符号和内部字符映射; +- 对连续的大型映射区间按结构和样本判定,不只靠语言正则; +- 抽查候选边界附近条目,避免把技术区切进译文组。 + +生成所有非候选条目的直通写回文件: + +```bash + -m ainiee_translate.mtool passthrough \ + --cache "$WORK/work/cache.json" \ + --selection "$WORK/work/candidate_indices.json" \ + --output "$WORK/work/mtool_passthrough.json" + + -m ainiee_translate.batch write \ + "$WORK/work/cache.json" \ + "$WORK/work/mtool_passthrough.json" +``` + +只允许主代理执行 `batch write`。写后重新读取缓存状态,确认未翻译数恰好等于候选数,再拆组。 + +## 阶段 3:小批试跑与估时 + +先翻译 1–2 个小组并锁定风格。MTool 字符串通常短而碎,不按固定 500 行起步;推荐初始目标: + +- 每组 100–200 条; +- 同时限制源文总字符数在约 2,000–6,000; +- 同一语境连续片段尽量放在同组; +- 第一批只开 1–2 个代理,验收质量和实际耗时后再扩大并发。 + +使用真实数据估时: + +```text +预计剩余时间 ≈ 候选总行数 ÷ 已验收行吞吐率 ÷ 实际可用并发槽位 +``` + +按完成试跑组的墙钟时间、有效行数和重做率计算。若预计耗时明显超出用户预期,暂停并报告,不要用更多排队任务掩盖吞吐不足。 + +## 阶段 4:拆组与子代理调度 + +在缓存已完成直通写回后拆分剩余未译项: + +```bash + -m ainiee_translate.mtool split \ + --cache "$WORK/work/cache.json" \ + --output-dir "$WORK/work/mtool_groups" \ + --size 150 \ + --max-chars 5000 +``` + +脚本生成 `mtool_NNN_src.json` 和 `manifest.json`。不要让多个代理处理同一组。 + +### 主代理职责 + +- 持有唯一任务队列和唯一写回权限;禁止子代理再创建子代理。 +- 先逐个尝试创建代理以发现当前真实并发上限;达到 `agent thread limit reached` 后停止重试,把余下组放入队列。 +- 同时运行数以实测上限为准,不把“用户想投放 10 个”误报成“10 个同时运行”。分别报告总分组数、运行槽位数和排队数。 +- 已完成代理仍占槽位;验收完成后立即关闭,再投放下一组。 +- 不因 `wait` 超时而中断代理;超时只表示尚未完成。 +- 仅在最终文件存在、内容稳定且验证通过后,才允许关闭一个仍在收尾的代理。 + +### 子代理唯一写入范围 + +每个子代理只能写自己的: + +- `mtool_NNN_trans.json.tmp`(构建中); +- 验证通过后原子重命名为 `mtool_NNN_trans.json`; +- 可选 `mtool_NNN_newterms.json`。 + +禁止子代理修改原始 MTool JSON、`cache.json`、锁定词汇表、项目提示词、风格指南或其他组文件;禁止执行 `batch write`、`export` 或合并。 + +### 原子交付 + +不要把正在增长的正式 JSON 文件当成进度。子代理先写 `.tmp`,完整构建后运行: + +```bash + -m ainiee_translate.mtool validate \ + --source "$WORK/work/mtool_groups/mtool_NNN_src.json" \ + --translation "$WORK/work/mtool_groups/mtool_NNN_trans.json.tmp" +``` + +验证成功后再把 `.tmp` 原子改名为正式 `_trans.json`。主代理只读取正式文件。 + +### 进度口径 + +```bash + -m ainiee_translate.mtool progress \ + --groups-dir "$WORK/work/mtool_groups" +``` + +只用“已通过验证的译文行数 ÷ 候选总行数”作为主进度。完成代理数/总分组数只作辅助,因为最后一组大小可能不同;创建成功、出现临时文件、代理口头报告或文件非空都不计进度。 + +## 阶段 5:验收、写回与术语收口 + +每个正式输出都必须通过: + +```bash + -m ainiee_translate.mtool validate \ + --source "$WORK/work/mtool_groups/mtool_NNN_src.json" \ + --translation "$WORK/work/mtool_groups/mtool_NNN_trans.json" +``` + +验证至少覆盖:JSON 可解析、UTF-8 BOM 兼容、条数相等、`text_index` 集合与顺序完全一致、可选 `source_text` 未被改动、译文非空、换行/转义/占位符/标签计数一致。 + +先由主代理审阅各组 `newterms`,统一后再更新锁定表。子代理提出的术语只是候选,不能自动成为唯一真相源。 + +大型缓存不要让每个子代理各自写回。可把一个已验收波次合并成一个文件,再由主代理执行一次串行写回: + +```bash + -m ainiee_translate.mtool merge \ + --groups-dir "$WORK/work/mtool_groups" \ + --output "$WORK/work/mtool_all_translations.json" + + -m ainiee_translate.batch write \ + "$WORK/work/cache.json" \ + "$WORK/work/mtool_all_translations.json" +``` + +`merge` 默认要求所有源组都有通过验证的正式输出。若按波次写回,先把该波次文件放入独立目录再合并;无论采用逐组还是逐波次,所有缓存写入都必须串行且由主代理执行。 + +## 阶段 6:导出与最终验证 + +完成翻译后先确认无未译候选: + +```bash + -m ainiee_translate.batch read "$WORK/work/cache.json" --size 1 +``` + +结果必须是 `[]`。随后导出: + +```bash + -m ainiee_translate.export \ + --cache "$WORK/work/cache.json" \ + --output "$WORK/out" \ + --input /path/to/game.json +``` + +最后验证 MTool 左右栏契约: + +```bash + -m ainiee_translate.mtool verify-output \ + --input /path/to/game.json \ + --output "$WORK/out/game_translated.json" +``` + +只有在以下全部成立时才报告完成: + +- 输入/输出条目数相同; +- key 集合和顺序完全相同; +- 所有 value 都是字符串,非预期空值为零; +- 控制标记检查通过; +- 原始输入文件仍未被覆盖; +- 最终导出文件可解析且路径明确。 + +## 取消与异常恢复 + +- 用户取消时立即停止投放、终止等待并关闭所有运行代理;不要写回尚未验收的结果,也不要导出“部分成品”。 +- 保留已验收分组和项目缓存,除非用户明确要求删除;说明哪些内容可复用、哪些只是临时文件。 +- `not_found`、`shutdown`、工具超时或客户端更新都属于不确定状态。先检查正式输出、`.tmp`、代理通知和文件稳定性,再决定重派;不要立即复制任务导致同组并发写入。 +- 正式文件若缺行、JSON 解析失败或仍在变化,按未完成处理,不得计入进度或写回。 +- 重派只重派失败组,不重跑已验收组。 + +## 子代理提示词模板 + +> 你是 MTool 游戏文本翻译执行代理,只负责 `{GROUP}`。完整阅读 `{STYLE_GUIDE}`、`{USER_PROMPT}`、`{GLOSSARY}` 和技能的 `references/translation_rules.md`。 +> +> MTool 契约:左侧原文/key 永远不变。输入 `{SRC}` 是 `{text_index, source_text}` 数组;只生成等长、同顺序的 `{text_index, translated_text}` 数组。不得修改 `text_index`、`source_text`、条目顺序、换行、转义、占位符、标签、文件名、路径或控制码。遇到半句只翻译已有片段,不补上下文。术语以锁定表为准,表外新术语记录到 `{NEWTERMS}`,不要修改锁定表。 +> +> 先把完整结果写到 `{TRANS}.tmp`,运行 `python -m ainiee_translate.mtool validate --source "{SRC}" --translation "{TRANS}.tmp"`;只有验证成功后才原子改名为 `{TRANS}`。不要改原始 JSON、`cache.json`、规则文件或其他组,不要执行 `batch write`/`export`,不要创建子代理。 +> +> 最终只报告正式输出路径、通过验证的条数和新术语数。 diff --git a/ainiee-translate/references/parallel_translation.md b/ainiee-translate/references/parallel_translation.md index d00543a..37dbf9b 100644 --- a/ainiee-translate/references/parallel_translation.md +++ b/ainiee-translate/references/parallel_translation.md @@ -1,5 +1,7 @@ # 多 agent 并行翻译(大书加速) +> 本页针对书籍/章节型输入。扁平 MTool JSON 必须改读 [`mtool_translation.md`](mtool_translation.md):它要求左 key 永久不变、技术项完整直通、输出原子落盘,并按已验收行数而不是代理通知计算进度。 + 把步骤 5 的逐章串行翻译,改为**多个 subagent 同时翻译不同章节范围**,墙钟时间≈最慢的那个 agent。 本流程在一本 ~2400 段的小说上实测:11 章 1704 段由 7 个 agent 并发,约 9 分钟译完(串行需十几轮)。 diff --git a/ainiee-translate/references/source-aware-qa.md b/ainiee-translate/references/source-aware-qa.md new file mode 100644 index 0000000..3010a43 --- /dev/null +++ b/ainiee-translate/references/source-aware-qa.md @@ -0,0 +1,12 @@ +# Source-Aware Translation QA + +Use `audit_translation.py` after batch validation and before export. It compares +each translated row with its exact source occurrence and reports control/tag +tokens, placeholders, newline sequence, Japanese dialogue quotes, and locked +term omissions. The audit is fail-closed for structural mismatches. + +The output is a review manifest, not an automatic repair. Add a `replacement` +only after confirming the intended text. `apply_repairs.py` requires the exact +`expected_before` value, writes a separate output file, and refuses stale rows. +Run the audit again after repair. Do not use this workflow to bypass Unity +occurrence manifests or MTool key immutability. diff --git a/ainiee-translate/scripts/ainiee_translate/apply_repairs.py b/ainiee-translate/scripts/ainiee_translate/apply_repairs.py new file mode 100644 index 0000000..da0ad16 --- /dev/null +++ b/ainiee-translate/scripts/ainiee_translate/apply_repairs.py @@ -0,0 +1,61 @@ +"""Apply only explicitly reviewed source-aware repairs.""" + +from __future__ import annotations + +import argparse +import json +from pathlib import Path +from typing import Any + + +def read_json(path: Path) -> Any: + with path.open("r", encoding="utf-8-sig") as handle: + return json.load(handle) + + +def main() -> int: + parser = argparse.ArgumentParser(description="Apply reviewed expected_before repair entries") + parser.add_argument("--translation", required=True, type=Path) + parser.add_argument("--manifest", required=True, type=Path) + parser.add_argument("--output", required=True, type=Path) + args = parser.parse_args() + translation = read_json(args.translation) + if not isinstance(translation, list): + parser.error("translation must be a JSON row list") + manifest = read_json(args.manifest) + entries = manifest.get("items", []) if isinstance(manifest, dict) else [] + replacements: dict[str, dict[str, Any]] = {} + for item in entries: + if not isinstance(item, dict) or "replacement" not in item: + continue + index = str(item.get("text_index")) + if index in replacements: + parser.error(f"duplicate repair text_index {item.get('text_index')}") + replacements[index] = item + row_indexes = [str(row.get("text_index")) for row in translation if isinstance(row, dict)] + if len(row_indexes) != len(set(row_indexes)): + parser.error("translation contains duplicate text_index values") + missing = sorted(set(replacements) - set(row_indexes)) + if missing: + parser.error(f"repair text_index values not found: {', '.join(missing[:20])}") + changed = 0 + for row in translation: + if not isinstance(row, dict): + continue + item = replacements.get(str(row.get("text_index"))) + if not item: + continue + current = str(row.get("translated_text", "") or "") + expected_before = str(item.get("expected_before", "") or "") + if current != expected_before: + parser.error(f"stale expected_before for text_index {row.get('text_index')}") + row["translated_text"] = str(item["replacement"]) + changed += 1 + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(translation, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") + print(json.dumps({"changed": changed, "output": str(args.output)}, ensure_ascii=False, indent=2)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/ainiee-translate/scripts/ainiee_translate/audit_translation.py b/ainiee-translate/scripts/ainiee_translate/audit_translation.py new file mode 100644 index 0000000..87fb6ef --- /dev/null +++ b/ainiee-translate/scripts/ainiee_translate/audit_translation.py @@ -0,0 +1,124 @@ +"""Audit source-aware translation invariants and emit a repair manifest.""" + +from __future__ import annotations + +import argparse +import json +import re +from collections import Counter +from pathlib import Path +from typing import Any + + +TOKEN_RE = re.compile( + r"\\[A-Za-z]+\[[^\]\r\n]*\]|\\[.!|^]|\$[A-Za-z_][A-Za-z0-9_]*|%[A-Za-z_][A-Za-z0-9_]*%|\{\{[^{}\r\n]+\}\}|\{\d+\}|<[/A-Za-z][^<>\r\n]*>" +) +PLACEHOLDER_RE = re.compile(r"%(?:\d+\$)?[+#0\- ]*\d*(?:\.\d+)?[A-Za-z]|\$\{[^}\r\n]+\}") +JP_QUOTE_CHARS = "「」『』" + + +def read_json(path: Path) -> Any: + with path.open("r", encoding="utf-8-sig") as handle: + return json.load(handle) + + +def rows(value: Any) -> list[dict[str, Any]]: + if isinstance(value, list): + return [row for row in value if isinstance(row, dict)] + if isinstance(value, dict) and isinstance(value.get("files"), dict): + result = [] + for file_data in value["files"].values(): + if isinstance(file_data, dict) and isinstance(file_data.get("items"), list): + result.extend(row for row in file_data["items"] if isinstance(row, dict)) + return result + raise ValueError("input must be a row list or AiNiee cache object") + + +def signature(text: str) -> dict[str, Any]: + return { + "tokens": dict(Counter(TOKEN_RE.findall(text))), + "placeholders": dict(Counter(PLACEHOLDER_RE.findall(text))), + "newlines": re.findall(r"\r\n|\r|\n", text), + "quotes": {char: text.count(char) for char in JP_QUOTE_CHARS if char in text}, + } + + +def glossary_issues(source: str, target: str, glossary: dict[str, Any] | None) -> list[dict[str, Any]]: + if not glossary: + return [] + issues = [] + for term in glossary.get("terms", []): + if not isinstance(term, dict) or term.get("keep_source"): + continue + src, dst = str(term.get("src", "")), str(term.get("dst", "")) + if src and dst and src in source and dst not in target: + issues.append({"kind": "term_missing", "detail": f"{src} -> {dst} not found in translation"}) + return issues + + +def index_rows(values: list[dict[str, Any]], label: str) -> tuple[dict[str, dict[str, Any]], list[dict[str, Any]]]: + indexed: dict[str, dict[str, Any]] = {} + issues: list[dict[str, Any]] = [] + for position, row in enumerate(values, 1): + raw = row.get("text_index") + if raw is None or isinstance(raw, bool): + issues.append({"kind": f"invalid_{label}_text_index", "position": position, "value": raw}) + continue + index = str(raw) + if index in indexed: + issues.append({"kind": f"duplicate_{label}_text_index", "text_index": raw}) + continue + indexed[index] = row + return indexed, issues + + +def audit(source_rows: list[dict[str, Any]], translation_rows: list[dict[str, Any]], glossary: dict[str, Any] | None) -> dict[str, Any]: + source_by_index, source_index_issues = index_rows(source_rows, "source") + translation_by_index, translation_index_issues = index_rows(translation_rows, "translation") + issues: list[dict[str, Any]] = [*source_index_issues, *translation_index_issues] + for index in sorted(set(translation_by_index) - set(source_by_index)): + issues.append({"kind": "extra_translation_row", "text_index": translation_by_index[index].get("text_index")}) + for index, source_row in source_by_index.items(): + target_row = translation_by_index.get(index) + if target_row is None: + issues.append({"kind": "missing_row", "text_index": source_row.get("text_index")}) + continue + source = str(source_row.get("source_text", "")) + target = str(target_row.get("translated_text", target_row.get("translation", "")) or "") + if not target.strip() and source.strip(): + issues.append({"kind": "empty_translation", "text_index": source_row.get("text_index"), "expected_before": target}) + continue + expected = signature(source) + actual = signature(target) + if expected["tokens"] != actual["tokens"]: + issues.append({"kind": "control_or_tag_mismatch", "text_index": source_row.get("text_index"), "expected": expected["tokens"], "actual": actual["tokens"], "expected_before": target}) + if expected["placeholders"] != actual["placeholders"]: + issues.append({"kind": "placeholder_mismatch", "text_index": source_row.get("text_index"), "expected": expected["placeholders"], "actual": actual["placeholders"], "expected_before": target}) + if expected["newlines"] != actual["newlines"]: + issues.append({"kind": "newline_mismatch", "text_index": source_row.get("text_index"), "expected": expected["newlines"], "actual": actual["newlines"], "expected_before": target}) + if expected["quotes"] and expected["quotes"] != actual["quotes"]: + issues.append({"kind": "dialogue_quote_mismatch", "text_index": source_row.get("text_index"), "expected": expected["quotes"], "actual": actual["quotes"], "expected_before": target}) + for issue in glossary_issues(source, target, glossary): + issue.update({"text_index": source_row.get("text_index"), "expected_before": target}) + issues.append(issue) + return {"valid": not issues, "source_rows": len(source_rows), "translation_rows": len(translation_rows), "issue_count": len(issues), "issues": issues} + + +def main() -> int: + parser = argparse.ArgumentParser(description="Audit source-aware translation invariants") + parser.add_argument("--source", required=True, type=Path) + parser.add_argument("--translation", required=True, type=Path) + parser.add_argument("--glossary", type=Path) + parser.add_argument("--repair-manifest", type=Path) + args = parser.parse_args() + glossary = read_json(args.glossary) if args.glossary else None + report = audit(rows(read_json(args.source)), rows(read_json(args.translation)), glossary) + if args.repair_manifest: + args.repair_manifest.parent.mkdir(parents=True, exist_ok=True) + args.repair_manifest.write_text(json.dumps({"format": "ainiee-source-repair-v1", "items": report["issues"]}, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") + print(json.dumps(report, ensure_ascii=False, indent=2)) + return 0 if report["valid"] else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/ainiee-translate/scripts/ainiee_translate/curate_terms.py b/ainiee-translate/scripts/ainiee_translate/curate_terms.py new file mode 100644 index 0000000..71f689a --- /dev/null +++ b/ainiee-translate/scripts/ainiee_translate/curate_terms.py @@ -0,0 +1,71 @@ +"""Export reviewed candidates as an AiNiee {src, dst, info} term table.""" + +from __future__ import annotations + +import argparse +import json +from pathlib import Path +from typing import Any + + +def read_json(path: Path) -> Any: + with path.open("r", encoding="utf-8-sig") as handle: + return json.load(handle) + + +def translation_rows(value: Any) -> list[dict[str, Any]]: + if isinstance(value, dict): + return [{"src": source, "dst": target} for source, target in value.items()] + if isinstance(value, list): + return [row for row in value if isinstance(row, dict)] + raise ValueError("translations must be a source-to-target object or a row list") + + +def main() -> int: + parser = argparse.ArgumentParser(description="Curate reviewed term candidates into AiNiee format") + parser.add_argument("--candidates", required=True, type=Path) + parser.add_argument("--translations", required=True, type=Path) + parser.add_argument("--output", required=True, type=Path) + parser.add_argument("--min-frequency", type=int, default=2) + parser.add_argument("--include-medium", action="store_true") + args = parser.parse_args() + candidate_data = read_json(args.candidates) + candidates = candidate_data.get("candidates", []) if isinstance(candidate_data, dict) else [] + known = {str(row.get("source")): row for row in candidates if isinstance(row, dict)} + mappings: dict[str, set[str]] = {} + owners: dict[str, set[str]] = {} + for row in translation_rows(read_json(args.translations)): + source = str(row.get("src", row.get("source", ""))).strip() + target = str(row.get("dst", row.get("target", ""))).strip() + if not source or not target: + continue + mappings.setdefault(source, set()).add(target) + for alias in [source, *list(row.get("aliases") or [])]: + owners.setdefault(str(alias), set()).add(target) + conflicts = {source: sorted(targets) for source, targets in mappings.items() if len(targets) > 1} + alias_conflicts = {alias: sorted(targets) for alias, targets in owners.items() if len(targets) > 1} + if conflicts or alias_conflicts: + print(json.dumps({"status": "blocked", "conflicts": conflicts, "alias_conflicts": alias_conflicts}, ensure_ascii=False, indent=2)) + return 1 + output = [] + skipped_unknown = [] + for source, targets in mappings.items(): + candidate = known.get(source) + if not candidate: + skipped_unknown.append(source) + continue + if int(candidate.get("frequency", 0)) < args.min_frequency: + continue + if candidate.get("confidence") != "high" and not args.include_medium: + continue + categories = ", ".join(candidate.get("categories", [])) + output.append({"src": source, "dst": next(iter(targets)), "info": f"{categories}; source rows: {candidate.get('frequency', 0)}"}) + output.sort(key=lambda row: (-len(row["src"]), row["src"])) + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(output, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") + print(json.dumps({"status": "written", "entries": len(output), "skipped_unknown": skipped_unknown[:20], "output": str(args.output)}, ensure_ascii=False, indent=2)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/ainiee-translate/scripts/ainiee_translate/extract_terms.py b/ainiee-translate/scripts/ainiee_translate/extract_terms.py new file mode 100644 index 0000000..69325de --- /dev/null +++ b/ainiee-translate/scripts/ainiee_translate/extract_terms.py @@ -0,0 +1,153 @@ +"""Extract conservative terminology candidates from an AiNiee cache. + +This is a review aid only. It never assigns translations and never edits the +cache. Candidates are derived from source text shape and repetition so the +agent can confirm them against context before locking a glossary entry. +""" + +from __future__ import annotations + +import argparse +import json +import re +from collections import defaultdict +from pathlib import Path +from typing import Any, Iterable + + +KANJI = r"一-龯々ヶ" +KANA = r"ぁ-ゖァ-ヺー" +READING_RE = re.compile(rf"([{KANJI}]{{1,20}})[((]([{KANA}]{{1,30}})[))]") +KATAKANA_RE = re.compile(rf"[{r'ァ-ヺー'}]{{2,24}}") +JAPANESE_RE = re.compile(rf"[{KANJI}{KANA}]") +TRAILING_VARIANT_RE = re.compile(r"(?:[++?!?!~~]|[_-]?[0-9]{1,3}|[A-Z])$") +PUNCTUATION_RE = re.compile(r"[。!?!?、,,。..「」『』【】()()::;;…]+") + + +def read_json(path: Path) -> Any: + with path.open("r", encoding="utf-8-sig") as handle: + return json.load(handle) + + +def iter_items(value: Any) -> Iterable[dict[str, Any]]: + if isinstance(value, list): + yield from (item for item in value if isinstance(item, dict)) + return + if not isinstance(value, dict): + raise ValueError("input must be an AiNiee cache object or a list of rows") + files = value.get("files") + if not isinstance(files, dict): + raise ValueError("cache object must contain a files mapping") + for file_data in files.values(): + if not isinstance(file_data, dict): + continue + items = file_data.get("items", []) + if isinstance(items, list): + yield from (item for item in items if isinstance(item, dict)) + + +def normalize_term(value: str) -> str: + value = value.strip(" \t ") + while value: + previous = value + value = re.sub(r"[((][^()()\r\n]{0,30}[))]$", "", value) + value = TRAILING_VARIANT_RE.sub("", value) + value = value.strip(" \t ") + if value == previous: + break + return value + + +def safe_label(value: str) -> bool: + value = value.strip() + return ( + 2 <= len(value) <= 32 + and bool(JAPANESE_RE.search(value)) + and not PUNCTUATION_RE.search(value) + and "\\" not in value + and "<" not in value + and ">" not in value + ) + + +def add(book: dict[str, dict[str, Any]], term: str, category: str, item: dict[str, Any]) -> None: + term = normalize_term(term) + if not safe_label(term): + return + entry = book.setdefault( + term, + { + "source": term, + "normalized": term, + "categories": set(), + "occurrences": [], + "row_keys": set(), + }, + ) + entry["categories"].add(category) + row_key = str(item.get("text_index", len(entry["row_keys"]))) + if row_key not in entry["row_keys"]: + entry["row_keys"].add(row_key) + if len(entry["occurrences"]) < 5: + entry["occurrences"].append( + {"text_index": item.get("text_index"), "source_text": item.get("source_text", "")} + ) + + +def extract(items: Iterable[dict[str, Any]], min_frequency: int) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]: + book: dict[str, dict[str, Any]] = {} + resource_hints: dict[str, dict[str, Any]] = defaultdict(lambda: {"resource": "", "frequency": 0}) + for item in items: + source = str(item.get("source_text", "")) + if not source.strip(): + continue + for match in READING_RE.finditer(source): + add(book, match.group(0), "reading_form", item) + for match in KATAKANA_RE.finditer(source): + add(book, match.group(0), "katakana", item) + if safe_label(source): + add(book, source, "short_label", item) + if re.fullmatch(r"[A-Za-z0-9_./\\-]{3,160}", source.strip()) and ("/" in source or "\\" in source or "." in source): + hint = resource_hints[source.strip()] + hint["resource"] = source.strip() + hint["frequency"] += 1 + + candidates = [] + for entry in book.values(): + frequency = len(entry["row_keys"]) + if frequency < min_frequency: + continue + candidates.append( + { + "source": entry["source"], + "normalized": entry["normalized"], + "categories": sorted(entry["categories"]), + "confidence": "high" if "reading_form" in entry["categories"] or "short_label" in entry["categories"] else "medium", + "frequency": frequency, + "requires_review": True, + "occurrences": entry["occurrences"], + } + ) + candidates.sort(key=lambda item: (-({"high": 3, "medium": 2}[item["confidence"]]), -item["frequency"], item["source"])) + hints = sorted(resource_hints.values(), key=lambda item: (-item["frequency"], item["resource"])) + return candidates, hints + + +def main() -> int: + parser = argparse.ArgumentParser(description="Extract conservative glossary candidates from an AiNiee cache") + parser.add_argument("--input", required=True, type=Path) + parser.add_argument("--output", required=True, type=Path) + parser.add_argument("--min-frequency", type=int, default=2) + args = parser.parse_args() + if args.min_frequency < 1: + parser.error("--min-frequency must be positive") + candidates, hints = extract(iter_items(read_json(args.input)), args.min_frequency) + payload = {"candidates": candidates, "resource_hints": hints, "requires_review": True} + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(payload, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") + print(json.dumps({"candidates": len(candidates), "resource_hints": len(hints), "output": str(args.output)}, ensure_ascii=False, indent=2)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/ainiee-translate/scripts/ainiee_translate/mtool.py b/ainiee-translate/scripts/ainiee_translate/mtool.py new file mode 100644 index 0000000..08ab4c8 --- /dev/null +++ b/ainiee-translate/scripts/ainiee_translate/mtool.py @@ -0,0 +1,494 @@ +"""Deterministic helpers for flat MTool JSON translation projects. + +The MTool contract is ``{source_key: translated_value}``: source keys are +immutable, while only values may change. This module does not translate text +and never writes cache.json; it prepares auditable files for the primary agent. +""" + +from __future__ import annotations + +import argparse +import json +import re +from collections import Counter +from pathlib import Path +from typing import Any + +CONTROL_TOKEN_RE = re.compile( + r"\\[A-Za-z]+\[[^\]\r\n]*\]" # RPG-style control code, e.g. \V[1] + r"|\{[^{}\r\n]+\}" # brace placeholder + r"|%(?:\d+\$)?[-+#0-9.]*[A-Za-z%]" # printf-style placeholder + r"|\$[A-Za-z_][A-Za-z0-9_]*" # dollar variable + r"|\r\n]*?>" # markup tag; <なし> is visible text, not a tag +) + + +def read_json(path: str | Path) -> Any: + return json.loads(Path(path).read_text(encoding="utf-8-sig")) + + +def write_json_atomic(path: str | Path, data: Any) -> None: + target = Path(path) + target.parent.mkdir(parents=True, exist_ok=True) + temp = target.with_name(target.name + ".tmp") + temp.write_text( + json.dumps(data, ensure_ascii=False, indent=2) + "\n", + encoding="utf-8", + ) + temp.replace(target) + + +def require_flat_mtool(data: Any, label: str) -> dict[str, str]: + if not isinstance(data, dict): + raise ValueError(f"{label} must be a top-level JSON object") + bad = [k for k, v in data.items() if not isinstance(k, str) or not isinstance(v, str)] + if bad: + raise ValueError(f"{label} must contain only string keys and string values") + return data + + +def load_selection(path: str | Path) -> set[int]: + data = read_json(path) + if isinstance(data, dict): + for key in ("candidate_indices", "indices", "text_indices"): + if key in data: + data = data[key] + break + if not isinstance(data, list): + raise ValueError("selection must be a JSON array or contain candidate_indices/indices") + indices: list[int] = [] + for item in data: + value = item.get("text_index") if isinstance(item, dict) else item + indices.append(int(value)) + if len(indices) != len(set(indices)): + raise ValueError("selection contains duplicate text_index values") + return set(indices) + + +def load_cache_items(path: str | Path) -> list[dict[str, Any]]: + data = read_json(path) + if not isinstance(data, dict) or not isinstance(data.get("files"), dict): + raise ValueError("cache must be an AiNiee CacheProject JSON object with a files mapping") + items: list[dict[str, Any]] = [] + for cache_file in data["files"].values(): + if not isinstance(cache_file, dict) or not isinstance(cache_file.get("items"), list): + raise ValueError("every cache file must contain an items array") + for item in cache_file["items"]: + if not isinstance(item, dict): + raise ValueError("cache items must be JSON objects") + items.append(item) + return items + + +def marker_signature(text: str) -> dict[str, Any]: + crlf = text.count("\r\n") + lf = text.count("\n") - crlf + cr = text.count("\r") - crlf + tokens = Counter(CONTROL_TOKEN_RE.findall(text)) + return { + "actual_crlf": crlf, + "actual_lf": lf, + "actual_cr": cr, + "literal_crlf": text.count(r"\r\n"), + "literal_lf": text.count(r"\n") - text.count(r"\r\n"), + "literal_cr": text.count(r"\r") - text.count(r"\r\n"), + "dollar": text.count("$"), + "percent": text.count("%"), + "tokens": sorted(tokens.items()), + } + + +def validate_records(source: Any, translation: Any) -> dict[str, Any]: + errors: list[str] = [] + marker_issues: list[dict[str, Any]] = [] + unchanged = 0 + + if not isinstance(source, list): + return {"valid": False, "errors": ["source is not a JSON array"]} + if not isinstance(translation, list): + return {"valid": False, "errors": ["translation is not a JSON array"]} + + try: + source_ids = [int(row["text_index"]) for row in source] + translation_ids = [int(row["text_index"]) for row in translation] + except (KeyError, TypeError, ValueError): + return {"valid": False, "errors": ["every row must contain an integer text_index"]} + + if len(source_ids) != len(set(source_ids)): + errors.append("source contains duplicate text_index values") + if len(translation_ids) != len(set(translation_ids)): + errors.append("translation contains duplicate text_index values") + if source_ids != translation_ids: + errors.append( + f"text_index/order mismatch: source={len(source_ids)} translation={len(translation_ids)}" + ) + + source_by_id = { + int(row["text_index"]): row + for row in source + if isinstance(row, dict) and "text_index" in row + } + empty_ids: list[int] = [] + changed_source_ids: list[int] = [] + invalid_value_ids: list[int] = [] + + for row in translation: + if not isinstance(row, dict) or "text_index" not in row: + continue + text_index = int(row["text_index"]) + translated = row.get("translated_text") + if not isinstance(translated, str): + invalid_value_ids.append(text_index) + continue + if not translated.strip(): + empty_ids.append(text_index) + src = source_by_id.get(text_index) + if src is None: + continue + source_text = src.get("source_text") + if not isinstance(source_text, str): + errors.append(f"source_text is not a string at text_index {text_index}") + continue + if "source_text" in row and row["source_text"] != source_text: + changed_source_ids.append(text_index) + if translated == source_text: + unchanged += 1 + source_markers = marker_signature(source_text) + translated_markers = marker_signature(translated) + if source_markers != translated_markers: + marker_issues.append( + { + "text_index": text_index, + "source": source_markers, + "translation": translated_markers, + } + ) + + if invalid_value_ids: + errors.append(f"translated_text is missing or non-string at {len(invalid_value_ids)} row(s)") + if empty_ids: + errors.append(f"empty translated_text at {len(empty_ids)} row(s)") + if changed_source_ids: + errors.append(f"source_text was changed at {len(changed_source_ids)} row(s)") + if marker_issues: + errors.append(f"control marker mismatch at {len(marker_issues)} row(s)") + + return { + "valid": not errors, + "source_rows": len(source), + "translation_rows": len(translation), + "unchanged_rows": unchanged, + "errors": errors, + "empty_text_indices": empty_ids[:20], + "changed_source_indices": changed_source_ids[:20], + "invalid_value_indices": invalid_value_ids[:20], + "marker_issue_count": len(marker_issues), + "marker_issues": marker_issues[:20], + } + + +def group_paths(groups_dir: Path, prefix: str) -> list[Path]: + return sorted(groups_dir.glob(f"{prefix}_*_src.json")) + + +def translation_path(source_path: Path) -> Path: + if not source_path.name.endswith("_src.json"): + raise ValueError(f"unexpected source filename: {source_path.name}") + return source_path.with_name(source_path.name[:-9] + "_trans.json") + + +def cmd_inspect(args: argparse.Namespace) -> None: + path = Path(args.input) + data = require_flat_mtool(read_json(path), str(path)) + report = { + "input": str(path), + "entries": len(data), + "key_equals_value": sum(k == v for k, v in data.items()), + "key_differs_from_value": sum(k != v for k, v in data.items()), + "blank_keys": sum(not k for k in data), + "blank_values": sum(not v for v in data.values()), + "source_characters": sum(len(k) for k in data), + "value_characters": sum(len(v) for v in data.values()), + } + print(json.dumps(report, ensure_ascii=False, indent=2)) + + +def cmd_passthrough(args: argparse.Namespace) -> None: + selection = load_selection(args.selection) + items = load_cache_items(args.cache) + all_ids = {int(item["text_index"]) for item in items} + unknown = sorted(selection - all_ids) + if unknown: + raise ValueError(f"selection contains unknown text_index values: {unknown[:20]}") + + rows = [] + filled_from_key = 0 + for item in items: + text_index = int(item["text_index"]) + if text_index in selection or int(item.get("translation_status", 0)) != 0: + continue + translated = item.get("translated_text", "") + if translated == "": + translated = item.get("source_text", "") + filled_from_key += 1 + rows.append({"text_index": text_index, "translated_text": translated}) + + write_json_atomic(args.output, rows) + print( + json.dumps( + { + "output": str(args.output), + "candidate_rows": len(selection), + "passthrough_rows": len(rows), + "blank_values_filled_from_key": filled_from_key, + }, + ensure_ascii=False, + indent=2, + ) + ) + + +def cmd_split(args: argparse.Namespace) -> None: + output_dir = Path(args.output_dir) + output_dir.mkdir(parents=True, exist_ok=True) + existing = list(output_dir.glob(f"{args.prefix}_*_src.json")) + existing += list(output_dir.glob(f"{args.prefix}_*_trans.json")) + existing += list(output_dir.glob(f"{args.prefix}_*.json.tmp")) + if existing: + raise ValueError("output directory already contains MTool group artifacts; use a new directory") + + pending = [ + {"text_index": int(item["text_index"]), "source_text": item.get("source_text", "")} + for item in load_cache_items(args.cache) + if int(item.get("translation_status", 0)) == 0 and item.get("source_text", "").strip() + ] + groups: list[list[dict[str, Any]]] = [] + current: list[dict[str, Any]] = [] + chars = 0 + for row in pending: + row_chars = len(row["source_text"]) + would_overflow_chars = args.max_chars > 0 and current and chars + row_chars > args.max_chars + if current and (len(current) >= args.size or would_overflow_chars): + groups.append(current) + current = [] + chars = 0 + current.append(row) + chars += row_chars + if current: + groups.append(current) + + manifest_groups = [] + for number, rows in enumerate(groups, 1): + name = f"{args.prefix}_{number:03d}_src.json" + write_json_atomic(output_dir / name, rows) + manifest_groups.append( + { + "group": number, + "source_file": name, + "rows": len(rows), + "characters": sum(len(row["source_text"]) for row in rows), + "first_text_index": rows[0]["text_index"], + "last_text_index": rows[-1]["text_index"], + } + ) + manifest = { + "cache": str(args.cache), + "prefix": args.prefix, + "group_size_limit": args.size, + "group_character_limit": args.max_chars, + "candidate_rows": len(pending), + "group_count": len(groups), + "groups": manifest_groups, + } + write_json_atomic(output_dir / "manifest.json", manifest) + print(json.dumps(manifest, ensure_ascii=False, indent=2)) + + +def cmd_validate(args: argparse.Namespace) -> None: + report = validate_records(read_json(args.source), read_json(args.translation)) + report["source"] = str(args.source) + report["translation"] = str(args.translation) + print(json.dumps(report, ensure_ascii=False, indent=2)) + if not report.get("valid"): + raise SystemExit(1) + + +def cmd_progress(args: argparse.Namespace) -> None: + groups_dir = Path(args.groups_dir) + sources = group_paths(groups_dir, args.prefix) + total_rows = 0 + valid_rows = 0 + valid_groups = 0 + missing: list[str] = [] + invalid: list[dict[str, Any]] = [] + for source_path in sources: + source = read_json(source_path) + total_rows += len(source) if isinstance(source, list) else 0 + trans_path = translation_path(source_path) + if not trans_path.exists(): + missing.append(trans_path.name) + continue + try: + report = validate_records(source, read_json(trans_path)) + except (OSError, json.JSONDecodeError, UnicodeError) as exc: + invalid.append({"file": trans_path.name, "error": str(exc)}) + continue + if report.get("valid"): + valid_groups += 1 + valid_rows += report["translation_rows"] + else: + invalid.append({"file": trans_path.name, "errors": report.get("errors", [])}) + temp_files = sorted(path.name for path in groups_dir.glob(f"{args.prefix}_*.tmp")) + report = { + "groups_total": len(sources), + "groups_valid": valid_groups, + "rows_total": total_rows, + "rows_valid": valid_rows, + "progress_percent": round(valid_rows * 100 / total_rows, 3) if total_rows else 100.0, + "missing_group_count": len(missing), + "missing_groups": missing[:20], + "invalid_group_count": len(invalid), + "invalid_groups": invalid, + "temporary_file_count": len(temp_files), + "temporary_files": temp_files, + } + print(json.dumps(report, ensure_ascii=False, indent=2)) + + +def cmd_merge(args: argparse.Namespace) -> None: + groups_dir = Path(args.groups_dir) + sources = group_paths(groups_dir, args.prefix) + if not sources: + raise ValueError("no source groups found") + merged: list[dict[str, Any]] = [] + seen: set[int] = set() + for source_path in sources: + trans_path = translation_path(source_path) + if not trans_path.exists(): + raise ValueError(f"missing translation: {trans_path.name}") + translation = read_json(trans_path) + report = validate_records(read_json(source_path), translation) + if not report.get("valid"): + raise ValueError(f"invalid translation {trans_path.name}: {report.get('errors', [])}") + for row in translation: + text_index = int(row["text_index"]) + if text_index in seen: + raise ValueError(f"duplicate text_index across groups: {text_index}") + seen.add(text_index) + merged.append({"text_index": text_index, "translated_text": row["translated_text"]}) + write_json_atomic(args.output, merged) + print( + json.dumps( + {"groups": len(sources), "rows": len(merged), "output": str(args.output)}, + ensure_ascii=False, + indent=2, + ) + ) + + +def cmd_verify_output(args: argparse.Namespace) -> None: + source = require_flat_mtool(read_json(args.input), str(args.input)) + output = require_flat_mtool(read_json(args.output), str(args.output)) + source_keys = list(source) + output_keys = list(output) + missing = [key for key in source_keys if key not in output] + extra = [key for key in output_keys if key not in source] + blank = [key for key, value in output.items() if not value] + new_blank = [key for key in blank if source.get(key, "") != ""] + marker_issues = [] + for key in source_keys: + if key not in output: + continue + expected = marker_signature(key) + actual = marker_signature(output[key]) + if expected != actual: + marker_issues.append({"key": key, "source": expected, "translation": actual}) + valid = ( + not missing + and not extra + and source_keys == output_keys + and (args.allow_empty_values or not new_blank) + and not marker_issues + ) + report = { + "valid": valid, + "input_entries": len(source), + "output_entries": len(output), + "same_key_order": source_keys == output_keys, + "missing_key_count": len(missing), + "extra_key_count": len(extra), + "blank_value_count": len(blank), + "new_blank_value_count": len(new_blank), + "marker_issue_count": len(marker_issues), + "missing_keys": missing[:20], + "extra_keys": extra[:20], + "blank_value_keys": blank[:20], + "new_blank_value_keys": new_blank[:20], + "marker_issues": marker_issues[:20], + } + print(json.dumps(report, ensure_ascii=False, indent=2)) + if not valid: + raise SystemExit(1) + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(description="MTool translation workflow helpers") + sub = parser.add_subparsers(dest="command", required=True) + + inspect_cmd = sub.add_parser("inspect", help="Inspect a flat MTool JSON file") + inspect_cmd.add_argument("input") + inspect_cmd.set_defaults(func=cmd_inspect) + + passthrough_cmd = sub.add_parser( + "passthrough", help="Build batch-write rows for every non-candidate untranslated item" + ) + passthrough_cmd.add_argument("--cache", required=True) + passthrough_cmd.add_argument("--selection", required=True) + passthrough_cmd.add_argument("--output", required=True) + passthrough_cmd.set_defaults(func=cmd_passthrough) + + split_cmd = sub.add_parser("split", help="Split remaining untranslated cache items") + split_cmd.add_argument("--cache", required=True) + split_cmd.add_argument("--output-dir", required=True) + split_cmd.add_argument("--size", type=int, default=150) + split_cmd.add_argument("--max-chars", type=int, default=5000) + split_cmd.add_argument("--prefix", default="mtool") + split_cmd.set_defaults(func=cmd_split) + + validate_cmd = sub.add_parser("validate", help="Validate one source/translation group") + validate_cmd.add_argument("--source", required=True) + validate_cmd.add_argument("--translation", required=True) + validate_cmd.set_defaults(func=cmd_validate) + + progress_cmd = sub.add_parser("progress", help="Report progress from valid final group files") + progress_cmd.add_argument("--groups-dir", required=True) + progress_cmd.add_argument("--prefix", default="mtool") + progress_cmd.set_defaults(func=cmd_progress) + + merge_cmd = sub.add_parser("merge", help="Validate and merge all completed groups") + merge_cmd.add_argument("--groups-dir", required=True) + merge_cmd.add_argument("--output", required=True) + merge_cmd.add_argument("--prefix", default="mtool") + merge_cmd.set_defaults(func=cmd_merge) + + verify_cmd = sub.add_parser("verify-output", help="Verify final MTool key/value integrity") + verify_cmd.add_argument("--input", required=True) + verify_cmd.add_argument("--output", required=True) + verify_cmd.add_argument("--allow-empty-values", action="store_true") + verify_cmd.set_defaults(func=cmd_verify_output) + return parser + + +def main(argv: list[str] | None = None) -> None: + parser = build_parser() + args = parser.parse_args(argv) + if getattr(args, "size", 1) <= 0: + parser.error("--size must be greater than zero") + try: + args.func(args) + except (OSError, json.JSONDecodeError, UnicodeError, ValueError) as exc: + parser.error(str(exc)) + + +if __name__ == "__main__": + main() diff --git a/tests/test_translation_skill_extensions.py b/tests/test_translation_skill_extensions.py new file mode 100644 index 0000000..d55d5e0 --- /dev/null +++ b/tests/test_translation_skill_extensions.py @@ -0,0 +1,139 @@ +from __future__ import annotations + +import contextlib +import hashlib +import importlib.util +import io +import json +import sys +import tempfile +import unittest +from pathlib import Path +from unittest import mock + + +ROOT = Path(__file__).resolve().parents[1] +AINIEE_SCRIPTS = ROOT / "ainiee-translate" / "scripts" +sys.path.insert(0, str(AINIEE_SCRIPTS)) + +from ainiee_translate import apply_repairs, audit_translation, curate_terms, extract_terms, mtool # noqa: E402 + + +def load_script(name: str, path: Path): + spec = importlib.util.spec_from_file_location(name, path) + assert spec and spec.loader + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +EMBEDDED = load_script( + "extract_embedded_csv", + ROOT / "unity-text-locator" / "scripts" / "extract_embedded_csv.py", +) +RUNTIME_DIFF = load_script( + "diff_autotranslator_files", + ROOT / "unity-text-locator" / "scripts" / "diff_autotranslator_files.py", +) + + +class AiNieeExtensionTests(unittest.TestCase): + def test_mtool_group_validation_preserves_control_markers(self) -> None: + source = [{"text_index": 1, "source_text": r"\C[23]愛菜\C[0]\n次"}] + translation = [{"text_index": 1, "translated_text": "爱菜"}] + report = mtool.validate_records(source, translation) + self.assertFalse(report["valid"]) + self.assertEqual(report["marker_issue_count"], 1) + + def test_term_candidates_merge_reading_forms_and_export_high_confidence(self) -> None: + source_rows = [ + {"text_index": 1, "source_text": "愛菜(まな)"}, + {"text_index": 2, "source_text": "愛菜(まな)"}, + ] + candidates, hints = extract_terms.extract(source_rows, 2) + self.assertEqual(hints, []) + self.assertEqual(len(candidates), 1) + self.assertEqual(candidates[0]["source"], "愛菜") + self.assertEqual(candidates[0]["confidence"], "high") + + def test_curated_term_conflict_blocks_output(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + candidates = root / "candidates.json" + translations = root / "translations.json" + output = root / "terms.json" + candidates.write_text( + json.dumps({"candidates": [{"source": "愛菜", "frequency": 2, "confidence": "high", "categories": ["reading_form"]}]}), + encoding="utf-8", + ) + translations.write_text( + json.dumps([{"src": "愛菜", "dst": "爱菜"}, {"src": "愛菜", "dst": "艾菜"}]), + encoding="utf-8", + ) + argv = ["curate_terms", "--candidates", str(candidates), "--translations", str(translations), "--output", str(output)] + with mock.patch.object(sys, "argv", argv), contextlib.redirect_stdout(io.StringIO()): + self.assertEqual(curate_terms.main(), 1) + self.assertFalse(output.exists()) + + def test_source_audit_rejects_duplicates_extra_rows_and_structure_changes(self) -> None: + source = [{"text_index": 1, "source_text": "「HPは%d」\n次"}] + translation = [ + {"text_index": 1, "translated_text": "『HP是%s』"}, + {"text_index": 1, "translated_text": "重复"}, + {"text_index": 2, "translated_text": "多余"}, + ] + report = audit_translation.audit(source, translation, None) + kinds = {issue["kind"] for issue in report["issues"]} + self.assertFalse(report["valid"]) + self.assertTrue({"duplicate_translation_text_index", "extra_translation_row", "placeholder_mismatch", "newline_mismatch", "dialogue_quote_mismatch"} <= kinds) + + def test_repair_manifest_rejects_missing_index_without_output(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + translation = root / "translation.json" + manifest = root / "manifest.json" + output = root / "output.json" + translation.write_text(json.dumps([{"text_index": 1, "translated_text": "旧译"}]), encoding="utf-8") + manifest.write_text(json.dumps({"items": [{"text_index": 2, "expected_before": "旧译", "replacement": "新译"}]}), encoding="utf-8") + argv = ["apply_repairs", "--translation", str(translation), "--manifest", str(manifest), "--output", str(output)] + with mock.patch.object(sys, "argv", argv), self.assertRaises(SystemExit): + apply_repairs.main() + self.assertFalse(output.exists()) + + +class UnityExtensionTests(unittest.TestCase): + def test_embedded_csv_manifest_preserves_duplicate_occurrences_and_hash(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + asset = root / "Game_Data" / "sharedassets0.assets" + asset.parent.mkdir(parents=True) + asset.write_bytes(b"prefix\x00" + "ID,名前,説明\n1,愛菜,主人公\n2,愛菜,案内役\n".encode("utf-8") + b"\x00\x00\x00\x00binary") + out_dir = root / "report" + argv = ["extract_embedded_csv", str(root), "--out-dir", str(out_dir)] + with mock.patch.object(sys, "argv", argv), contextlib.redirect_stdout(io.StringIO()): + self.assertEqual(EMBEDDED.main(), 0) + manifest = json.loads((out_dir / "embedded_csv_manifest.json").read_text(encoding="utf-8")) + duplicates = [row for table in manifest["tables"] for row in table["occurrences"] if row["original_flat"] == "愛菜"] + self.assertEqual(len(duplicates), 2) + self.assertEqual(manifest["tables"][0]["source_file"], "Game_Data/sharedassets0.assets") + self.assertEqual(manifest["tables"][0]["source_sha256"], hashlib.sha256(asset.read_bytes()).hexdigest()) + + def test_runtime_diff_blocks_duplicate_keys(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + old = root / "old.txt" + new = root / "new.txt" + output = root / "diff.json" + old.write_text("同じ=相同\n", encoding="utf-8") + new.write_text("同じ=相同\n同じ=另一译文\n", encoding="utf-8") + argv = ["diff_autotranslator_files", "--old", str(old), "--new", str(new), "--output", str(output)] + with mock.patch.object(sys, "argv", argv), contextlib.redirect_stdout(io.StringIO()): + self.assertEqual(RUNTIME_DIFF.main(), 1) + report = json.loads(output.read_text(encoding="utf-8")) + self.assertEqual(report["status"], "blocked") + self.assertEqual(report["new_duplicate_key_count"], 1) + self.assertEqual(report["runtime_merge"], "manual_review_required") + + +if __name__ == "__main__": + unittest.main() diff --git a/unity-text-locator/SKILL.md b/unity-text-locator/SKILL.md index 55c511b..f4d91de 100644 --- a/unity-text-locator/SKILL.md +++ b/unity-text-locator/SKILL.md @@ -1,6 +1,6 @@ --- name: unity-text-locator -description: Safely localize unknown Unity games, especially Japanese-to-Chinese projects. Use to discover Unity version/layout/backend, migrate translations across game versions, extract occurrence-mapped text from serialized assets, bytecode, UI, Localization StringTables, Utage, or middleware, bridge rows through AiNiee-style translation, validate and transactionally write approved candidates, audit residual and runtime-only text, diagnose TMP fonts and Addressables/IL2CPP failures, locate saves, or acceptance-test SHA-verified patches. +description: Safely localize unknown Unity games, especially Japanese-to-Chinese projects. Use to discover Unity version/layout/backend, migrate translations across game versions, extract occurrence-mapped text from serialized assets, embedded CSV, bytecode, UI, Localization StringTables, Utage, or middleware, bridge rows through AiNiee-style translation, validate and transactionally write approved candidates, audit residual and runtime-only text or AutoTranslator diffs, diagnose TMP fonts and Addressables/IL2CPP failures, locate saves, or acceptance-test SHA-verified patches. --- # Unity Translation Workflow @@ -34,6 +34,7 @@ Read [references/mono-raw-residual.md](references/mono-raw-residual.md) when Typ Read [references/font-asset-replacement.md](references/font-asset-replacement.md) before selecting, converting, or replacing any TMP font asset or font bundle. Read [references/runtime-text-adapters.md](references/runtime-text-adapters.md) before generating or testing an AutoTranslator candidate for IL2CPP, Naninovel, GameCreator, custom-rendered, or runtime-only text. Read [references/version-migration-and-release-audit.md](references/version-migration-and-release-audit.md) when porting a translation to a newer game version, separating scratch candidates from durable project evidence, selecting a runtime Locale/StringTable, auditing short bytecode choices or residual Japanese, choosing static CJK font coverage or smoke-test depth, or preparing a release-only patch baseline. +Read [references/embedded-csv-and-runtime-diff.md](references/embedded-csv-and-runtime-diff.md) when Naninovel-like CSV data is embedded in assets or when comparing runtime AutoTranslator captures. Public repository release notes belong in GitHub Releases. Do not add `CHANGELOG.md` to the GitHub repository. @@ -137,6 +138,7 @@ Legacy `original_flat,zh_cn` translation files remain accepted by validation/wri | TMP font bundle candidate for resource-level replacement | `trace_tmp_font_usage.py`, `inspect_tmp_font_bundle.py`, then one-font runtime canary | Visible-component identity and coverage gate | | Correct Unicode renders as another valid Chinese character | Verify CSV/code point, inspect populated TMP tables, then build a matched static set | Glyph-index conflict; not translation substitution | | IL2CPP game without a verified runtime loader | Inspect candidates or generate an exact-version TMP asset; do not use the Mono injector | Project-specific font workflow | +| Embedded CSV signatures in `sharedassets*.assets` or bundles | `extract_embedded_csv.py` | Discovery-only occurrence manifest; no generic writeback | ## Core Commands @@ -230,6 +232,24 @@ python scripts/generate_autotranslator_txt.py \ The default writes only `autotranslator_export_report.json`. Inspect it, then repeat with `--write` to create `_PreTranslated.txt`. Any row-count mismatch, conflicting or partly blank duplicate key, ambiguous plain-text key/value, or serialized-key collision blocks the whole candidate. The skill does not install BepInEx, XUnity.AutoTranslator, configs, or fonts; verify the external loader/plugin contract and complete a runtime canary as described in `runtime-text-adapters.md`. +For embedded CSV discovery, produce an occurrence-preserving audit before considering a runtime-specific writer: + +```bash +python scripts/extract_embedded_csv.py "GameFolder" \ + --out-dir "GameFolder/_translation/unity-text-report/text/embedded-csv" +``` + +For runtime candidate comparison, keep the diff separate from deployment: + +```bash +python scripts/diff_autotranslator_files.py \ + --old "GameFolder/_translation/unity-text-report/runtime/autotranslator/_PreTranslated.previous.txt" \ + --new "GameFolder/_translation/unity-text-report/runtime/autotranslator/_PreTranslated.txt" \ + --output "GameFolder/_translation/unity-text-report/runtime/autotranslator/runtime_diff.json" +``` + +Review removals and broad changes manually, then rerun the runtime canary. These adapters never merge or install runtime files. + For full-automatic translation, convert a Unity source CSV to an AiNiee cache, translate it with `ainiee-translate`, then convert it back to a Unity translation CSV: ```bash diff --git a/unity-text-locator/references/embedded-csv-and-runtime-diff.md b/unity-text-locator/references/embedded-csv-and-runtime-diff.md new file mode 100644 index 0000000..d34d569 --- /dev/null +++ b/unity-text-locator/references/embedded-csv-and-runtime-diff.md @@ -0,0 +1,16 @@ +# Embedded CSV and Runtime Diff Adapters + +`extract_embedded_csv.py` is a discovery adapter for Naninovel-like or custom +CSV data stored inside `sharedassets*.assets` and AssetBundles. It emits +occurrence rows with the game-relative source path, source SHA-256, byte +offset, record number, row id, and column. Keep duplicates. Treat the result +as an audit source until the game's serializer or runtime API proves a matching +writeback path; never use a global `original_flat -> translation` map for +occurrence-specific edits. + +`diff_autotranslator_files.py` compares two `_PreTranslated.txt` candidates and +reports added, removed, and changed keys. Use it after a new runtime capture or +before a manual deployment review. Duplicate keys in either input block the +report. The script does not merge files, install loaders, or claim that a +runtime candidate is safe. A removal or broad change requires manual review +and a runtime canary. diff --git a/unity-text-locator/scripts/diff_autotranslator_files.py b/unity-text-locator/scripts/diff_autotranslator_files.py new file mode 100644 index 0000000..7b3c165 --- /dev/null +++ b/unity-text-locator/scripts/diff_autotranslator_files.py @@ -0,0 +1,68 @@ +#!/usr/bin/env python3 +"""Diff two AutoTranslator files without merging or deploying either one.""" + +from __future__ import annotations + +import argparse +import json +from collections import OrderedDict +from pathlib import Path + + +def parse(path: Path) -> tuple[OrderedDict[str, str], int, list[str]]: + text = path.read_text(encoding="utf-8-sig") + entries: OrderedDict[str, str] = OrderedDict() + duplicates: list[str] = [] + for line in text.splitlines(): + line = line.strip() + if not line or line.startswith("//") or line.startswith("#") or "=" not in line: + continue + key, value = line.split("=", 1) + key = key.strip() + if key in entries: + duplicates.append(key) + entries[key] = value.strip() + return entries, len(text), duplicates + + +def main() -> int: + parser = argparse.ArgumentParser(description="Diff two _PreTranslated.txt files") + parser.add_argument("--old", required=True, type=Path) + parser.add_argument("--new", required=True, type=Path) + parser.add_argument("--output", type=Path) + parser.add_argument("--stats-only", action="store_true") + args = parser.parse_args() + old, old_chars, old_duplicates = parse(args.old) + new, new_chars, new_duplicates = parse(args.new) + old_keys, new_keys = set(old), set(new) + added = sorted(new_keys - old_keys) + removed = sorted(old_keys - new_keys) + changed = sorted(key for key in old_keys & new_keys if old[key] != new[key]) + blocked = bool(old_duplicates or new_duplicates) + report: dict[str, object] = { + "format": "unity-autotranslator-diff-v1", + "status": "blocked" if blocked else "ready", + "old": str(args.old), "new": str(args.new), + "old_entries": len(old), "new_entries": len(new), + "old_duplicate_key_count": len(old_duplicates), "new_duplicate_key_count": len(new_duplicates), + "old_duplicate_key_samples": old_duplicates[:20], "new_duplicate_key_samples": new_duplicates[:20], + "added": len(added), "removed": len(removed), "changed": len(changed), + "unchanged": len(old_keys & new_keys) - len(changed), + "old_chars": old_chars, "new_chars": new_chars, + "removed_warning": len(removed) > 10, + "runtime_merge": "manual_review_required", + } + if not args.stats_only: + report["added_samples"] = [{"key": key, "value": new[key]} for key in added[:20]] + report["removed_samples"] = [{"key": key, "value": old[key]} for key in removed[:20]] + report["changed_samples"] = [{"key": key, "old": old[key], "new": new[key]} for key in changed[:20]] + payload = json.dumps(report, ensure_ascii=False, indent=2) + "\n" + if args.output: + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(payload, encoding="utf-8") + print(payload, end="") + return 1 if blocked else 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/unity-text-locator/scripts/extract_embedded_csv.py b/unity-text-locator/scripts/extract_embedded_csv.py new file mode 100644 index 0000000..62f2b94 --- /dev/null +++ b/unity-text-locator/scripts/extract_embedded_csv.py @@ -0,0 +1,138 @@ +#!/usr/bin/env python3 +"""Extract occurrence-preserving CSV tables embedded in Unity binaries.""" + +from __future__ import annotations + +import argparse +import csv +import hashlib +import io +import json +import re +from pathlib import Path +from typing import Any + + +SIGNATURES = (b"ID,\xe5\x90\x8d\xe5\x89\x8d", b"ID,Name", b"ID,\xe5\x90\x8d\xe7\xa7\xb0", b"Name,Description") +TRANSLATABLE_COLUMNS = {"名前", "名称", "説明", "タイトル", "物語", "報酬", "Name", "Title", "Description", "Story", "Reward", "text", "label", "message"} +IGNORED_COLUMNS = {"ID", "id", "type", "Type", "price", "cost", "flag", "count", "amount", "売値", "買値", "効果"} +ROW_RE = re.compile(r"(?m)^(?:\"?\d+\"?|\"?[A-Za-z][A-Za-z0-9_-]{0,32}\"?),") + + +def file_sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def decode(data: bytes) -> str | None: + for encoding in ("utf-8", "cp932"): + try: + return data.replace(b"\x00", b"").decode(encoding) + except UnicodeDecodeError: + continue + return None + + +def extract_table(data: bytes, offset: int) -> str | None: + start = offset + while start > 0 and data[start - 1:start] not in (b"\n", b"\r", b"\x00"): + start -= 1 + end = min(len(data), offset + 65536) + null = data.find(b"\x00\x00\x00\x00", offset, end) + if null != -1: + end = null + text = decode(data[start:end]) + if not text or len(text.splitlines()) < 3 or not ROW_RE.search(text): + return None + return text.strip("\r\n") + + +def parse_table(text: str, source_file: str, offset: int, source_sha256: str) -> dict[str, Any] | None: + try: + records = list(csv.reader(io.StringIO(text, newline=""))) + except csv.Error: + return None + if len(records) < 2: + return None + header = records[0] + if len(header) < 2: + return None + columns = [] + for index, name in enumerate(header): + clean = name.strip() + if clean in TRANSLATABLE_COLUMNS or (clean not in IGNORED_COLUMNS and re.search(r"[ぁ-んァ-ン一-龯]", clean)): + columns.append(index) + if not columns: + return None + occurrences: list[dict[str, Any]] = [] + row_count = 0 + for record_number, row in enumerate(records[1:], 2): + if len(row) != len(header): + continue + row_count += 1 + row_id = row[0].strip() if row else str(record_number) + for column_index in columns: + value = row[column_index].strip() + if len(value) < 2: + continue + occurrences.append({"original_flat": value, "source_file": source_file, "embedded_offset": offset, "record_number": record_number, "row_id": row_id, "column": header[column_index].strip()}) + if not row_count or not occurrences: + return None + return {"source_file": source_file, "source_sha256": source_sha256, "embedded_offset": offset, "columns": header, "row_count": row_count, "occurrences": occurrences} + + +def scan_file(path: Path, game_root: Path) -> list[dict[str, Any]]: + data = path.read_bytes() + try: + source_file = path.relative_to(game_root).as_posix() + except ValueError: + source_file = path.as_posix() + source_hash = file_sha256(path) + results: list[dict[str, Any]] = [] + for signature in SIGNATURES: + cursor = 0 + while True: + offset = data.find(signature, cursor) + if offset < 0: + break + table = extract_table(data, offset) + if table: + parsed = parse_table(table, source_file, offset, source_hash) + if parsed: + parsed["signature"] = signature.decode("utf-8", errors="replace") + results.append(parsed) + cursor = offset + len(signature) + return results + + +def main() -> int: + parser = argparse.ArgumentParser(description="Extract embedded CSV occurrences from Unity assets") + parser.add_argument("game_root", type=Path) + parser.add_argument("--out-dir", type=Path, required=True) + args = parser.parse_args() + game_root = args.game_root.resolve() + candidates = sorted(set(game_root.rglob("sharedassets*.assets")) | set(game_root.rglob("*.bundle"))) + tables: list[dict[str, Any]] = [] + for path in candidates: + try: + tables.extend(scan_file(path, game_root)) + except OSError: + continue + occurrences = [row for table in tables for row in table["occurrences"]] + args.out_dir.mkdir(parents=True, exist_ok=True) + csv_path = args.out_dir / "embedded_csv_occurrences.csv" + with csv_path.open("w", encoding="utf-8-sig", newline="") as handle: + writer = csv.DictWriter(handle, fieldnames=["original_flat", "source_file", "embedded_offset", "record_number", "row_id", "column"]) + writer.writeheader() + writer.writerows(occurrences) + manifest = {"format": "unity-embedded-csv-occurrences-v1", "game_root": str(game_root), "tables": tables, "occurrence_count": len(occurrences), "duplicate_texts_preserved": True, "writeback": "not_supported_by_this_discovery_adapter"} + (args.out_dir / "embedded_csv_manifest.json").write_text(json.dumps(manifest, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") + print(json.dumps({"files_scanned": len(candidates), "tables": len(tables), "occurrences": len(occurrences), "csv": str(csv_path)}, ensure_ascii=False, indent=2)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main())