diff --git a/unity-text-locator/SKILL.md b/unity-text-locator/SKILL.md index e38a138..466dc48 100644 --- a/unity-text-locator/SKILL.md +++ b/unity-text-locator/SKILL.md @@ -1,6 +1,6 @@ --- name: unity-text-locator -description: Safely localize unknown Unity games, especially Japanese-to-Chinese projects. Use to discover Unity version/layout/backend, extract occurrence-mapped one-column CSVs, bridge them through AiNiee-style translation, validate and transactionally write approved rows into serialized assets or verified middleware adapters, audit raw strings, diagnose TMP fonts, install a licensed app-local Noto fallback for supported Mono builds, inspect Addressables/IL2CPP risks, locate saves, or build SHA-verified patch packages. +description: Adaptive Unity Japanese-to-Chinese translation workflow. Use when Codex needs to discover and extract Unity text from serialized assets, fixed-byte DMSL records, component UI fields, Unity Localization StringTables, Utage, or BansheeGz BGDatabase; validate one-column translation CSVs; write occurrence-mapped candidates; diagnose Addressables CRC/catalog black screens; trace visible TMP components through FontAsset/material/CAB references; distinguish tofu, atlas corruption, and valid-Han glyph-index mis-mapping; build matched static TMP font candidates; locate ES3 saves; or package verified binary patches. --- # Unity Translation Workflow @@ -14,7 +14,7 @@ Treat each Unity game as a separate format-discovery task. Do not assume that a 3. Extract project-named, one-column text files. 4. Ask for matching one-column translation files. 5. Validate row alignment and structural tokens; blank only row-addressable failures when the user elects partial writeback. -6. Dry-run, prove source-specific serialization/runtime safety, back up, then write only approved rows. +6. Dry-run, prove source-specific serialization/runtime safety, build candidates outside the game root, back up, then write only approved rows. 7. Run source-specific residual and runtime checks. 8. Build a SHA-verified binary patch package only after testing the translated files. @@ -24,8 +24,6 @@ Use three gates for every project: - Translator gate: expose only project-named one-column CSVs for row-mapped sources; keep broad `extracted_text.csv` as audit evidence, not as a writeback source. - Runtime gate: a clean CSV validation and a tool-level dry run are not enough for runtime-sensitive containers; prove the edited file loads through the same game path before distributing it. -Keep `raw-utf8-fallback` audit-only by default. Expose it with `--include-raw-candidates` and write it with `--allow-raw-fallback` only after an adapter-specific container review. Normal writes must complete in staging, pass exact object/file validation, and commit with automatic rollback. - Read [references/unity-text-map.md](references/unity-text-map.md) when locating candidate Unity text containers or middleware clues. Read [references/runtime-sensitive-adapters.md](references/runtime-sensitive-adapters.md) when handling BansheeGz databases, Addressables bundles/catalogs, ES3 saves, or runtime font compatibility. Read [references/workflow-modes.md](references/workflow-modes.md) before choosing semi-automatic or full-automatic translation. @@ -62,6 +60,8 @@ GameFolder/_translation/unity-text-report/ `scan_unity_text.py` and `extract_unity_text.py` default to this directory and ignore `_translation` while scanning. Put distributable patch archives outside the game directory only after verification. +Reports, manifests, and CSVs may live under `_translation`, but rebuilt bundles, catalogs, font assets, and other runtime binary candidates must be written to a separate candidate root outside the game directory. Applying a candidate is a distinct, backed-up step. + ## Translator File Contract Use one source file and one translation file per extracted Unity source. Name translator-facing files with the project folder name followed by a short semantic description, not the internal Unity asset path: @@ -120,9 +120,13 @@ Legacy `original_flat,zh_cn` translation files remain accepted by validation/wri | Utage `ALL_Texts.book` / `ALL_EventTexts.book` in scenario assets or embedded assets | `extract_utage_scenarios.py`, `writeback_utage_scenarios.py` | Automated one-column workflow after confirming text column/books | | `bansheegz_database` TextAsset plus `BGDatabase.dll`/`BansheeGz.BGDatabase` | Build a one-shot game-runtime export/write/save probe and reinsert the verified payload | Runtime-specific structured adapter | | Loose CSV/JSON/TXT or custom scenario string parsers | Inspect format and preserve its record/newline contract | Project-specific | +| `0x06 + uint32 LE + UTF-8` length-prefixed records, often near a `DMSL` signature | `extract_dmsl_text.py`, `writeback_dmsl_text.py` | Fixed-byte occurrence workflow; byte budget is blocking | +| TMP/UI `m_text`, label fields, or Unity Localization `m_TableData.*.m_Localized` | `extract_typetree_ui_text.py`, `writeback_typetree_ui_text.py` | TypeTree occurrence workflow with field-path verification | | Addressables catalogs/bundles or IL2CPP `global-metadata.dat` | Confirm exact runtime text, integrity metadata, and game-runtime loading before applying candidates | Project-specific | +| Rebuilt Addressables bundle loads to a black screen or logs CRC mismatch | `patch_addressables_catalog_crc.py`, then paired bundle/catalog canary | Binary catalog candidate; unique internal-name match only | | Chinese displayed as boxes/tofu in Mono/TMP games with JSON injection tables | `install_tmp_chinese_font_fix.py --dry-run`, then apply | Preflighted runtime fallback workflow | -| TMP font bundle candidate for resource-level replacement | `inspect_tmp_font_bundle.py`, then one-font runtime canary | Read-only compatibility and coverage gate | +| TMP font bundle candidate for resource-level replacement | `trace_tmp_font_usage.py`, `inspect_tmp_font_bundle.py`, then one-font runtime canary | Visible-component identity and coverage gate | +| Correct Unicode renders as another valid Chinese character | Verify CSV/code point, inspect populated TMP tables, then build a matched static set | Glyph-index conflict; not translation substitution | | IL2CPP game without a verified runtime loader | Inspect candidates or generate an exact-version TMP asset; do not use the Mono injector | Project-specific font workflow | ## Core Commands @@ -135,6 +139,46 @@ python scripts/scan_unity_text.py "path/to/GameFolder" This keeps the broad `extracted_text.csv` and `extraction_manifest.json` for audit, and creates translator-facing, occurrence-preserving files in `text/`. +For fixed-width DMSL operands, extract from explicit inputs or a constrained candidate root: + +```bash +python scripts/extract_dmsl_text.py "GameFolder" \ + --candidate-root "Game_Data/StreamingAssets/aa/StandaloneWindows64" \ + --out-dir "GameFolder/_translation/unity-text-report/text" +``` + +After validation, dry-run the fixed byte budgets, then write only to an external candidate root: + +```bash +python scripts/writeback_dmsl_text.py "GameFolder" \ + --manifest "GameFolder/_translation/unity-text-report/text/GameTitle_dmsl_text.manifest.json" \ + --translation-csv "GameFolder/_translation/unity-text-report/text/GameTitle_dmsl_text_translation.csv" \ + --report "CandidateReports/dmsl_dry_run.json" +``` + +Add `--base-root "PriorCandidateRoot" --out-dir "NextCandidateRoot" --write` only after the dry run is clean. Any over-budget row rejects the whole batch and must be shortened and revalidated; the writer never truncates UTF-8. + +Extract component UI fields and Unity Localization StringTables with: + +```bash +python scripts/extract_typetree_ui_text.py "GameFolder" \ + --scan-root "Game_Data/StreamingAssets/aa" \ + --out-dir "GameFolder/_translation/unity-text-report/text" +``` + +Dry-run `writeback_typetree_ui_text.py` with its manifest, source CSV, and translation CSV. Use `--base-root` to compose over a DMSL candidate, and add `--out-root "NextCandidateRoot" --write` only after every object identity and field path still matches. + +For a rebuilt Addressables bundle, generate the paired external catalog candidate from its internal name: + +```bash +python scripts/patch_addressables_catalog_crc.py \ + "GameFolder/Game_Data/StreamingAssets/aa/catalog.bin" \ + --bundle "CandidateRoot/Game_Data/StreamingAssets/aa/StandaloneWindows64/changed.bundle" \ + --game-root "GameFolder" --out-dir "CatalogCandidateRoot" --write +``` + +The catalog tool rejects unknown/duplicate records and byte-compares the result so only the selected CRC field can change. Apply and roll back the bundle/catalog pair together. + When a broad audit CSV must be narrowed for human translation review, produce a conservative visible-safe subset and audit the omissions: ```bash @@ -183,8 +227,6 @@ Use the sibling `ainiee-translate` skill to run the batch translation loop on th --out-csv "GameFolder/_translation/unity-text-report/text/GameTitle_text_translation.csv" ``` -This conversion is fail-closed: it verifies the source CSV fingerprint and rejects missing, duplicate, untranslated, or blank rows. `--allow-partial` is only for generating a review artifact and must not feed final writeback. - Then run `validate_translation_csv.py` exactly as in the semi-automatic path. When the user explicitly chooses to apply only structurally safe rows, filter row-addressable failures and then validate the filtered output again: @@ -275,24 +317,35 @@ For Utage, check both `StreamingAssets/Utage_AdvProject` and embedded books in ` For a BansheeGz/BGDatabase payload embedded in a TextAsset, do not infer its serialized field format from exported bytes. If the game's managed API can load and save it, use a temporary runtime probe to export occurrences, compare each selected source value, write translated field values, and save a verified replacement payload. Remove all probe registration and DLLs before smoke-testing the normal game. -For Addressables or IL2CPP metadata, require a backup and an integrity plan before edits. Catalog hashes/CRCs, bundle packing, or metadata offsets may make ordinary asset writeback unsuitable. A candidate that UnityPy can reopen is not approved unless the game runtime can load it without Addressables/bundle errors. +For Addressables or IL2CPP metadata, require a backup and an integrity plan before edits. Locate a binary catalog entry from the bundle's internal `AssetBundle.m_Name`, never from an opaque disk filename alone. When a uniquely identified CRC must be disabled, use `patch_addressables_catalog_crc.py` to change only that four-byte field in an external catalog candidate; do not guess `catalog.hash` or `bundleSize`. Treat bundle plus catalog as one canary and one rollback unit. A candidate that UnityPy can reopen is not approved unless the game runtime loads it without Addressables/bundle errors. + +After an Addressables edit causes a black screen, inspect `Player.log` before revising translations or fonts. Search for CRC mismatch, `Will not load AssetBundle`, bundle dependency/load errors, and catalog parse/hash/cache failures. Preserve the log and roll back the bundle/catalog pair before trying another candidate. ## Font Diagnosis -Treat remaining Japanese as a translation/extraction gap. Distinguish uniform tofu squares from repeated glyph fragments or boxed atlas snippets: tofu indicates missing glyphs, while fragments indicate a font-table, atlas, stream, material, or platform-texture mismatch. Inspect the runtime log and map the visible component to TMP or legacy UI before applying a fix. +Treat remaining Japanese as a translation/extraction gap. Distinguish uniform tofu squares from repeated glyph fragments or boxed atlas snippets: tofu indicates missing glyphs, while fragments indicate a font-table, atlas, stream, material, or platform-texture mismatch. A correct source character rendered as another valid Han character indicates a glyph-index conflict: verify the CSV and Unicode first, then inspect the TMP tables instead of changing the translation. Inspect the runtime log and map the visible component to TMP or legacy UI before applying a fix. + +Never trust a font package filename or guess the primary font by name. Use `trace_tmp_font_usage.py` from a visible failing component to establish the actual FontAsset/Material/CAB/bundle chain. Run `inspect_tmp_font_bundle.py` to read dynamic/static mode, populated tables, source Font PPtr/family identity, Unity version, glyph coverage, atlas metadata, and SHA-256. If a dynamic asset already has character/glyph tables, never replace only its source TTF. Static replacement must move the matched character table, glyph table, material parameters, and SDF atlas together with `build_static_tmp_fontasset.py`. Require a one-font, one-screen runtime canary before broad replacement. UnityPy reopening is structural evidence, not visual or GPU-runtime proof. Follow [references/font-asset-replacement.md](references/font-asset-replacement.md) for stop conditions and rollback rules. + +Trace from a visible string or component name while indexing dependency bundles: + +```bash +python scripts/trace_tmp_font_usage.py \ + --asset "ComponentBundle" --candidate-root "AddressablesBundleRoot" \ + --text-contains "Visible failing text" --out "CandidateReports/tmp_usage.json" +``` -Never trust a font package filename. Run `inspect_tmp_font_bundle.py` to read the internal font identity, Unity version, glyph coverage, atlas metadata, and SHA-256. Require a one-font, one-screen runtime canary before broad replacement. UnityPy reopening is structural evidence, not visual or GPU-runtime proof. Follow [references/font-asset-replacement.md](references/font-asset-replacement.md) for stop conditions and rollback rules. +When static replacement is justified, pass the traced target FontAsset, Material, and Texture2D PathIDs explicitly to `build_static_tmp_fontasset.py`, along with a donor, approved translation root, game root, external output, and report path. Exact Unity versions and real donor coverage are required by default; explicit mismatch or donor-alias opt-ins mark the result canary-only. For compatible Mono/TMP games with `ScriptingAssemblies.json` and `RuntimeInitializeOnLoads.json`, prefer the runtime fallback preflight: ```bash python scripts/install_tmp_chinese_font_fix.py "GameFolder" \ - --runtime-font-file "path/to/licensed/NotoSansCJKsc-Regular.otf" \ --out-dir "GameFolder/_translation/unity-text-report/font-fix" \ --dry-run ``` -The fixer compiles the DLL and validates injection files during preflight, then applies the DLL, JSON, and app-local runtime font as one rollback-capable change set. Prefer a legally redistributable Noto CJK/SC font; never publish `arialuni_sdf_u2019` without explicit redistribution rights. When no runtime font is supplied, the fixer may fall back to installed system CJK fonts. Use embedded replacement only after exact Unity/TMP compatibility review. +The fixer compiles the DLL and validates injection files during preflight, then applies all planned writes with backups only when run without `--dry-run`. Its default runtime behavior chooses an installed CJK font file, adds a dynamic TMP fallback, and changes legacy UI fonts only for strings with missing CJK glyphs; it does not replace every TMP font. Use `--patch-embedded-fonts --font "Microsoft YaHei"` only when a reviewed embedded legacy `Font` replacement is needed. After installation, launch the game and inspect `Player.log`. Require an initialization log and no `MissingMethodException`, `Unable to load font face`, or repeated missing-glyph failures. Some stripped Mono builds omit `Font.CreateDynamicFontFromOSFont` or `ParameterInfo.HasDefaultValue`; use the shipped file-path/reflection-compatible fixer rather than assuming those APIs exist. If the game directory permits content writes but rejects rename replacement, keep the backup, accept verified copy-overwrite only, and remove or report staged `.tmp` files. diff --git a/unity-text-locator/agents/openai.yaml b/unity-text-locator/agents/openai.yaml index abb06bd..824c10b 100644 --- a/unity-text-locator/agents/openai.yaml +++ b/unity-text-locator/agents/openai.yaml @@ -1,4 +1,4 @@ interface: display_name: "Unity Translation Pipeline" - short_description: "Safely extract, translate, write, and debug Unity game text." - default_prompt: "Use $unity-text-locator to inspect this Unity game, extract and validate project-named translation CSVs, choose the correct writeback adapter, inspect TMP font-bundle compatibility and glyph coverage, and verify runtime loading before packaging." + short_description: "Extract and localize Unity text, catalogs, and TMP fonts safely." + default_prompt: "Use $unity-text-locator to discover every visible Unity text source, including DMSL, component UI, and Localization StringTables; extract and validate occurrence-mapped translation CSVs; generate isolated writeback candidates; trace the actual TMP FontAsset used by failing components; diagnose Addressables CRC or glyph-index failures from logs and Unicode evidence; and require paired runtime canaries before packaging." diff --git a/unity-text-locator/references/ainiee-integration.md b/unity-text-locator/references/ainiee-integration.md index 5207d88..7de21c7 100644 --- a/unity-text-locator/references/ainiee-integration.md +++ b/unity-text-locator/references/ainiee-integration.md @@ -82,7 +82,22 @@ python unity-text-locator/scripts/validate_translation_csv.py \ ## Guardrails -- Cache writes are atomic and cooperatively locked, but subagents should still produce separate batch JSON files and let one coordinator write `cache.json`. +- Do not let subagents write the same `cache.json` concurrently. - Do not use AiNiee output directly for Unity writeback; always convert to `zh_cn` CSV and run Unity validation. -- Conversion verifies the source CSV content hash, index continuity, translation status, and nonblank output. Any incomplete row fails by default. -- `--allow-partial` emits blanks only for an explicitly requested review artifact. Do not pass such output to writeback. +- Blank translations remain intentional skips. +- If the translated cache has fewer rows than the source CSV, conversion emits blank rows for missing indexes and validation/reporting should catch the gap before writeback. + +## Fixed-Byte DMSL Sources + +Some DMSL containers store each string as `0x06 + uint32 little-endian length + UTF-8`, but the runtime or surrounding bundle still requires every replacement to fit the original byte span. Translation quality validation and structural-token validation do not prove this byte budget. + +Use this loop: + +1. Extract with `extract_dmsl_text.py`; keep its occurrence manifest and source SHA-256. +2. Translate through the normal AiNiee cache bridge and validate the returned one-column CSV. +3. Run `writeback_dmsl_text.py --dry-run`. The report must list every row whose UTF-8 translation exceeds its original byte budget and must not emit a candidate. +4. Shorten the reported Chinese rows without changing names, placeholders, tags, or meaning. Never silently truncate encoded bytes. +5. Re-run ordinary CSV validation and the DMSL dry run until every nonblank row fits. +6. Generate the candidate outside the game root, reopen it, and verify every occurrence and length prefix before any application step. + +When TypeTree UI and DMSL records overlap the same bundle, use the writer's `--base-root` to layer the second candidate on the first. The final writer must still verify each field/record against the manifest rather than treating a prior candidate as trusted. diff --git a/unity-text-locator/references/font-asset-replacement.md b/unity-text-locator/references/font-asset-replacement.md index 2a84cef..173cce3 100644 --- a/unity-text-locator/references/font-asset-replacement.md +++ b/unity-text-locator/references/font-asset-replacement.md @@ -4,27 +4,30 @@ Treat every font package as untrusted until its internal Unity version, TMP obje Keep inspection reports in the game workspace, not in the shared skill repository, because reports echo user-supplied local paths. Do not commit font binaries unless their redistribution license has been reviewed and repository inclusion is intentional. -## Preflight +## Preflight And Usage Tracing -Run the read-only inspector before selecting a candidate: +Start from a text component that visibly reproduces the problem. Run `trace_tmp_font_usage.py` to follow its `m_fontAsset` and material PPtrs through External FileIDs/CAB names to the real bundle and PathIDs. Do not infer the "main font" from bundle filenames, object names, character coverage, or discovery order. A title screen, dialogue view, and settings page may use different TMP assets even when their styling looks similar. + +Run the read-only inspector before selecting a target or donor: ```bash python scripts/inspect_tmp_font_bundle.py "FontBundle" \ --target-asset "GameFolder/Game_Data/sharedassets0.assets" \ --translation-root "GameFolder/_translation/unity-text-report/validation" \ - --out "GameFolder/_translation/unity-text-report/font-fix/font_inspection.json" + --out "CandidateRoot/font_inspection.json" ``` Verify: -- the internal TMP font, material, and atlas names; +- the internal TMP face family, source Font identity, material, and atlas names; - the serialized Unity version inside the bundle and target asset; -- character and glyph counts; +- static/dynamic population mode and existing character/glyph table counts; +- source Font PPtr and any family-identity conflict; - atlas dimensions, format, stream path, platform blob, and color space; - coverage of characters actually used by the validated translation; - the package SHA-256 and size. -Prefer an exact engine version. Treat the same Unity major/minor as a candidate requiring a runtime canary. Treat cross-major/minor raw MonoBehaviour replacement as high risk even if UnityPy can parse both files. +Prefer an exact engine version. Treat the same Unity major/minor as a candidate requiring a runtime canary. Treat cross-version raw MonoBehaviour replacement as high risk even if UnityPy can parse both files. ## Diagnose The Symptom First @@ -35,40 +38,49 @@ Prefer an exact engine version. Treat the same Unity major/minor as a candidate | Pink/magenta text or rectangles | Missing/incompatible shader or material | Restore the target shader and validate material properties and atlas reference | | `corrupted`, `Position out of bounds`, or load failure | Serialization/container mismatch | Roll back; do not continue from the candidate | | Dialogue is correct but menus are not | Multiple font assets or UI systems | Map each visible component to its actual TMP or legacy font before changing more assets | +| Correct CSV/Unicode displays as another valid CJK character | Stored character/glyph index now points into a different source font | Compare CSV bytes and code points first; treat this as a glyph-index conflict, not a translation error | + +Do not call glyph fragments "missing characters." Do not infer runtime texture orientation from exported-image coordinates; Unity runtime rendering is the authority. + +## Dynamic TMP Source-Font Trap + +Inspect `m_AtlasPopulationMode`, `m_CharacterTable`, `m_GlyphTable`, and `m_SourceFontFile` together. A dynamic TMP FontAsset can still serialize populated character and glyph tables. If its tables are populated, replacing only the referenced TTF/Font object is forbidden: character entries retain old glyph indices, while the new source font can assign those indices to unrelated glyphs. The result can be plausible but wrong Chinese, with one valid Han character consistently rendering as another. -Do not call glyph fragments “missing characters.” Do not infer runtime texture orientation from PIL or exported-image coordinates; Unity runtime rendering is the authority. +First prove the serialized CSV contains the intended Unicode code point. If it does, preserve the translation and repair the font asset. Do not add character aliases to hide missing donor coverage; aliases deliberately render a different glyph and make coverage reports misleading. ## Choose The Least Invasive Adapter 1. For Mono/TMP games with supported injection tables, prefer the preflighted runtime fallback workflow. It preserves original materials and atlases. 2. For IL2CPP games, do not claim the Mono fallback installer applies. Use a compatible IL2CPP runtime plugin only when its loader and interop generation are independently verified. -3. Use resource-level TMP replacement only when the target asset, container rules, and runtime test path are known. +3. Use resource-level TMP replacement only when the visible component, exact target objects, container rules, and runtime test path are known. 4. If no compatible package exists, generate a TMP FontAsset with the exact target Unity/TMP version. Treat editor installation as a build-time dependency, not a game runtime dependency. -## Resource-Level Replacement Contract +## Static Resource Replacement Contract Before writing: -1. Identify the exact visible text component and its font asset. Do not replace every discovered font as the first experiment. -2. Back up every modified asset and record its SHA-256. -3. Inspect the candidate with `inspect_tmp_font_bundle.py` and save the report. -4. Prove required-character coverage against validated translations. -5. Preserve the target MonoBehaviour script, object name, material and atlas PathIDs, and target shader. -6. When materializing a streamed atlas, copy the full payload and all relevant Texture2D fields, including platform blob, format, dimensions, mip settings, preprocessing flags, color space, and stream state. A correct byte count alone is insufficient. -7. Reopen the candidate with UnityPy and validate object names, references, glyph/character counts, atlas dimensions, and payload length. -8. Apply one-font, one-screen canary first. Do not broaden the replacement until the canary renders correctly. +1. Identify the exact visible text component and its FontAsset, Material, and Texture2D PathIDs. Never replace every discovered font as the first experiment. +2. Back up every file that will eventually be applied and record its SHA-256. Build the candidate outside the game directory. +3. Inspect both target and donor and save the reports. +4. Prove required-character coverage against approved translation CSVs. Missing characters and implicit glyph aliases are blocking failures. +5. Preserve the target MonoBehaviour script, object name, FontAsset/Material/Texture PathIDs, and target shader. +6. Copy the donor character table, glyph table, font metrics, matching SDF atlas, and material parameters as one matched set. Never combine a donor table with a target atlas or retain target glyph indices against a donor source font. +7. When materializing a streamed atlas, copy the full payload and all Texture2D fields, including platform blob, format, dimensions, mip settings, preprocessing flags, color space, and stream state. A correct byte count alone is insufficient. +8. Require exact serialized Unity versions by default. A mismatch needs explicit opt-in and produces a canary-only candidate; it is not release evidence. +9. Reopen the candidate with UnityPy and validate names, PathIDs, script/shader identity, FontAsset material/atlas references, glyph/character counts, atlas dimensions, inline payload, and required-character coverage. +10. Apply one-font, one-screen canary first. Do not broaden the replacement until the canary renders correctly. -UnityPy reopening proves structural readability only. It does not prove GPU texture upload, shader compatibility, glyph coordinates, bundle integrity, or game-runtime loading. +Use `build_static_tmp_fontasset.py` only after all explicit target PathIDs are known. UnityPy reopening proves structural readability only; it does not prove GPU texture upload, shader compatibility, glyph coordinates, bundle integrity, or game-runtime loading. ## Runtime Canary -Use a screen containing representative simplified Chinese, punctuation, Latin text, heart/music symbols, and any retained Japanese names. Fully exit and restart the game between candidates so an old atlas cannot remain in memory. +Use a screen containing representative simplified Chinese, punctuation, Latin text, symbols, and any retained Japanese names. Fully exit and restart the game between candidates so an old atlas cannot remain in memory. Require: -- correct glyph shapes without atlas fragments; +- correct glyph shapes without atlas fragments or wrong valid Han characters; - acceptable outline, spacing, clipping, and alignment; -- no new asset corruption, missing font, shader, or repeated missing-glyph errors in the log; +- no new asset corruption, missing font, shader, or repeated missing-glyph errors in `Player.log`; - coverage checks for menus, settings, shop/tutorial screens, dialogue, and at least one late-game scene when available. On the first visual regression, stop, preserve the screenshot and candidate report, and restore the backup. Do not stack another speculative transformation on the broken candidate. diff --git a/unity-text-locator/references/runtime-sensitive-adapters.md b/unity-text-locator/references/runtime-sensitive-adapters.md index dc2d56a..8db362f 100644 --- a/unity-text-locator/references/runtime-sensitive-adapters.md +++ b/unity-text-locator/references/runtime-sensitive-adapters.md @@ -29,10 +29,16 @@ Text-table rows inside a bundle may be readable and writable while the resulting Required gates: 1. Preserve original bundle packing/compression when rebuilding where possible. -2. Determine whether catalog entries carry CRC, hash, size, cache, or local-load options. -3. Treat catalog edits as integrity changes: keep an untouched original and write a candidate first. -4. Validate the rebuilt bundle in the Unity game runtime, ideally through the same loading route used by Addressables. -5. If runtime load fails or cannot be proven, do not apply the translated bundle or catalog even when translation CSV validation passed. +2. Read the rebuilt bundle's internal `AssetBundle.m_Name`; use that exact value to locate its binary catalog record. Do not match by an opaque on-disk filename alone. +3. Determine whether catalog entries carry CRC, hash, size, cache, or local-load options. Catalog layouts vary, so unknown or duplicate internal-name records are blocking failures. +4. For the supported binary layout, use `patch_addressables_catalog_crc.py` to create an external catalog candidate. By default, set only the uniquely identified record's four-byte CRC to zero and byte-compare the result. Do not guess `catalog.hash`, and do not update `bundleSize` without a separately proven layout and runtime need. +5. Treat the rebuilt bundle and matching catalog candidate as one canary unit. Apply both together after backing up both; if either copy fails, restore both. Never leave a new bundle paired with the old CRC catalog. +6. Validate the pair in the Unity game runtime through the same loading route used by Addressables. +7. If runtime load fails or cannot be proven, roll back both files even when CSV validation and UnityPy reopening passed. + +If a rebuilt Addressables bundle produces a black screen, inspect `Player.log` before changing text or fonts again. Search first for `CRC Mismatch`, `Will not load AssetBundle`, bundle load exceptions, catalog parse errors, missing dependencies, and hash/cache failures. A CRC mismatch is an integrity-pair failure, not evidence that the translated text or TMP font is bad. + +`patch_addressables_catalog_crc.py` is a candidate generator, not an installer. Its output and JSON byte-diff report must stay outside the game root. Application and rollback remain explicit, backed-up operations. Translation structure validation answers whether rows are safe text. It does not prove resource-container integrity. diff --git a/unity-text-locator/references/unity-text-map.md b/unity-text-locator/references/unity-text-map.md index fb32123..7c48900 100644 --- a/unity-text-locator/references/unity-text-map.md +++ b/unity-text-locator/references/unity-text-map.md @@ -1,13 +1,5 @@ # Unity Text Location Map -## Contents - -- Common locations and middleware clues -- Discovery order and tool choice -- Per-source CSV/manifest contract -- Safe writeback and validation -- Font fallback and completion evidence - ## Common Locations | Location | What it may contain | First action | @@ -35,6 +27,7 @@ | `I2Languages`, `I2Loc`, `LanguageSource` | I2 Localization | Export ScriptableObjects/MonoBehaviours | | `LocalizeStringEvent`, `StringTableCollection` | Unity Localization package | Export string tables and Addressables | | `TextMeshPro`, `TMP_Text` | TextMeshPro UI text | Inspect prefabs/scenes/MonoBehaviours | +| `DMSL` near `0x06 + uint32 LE + UTF-8` operands | Custom length-prefixed scenario bytecode | Use the fixed-byte DMSL extractor and retain every occurrence/byte budget | | `bansheegz_database`, `BGDatabase.dll`, `BGRepo` | BansheeGz BGDatabase | Use a game-runtime load/export/save probe; do not raw-patch payload bytes | | `ES3`, `ES3Defaults`, `SaveFile.es3`, `AutoSave.es3` | Easy Save 3 persistence | Inspect defaults resource and confirm actual save path/timestamps | @@ -119,6 +112,8 @@ searching for translated bytes after the write rather than trusting old offsets. For raw fallback byte spans, write back only when the UTF-8 translated byte length does not exceed the original byte span; pad shorter replacements with spaces. +For visible UI missed by scenario extraction, inspect MonoBehaviour TypeTrees for `m_text`, label/caption/title-like fields, and Unity Localization `m_TableData[*].m_Localized`. Preserve serialized-file identity, PathID, a typed field path, source field hash, and duplicate occurrence metadata; PathID alone is not unique across serialized files inside a bundle. + Before writing: - create timestamped backups under the report `writeback//backups/` directory, or pass an explicit `--backup-dir` when a project needs a different backup root; @@ -130,11 +125,15 @@ is only a structural check. Preserve compression/integrity metadata where applicable and require a game-runtime bundle/Addressables load test before applying the bundle or its catalog changes. +For a binary catalog, derive the lookup key from `AssetBundle.m_Name`, require one plausible record, and audit an exact four-byte CRC field change. Do not infer a record from the external bundle filename, and do not guess the catalog hash or stored bundle size. + ## Chinese Font Fix If translated Chinese renders as square boxes, the likely cause is that the game uses embedded Japanese pixel fonts or a TMP font asset without Chinese glyphs. +If the serialized translation contains the correct Unicode but the screen shows a different valid Han character, this is not tofu and usually is not a translation error. Trace the visible component to its actual TMP FontAsset and check for a populated dynamic character/glyph table paired with a replaced source Font. + Preferred runtime font files on Windows, in order of availability: - Microsoft YaHei (`msyh.ttc`/`msyh.ttf`) diff --git a/unity-text-locator/scripts/build_static_tmp_fontasset.py b/unity-text-locator/scripts/build_static_tmp_fontasset.py new file mode 100644 index 0000000..2e46a28 --- /dev/null +++ b/unity-text-locator/scripts/build_static_tmp_fontasset.py @@ -0,0 +1,367 @@ +#!/usr/bin/env python3 +"""Build a static TMP FontAsset candidate while preserving target identities.""" + +from __future__ import annotations + +import argparse +import copy +import csv +import hashlib +import json +import os +import re +import tempfile +import unicodedata +from pathlib import Path +from typing import Any, Iterable + +import UnityPy + + +TAG_RE = re.compile(r"<[^>]*>") + + +def sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as stream: + for chunk in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def inside(path: Path, root: Path) -> bool: + try: + path.resolve().relative_to(root.resolve()) + return True + except ValueError: + return False + + +def versions(env: Any) -> list[str]: + return sorted( + { + str(getattr(obj.assets_file, "unity_version", "")) + for obj in env.objects + if getattr(obj.assets_file, "unity_version", "") + } + ) + + +def object_at(env: Any, type_name: str, path_id: int) -> tuple[Any, dict[str, Any]]: + matches = [obj for obj in env.objects if obj.type.name == type_name and int(obj.path_id) == path_id] + if len(matches) != 1: + raise RuntimeError(f"expected exactly one {type_name} PathID {path_id}, found {len(matches)}") + return matches[0], matches[0].read_typetree() + + +def infer_donor_object(env: Any, type_name: str, path_id: int | None) -> tuple[Any, dict[str, Any]]: + if path_id is not None: + return object_at(env, type_name, path_id) + matches = [obj for obj in env.objects if obj.type.name == type_name] + if len(matches) != 1: + raise RuntimeError( + f"donor has {len(matches)} {type_name} objects; pass the matching --donor-{type_name.lower()}-path-id" + ) + return matches[0], matches[0].read_typetree() + + +def walk_files(file: Any, name: str = "") -> Iterable[tuple[str, Any]]: + yield name, file + children = getattr(file, "files", None) + if isinstance(children, dict): + for child_name, child in children.items(): + yield from walk_files(child, str(child_name)) + + +def top_file(env: Any) -> Any: + roots = list(env.files.values()) + if len(roots) != 1: + raise RuntimeError(f"expected one top-level Unity container, found {len(roots)}") + return roots[0] + + +def stream_bytes(env: Any, texture: dict[str, Any]) -> bytes: + inline = bytes(texture.get("image data", b"")) + stream = texture.get("m_StreamData") or {} + stream_path = str(stream.get("path", "")) + stream_size = int(stream.get("size", 0)) + stream_offset = int(stream.get("offset", 0)) + if stream_path and stream_size: + wanted = Path(stream_path.replace("\\", "/")).name.lower() + matches = [] + for child_name, file in walk_files(top_file(env)): + object_name = str(getattr(file, "name", "") or getattr(file, "path", "")) + candidate_name = child_name or object_name + if Path(candidate_name.replace("\\", "/")).name.lower() == wanted: + matches.append(file) + if len(matches) != 1: + raise RuntimeError(f"expected one streamed atlas {stream_path!r}, found {len(matches)}") + source = matches[0] + source.seek(stream_offset) + payload = source.read_bytes(stream_size) + else: + payload = inline + expected = int(texture.get("m_CompleteImageSize", len(payload))) + if not payload or len(payload) != expected: + raise RuntimeError(f"atlas payload length {len(payload)} does not match m_CompleteImageSize {expected}") + return payload + + +def rewrite_pptr(value: Any, source_path_id: int, target_path_id: int) -> None: + if isinstance(value, dict): + if "m_FileID" in value and "m_PathID" in value and int(value["m_PathID"]) == source_path_id: + value["m_FileID"] = 0 + value["m_PathID"] = target_path_id + for child in value.values(): + rewrite_pptr(child, source_path_id, target_path_id) + elif isinstance(value, (list, tuple)): + for child in value: + rewrite_pptr(child, source_path_id, target_path_id) + + +def translation_files(root: Path) -> list[Path]: + if root.is_file(): + return [root] + return sorted(path for path in root.rglob("*.csv") if "translation" in path.stem.lower()) + + +def required_characters(root: Path) -> set[str]: + files = translation_files(root) + if not files: + raise RuntimeError(f"no translation CSV files found under {root}") + result: set[str] = set() + for path in files: + with path.open("r", encoding="utf-8-sig", newline="") as stream: + rows = list(csv.reader(stream)) + if not rows: + continue + start = 1 if len(rows[0]) == 1 and rows[0][0].strip().lower() in {"zh_cn", "translation"} else 0 + for row in rows[start:]: + if not row: + continue + value = TAG_RE.sub("", row[-1]) + result.update( + char + for char in value + if not char.isspace() and not unicodedata.category(char).startswith("C") + ) + if not result: + raise RuntimeError("translation inputs contain no visible characters; coverage cannot be proven") + return result + + +def pptr(tree: dict[str, Any], key: str) -> tuple[int, int]: + value = tree.get(key) or {} + return int(value.get("m_FileID", 0)), int(value.get("m_PathID", 0)) + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--target", type=Path, required=True, help="Target bundle or asset to clone") + parser.add_argument("--target-font-path-id", type=int, required=True) + parser.add_argument("--target-material-path-id", type=int, required=True) + parser.add_argument("--target-texture-path-id", type=int, required=True) + parser.add_argument("--donor", type=Path, required=True, help="Static TMP donor bundle") + parser.add_argument("--donor-font-path-id", type=int) + parser.add_argument("--donor-material-path-id", type=int) + parser.add_argument("--donor-texture-path-id", type=int) + parser.add_argument("--translation-root", type=Path, required=True) + parser.add_argument("--game-root", type=Path, required=True) + parser.add_argument("--out", type=Path, required=True) + parser.add_argument("--report", type=Path, required=True) + parser.add_argument("--force", action="store_true", help="replace an existing external candidate/report") + parser.add_argument( + "--allow-version-mismatch", + action="store_true", + help="Allow a mismatched serialized Unity version and mark the result as canary-only", + ) + parser.add_argument( + "--allow-donor-glyph-aliases", + action="store_true", + help="Allow required donor characters that share a glyph index; mark the result as canary-only", + ) + args = parser.parse_args() + + target = args.target.resolve() + donor = args.donor.resolve() + translation_root = args.translation_root.resolve() + game_root = args.game_root.resolve() + out = args.out.resolve() + report_path = args.report.resolve() + for source in (target, donor): + if not source.is_file(): + raise SystemExit(f"input file not found: {source}") + if not game_root.is_dir(): + raise SystemExit(f"game root not found: {game_root}") + if out == target or inside(out, game_root) or inside(report_path, game_root): + raise SystemExit("candidate and report must be outside the game root and must not replace the target") + for destination in (out, report_path): + if destination.exists() and not destination.is_file(): + raise SystemExit(f"output path exists and is not a file: {destination}") + existing = [str(path) for path in (out, report_path) if path.exists()] + if existing and not args.force: + raise SystemExit("external output already exists; pass --force to replace: " + ", ".join(existing)) + + target_env = UnityPy.load(str(target)) + donor_env = UnityPy.load(str(donor)) + target_versions = versions(target_env) + donor_versions = versions(donor_env) + if target_versions != donor_versions and not args.allow_version_mismatch: + raise SystemExit( + f"Unity version mismatch: target={target_versions}, donor={donor_versions}; " + "use --allow-version-mismatch only for an isolated runtime canary" + ) + + target_font_obj, target_font = object_at(target_env, "MonoBehaviour", args.target_font_path_id) + target_material_obj, target_material = object_at(target_env, "Material", args.target_material_path_id) + target_texture_obj, target_texture = object_at(target_env, "Texture2D", args.target_texture_path_id) + donor_font_obj, donor_font = infer_donor_object(donor_env, "MonoBehaviour", args.donor_font_path_id) + donor_material_obj, donor_material = infer_donor_object(donor_env, "Material", args.donor_material_path_id) + donor_texture_obj, donor_texture = infer_donor_object(donor_env, "Texture2D", args.donor_texture_path_id) + if not donor_font.get("m_CharacterTable") or not donor_font.get("m_GlyphTable"): + raise SystemExit("donor is not a populated static TMP FontAsset") + + required = required_characters(translation_root) + donor_codepoints = {int(entry["m_Unicode"]) for entry in donor_font["m_CharacterTable"]} + missing = sorted((char, f"U+{ord(char):04X}") for char in required if ord(char) not in donor_codepoints) + if missing: + preview = ", ".join(f"{char}({codepoint})" for char, codepoint in missing[:20]) + raise SystemExit(f"donor misses {len(missing)} required characters; glyph aliases are forbidden: {preview}") + codepoints_by_glyph: dict[int, set[int]] = {} + for entry in donor_font["m_CharacterTable"]: + codepoints_by_glyph.setdefault(int(entry.get("m_GlyphIndex", -1)), set()).add(int(entry["m_Unicode"])) + donor_entries = {int(entry["m_Unicode"]): entry for entry in donor_font["m_CharacterTable"]} + aliases = { + f"U+{codepoint:04X}": [f"U+{other:04X}" for other in sorted(codepoints_by_glyph[glyph]) if other != codepoint] + for codepoint in sorted(ord(char) for char in required) + if len( + codepoints_by_glyph[ + (glyph := int(donor_entries[codepoint].get("m_GlyphIndex", -1))) + ] + ) + > 1 + } + if aliases and not args.allow_donor_glyph_aliases: + preview = ", ".join(f"{key}->{value}" for key, value in list(aliases.items())[:10]) + raise SystemExit( + f"donor uses shared glyph indices for {len(aliases)} required characters; " + f"use --allow-donor-glyph-aliases only for a reviewed canary: {preview}" + ) + + atlas = stream_bytes(donor_env, donor_texture) + new_texture = copy.deepcopy(donor_texture) + new_texture["m_Name"] = target_texture.get("m_Name", "") + new_texture["image data"] = atlas + new_texture["m_StreamData"] = {"offset": 0, "size": 0, "path": ""} + target_texture_obj.save_typetree(new_texture) + + new_material = copy.deepcopy(donor_material) + new_material["m_Name"] = target_material.get("m_Name", "") + new_material["m_Shader"] = copy.deepcopy(target_material.get("m_Shader")) + rewrite_pptr(new_material, int(donor_texture_obj.path_id), int(target_texture_obj.path_id)) + target_material_obj.save_typetree(new_material) + + new_font = copy.deepcopy(donor_font) + for key in ("m_GameObject", "m_Enabled", "m_Script", "m_Name", "m_EditorClassIdentifier"): + if key in target_font: + new_font[key] = copy.deepcopy(target_font[key]) + new_font["m_Material"] = {"m_FileID": 0, "m_PathID": int(target_material_obj.path_id)} + new_font["m_AtlasTextures"] = [{"m_FileID": 0, "m_PathID": int(target_texture_obj.path_id)}] + new_font["m_SourceFontFile"] = {"m_FileID": 0, "m_PathID": 0} + new_font["m_AtlasPopulationMode"] = 0 + if "m_IsMultiAtlasTexturesEnabled" in new_font: + new_font["m_IsMultiAtlasTexturesEnabled"] = 0 + target_font_obj.save_typetree(new_font) + + out.parent.mkdir(parents=True, exist_ok=True) + report_path.parent.mkdir(parents=True, exist_ok=True) + fd, temp_name = tempfile.mkstemp(prefix=out.name + ".", suffix=".tmp", dir=str(out.parent)) + os.close(fd) + temp = Path(temp_name) + try: + temp.write_bytes(top_file(target_env).save()) + # Loading from bytes prevents UnityPy from retaining a Windows file + # handle that would block the atomic os.replace below. + check_env = UnityPy.load(temp.read_bytes()) + check_font_obj, check_font = object_at(check_env, "MonoBehaviour", args.target_font_path_id) + check_material_obj, check_material = object_at(check_env, "Material", args.target_material_path_id) + check_texture_obj, check_texture = object_at(check_env, "Texture2D", args.target_texture_path_id) + reopened_codepoints = {int(entry["m_Unicode"]) for entry in check_font.get("m_CharacterTable", [])} + reopened_missing = sorted(f"U+{ord(char):04X}" for char in required if ord(char) not in reopened_codepoints) + invariants = { + "font_path_id": int(check_font_obj.path_id) == int(target_font_obj.path_id), + "material_path_id": int(check_material_obj.path_id) == int(target_material_obj.path_id), + "texture_path_id": int(check_texture_obj.path_id) == int(target_texture_obj.path_id), + "font_name": check_font.get("m_Name") == target_font.get("m_Name"), + "material_name": check_material.get("m_Name") == target_material.get("m_Name"), + "texture_name": check_texture.get("m_Name") == target_texture.get("m_Name"), + "script": check_font.get("m_Script") == target_font.get("m_Script"), + "shader": check_material.get("m_Shader") == target_material.get("m_Shader"), + "font_material_ref": pptr(check_font, "m_Material") == (0, args.target_material_path_id), + "font_atlas_ref": bool(check_font.get("m_AtlasTextures")) + and int(check_font["m_AtlasTextures"][0].get("m_PathID", 0)) == args.target_texture_path_id, + "atlas_inline": len(check_texture.get("image data", b"")) == len(atlas), + "source_font_cleared": pptr(check_font, "m_SourceFontFile") == (0, 0), + "static_population": int(check_font.get("m_AtlasPopulationMode", -1)) == 0, + "coverage_complete": not reopened_missing, + } + if not all(invariants.values()): + failed = [name for name, passed in invariants.items() if not passed] + raise RuntimeError(f"candidate reopen verification failed: {failed}") + if out.exists() and not args.force: + raise RuntimeError(f"candidate appeared during build; refusing to replace it: {out}") + os.replace(temp, out) + finally: + if temp.exists(): + temp.unlink() + + report = { + "candidate": str(out), + "source_target": str(target), + "donor": str(donor), + "sha256": {"target": sha256(target), "donor": sha256(donor), "candidate": sha256(out)}, + "unity_versions": {"target": target_versions, "donor": donor_versions}, + "version_mismatch_allowed": bool(args.allow_version_mismatch and target_versions != donor_versions), + "canary_only": bool( + (args.allow_version_mismatch and target_versions != donor_versions) + or (args.allow_donor_glyph_aliases and aliases) + ), + "target_ids": { + "font": args.target_font_path_id, + "material": args.target_material_path_id, + "texture": args.target_texture_path_id, + }, + "donor_ids": { + "font": int(donor_font_obj.path_id), + "material": int(donor_material_obj.path_id), + "texture": int(donor_texture_obj.path_id), + }, + "identity_and_reopen_checks": invariants, + "font": { + "character_count": len(check_font.get("m_CharacterTable", [])), + "glyph_count": len(check_font.get("m_GlyphTable", [])), + "atlas_population_mode": int(check_font.get("m_AtlasPopulationMode", -1)), + }, + "texture": { + "width": int(check_texture.get("m_Width", 0)), + "height": int(check_texture.get("m_Height", 0)), + "format": int(check_texture.get("m_TextureFormat", -1)), + "inline_bytes": len(check_texture.get("image data", b"")), + }, + "coverage": { + "required_unique": len(required), + "covered_unique": len(required), + "missing": reopened_missing, + "glyph_aliases": aliases, + "donor_glyph_aliases_allowed": bool(args.allow_donor_glyph_aliases and aliases), + }, + "not_applied_to_game": True, + "runtime_canary_required": True, + } + report_path.write_text(json.dumps(report, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") + print(json.dumps(report, ensure_ascii=False, indent=2)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/unity-text-locator/scripts/extract_dmsl_text.py b/unity-text-locator/scripts/extract_dmsl_text.py new file mode 100644 index 0000000..cbb0796 --- /dev/null +++ b/unity-text-locator/scripts/extract_dmsl_text.py @@ -0,0 +1,378 @@ +#!/usr/bin/env python3 +"""Extract fixed-width DMSL UTF-8 operands from Unity MonoBehaviours. + +The supported operand is ``0x06 + uint32 little-endian byte length + UTF-8``. +Every duplicate is retained as a separate occurrence in a sidecar manifest. +This command is read-only with respect to Unity inputs. +""" +from __future__ import annotations + +import argparse +import csv +import hashlib +import json +import os +import re +import struct +import sys +from collections import defaultdict +from pathlib import Path +from typing import Any, Iterable, Iterator, Sequence + +import UnityPy + + +FORMAT = "dmsl-0x06-u32le-utf8-v3" +KANA_RE = re.compile(r"[\u3040-\u30ff]") +JP_RE = re.compile(r"[\u3040-\u30ff\u3400-\u4dbf\u4e00-\u9fff]") +PATHISH_RE = re.compile( + r"(?:^|[/\\])(?:assets?|resources?|scenario|script|voice|audio|bgm|se|cg|image|sprite)" + r"(?:[/\\]|$)|\.(?:asset|prefab|png|jpe?g|webp|wav|ogg|mp3|mp4|anim|controller|mat|shader)$", + re.IGNORECASE, +) +INVALID_STEM_RE = re.compile(r"[<>:\"/\\|?*\x00-\x1f]") +IDENTIFIER_RE = re.compile(r"[\w./\\:@+\-]+", re.UNICODE) + + +def sha256_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as stream: + for block in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(block) + return digest.hexdigest() + + +def sha256_bytes(data: bytes) -> str: + return hashlib.sha256(data).hexdigest() + + +def flatten_text(text: str) -> str: + """Represent control whitespace without putting physical newlines in CSV.""" + return ( + text.replace("\\", "\\\\") + .replace("\r\n", "\n") + .replace("\r", "\n") + .replace("\n", "\\n") + .replace("\t", "\\t") + ) + + +def safe_stem(value: str) -> str: + stem = INVALID_STEM_RE.sub("_", value).strip().rstrip(".") + if not stem: + raise ValueError("project name is empty after filename sanitization") + return stem + + +def is_within(path: Path, root: Path) -> bool: + return path == root or root in path.parents + + +def resolve_under(root: Path, value: Path) -> Path: + path = (root / value).resolve() if not value.is_absolute() else value.resolve() + if not is_within(path, root): + raise ValueError(f"input must be within game_root: {path}") + return path + + +def unwrap_byte_array(value: Any) -> tuple[list[Any] | None, tuple[str, ...]]: + suffix: list[str] = [] + seen: set[int] = set() + while isinstance(value, dict): + identity = id(value) + if identity in seen: + return None, () + seen.add(identity) + key = next((candidate for candidate in ("data", "Array", "bytecode") if candidate in value), None) + if key is None: + return None, () + suffix.append(key) + value = value[key] + return (value, tuple(suffix)) if isinstance(value, list) else (None, ()) + + +def iter_bytecode_arrays(value: Any, path: tuple[Any, ...] = ()) -> Iterator[tuple[tuple[Any, ...], list[Any]]]: + """Yield exact paths to fields named bytecode, including list nesting.""" + if isinstance(value, dict): + for key, child in value.items(): + child_path = path + (key,) + if key.lower() == "bytecode": + array, suffix = unwrap_byte_array(child) + if array is not None: + yield child_path + suffix, array + continue + if isinstance(child, (dict, list)): + yield from iter_bytecode_arrays(child, child_path) + elif isinstance(value, list): + for index, child in enumerate(value): + if isinstance(child, (dict, list)): + yield from iter_bytecode_arrays(child, path + (index,)) + + +def visible_japanese(text: str) -> bool: + stripped = text.strip() + if not KANA_RE.search(stripped) or len(JP_RE.findall(stripped)) < 2: + return False + if PATHISH_RE.search(stripped): + return False + if IDENTIFIER_RE.fullmatch(stripped) and not re.search(r"[、。!?!?…「」『』\s]", stripped): + return False + return valid_visible_text(text) + + +def valid_visible_text(text: str) -> bool: + if not text or "\x00" in text: + return False + if any(ord(char) < 0x20 and char not in "\n\r\t" for char in text): + return False + return any(not char.isspace() for char in text) + + +def parse_operands(data: bytes, include_all_utf8: bool) -> Iterator[tuple[int, int, bytes, str]]: + """Yield syntactically valid operands without deduplicating occurrences.""" + for offset in range(max(0, len(data) - 5)): + if data[offset] != 0x06: + continue + byte_length = struct.unpack_from(" len(data): + continue + payload = data[start:end] + try: + text = payload.decode("utf-8", errors="strict") + except UnicodeDecodeError: + continue + if valid_visible_text(text) and (include_all_utf8 or visible_japanese(text)): + yield offset, byte_length, payload, text + + +def object_name(tree: dict[str, Any], path_id: int) -> str: + for key in ("m_Name", "name", "Name"): + value = tree.get(key) + if isinstance(value, str) and value: + return value + return f"MonoBehaviour_{path_id}" + + +def serialized_file_name(obj: Any) -> str: + assets_file = getattr(obj, "assets_file", None) + name = getattr(assets_file, "name", None) + if not isinstance(name, str) or not name: + raise ValueError(f"path_id {obj.path_id} has no stable serialized file name") + return name + + +def normalize_byte_array(raw: Sequence[Any]) -> bytes: + try: + return bytes(int(value) & 0xFF for value in raw) + except (TypeError, ValueError) as exc: + raise ValueError("bytecode field is not an integer byte array") from exc + + +def collect_candidates( + game_root: Path, + inputs: Sequence[Path], + candidate_roots: Sequence[Path], + patterns: Sequence[str], +) -> list[Path]: + found: dict[str, Path] = {} + + def add(path: Path) -> None: + resolved = resolve_under(game_root, path) + if not resolved.is_file(): + raise ValueError(f"input file does not exist: {resolved}") + key = str(resolved).casefold() + found[key] = resolved + + for value in inputs: + resolved = resolve_under(game_root, value) + if resolved.is_file(): + add(resolved) + elif resolved.is_dir(): + for pattern in patterns: + for path in resolved.rglob(pattern): + if path.is_file(): + add(path) + else: + raise ValueError(f"input does not exist: {resolved}") + for value in candidate_roots: + root = resolve_under(game_root, value) + if not root.is_dir(): + raise ValueError(f"candidate root is not a directory: {root}") + for pattern in patterns: + for path in root.rglob(pattern): + if path.is_file(): + add(path) + return sorted(found.values(), key=lambda item: item.as_posix().casefold()) + + +def scan_source(path: Path, game_root: Path, include_all_utf8: bool) -> tuple[list[dict[str, Any]], list[str]]: + rows: list[dict[str, Any]] = [] + warnings: list[str] = [] + env = UnityPy.load(str(path)) + for obj in env.objects: + if obj.type.name != "MonoBehaviour": + continue + serialized_file = serialized_file_name(obj) + try: + tree = obj.read_typetree() + except Exception as exc: + warnings.append(f"path_id {obj.path_id}: typetree: {type(exc).__name__}: {exc}") + continue + if not isinstance(tree, dict): + continue + for field_path, raw in iter_bytecode_arrays(tree): + try: + data = normalize_byte_array(raw) + except ValueError as exc: + warnings.append(f"path_id {obj.path_id} field {list(field_path)!r}: {exc}") + continue + for offset, width, payload, text in parse_operands(data, include_all_utf8): + rows.append( + { + "source_path": path.relative_to(game_root).as_posix(), + "serialized_file": serialized_file, + "path_id": int(obj.path_id), + "object_type": obj.type.name, + "object_name": object_name(tree, int(obj.path_id)), + "bytecode_field_path": list(field_path), + "bytecode_offset": offset, + "byte_length": width, + "original_utf8_sha256": sha256_bytes(payload), + "original_flat": flatten_text(text), + } + ) + return rows, warnings + + +def write_source_csv(path: Path, rows: Sequence[dict[str, Any]]) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + with path.open("w", encoding="utf-8-sig", newline="") as stream: + writer = csv.writer(stream, lineterminator="\n") + writer.writerow(["original_flat"]) + writer.writerows([row["original_flat"]] for row in rows) + + +def parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("game_root", type=Path, help="root used for portable manifest-relative source paths") + parser.add_argument("--input", type=Path, action="append", default=[], help="explicit file or directory; repeatable") + parser.add_argument( + "--candidate-root", type=Path, action="append", default=[], + help="directory to scan recursively below game_root; repeatable", + ) + parser.add_argument("--glob", dest="patterns", action="append", help="candidate filename glob; repeatable") + parser.add_argument("--project-name", help="output filename prefix (default: game_root folder name)") + parser.add_argument("--out-dir", type=Path, help="report directory (default: game_root/_translation/.../text)") + parser.add_argument( + "--include-all-utf8", action="store_true", + help="include every printable UTF-8 operand instead of conservative Japanese-visible rows", + ) + parser.add_argument( + "--allow-load-errors", action="store_true", + help="write an incomplete extraction and return success when some Unity inputs fail to load", + ) + return parser.parse_args(argv) + + +def main(argv: Sequence[str] | None = None) -> int: + args = parse_args(argv) + game_root = args.game_root.resolve() + if not game_root.is_dir(): + raise SystemExit(f"game_root is not a directory: {game_root}") + if not args.input and not args.candidate_root: + raise SystemExit("provide at least one --input or --candidate-root") + patterns = args.patterns or ["*.bundle"] + try: + candidates = collect_candidates(game_root, args.input, args.candidate_root, patterns) + project_name = safe_stem(args.project_name or game_root.name) + except ValueError as exc: + raise SystemExit(str(exc)) from exc + if not candidates: + raise SystemExit("no candidate files matched") + + out_dir = (args.out_dir or game_root / "_translation" / "unity-text-report" / "text").resolve() + csv_path = out_dir / f"{project_name}_dmsl_text.csv" + manifest_path = out_dir / f"{project_name}_dmsl_text.manifest.json" + all_rows: list[dict[str, Any]] = [] + errors: list[dict[str, str]] = [] + warnings: list[dict[str, Any]] = [] + source_hashes: dict[str, str] = {} + + for index, source in enumerate(candidates, 1): + relative = source.relative_to(game_root).as_posix() + try: + rows, source_warnings = scan_source(source, game_root, args.include_all_utf8) + source_hashes[relative] = sha256_file(source) + all_rows.extend(rows) + if source_warnings: + warnings.append({"source_path": relative, "messages": source_warnings}) + print(f"[{index}/{len(candidates)}] {relative}: {len(rows)}", file=sys.stderr) + except Exception as exc: + errors.append({"source_path": relative, "error": f"{type(exc).__name__}: {exc}"}) + + locations: set[tuple[Any, ...]] = set() + occurrence_counts: defaultdict[str, int] = defaultdict(int) + manifest_rows: list[dict[str, Any]] = [] + for row_index, row in enumerate(all_rows): + location = ( + row["source_path"], row["serialized_file"], row["path_id"], row["object_type"], + json.dumps(row["bytecode_field_path"], ensure_ascii=True), row["bytecode_offset"], + ) + if location in locations: + raise SystemExit(f"duplicate extracted location: {location!r}") + locations.add(location) + original_key = row["original_utf8_sha256"] + occurrence_index = occurrence_counts[original_key] + occurrence_counts[original_key] += 1 + occurrence_seed = json.dumps(location, ensure_ascii=True, separators=(",", ":")).encode("utf-8") + manifest_rows.append( + { + "row_index": row_index, + "occurrence_index": occurrence_index, + "occurrence_id": sha256_bytes(occurrence_seed)[:24], + **{key: value for key, value in row.items() if key != "original_flat"}, + } + ) + + write_source_csv(csv_path, all_rows) + manifest = { + "format": FORMAT, + "text_escape": "backslash-controls-v1", + "fixed_width": True, + "source_csv_name": csv_path.name, + "source_csv_sha256": sha256_file(csv_path), + "source_hashes_sha256": source_hashes, + "rows": manifest_rows, + "scan": { + "candidate_count": len(candidates), + "matched_occurrences": len(manifest_rows), + "load_errors": errors, + "warnings": warnings, + "filter": "all-printable-utf8" if args.include_all_utf8 else "conservative-visible-japanese", + }, + } + manifest_path.parent.mkdir(parents=True, exist_ok=True) + manifest_path.write_text(json.dumps(manifest, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") + summary = { + "csv": str(csv_path), + "manifest": str(manifest_path), + "rows": len(manifest_rows), + "sources": len(source_hashes), + "load_errors": len(errors), + } + print(json.dumps(summary, ensure_ascii=False)) + if errors and not args.allow_load_errors: + return 1 + return 0 + + +if __name__ == "__main__": + exit_code = main() + # Some UnityPy native dependencies crash in CPython finalization after a + # large multi-bundle scan. This is a single-shot read-only CLI, so flush + # completed outputs and bypass that native teardown path. + sys.stdout.flush() + sys.stderr.flush() + os._exit(exit_code) diff --git a/unity-text-locator/scripts/extract_typetree_ui_text.py b/unity-text-locator/scripts/extract_typetree_ui_text.py new file mode 100644 index 0000000..1052070 --- /dev/null +++ b/unity-text-locator/scripts/extract_typetree_ui_text.py @@ -0,0 +1,376 @@ +#!/usr/bin/env python3 +"""Extract Japanese UI text from Unity TypeTrees without modifying the game. + +The extractor records every occurrence separately. A row is addressed by the +bundle path, serialized asset file, object PathID, and a typed field path; this +avoids conflating equal strings or equal PathIDs in different serialized files. +""" +from __future__ import annotations + +import argparse +import csv +import hashlib +import json +import os +import re +import sys +from collections import Counter, defaultdict +from pathlib import Path +from typing import Any, Iterable + +try: + import UnityPy +except ImportError as exc: # pragma: no cover - depends on the host environment + raise SystemExit("UnityPy is required: install it in the active Python environment") from exc + + +FORMAT = "unity-typetree-ui-text-v2" +JP_RE = re.compile(r"[\u3040-\u30ff\u3400-\u4dbf\u4e00-\u9fff]") +KANA_RE = re.compile(r"[\u3040-\u30ff]") +KEYISH_RE = re.compile(r"^[A-Za-z0-9_./\\:@+\-]+$") +EXT_RE = re.compile( + r"\.(?:png|jpe?g|webp|wav|ogg|mp3|mp4|asset|prefab|anim|controller|mat|shader)$", + re.IGNORECASE, +) +TOKEN_RE = re.compile( + r"<[^<>]+>|\\[nrt]|\{[^{}\r\n]+\}|" + r"%(?:\d+\$)?[-+#0 ]*\d*(?:\.\d+)?[diouxXeEfFgGcrsa%]" +) +TEXT_LEAVES = {"text", "m_text"} +LABEL_HINTS = ("label", "caption", "title", "description", "option", "tooltip") +VALUE_LEAVES = {"m_value", "string", "value"} + + +def file_sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as stream: + for block in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(block) + return digest.hexdigest() + + +def text_sha256(value: str) -> str: + return hashlib.sha256(value.encode("utf-8")).hexdigest() + + +def flatten_text(value: str) -> str: + return value.replace("\r\n", "\\n").replace("\n", "\\n").replace("\r", "\\n") + + +def newline_events(value: str) -> list[str]: + events: list[str] = [] + index = 0 + while index < len(value): + if value.startswith("\r\n", index): + events.append("CRLF") + index += 2 + elif value[index] == "\n": + events.append("LF") + index += 1 + elif value[index] == "\r": + events.append("CR") + index += 1 + else: + index += 1 + return events + + +def token_info(value: str) -> dict[str, Any]: + ordered = TOKEN_RE.findall(value) + return {"ordered": ordered, "count": len(ordered)} + + +def visible_japanese(value: str) -> bool: + stripped = value.strip() + if not stripped or "\x00" in stripped or not JP_RE.search(stripped): + return False + if any(ord(char) < 0x20 and char not in "\r\n\t" for char in stripped): + return False + if EXT_RE.search(stripped) or KEYISH_RE.fullmatch(stripped): + return False + # A lone CJK codepoint is commonly an enum/key artifact. Kana or at least + # two Japanese/CJK codepoints is a conservative default for visible UI. + return bool(KANA_RE.search(stripped) or len(JP_RE.findall(stripped)) >= 2) + + +def safe_nonempty_text(value: str) -> bool: + stripped = value.strip() + return bool(stripped) and "\x00" not in stripped and not any( + ord(char) < 0x20 and char not in "\r\n\t" for char in stripped + ) + + +def iter_strings(value: Any, path: tuple[str | int, ...] = ()) -> Iterable[tuple[tuple[str | int, ...], str]]: + if isinstance(value, dict): + for key, child in value.items(): + yield from iter_strings(child, path + (str(key),)) + elif isinstance(value, list): + for index, child in enumerate(value): + yield from iter_strings(child, path + (index,)) + elif isinstance(value, str): + yield path, value + + +def normalized_name(value: str | int) -> str: + if isinstance(value, int): + return "" + return value.lower().replace("_", "") + + +def classify(path: tuple[str | int, ...]) -> str | None: + names = [part for part in path if isinstance(part, str)] + if not names: + return None + normalized = [normalized_name(part) for part in names] + leaf = normalized[-1] + if "mtabledata" in normalized and leaf == "mlocalized": + return "localization_string_table" + if leaf in {normalized_name(item) for item in TEXT_LEAVES}: + return "component_text" + if any(hint in leaf for hint in LABEL_HINTS): + return "ui_label_field" + if leaf in {normalized_name(item) for item in VALUE_LEAVES} and any( + any(hint in parent for hint in LABEL_HINTS) for parent in normalized[:-1] + ): + return "ui_label_field" + return None + + +def display_field_path(path: tuple[str | int, ...]) -> str: + rendered = "" + for part in path: + if isinstance(part, int): + rendered += f"[{part}]" + else: + rendered += ("." if rendered else "") + part + return rendered + + +def serialized_file_name(obj: Any) -> str: + assets_file = getattr(obj, "assets_file", None) + name = getattr(assets_file, "name", None) + return str(name or "") + + +def object_name(tree: dict[str, Any], obj: Any) -> str: + for key in ("m_Name", "name", "Name"): + value = tree.get(key) + if isinstance(value, str) and value: + return value + return f"{obj.type.name}_{obj.path_id}" + + +def container_paths(env: Any) -> dict[tuple[str, int], list[str]]: + result: dict[tuple[str, int], list[str]] = defaultdict(list) + try: + entries = env.container.items() + except Exception: + return result + for path, pointer in entries: + try: + obj = pointer.get_obj() if hasattr(pointer, "get_obj") else pointer + key = (serialized_file_name(obj), int(obj.path_id)) + result[key].append(str(path)) + except Exception: + continue + return result + + +def scan_bundle(path: Path, root: Path, include_all_text: bool) -> tuple[list[dict[str, Any]], str | None]: + try: + env = UnityPy.load(path.read_bytes()) + except Exception as exc: + return [], f"load: {type(exc).__name__}: {exc}" + + rows: list[dict[str, Any]] = [] + containers = container_paths(env) + for obj in env.objects: + if obj.type.name != "MonoBehaviour": + continue + try: + tree = obj.read_typetree() + except Exception: + continue + if not isinstance(tree, dict): + continue + asset_file = serialized_file_name(obj) + name = object_name(tree, obj) + for field_path, original in iter_strings(tree): + category = classify(field_path) + if category is None or not ( + safe_nonempty_text(original) if include_all_text else visible_japanese(original) + ): + continue + relative = path.relative_to(root).as_posix() + events = newline_events(original) + rows.append( + { + "original_flat": flatten_text(original), + "original": original, + "bundle_path": relative, + "serialized_file": asset_file, + "container_paths": sorted(containers.get((asset_file, int(obj.path_id)), [])), + "path_id": str(obj.path_id), + "object_type": obj.type.name, + "object_name": name, + "field_path": list(field_path), + "field_path_display": display_field_path(field_path), + "category": category, + "source_hash_sha256": text_sha256(original), + "newlines": {"count": len(events), "ordered": events}, + "tokens": token_info(original), + } + ) + return rows, None + + +def resolve_scan_roots(game_root: Path, requested: list[Path]) -> list[Path]: + if not requested: + return [game_root] + roots: list[Path] = [] + for value in requested: + resolved = value.resolve() if value.is_absolute() else (game_root / value).resolve() + if resolved != game_root and game_root not in resolved.parents: + raise ValueError(f"scan root must be inside game_root: {resolved}") + if not resolved.is_dir(): + raise FileNotFoundError(f"scan root is not a directory: {resolved}") + roots.append(resolved) + return roots + + +def discover_bundles(game_root: Path, scan_roots: list[Path], patterns: list[str]) -> list[Path]: + found: set[Path] = set() + for scan_root in scan_roots: + for pattern in patterns: + for candidate in scan_root.glob(pattern): + resolved = candidate.resolve() + if resolved.is_file() and (resolved == game_root or game_root in resolved.parents): + found.add(resolved) + return sorted(found, key=lambda item: item.relative_to(game_root).as_posix().lower()) + + +def safe_stem(value: str) -> str: + cleaned = re.sub(r"[<>:\"/\\|?*\x00-\x1f]", "_", value).strip(" ._") + return cleaned or "unity" + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("game_root", type=Path, help="Unity game root (read-only)") + parser.add_argument( + "--scan-root", + type=Path, + action="append", + default=[], + help="directory inside game_root to scan; repeatable (default: game_root)", + ) + parser.add_argument( + "--glob", + action="append", + default=[], + help="recursive or direct glob relative to each scan root; repeatable " + "(defaults: **/*.bundle and **/*.assets)", + ) + parser.add_argument("--out-dir", type=Path, required=True, help="directory for CSV and manifest") + parser.add_argument("--stem", help="output stem (default: _ui_text)") + parser.add_argument( + "--include-all-text", + action="store_true", + help="include classified non-Japanese fields as well as Japanese text", + ) + parser.add_argument( + "--strict-load-errors", + action="store_true", + help="return a failure when any matched file cannot be loaded by UnityPy", + ) + return parser.parse_args() + + +def main() -> int: + args = parse_args() + root = args.game_root.resolve() + if not root.is_dir(): + raise SystemExit(f"game_root is not a directory: {root}") + try: + scan_roots = resolve_scan_roots(root, args.scan_root) + except (ValueError, FileNotFoundError) as exc: + raise SystemExit(str(exc)) from exc + patterns = args.glob or ["**/*.bundle", "**/*.assets"] + bundles = discover_bundles(root, scan_roots, patterns) + if not bundles: + raise SystemExit("no candidate bundles matched the requested scan roots and globs") + + all_rows: list[dict[str, Any]] = [] + errors: list[dict[str, str]] = [] + bundle_hashes: dict[str, str] = {} + for index, bundle in enumerate(bundles, 1): + rows, error = scan_bundle(bundle, root, args.include_all_text) + relative = bundle.relative_to(root).as_posix() + if rows: + all_rows.extend(rows) + bundle_hashes[relative] = file_sha256(bundle) + print(f"[{index}/{len(bundles)}] {relative}: {len(rows)} rows", file=sys.stderr) + if error: + errors.append({"bundle_path": relative, "error": error}) + + occurrence_seen: Counter[str] = Counter() + occurrence_totals = Counter(row["original"] for row in all_rows) + rows_for_manifest: list[dict[str, Any]] = [] + for row_index, row in enumerate(all_rows): + original = row["original"] + occurrence_seen[original] += 1 + manifest_row = {key: value for key, value in row.items() if key != "original_flat"} + manifest_row.update( + { + "row_index": row_index, + "occurrence": occurrence_seen[original], + "occurrence_count": occurrence_totals[original], + } + ) + rows_for_manifest.append(manifest_row) + + out_dir = args.out_dir.resolve() + out_dir.mkdir(parents=True, exist_ok=True) + stem = safe_stem(args.stem or f"{root.name}_ui_text") + csv_path = out_dir / f"{stem}.csv" + manifest_path = out_dir / f"{stem}.manifest.json" + with csv_path.open("w", encoding="utf-8-sig", newline="") as stream: + writer = csv.writer(stream, lineterminator="\n") + writer.writerow(["original_flat"]) + writer.writerows([row["original_flat"]] for row in all_rows) + + manifest = { + "format": FORMAT, + "mode": "read-only-extraction", + "source_csv_name": csv_path.name, + "source_csv_sha256": file_sha256(csv_path), + "source": { + "game_root": str(root), + "bundle_hashes_sha256": bundle_hashes, + }, + "rows": rows_for_manifest, + "scan": { + "patterns": patterns, + "candidate_count": len(bundles), + "matched_occurrences": len(all_rows), + "bundles_with_rows": len(bundle_hashes), + "load_errors": errors, + }, + } + manifest_path.write_text(json.dumps(manifest, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") + summary = { + "csv": str(csv_path), + "manifest": str(manifest_path), + "rows": len(all_rows), + "bundles_with_rows": len(bundle_hashes), + "load_errors": len(errors), + } + print(json.dumps(summary, ensure_ascii=False)) + return 1 if args.strict_load_errors and errors else 0 + + +if __name__ == "__main__": + exit_code = main() + sys.stdout.flush() + sys.stderr.flush() + os._exit(exit_code) diff --git a/unity-text-locator/scripts/inspect_tmp_font_bundle.py b/unity-text-locator/scripts/inspect_tmp_font_bundle.py index ca58a1b..7a8f892 100644 --- a/unity-text-locator/scripts/inspect_tmp_font_bundle.py +++ b/unity-text-locator/scripts/inspect_tmp_font_bundle.py @@ -81,6 +81,14 @@ def pptr(value: Any) -> dict[str, int] | None: } +def normalized_family(value: str) -> str: + return re.sub(r"[^a-z0-9]", "", value.casefold()) + + +def population_mode_name(value: int) -> str: + return {0: "static", 1: "dynamic", 2: "dynamic-os"}.get(value, f"unknown-{value}") + + def collect_translation_chars(path: Path) -> Counter[str]: files = [path] if path.is_file() else sorted(path.rglob("*.csv")) chars: Counter[str] = Counter() @@ -112,16 +120,19 @@ def main() -> int: env = UnityPy.load(str(bundle)) materials: list[dict[str, Any]] = [] textures: list[dict[str, Any]] = [] + source_fonts: list[dict[str, Any]] = [] fonts: list[dict[str, Any]] = [] for obj in env.objects: - if obj.type.name not in {"MonoBehaviour", "Material", "Texture2D"}: + if obj.type.name not in {"MonoBehaviour", "Material", "Texture2D", "Font"}: continue try: data, name = read_name(obj) except Exception: continue - if obj.type.name == "Material": + if obj.type.name == "Font": + source_fonts.append({"name": name, "path_id": int(obj.path_id)}) + elif obj.type.name == "Material": materials.append({"name": name, "path_id": int(obj.path_id)}) elif obj.type.name == "Texture2D": stream = getattr(data, "m_StreamData", None) @@ -145,6 +156,11 @@ def main() -> int: if glyphs is None or characters is None: continue codepoints = {int(getattr(char, "m_Unicode", -1)) for char in characters} + population_mode = int(getattr(data, "m_AtlasPopulationMode", -1)) + source_font = pptr(getattr(data, "m_SourceFontFile", None)) + face_info = getattr(data, "m_FaceInfo", None) + face_family = str(getattr(face_info, "m_FamilyName", "")) if face_info else "" + face_style = str(getattr(face_info, "m_StyleName", "")) if face_info else "" fonts.append( { "name": name, @@ -153,13 +169,67 @@ def main() -> int: "glyph_count": len(glyphs), "atlas_width": int(getattr(data, "m_AtlasWidth", 0)), "atlas_height": int(getattr(data, "m_AtlasHeight", 0)), - "atlas_population_mode": int(getattr(data, "m_AtlasPopulationMode", -1)), + "atlas_population_mode": population_mode, + "population_mode": population_mode_name(population_mode), + "source_font_file": source_font, + "face_family": face_family, + "face_style": face_style, "material": pptr(getattr(data, "m_Material", None) or getattr(data, "material", None)), "atlas_textures": [pptr(item) for item in (getattr(data, "m_AtlasTextures", None) or [])], "codepoints": codepoints, } ) + sources_by_id = {item["path_id"]: item for item in source_fonts} + for font in fonts: + source_ref = font.get("source_font_file") or {} + source_name = "" + source_file_id = int(source_ref.get("file_id", 0)) + source_path_id = int(source_ref.get("path_id", 0)) + if source_file_id == 0: + source = sources_by_id.get(int(source_ref.get("path_id", 0))) + source_name = str((source or {}).get("name", "")) + if source_file_id: + source_status = "external-unresolved" + elif source_path_id == 0: + source_status = "none" + elif source_name: + source_status = "local-resolved" + else: + source_status = "local-unresolved" + face_family = str(font.get("face_family", "")) + identity_conflict: bool | None + if source_status != "local-resolved" or not face_family: + identity_conflict = None + else: + identity_conflict = bool( + normalized_family(source_name) != normalized_family(face_family) + and normalized_family(source_name) not in normalized_family(face_family) + and normalized_family(face_family) not in normalized_family(source_name) + ) + populated_dynamic = ( + str(font.get("population_mode", "")).startswith("dynamic") + and int(font.get("character_count", 0)) > 0 + and int(font.get("glyph_count", 0)) > 0 + ) + font["source_font_name"] = source_name + font["source_font_resolution"] = source_status + font["font_family_identity_conflict"] = identity_conflict + font["source_font_only_replacement_risk"] = populated_dynamic + font["risk_notes"] = [] + if identity_conflict is True: + font["risk_notes"].append( + "The TMP face family and referenced source Font identity differ; verify the visible component chain." + ) + elif source_status.endswith("unresolved"): + font["risk_notes"].append( + "The source Font identity is unresolved in this file; trace the visible component/CAB chain before comparing families." + ) + if populated_dynamic: + font["risk_notes"].append( + "BLOCK: this dynamic TMP asset already has character/glyph tables. Replacing only its source TTF can map valid Unicode to wrong glyph indices." + ) + used_chars = collect_translation_chars(args.translation_root.resolve()) if args.translation_root else Counter() for font in fonts: codepoints = font.pop("codepoints") @@ -184,6 +254,7 @@ def main() -> int: "sha256": sha256(bundle), "unity_versions": source_versions, "fonts": fonts, + "source_fonts": source_fonts, "materials": materials, "textures": textures, "runtime_canary_required": True, diff --git a/unity-text-locator/scripts/patch_addressables_catalog_crc.py b/unity-text-locator/scripts/patch_addressables_catalog_crc.py new file mode 100644 index 0000000..dd8ff48 --- /dev/null +++ b/unity-text-locator/scripts/patch_addressables_catalog_crc.py @@ -0,0 +1,370 @@ +#!/usr/bin/env python3 +"""Prepare an external binary Addressables catalog candidate with selected CRCs patched. + +Bundle records are located from each bundle's AssetBundle.m_Name. The source +catalog, catalog.hash, and bundleSize fields are never modified. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import struct +import sys +from pathlib import Path +from typing import Any + + +RECORD = struct.Struct("<5I") + + +def sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def print_json(value: Any) -> None: + # ASCII-escaped stdout stays one valid JSON document even on legacy + # Windows code pages. UTF-8 paths remain intact in the report file. + print(json.dumps(value, ensure_ascii=True)) + + +def all_offsets(data: bytes, needle: bytes) -> list[int]: + offsets: list[int] = [] + start = 0 + while True: + offset = data.find(needle, start) + if offset < 0: + return offsets + offsets.append(offset) + start = offset + 1 + + +def normalized(path: Path) -> str: + return os.path.normcase(str(path.resolve())) + + +def is_within(path: Path, root: Path) -> bool: + path_value = normalized(path) + root_value = normalized(root) + try: + return os.path.commonpath([path_value, root_value]) == root_value + except ValueError: + return False + + +def infer_game_root(catalog: Path) -> Path | None: + for parent in catalog.parents: + if parent.name.casefold().endswith("_data"): + return parent.parent.resolve() + return None + + +def parse_u32(value: str) -> int: + try: + parsed = int(value, 0) + except ValueError as exc: + raise argparse.ArgumentTypeError("CRC must be an integer such as 0 or 0x1234abcd") from exc + if not 0 <= parsed <= 0xFFFFFFFF: + raise argparse.ArgumentTypeError("CRC must fit an unsigned 32-bit integer") + return parsed + + +def list_entry_path(item: Any) -> str: + if isinstance(item, str): + return item + if isinstance(item, dict): + for key in ("target", "bundle", "path", "file"): + value = item.get(key) + if isinstance(value, str): + return value + raise ValueError(f"bundle-list entry has no path: {item!r}") + + +def load_bundle_list(path: Path) -> list[Path]: + base = path.resolve().parent + if path.suffix.casefold() == ".json": + raw = json.loads(path.read_text(encoding="utf-8-sig")) + if isinstance(raw, dict): + for key in ("files", "bundles", "targets", "entries"): + if key in raw: + raw = raw[key] + break + if not isinstance(raw, list): + raise ValueError("JSON bundle list must be an array or contain files/bundles/targets/entries") + values = [list_entry_path(item) for item in raw] + else: + values = [ + line.strip() + for line in path.read_text(encoding="utf-8-sig").splitlines() + if line.strip() and not line.lstrip().startswith("#") + ] + return [(Path(value) if Path(value).is_absolute() else base / value).resolve() for value in values] + + +def collect_bundles(args: argparse.Namespace) -> list[Path]: + paths = [Path(value).resolve() for value in args.bundle] + for list_name in args.bundle_list: + list_path = Path(list_name).resolve() + if not list_path.is_file(): + raise FileNotFoundError(f"bundle list not found: {list_path}") + paths.extend(load_bundle_list(list_path)) + for root_name in args.candidate_root: + root = Path(root_name).resolve() + if not root.is_dir(): + raise FileNotFoundError(f"candidate root not found: {root}") + paths.extend(sorted(path.resolve() for path in root.rglob("*.bundle") if path.is_file())) + + unique: dict[str, Path] = {} + for path in paths: + unique[normalized(path)] = path + if not unique: + raise ValueError("provide at least one --bundle, --bundle-list, or --candidate-root") + for path in unique.values(): + if not path.is_file(): + raise FileNotFoundError(f"bundle not found: {path}") + return list(unique.values()) + + +def read_internal_bundle_name(path: Path) -> str: + try: + import UnityPy # type: ignore + except ImportError as exc: + raise RuntimeError("UnityPy is required to read AssetBundle.m_Name") from exc + + env = UnityPy.load(str(path)) + names: list[str] = [] + for obj in env.objects: + if obj.type.name != "AssetBundle": + continue + value = getattr(obj.read(check_read=False), "m_Name", None) + if isinstance(value, str) and value: + names.append(value) + if len(names) != 1: + raise ValueError(f"{path}: expected one non-empty AssetBundle.m_Name, found {names!r}") + return names[0] + + +def catalog_key(internal_name: str) -> bytes: + value = internal_name[:-7] if internal_name.casefold().endswith(".bundle") else internal_name + try: + return value.encode("utf-8") + except UnicodeEncodeError as exc: + raise ValueError(f"internal bundle name is not valid UTF-8 text: {internal_name!r}") from exc + + +def locate_record(catalog: bytes, internal_name: str, external_size: int) -> dict[str, Any]: + key = catalog_key(internal_name) + string_offsets = all_offsets(catalog, key) + if len(string_offsets) != 1: + raise ValueError( + f"catalog key {key!r} for {internal_name!r} occurs {len(string_offsets)} times; expected 1" + ) + string_offset = string_offsets[0] + pointer_offsets = all_offsets(catalog, struct.pack(" len(catalog): + continue + values = RECORD.unpack_from(catalog, record_offset) + hash_id, bundle_name_id, _crc, bundle_size, common_id = values + if ( + bundle_name_id == string_offset + and 0 <= hash_id <= len(catalog) - 16 + and 0 <= common_id <= len(catalog) - 4 + and bundle_size > 0 + ): + candidates.append((record_offset, values)) + if len(candidates) != 1: + raise ValueError( + f"catalog key {internal_name!r} resolves to {len(candidates)} plausible records; expected 1" + ) + + record_offset, values = candidates[0] + if record_offset != string_offset + len(key): + raise ValueError( + f"catalog record for {internal_name!r} is not adjacent to its key; refusing format guess" + ) + hash_id, bundle_name_id, crc, bundle_size, common_id = values + return { + "catalog_key": key.decode("utf-8"), + "catalog_string_offset": string_offset, + "record_offset": record_offset, + "crc_offset": record_offset + 8, + "hash_id": hash_id, + "bundle_name_id": bundle_name_id, + "original_crc": crc, + "catalog_bundle_size": bundle_size, + "external_bundle_size": external_size, + "size_matches_catalog": external_size == bundle_size, + "common_id": common_id, + } + + +def expected_byte_changes(source: bytes, entries: list[dict[str, Any]], new_crc: int) -> set[int]: + expected: set[int] = set() + replacement = struct.pack(" None: + temporary = path.with_name(path.name + ".tmp") + if temporary.exists(): + temporary.unlink() + try: + temporary.write_bytes(data) + temporary.replace(path) + finally: + if temporary.exists(): + temporary.unlink() + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("catalog", help="untouched source binary catalog") + parser.add_argument("--bundle", action="append", default=[], help="candidate bundle (repeatable)") + parser.add_argument("--bundle-list", action="append", default=[], help="text or JSON bundle list") + parser.add_argument( + "--candidate-root", action="append", default=[], help="recursively include *.bundle files" + ) + parser.add_argument("--game-root", help="game root; inferred from a *_Data ancestor when omitted") + parser.add_argument("--out-dir", required=True, help="game-external output directory") + parser.add_argument("--new-crc", type=parse_u32, default=0, help="replacement uint32 CRC (default: 0)") + parser.add_argument("--write", action="store_true", help="write an external catalog candidate") + parser.add_argument("--force", action="store_true", help="replace existing report/candidate files") + args = parser.parse_args() + + try: + catalog_path = Path(args.catalog).resolve() + if not catalog_path.is_file(): + raise FileNotFoundError(f"catalog not found: {catalog_path}") + game_root = Path(args.game_root).resolve() if args.game_root else infer_game_root(catalog_path) + if game_root is None: + raise ValueError("cannot infer game root from catalog path; pass --game-root") + if not game_root.is_dir(): + raise FileNotFoundError(f"game root not found: {game_root}") + if not is_within(catalog_path, game_root): + raise ValueError(f"catalog is outside --game-root: {catalog_path}") + + out_dir = Path(args.out_dir).resolve() + if is_within(out_dir, game_root): + raise ValueError(f"--out-dir must be outside the game root: {game_root}") + + bundles = collect_bundles(args) + source = catalog_path.read_bytes() + entries: list[dict[str, Any]] = [] + seen_names: set[str] = set() + seen_crc_offsets: set[int] = set() + for bundle in bundles: + internal_name = read_internal_bundle_name(bundle) + folded = internal_name.casefold() + if folded in seen_names: + raise ValueError(f"duplicate AssetBundle.m_Name in input: {internal_name!r}") + seen_names.add(folded) + located = locate_record(source, internal_name, bundle.stat().st_size) + crc_offset = int(located["crc_offset"]) + if crc_offset in seen_crc_offsets: + raise ValueError(f"multiple input bundles resolve to CRC offset {crc_offset}") + seen_crc_offsets.add(crc_offset) + entries.append( + { + "external_file": str(bundle), + "external_sha256": sha256(bundle), + "internal_name": internal_name, + "new_crc": args.new_crc, + **located, + } + ) + + report_name = "catalog_crc_write_report.json" if args.write else "catalog_crc_dry_run.json" + report_path = out_dir / report_name + candidate_path = out_dir / catalog_path.name + outputs = [report_path] + ([candidate_path] if args.write else []) + conflicts = [str(path) for path in outputs if path.exists()] + if conflicts and not args.force: + raise FileExistsError("output already exists; pass --force to replace: " + ", ".join(conflicts)) + + hash_path = catalog_path.with_name("catalog.hash") + report: dict[str, Any] = { + "format": "addressables-binary-catalog-crc-patch-v1", + "mode": "write-candidate" if args.write else "dry-run", + "source_catalog": str(catalog_path), + "source_catalog_sha256": sha256(catalog_path), + "game_root": str(game_root), + "new_crc": args.new_crc, + "target_count": len(entries), + "entries": entries, + "catalog_hash": { + "path": str(hash_path), + "exists": hash_path.is_file(), + "modified": False, + "policy": "Never guessed or changed by this tool.", + }, + "bundle_size_modified": False, + } + + out_dir.mkdir(parents=True, exist_ok=True) + if args.write: + candidate = bytearray(source) + for entry in entries: + struct.pack_into(" None: + # Keep stdout machine-readable on legacy Windows code pages. `--out` + # retains the original Unicode in a UTF-8 report. + print(json.dumps(value, ensure_ascii=True, indent=2 if pretty else None)) + + +def normalized(path: Path) -> str: + return os.path.normcase(str(path.resolve())) + + +def dedupe_paths(paths: Iterable[Path]) -> list[Path]: + unique: dict[str, Path] = {} + for path in paths: + resolved = path.resolve() + unique[normalized(resolved)] = resolved + return list(unique.values()) + + +def collect_inputs(args: argparse.Namespace) -> tuple[list[Path], list[Path]]: + explicit = [Path(value).resolve() for value in args.asset] + for path in explicit: + if not path.is_file(): + raise FileNotFoundError(f"asset not found: {path}") + + rooted: list[Path] = [] + for value in args.candidate_root: + root = Path(value).resolve() + if not root.is_dir(): + raise FileNotFoundError(f"candidate root not found: {root}") + rooted.extend(sorted(path.resolve() for path in root.rglob("*.bundle") if path.is_file())) + + index_files = dedupe_paths([*explicit, *rooted]) + if not index_files: + raise ValueError("provide at least one --asset or --candidate-root") + source_files = dedupe_paths(explicit) if explicit else index_files + return source_files, index_files + + +def pptr(value: Any) -> dict[str, int] | None: + if isinstance(value, dict): + file_id = value.get("m_FileID") + path_id = value.get("m_PathID") + else: + file_id = getattr(value, "m_FileID", None) + path_id = getattr(value, "m_PathID", None) + if file_id is None or path_id is None: + return None + return {"file_id": int(file_id), "path_id": int(path_id)} + + +def pptrs_from_value(value: Any, field_path: str) -> list[dict[str, Any]]: + direct = pptr(value) + if direct is not None: + return [{"field_path": field_path, **direct}] + if isinstance(value, list): + found: list[dict[str, Any]] = [] + for index, item in enumerate(value): + found.extend(pptrs_from_value(item, f"{field_path}[{index}]")) + return found + return [] + + +def serialized_name(obj: Any) -> str: + return str(getattr(obj.assets_file, "name", "") or "") + + +def external_values(external: Any) -> tuple[str, str]: + if isinstance(external, dict): + name = str(external.get("name", "") or "") + path = str(external.get("path", "") or "") + else: + name = str(getattr(external, "name", "") or "") + path = str(getattr(external, "path", "") or "") + if not name and path: + name = path.rstrip("/\\").rsplit("/", 1)[-1].rsplit("\\", 1)[-1] + return name, path + + +def externals_for(obj: Any) -> list[dict[str, Any]]: + result: list[dict[str, Any]] = [] + for index, external in enumerate(getattr(obj.assets_file, "externals", []) or [], start=1): + name, path = external_values(external) + result.append({"file_id": index, "name": name, "path": path}) + return result + + +def object_name(tree: dict[str, Any], obj: Any, local_objects: dict[tuple[int, int], Any]) -> str: + own_name = tree.get("m_Name") + if isinstance(own_name, str) and own_name: + return own_name + game_object = pptr(tree.get("m_GameObject")) + if game_object and game_object["file_id"] == 0 and game_object["path_id"]: + target = local_objects.get((id(obj.assets_file), game_object["path_id"])) + if target is not None: + try: + name = getattr(target.read(check_read=False), "m_Name", "") + if isinstance(name, str) and name: + return name + except Exception: + pass + return f"{obj.type.name}_{obj.path_id}" + + +def matches_filters( + component: dict[str, Any], + path_ids: set[int], + text_needles: list[str], + name_needles: list[str], +) -> bool: + if path_ids and component["path_id"] not in path_ids: + return False + if text_needles and not any(needle in component["text"] for needle in text_needles): + return False + folded_name = component["object_name"].casefold() + if name_needles and not any(needle.casefold() in folded_name for needle in name_needles): + return False + return True + + +def resolve_pointer( + pointer: dict[str, Any], + source_serialized_file: str, + source_bundle: str, + externals: list[dict[str, Any]], + cab_index: dict[str, set[str]], + object_index: dict[tuple[str, int], list[dict[str, Any]]], +) -> dict[str, Any]: + file_id = int(pointer["file_id"]) + path_id = int(pointer["path_id"]) + external_name: str | None = None + external_path: str | None = None + error: str | None = None + if file_id == 0: + target_serialized_file = source_serialized_file + elif 1 <= file_id <= len(externals): + external = externals[file_id - 1] + external_name = external["name"] + external_path = external["path"] + target_serialized_file = external_name + else: + target_serialized_file = "" + error = f"FileID {file_id} is outside the {len(externals)}-entry external table" + + key = target_serialized_file.casefold() + locators = object_index.get((key, path_id), []) if key and path_id else [] + bundles = sorted({item["bundle"] for item in locators}) + if not bundles and key: + bundles = sorted(cab_index.get(key, set())) + if file_id == 0 and source_bundle not in bundles: + bundles.insert(0, source_bundle) + bundles = sorted(set(bundles)) + + return { + **pointer, + "external_index": file_id - 1 if file_id > 0 else None, + "external_name": external_name, + "external_path": external_path, + "target_serialized_file": target_serialized_file or None, + "actual_bundle": bundles[0] if len(bundles) == 1 else None, + "bundle_candidates": bundles, + "target_object_found": bool(locators), + "target_object_types": sorted({item["type"] for item in locators}), + "resolution_error": error or ("ambiguous CAB-to-bundle mapping" if len(bundles) > 1 else None), + } + + +def scan_bundle_payload(asset_path: Path, scan_components: bool) -> dict[str, Any]: + try: + import UnityPy # type: ignore + except ImportError as exc: + raise RuntimeError("UnityPy is required to trace TMP references") from exc + + asset_string = str(asset_path) + env = UnityPy.load(asset_string) + objects = list(env.objects) + if not objects: + raise ValueError(f"asset contains no Unity objects: {asset_path}") + local_objects = {(id(obj.assets_file), int(obj.path_id)): obj for obj in objects} + serialized_files = sorted({serialized_name(obj) for obj in objects if serialized_name(obj)}) + object_entries = [ + { + "bundle": asset_string, + "serialized_file": serialized_name(obj), + "path_id": int(obj.path_id), + "type": obj.type.name, + } + for obj in objects + ] + raw_components: list[dict[str, Any]] = [] + typetree_errors: Counter[str] = Counter() + if scan_components: + for obj in objects: + if obj.type.name != "MonoBehaviour": + continue + try: + tree = obj.read_typetree() + except Exception as exc: + typetree_errors[type(exc).__name__] += 1 + continue + if not isinstance(tree.get("m_text"), str) or "m_fontAsset" not in tree: + continue + materials: list[dict[str, Any]] = [] + for field in MATERIAL_FIELDS: + if field in tree: + materials.extend(pptrs_from_value(tree[field], field)) + raw_components.append( + { + "source_bundle": asset_string, + "serialized_file": serialized_name(obj), + "path_id": int(obj.path_id), + "component_type": obj.type.name, + "object_name": object_name(tree, obj, local_objects), + "text": tree["m_text"], + "font_asset": pptr(tree.get("m_fontAsset")), + "materials": [item for item in materials if item["path_id"] != 0], + "externals": externals_for(obj), + } + ) + payload = { + "file": { + "asset": asset_string, + "source_component_scan": scan_components, + "object_count": len(objects), + "serialized_files": serialized_files, + }, + "object_entries": object_entries, + "raw_components": raw_components, + "typetree_errors": dict(typetree_errors), + } + # Keep UnityPy/native decompression objects out of the returned payload. + obj = None + tree = None + local_objects.clear() + objects.clear() + del local_objects, objects, env + return payload + + +def isolated_component_scan(asset_path: Path) -> dict[str, Any]: + command = [sys.executable, str(Path(__file__).resolve()), "--_worker-scan", str(asset_path)] + completed = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=False) + stdout = completed.stdout.decode("utf-8", errors="replace").splitlines() + if completed.returncode != 0 or not stdout: + stderr = completed.stderr.decode("utf-8", errors="replace").strip() + raise RuntimeError( + f"isolated scan failed for {asset_path} (exit {completed.returncode}): {stderr}" + ) + try: + payload = json.loads(stdout[-1]) + except json.JSONDecodeError as exc: + raise RuntimeError(f"isolated scan returned invalid JSON for {asset_path}") from exc + if not isinstance(payload, dict) or "raw_components" not in payload: + raise RuntimeError(f"isolated scan returned an invalid payload for {asset_path}") + return payload + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--asset", + action="append", + default=[], + help="component asset/bundle to scan (repeatable); roots then index dependencies only", + ) + parser.add_argument( + "--candidate-root", + action="append", + default=[], + help="recursively index *.bundle files; also scan them when --asset is omitted", + ) + parser.add_argument("--path-id", action="append", type=int, default=[], help="component PathID") + parser.add_argument( + "--text-contains", action="append", default=[], help="keep components containing any value" + ) + parser.add_argument( + "--object-name-contains", + action="append", + default=[], + help="keep components whose GameObject/component name contains any value", + ) + parser.add_argument("--out", type=Path, help="optional UTF-8 JSON output path") + parser.add_argument("--_worker-scan", help=argparse.SUPPRESS) + args = parser.parse_args() + + try: + if args._worker_scan: + payload = scan_bundle_payload(Path(args._worker_scan).resolve(), True) + print(json.dumps(payload, ensure_ascii=True, separators=(",", ":"))) + return 0 + + source_files, index_files = collect_inputs(args) + source_keys = {normalized(path) for path in source_files} + cab_index: dict[str, set[str]] = defaultdict(set) + object_index: dict[tuple[str, int], list[dict[str, Any]]] = defaultdict(list) + raw_components: list[dict[str, Any]] = [] + files: list[dict[str, Any]] = [] + scan_errors: list[dict[str, Any]] = [] + typetree_error_counts: Counter[str] = Counter() + tmp_component_count = 0 + + isolate_sources = len(source_files) > 1 + for asset_path in index_files: + asset_string = str(asset_path) + is_source = normalized(asset_path) in source_keys + try: + payload = ( + isolated_component_scan(asset_path) + if is_source and isolate_sources + else scan_bundle_payload(asset_path, is_source) + ) + except Exception as exc: + error = {"asset": asset_string, "stage": "load", "error": str(exc)} + if is_source: + raise RuntimeError(f"failed to load source asset {asset_path}: {exc}") from exc + scan_errors.append(error) + continue + file_entry = payload["file"] + file_entry["source_component_scan"] = is_source + for name in file_entry["serialized_files"]: + cab_index[name.casefold()].add(asset_string) + for entry in payload["object_entries"]: + object_index[(entry["serialized_file"].casefold(), int(entry["path_id"]))].append( + { + "bundle": entry["bundle"], + "serialized_file": entry["serialized_file"], + "type": entry["type"], + } + ) + files.append(file_entry) + raw_components.extend(payload["raw_components"]) + tmp_component_count += len(payload["raw_components"]) + typetree_error_counts.update(payload["typetree_errors"]) + + selected = [ + item + for item in raw_components + if matches_filters( + item, + set(args.path_id), + list(args.text_contains), + list(args.object_name_contains), + ) + ] + components: list[dict[str, Any]] = [] + for item in selected: + externals = item.pop("externals") + font = item.pop("font_asset") + materials = item.pop("materials") + components.append( + { + **item, + "external_count": len(externals), + "font_asset": resolve_pointer( + font, + item["serialized_file"], + item["source_bundle"], + externals, + cab_index, + object_index, + ) + if font + else None, + "materials": [ + { + "field_path": pointer["field_path"], + **resolve_pointer( + pointer, + item["serialized_file"], + item["source_bundle"], + externals, + cab_index, + object_index, + ), + } + for pointer in materials + ], + } + ) + + usage: dict[tuple[str, int, tuple[str, ...]], dict[str, Any]] = {} + for component in components: + font = component["font_asset"] + if not font: + key = ("", 0, ()) + else: + key = ( + str(font["target_serialized_file"] or ""), + int(font["path_id"]), + tuple(font["bundle_candidates"]), + ) + if key not in usage: + usage[key] = { + "target_serialized_file": key[0], + "path_id": key[1], + "actual_bundle": key[2][0] if len(key[2]) == 1 else None, + "bundle_candidates": list(key[2]), + "component_count": 0, + } + usage[key]["component_count"] += 1 + + report = { + "format": "tmp-font-usage-trace-v1", + "mode": "read-only", + "source_assets": [str(path) for path in source_files], + "index_assets": [str(path) for path in index_files], + "filters": { + "path_ids": args.path_id, + "text_contains_any": args.text_contains, + "object_name_contains_any": args.object_name_contains, + "cross_category_semantics": "AND", + }, + "source_asset_count": len(source_files), + "index_asset_count": len(index_files), + "indexed_serialized_file_count": len(cab_index), + "tmp_component_count_before_filters": tmp_component_count, + "matched_component_count": len(components), + "font_usage": sorted( + usage.values(), + key=lambda item: (-item["component_count"], item["target_serialized_file"], item["path_id"]), + ), + "components": components, + "files": files, + "scan_errors": scan_errors, + "typetree_errors": dict(sorted(typetree_error_counts.items())), + } + if args.out: + out = args.out.resolve() + out.parent.mkdir(parents=True, exist_ok=True) + out.write_text(json.dumps(report, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") + print_json( + { + "ok": bool(components), + "report": str(out), + "matched_component_count": len(components), + "font_asset_count": len(usage), + } + ) + else: + print_json(report, pretty=True) + return 0 if components else 3 + except Exception as exc: + print_json({"ok": False, "error": str(exc)}) + return 2 + + +if __name__ == "__main__": + exit_code = main() + if "--_worker-scan" in sys.argv: + # A worker is a single-shot UnityPy process. Flush its JSON payload and + # bypass native finalizers that can crash after large TypeTree scans. + sys.stdout.flush() + sys.stderr.flush() + os._exit(exit_code) + raise SystemExit(exit_code) diff --git a/unity-text-locator/scripts/writeback_dmsl_text.py b/unity-text-locator/scripts/writeback_dmsl_text.py new file mode 100644 index 0000000..01ebb2e --- /dev/null +++ b/unity-text-locator/scripts/writeback_dmsl_text.py @@ -0,0 +1,663 @@ +#!/usr/bin/env python3 +"""Validate and stage fixed-width DMSL translations into Unity candidates. + +Dry-run is the default. The command performs a whole-batch preflight and writes +nothing when any row is too long, stale, duplicated, structurally invalid, or +no longer located at the manifest field path. Source game files are never +modified. ``--base-root`` composes onto an earlier external candidate tree. +""" +from __future__ import annotations + +import argparse +import csv +import json +import os +import re +import struct +import sys +import uuid +from collections import defaultdict +from pathlib import Path +from typing import Any, Sequence + +import UnityPy + +from extract_dmsl_text import ( + FORMAT, + normalize_byte_array, + object_name, + serialized_file_name, + sha256_bytes, + sha256_file, +) + + +SOURCE_HEADERS = ("original_flat", "source", "original") +TRANSLATION_HEADERS = ( + "zh_cn", "translated_flat", "translation", "translated_text", "translated", "text", "original_flat" +) +TOKEN_RE = re.compile( + r"<[^<>]+>|\{(?:\d+|[A-Za-z_][\w.]*)\}|" + r"%(?:\d+\$)?[-+#0 ]*\d*(?:\.\d+)?[diouxXeEfFgGcrsa%]" +) + + +def is_within(path: Path, root: Path) -> bool: + return path == root or root in path.parents + + +def safe_join(root: Path, relative: str) -> Path: + rel = Path(relative) + if rel.is_absolute() or not relative or any(part == ".." for part in rel.parts): + raise ValueError(f"unsafe manifest source_path: {relative!r}") + path = (root / rel).resolve() + if not is_within(path, root): + raise ValueError(f"manifest source_path escapes root: {relative!r}") + return path + + +def unflatten_text(value: str) -> str: + """Decode the extractor's backslash-controls-v1 representation.""" + output: list[str] = [] + index = 0 + escapes = {"n": "\n", "r": "\r", "t": "\t", "\\": "\\"} + while index < len(value): + char = value[index] + if char == "\\" and index + 1 < len(value) and value[index + 1] in escapes: + output.append(escapes[value[index + 1]]) + index += 2 + else: + output.append(char) + index += 1 + return "".join(output) + + +def read_one_column( + path: Path, + known_headers: Sequence[str], + *, + require_header: bool, +) -> tuple[str | None, list[str]]: + with path.open("r", encoding="utf-8-sig", newline="") as stream: + table = list(csv.reader(stream)) + if not table: + raise ValueError(f"CSV is empty: {path}") + # A physical blank row is the ordinary hand-edited form of one empty cell. + table = [[""] if not row else row for row in table] + widths = sorted({len(row) for row in table}) + if widths != [1]: + raise ValueError(f"CSV must contain exactly one column; observed row widths {widths}") + first = table[0][0] + if first in known_headers: + return first, [row[0] for row in table[1:]] + if require_header: + raise ValueError(f"CSV header must be one of {tuple(known_headers)!r}, got {first!r}") + return None, [row[0] for row in table] + + +def get_tree_path(tree: Any, field_path: Sequence[Any]) -> list[Any]: + value = tree + for segment in field_path: + if isinstance(segment, str) and isinstance(value, dict) and segment in value: + value = value[segment] + elif isinstance(segment, int) and isinstance(value, list) and 0 <= segment < len(value): + value = value[segment] + else: + raise ValueError(f"bytecode field path drifted at segment {segment!r}") + if not isinstance(value, list): + raise ValueError("bytecode field path no longer resolves to a list") + return value + + +def load_unique_object(env: Any, serialized_file: str, path_id: int, object_type: str) -> Any: + matches = [ + obj + for obj in env.objects + if serialized_file_name(obj) == serialized_file + and int(obj.path_id) == path_id + and obj.type.name == object_type + ] + if len(matches) != 1: + raise ValueError( + f"serialized_file={serialized_file!r}, path_id={path_id}, object_type={object_type!r} " + f"matched {len(matches)} objects" + ) + return matches[0] + + +def verify_object_name(tree: Any, path_id: int, expected_name: str) -> None: + if not isinstance(tree, dict): + raise ValueError("object TypeTree is not an object") + actual_name = object_name(tree, path_id) + if actual_name != expected_name: + raise ValueError(f"object name drift: expected={expected_name!r}, actual={actual_name!r}") + + +def inspect_operand(data: bytes, row: dict[str, Any]) -> tuple[bytes, str]: + offset = int(row["bytecode_offset"]) + width = int(row["byte_length"]) + if offset < 0 or width <= 0 or offset + 5 + width > len(data): + raise ValueError("operand is outside the bytecode field") + if data[offset] != 0x06: + raise ValueError(f"opcode drift: expected 0x06, got 0x{data[offset]:02x}") + actual_width = struct.unpack_from(" list[str]: + issues: list[str] = [] + original_tokens = TOKEN_RE.findall(original) + translated_tokens = TOKEN_RE.findall(translated) + if original_tokens != translated_tokens: + issues.append(f"tag/placeholder mismatch: {original_tokens!r} != {translated_tokens!r}") + if original.count("\n") != translated.count("\n"): + issues.append( + f"newline count mismatch: {original.count(chr(10))} != {translated.count(chr(10))}" + ) + return issues + + +def verify_environment( + env: Any, + expected: dict[tuple[str, int, str, str, int], tuple[list[Any], bytes, str]], +) -> None: + for (serialized_file, path_id, object_type, path_key, offset), ( + field_path, wanted, expected_name, + ) in expected.items(): + obj = load_unique_object(env, serialized_file, path_id, object_type) + tree = obj.read_typetree() + verify_object_name(tree, path_id, expected_name) + raw = get_tree_path(tree, field_path) + data = normalize_byte_array(raw) + if data[offset] != 0x06: + raise ValueError(f"reopen verification failed opcode at path_id {path_id}, field {path_key}") + width = struct.unpack_from(" None: + report["issues"].append(issue) + report["counts"]["blocking"] += 1 + + +def parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("game_root", type=Path, help="original game root used by manifest source_path values") + parser.add_argument("--manifest", type=Path, required=True) + parser.add_argument("--source-csv", type=Path, help="default: source_csv_name beside the manifest") + parser.add_argument("--translation-csv", type=Path, required=True) + parser.add_argument( + "--base-root", type=Path, + help="external candidate tree to compose onto; missing files fall back to original game files", + ) + parser.add_argument("--out-dir", type=Path, help="external candidate root; required with --write") + parser.add_argument("--report", type=Path, help="optional JSON report path") + parser.add_argument("--write", action="store_true", help="write candidates after a clean whole-batch preflight") + parser.add_argument("--overwrite", action="store_true", help="replace existing external candidate files") + return parser.parse_args(argv) + + +def main(argv: Sequence[str] | None = None) -> int: + args = parse_args(argv) + game_root = args.game_root.resolve() + if not game_root.is_dir(): + raise SystemExit(f"game_root is not a directory: {game_root}") + if args.write and args.out_dir is None: + raise SystemExit("--write requires --out-dir") + + out_root = args.out_dir.resolve() if args.out_dir else None + base_root = args.base_root.resolve() if args.base_root else None + if out_root and is_within(out_root, game_root): + raise SystemExit("--out-dir must be outside game_root") + if base_root: + if not base_root.is_dir(): + raise SystemExit(f"--base-root is not a directory: {base_root}") + if is_within(base_root, game_root): + raise SystemExit("--base-root must be outside game_root") + if out_root == base_root: + raise SystemExit("--out-dir must differ from --base-root") + + manifest_path = args.manifest.resolve() + try: + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + except Exception as exc: + raise SystemExit(f"cannot read manifest: {type(exc).__name__}: {exc}") from exc + if manifest.get("format") != FORMAT: + raise SystemExit(f"unsupported manifest format: {manifest.get('format')!r}") + if manifest.get("text_escape") != "backslash-controls-v1" or not manifest.get("fixed_width"): + raise SystemExit("manifest does not declare the supported fixed-width text encoding") + + source_csv = ( + args.source_csv.resolve() + if args.source_csv + else manifest_path.parent / str(manifest.get("source_csv_name", "")) + ) + report: dict[str, Any] = { + "mode": "write-candidates" if args.write else "dry-run", + "whole_batch_policy": "any preflight issue rejects every candidate write", + "fixed_width_policy": "UTF-8 bytes must fit; shorter payloads are right-padded with ASCII spaces", + "source_game_files_modified": False, + "write_performed": False, + "counts": { + "rows": 0, "planned": 0, "preserved_blank": 0, "unchanged": 0, + "blocking": 0, "candidate_files": 0, + }, + "bundles": [], + "issues": [], + "candidate_transaction": { + "status": "not_started", + "staged_files": 0, + "published_files": 0, + "rolled_back": False, + }, + } + + try: + expected_csv_hash = manifest.get("source_csv_sha256") + actual_csv_hash = sha256_file(source_csv) + if not expected_csv_hash or actual_csv_hash.lower() != str(expected_csv_hash).lower(): + raise ValueError( + f"source CSV SHA-256 mismatch: expected={expected_csv_hash}, actual={actual_csv_hash}" + ) + _, original_flat = read_one_column(source_csv, SOURCE_HEADERS, require_header=True) + translation_header, translated_flat = read_one_column( + args.translation_csv.resolve(), TRANSLATION_HEADERS, require_header=False + ) + except Exception as exc: + add_issue(report, scope="input", reason=str(exc)) + original_flat = [] + translated_flat = [] + translation_header = None + + rows = manifest.get("rows") + if not isinstance(rows, list): + add_issue(report, scope="manifest", reason="rows must be a list") + rows = [] + report["counts"]["rows"] = len(rows) + report["translation_column"] = translation_header or "headerless" + if len(rows) != len(original_flat) or len(rows) != len(translated_flat): + add_issue( + report, + scope="alignment", + reason=( + f"row count mismatch: manifest={len(rows)}, source_csv={len(original_flat)}, " + f"translation_csv={len(translated_flat)}" + ), + ) + if [row.get("row_index") for row in rows if isinstance(row, dict)] != list(range(len(rows))): + add_issue(report, scope="manifest", reason="row_index values must be contiguous and ordered") + + locations: set[tuple[Any, ...]] = set() + occurrence_ids: set[str] = set() + intervals: defaultdict[tuple[str, str, int, str, str], list[tuple[int, int, int]]] = defaultdict(list) + grouped: defaultdict[str, list[tuple[dict[str, Any], str, str]]] = defaultdict(list) + if len(rows) == len(original_flat) == len(translated_flat): + for row, original_value, translated_value in zip(rows, original_flat, translated_flat): + if not isinstance(row, dict): + add_issue(report, scope="manifest", reason="each row must be an object") + continue + try: + row_index = int(row["row_index"]) + source_path = str(row["source_path"]) + serialized_file = str(row["serialized_file"]) + path_id = int(row["path_id"]) + object_type = str(row["object_type"]) + expected_name = str(row["object_name"]) + field_path = row["bytecode_field_path"] + offset = int(row["bytecode_offset"]) + width = int(row["byte_length"]) + occurrence_id = str(row["occurrence_id"]) + if not serialized_file: + raise ValueError("serialized_file must be a non-empty string") + if not object_type: + raise ValueError("object_type must be a non-empty string") + if not expected_name: + raise ValueError("object_name must be a non-empty string") + if not isinstance(field_path, list) or not field_path: + raise ValueError("bytecode_field_path must be a non-empty list") + if any(not isinstance(part, (str, int)) or isinstance(part, bool) for part in field_path): + raise ValueError("bytecode_field_path contains an unsupported segment") + if offset < 0 or width <= 0: + raise ValueError("bytecode_offset and byte_length must be positive coordinates") + path_key = json.dumps(field_path, ensure_ascii=True, separators=(",", ":")) + location = (source_path, serialized_file, path_id, object_type, path_key, offset) + if location in locations: + raise ValueError("duplicate manifest location") + locations.add(location) + if not occurrence_id or occurrence_id in occurrence_ids: + raise ValueError("missing or duplicate occurrence_id") + occurrence_ids.add(occurrence_id) + interval_key = (source_path, serialized_file, path_id, object_type, path_key) + for existing_start, existing_end, existing_row in intervals[interval_key]: + if max(offset, existing_start) < min(offset + 5 + width, existing_end): + raise ValueError(f"operand overlaps row {existing_row}") + intervals[interval_key].append((offset, offset + 5 + width, row_index)) + + original = unflatten_text(original_value) + original_bytes = original.encode("utf-8", errors="strict") + if len(original_bytes) != width: + raise ValueError( + f"source CSV byte width {len(original_bytes)} != manifest byte_length {width}" + ) + expected_original_hash = str(row.get("original_utf8_sha256", "")) + if sha256_bytes(original_bytes) != expected_original_hash: + raise ValueError("source CSV text hash differs from manifest") + grouped[source_path].append((row, original, translated_value)) + except Exception as exc: + add_issue(report, row_index=row.get("row_index"), scope="manifest-row", reason=str(exc)) + + plans: list[dict[str, Any]] = [] + source_hashes = manifest.get("source_hashes_sha256", {}) + if not isinstance(source_hashes, dict): + add_issue(report, scope="manifest", reason="source_hashes_sha256 must be an object") + source_hashes = {} + + for source_path, work in sorted(grouped.items()): + bundle_report: dict[str, Any] = { + "source_path": source_path, + "base": None, + "base_sha256": None, + "planned": 0, + "candidate": None, + "serialized_reopen": False, + "disk_reopen": False, + } + report["bundles"].append(bundle_report) + try: + source = safe_join(game_root, source_path) + if not source.is_file(): + raise FileNotFoundError(f"original source missing: {source}") + expected_source_hash = source_hashes.get(source_path) + actual_source_hash = sha256_file(source) + if not expected_source_hash or actual_source_hash.lower() != str(expected_source_hash).lower(): + raise ValueError( + f"source SHA-256 mismatch: expected={expected_source_hash}, actual={actual_source_hash}" + ) + + base = source + if base_root: + candidate_base = safe_join(base_root, source_path) + if candidate_base.exists(): + if not candidate_base.is_file(): + raise ValueError(f"base candidate is not a file: {candidate_base}") + base = candidate_base + bundle_report["base"] = "candidate" if base != source else "original" + bundle_report["base_sha256"] = sha256_file(base) + env = UnityPy.load(str(base)) + expected_payloads: dict[ + tuple[str, int, str, str, int], tuple[list[Any], bytes, str] + ] = {} + object_work: defaultdict[ + tuple[str, int, str], list[tuple[dict[str, Any], str, str]] + ] = defaultdict(list) + for item in work: + row = item[0] + object_key = ( + str(row["serialized_file"]), int(row["path_id"]), str(row["object_type"]) + ) + object_work[object_key].append(item) + + for (serialized_file, path_id, object_type), object_rows in object_work.items(): + try: + obj = load_unique_object(env, serialized_file, path_id, object_type) + tree = obj.read_typetree() + expected_names = {str(item[0]["object_name"]) for item in object_rows} + if len(expected_names) != 1: + raise ValueError(f"manifest object_name values disagree: {sorted(expected_names)!r}") + expected_name = next(iter(expected_names)) + verify_object_name(tree, path_id, expected_name) + fields: dict[str, tuple[list[Any], bytearray]] = {} + changed = False + for row, original, translated_value in object_rows: + row_index = int(row["row_index"]) + field_path = row["bytecode_field_path"] + path_key = json.dumps(field_path, ensure_ascii=True, separators=(",", ":")) + try: + if path_key not in fields: + raw = get_tree_path(tree, field_path) + fields[path_key] = (raw, bytearray(normalize_byte_array(raw))) + raw, data = fields[path_key] + payload, actual_text = inspect_operand(bytes(data), row) + if actual_text != original: + raise ValueError("field text differs from source CSV (field drift or stale base)") + if sha256_bytes(payload) != str(row["original_utf8_sha256"]): + raise ValueError("field payload hash differs from manifest") + + if translated_value == "": + report["counts"]["preserved_blank"] += 1 + continue + translated = unflatten_text(translated_value) + structural = structural_issues(original, translated) + if structural: + raise ValueError("; ".join(structural)) + encoded = translated.encode("utf-8", errors="strict") + width = int(row["byte_length"]) + if len(encoded) > width: + add_issue( + report, + row_index=row_index, + occurrence_id=row["occurrence_id"], + scope="byte-budget", + reason="translation_too_long", + bytes=len(encoded), + limit=width, + over_by=len(encoded) - width, + translated_flat=translated_value, + ) + continue + padded = encoded + b" " * (width - len(encoded)) + if padded == payload: + report["counts"]["unchanged"] += 1 + continue + start = int(row["bytecode_offset"]) + 5 + data[start:start + width] = padded + expected_payloads[ + ( + serialized_file, path_id, object_type, path_key, + int(row["bytecode_offset"]), + ) + ] = ( + field_path, padded, expected_name + ) + report["counts"]["planned"] += 1 + bundle_report["planned"] += 1 + changed = True + except Exception as exc: + add_issue( + report, + row_index=row_index, + occurrence_id=row.get("occurrence_id"), + source_path=source_path, + scope="row-preflight", + reason=str(exc), + ) + if changed: + for raw, data in fields.values(): + raw[:] = list(data) + obj.save_typetree(tree) + except Exception as exc: + add_issue( + report, source_path=source_path, serialized_file=serialized_file, + path_id=path_id, object_type=object_type, + scope="object-preflight", reason=str(exc), + ) + + candidate = safe_join(out_root, source_path) if out_root else None + if candidate: + if candidate.resolve() == source.resolve() or candidate.resolve() == base.resolve(): + raise ValueError("candidate path resolves to an input file") + if candidate.exists() and not candidate.is_file(): + raise ValueError(f"candidate path exists but is not a file: {candidate}") + if candidate.exists() and not args.overwrite: + raise FileExistsError(f"candidate already exists; pass --overwrite: {candidate}") + bundle_report["candidate"] = str(candidate) + plans.append( + { + "source_path": source_path, + "env": env, + "expected": expected_payloads, + "candidate": candidate, + "report": bundle_report, + } + ) + except Exception as exc: + add_issue(report, source_path=source_path, scope="bundle-preflight", reason=str(exc)) + + serialized_plans: list[dict[str, Any]] = [] + if report["counts"]["blocking"] == 0: + for plan in plans: + if not plan["expected"]: + continue + try: + serialized = plan["env"].file.save() + reopened = UnityPy.load(serialized) + verify_environment(reopened, plan["expected"]) + plan["report"]["serialized_reopen"] = True + plan["serialized"] = serialized + serialized_plans.append(plan) + except Exception as exc: + add_issue( + report, source_path=plan["source_path"], + scope="serialized-reopen", reason=f"{type(exc).__name__}: {exc}", + ) + + if args.write and report["counts"]["blocking"] == 0: + transaction = report["candidate_transaction"] + staged_plans: list[dict[str, Any]] = [] + transaction["status"] = "staging" + + # Stage and reopen every candidate before publishing any of them. + try: + for plan in serialized_plans: + candidate: Path = plan["candidate"] + candidate.parent.mkdir(parents=True, exist_ok=True) + temporary = candidate.with_name(f".{candidate.name}.tmp-dmsl-{uuid.uuid4().hex}") + plan["temporary"] = temporary + plan["rollback"] = None + plan["published"] = False + temporary.write_bytes(plan["serialized"]) + reopened = UnityPy.load(temporary.read_bytes()) + verify_environment(reopened, plan["expected"]) + staged_plans.append(plan) + transaction["staged_files"] += 1 + except Exception as exc: + add_issue( + report, + source_path=plan.get("source_path") if "plan" in locals() else None, + scope="candidate-stage", + reason=f"{type(exc).__name__}: {exc}", + ) + transaction["status"] = "stage_failed" + + if report["counts"]["blocking"] == 0: + transaction["status"] = "publishing" + try: + for plan in staged_plans: + candidate = plan["candidate"] + if candidate.exists(): + if not candidate.is_file(): + raise ValueError(f"candidate path became a non-file: {candidate}") + if not args.overwrite: + raise FileExistsError(f"candidate appeared during write: {candidate}") + rollback = candidate.with_name( + f".{candidate.name}.rollback-dmsl-{uuid.uuid4().hex}" + ) + os.replace(candidate, rollback) + plan["rollback"] = rollback + os.replace(plan["temporary"], candidate) + plan["published"] = True + transaction["published_files"] += 1 + + # Read bytes before reopening so UnityPy cannot retain a Windows file handle. + for plan in staged_plans: + final_env = UnityPy.load(plan["candidate"].read_bytes()) + verify_environment(final_env, plan["expected"]) + + for plan in staged_plans: + plan["report"]["disk_reopen"] = True + report["counts"]["candidate_files"] = len(staged_plans) + report["write_performed"] = bool(staged_plans) + transaction["status"] = "committed" + except Exception as exc: + add_issue( + report, + source_path=plan.get("source_path") if "plan" in locals() else None, + scope="candidate-publish", + reason=f"{type(exc).__name__}: {exc}", + ) + rollback_errors: list[dict[str, str]] = [] + for rollback_plan in reversed(staged_plans): + candidate = rollback_plan["candidate"] + rollback = rollback_plan.get("rollback") + try: + if rollback is not None and rollback.exists(): + os.replace(rollback, candidate) + elif rollback_plan.get("published") and candidate.exists(): + candidate.unlink() + except Exception as rollback_exc: + rollback_errors.append( + { + "source_path": rollback_plan["source_path"], + "reason": f"{type(rollback_exc).__name__}: {rollback_exc}", + } + ) + rollback_plan["report"]["disk_reopen"] = False + report["counts"]["candidate_files"] = 0 + report["write_performed"] = False + transaction["rolled_back"] = not rollback_errors + transaction["status"] = "rolled_back" if not rollback_errors else "rollback_failed" + for rollback_error in rollback_errors: + add_issue(report, scope="candidate-rollback", **rollback_error) + + # A stage failure or a successful/failed publication can all leave unused temp files. + cleanup_errors: list[dict[str, str]] = [] + for cleanup_plan in serialized_plans: + temporary = cleanup_plan.get("temporary") + if temporary is not None and temporary.exists(): + try: + temporary.unlink() + except Exception as cleanup_exc: + cleanup_errors.append( + { + "source_path": cleanup_plan["source_path"], + "reason": f"temp cleanup failed: {type(cleanup_exc).__name__}: {cleanup_exc}", + } + ) + rollback = cleanup_plan.get("rollback") + if rollback is not None and rollback.exists() and transaction["status"] != "rollback_failed": + try: + rollback.unlink() + except Exception as cleanup_exc: + cleanup_errors.append( + { + "source_path": cleanup_plan["source_path"], + "reason": f"rollback cleanup failed: {type(cleanup_exc).__name__}: {cleanup_exc}", + } + ) + for cleanup_error in cleanup_errors: + add_issue(report, scope="candidate-cleanup", **cleanup_error) + + if report["counts"]["blocking"]: + report["result"] = "rejected" + report["write_performed"] = False if report["counts"]["candidate_files"] == 0 else report["write_performed"] + elif args.write: + report["result"] = "candidates_written" + else: + report["result"] = "dry_run_clean" + + rendered = json.dumps(report, ensure_ascii=False, indent=2) + "\n" + if args.report: + report_path = args.report.resolve() + report_path.parent.mkdir(parents=True, exist_ok=True) + report_path.write_text(rendered, encoding="utf-8") + print(rendered, end="") + return 1 if report["counts"]["blocking"] else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/unity-text-locator/scripts/writeback_typetree_ui_text.py b/unity-text-locator/scripts/writeback_typetree_ui_text.py new file mode 100644 index 0000000..459a14e --- /dev/null +++ b/unity-text-locator/scripts/writeback_typetree_ui_text.py @@ -0,0 +1,685 @@ +#!/usr/bin/env python3 +"""Validate or stage TypeTree UI translations as external Unity candidates. + +Dry-run is the default. ``--write`` uses a two-phase transaction: every source +bundle and field is preflighted first, then candidates are built in a temporary +root, reopened with UnityPy, and verified field by field before promotion. +Addressables catalogs are deliberately outside this tool's scope. +""" +from __future__ import annotations + +import argparse +import csv +import gc +import hashlib +import json +import os +import re +import shutil +import sys +import tempfile +from collections import Counter, defaultdict +from pathlib import Path +from typing import Any + +try: + import UnityPy +except ImportError as exc: # pragma: no cover - depends on the host environment + raise SystemExit("UnityPy is required: install it in the active Python environment") from exc + + +FORMAT = "unity-typetree-ui-text-v2" +TOKEN_RE = re.compile( + r"<[^<>]+>|\\[nrt]|\{[^{}\r\n]+\}|" + r"%(?:\d+\$)?[-+#0 ]*\d*(?:\.\d+)?[diouxXeEfFgGcrsa%]" +) +ALLOWED_CATEGORIES = {"component_text", "ui_label_field", "localization_string_table"} +TEXT_LEAVES = {"text", "mtext"} +VALUE_LEAVES = {"mvalue", "string", "value"} +LABEL_HINTS = ("label", "caption", "title", "description", "option", "tooltip") + + +class ValidationError(ValueError): + """A deterministic manifest, source, or translation validation failure.""" + + +def file_sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as stream: + for block in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(block) + return digest.hexdigest() + + +def text_sha256(value: str) -> str: + return hashlib.sha256(value.encode("utf-8")).hexdigest() + + +def flatten_text(value: str) -> str: + return value.replace("\r\n", "\\n").replace("\n", "\\n").replace("\r", "\\n") + + +def newline_events(value: str) -> list[str]: + events: list[str] = [] + index = 0 + while index < len(value): + if value.startswith("\r\n", index): + events.append("CRLF") + index += 2 + elif value[index] == "\n": + events.append("LF") + index += 1 + elif value[index] == "\r": + events.append("CR") + index += 1 + else: + index += 1 + return events + + +def source_markers(value: str) -> list[str]: + """Return real newline styles and literal backslash-n markers in order.""" + markers: list[str] = [] + index = 0 + while index < len(value): + if value.startswith("\r\n", index): + markers.append("\r\n") + index += 2 + elif value[index] == "\n": + markers.append("\n") + index += 1 + elif value[index] == "\r": + markers.append("\r") + index += 1 + elif value.startswith("\\n", index): + markers.append("\\n") + index += 2 + else: + index += 1 + return markers + + +def split_translation_markers(value: str) -> tuple[list[str], int]: + """Split translated CSV text at real newlines or literal backslash-n.""" + pieces: list[str] = [] + start = 0 + index = 0 + count = 0 + while index < len(value): + width = 0 + if value.startswith("\r\n", index) or value.startswith("\\n", index): + width = 2 + elif value[index] in "\r\n": + width = 1 + if width: + pieces.append(value[start:index]) + count += 1 + index += width + start = index + else: + index += 1 + pieces.append(value[start:]) + return pieces, count + + +def decode_translation(value: str, source: str) -> str: + """Restore the source field's exact newline/literal-marker conventions.""" + if value == flatten_text(source) or value == source: + return source + markers = source_markers(source) + pieces, marker_count = split_translation_markers(value) + if marker_count != len(markers): + raise ValidationError( + f"newline/\\n marker count mismatch: source={len(markers)}, translation={marker_count}" + ) + rebuilt = pieces[0] + for marker, piece in zip(markers, pieces[1:]): + rebuilt += marker + piece + return rebuilt + + +def read_one_column_csv(path: Path) -> tuple[str, list[str]]: + with path.open("r", encoding="utf-8-sig", newline="") as stream: + reader = csv.DictReader(stream) + if not reader.fieldnames or len(reader.fieldnames) != 1: + raise ValidationError(f"CSV must have exactly one column: {path}") + column = reader.fieldnames[0] + values: list[str] = [] + for row_number, row in enumerate(reader, 2): + if None in row: + raise ValidationError(f"CSV row {row_number} has extra columns: {path}") + values.append(row.get(column) or "") + return column, values + + +def inside(path: Path, parent: Path) -> bool: + return path == parent or parent in path.parents + + +def safe_relative_path(value: Any) -> Path: + if not isinstance(value, str) or not value: + raise ValidationError("bundle_path must be a non-empty string") + path = Path(value) + if path.is_absolute() or path.drive or any(part == ".." for part in path.parts): + raise ValidationError(f"unsafe bundle_path: {value!r}") + return path + + +def normalized_name(value: str | int) -> str: + return "" if isinstance(value, int) else value.lower().replace("_", "") + + +def validate_and_tuple_path(row: dict[str, Any]) -> tuple[str | int, ...]: + raw = row.get("field_path") + if not isinstance(raw, list) or not raw: + raise ValidationError("field_path must be a non-empty typed list") + path: list[str | int] = [] + for part in raw: + if isinstance(part, bool) or not isinstance(part, (str, int)): + raise ValidationError(f"invalid field_path segment: {part!r}") + if isinstance(part, int) and part < 0: + raise ValidationError(f"negative field_path index: {part}") + if isinstance(part, str) and not part: + raise ValidationError("empty field_path key") + path.append(part) + + category = row.get("category") + if category not in ALLOWED_CATEGORIES: + raise ValidationError(f"unsupported category: {category!r}") + names = [normalized_name(part) for part in path if isinstance(part, str)] + leaf = names[-1] if names else "" + is_localization = "mtabledata" in names and leaf == "mlocalized" + is_component = leaf in TEXT_LEAVES + is_label = any(hint in leaf for hint in LABEL_HINTS) or ( + leaf in VALUE_LEAVES + and any(any(hint in parent for hint in LABEL_HINTS) for parent in names[:-1]) + ) + if category == "localization_string_table" and not is_localization: + raise ValidationError("Localization row does not end in m_TableData.*.m_Localized") + if category == "component_text" and not is_component: + raise ValidationError("component_text row does not target m_text/text") + if category == "ui_label_field" and not is_label: + raise ValidationError("ui_label_field row does not target an allowed UI label field") + return tuple(path) + + +def get_field(tree: Any, path: tuple[str | int, ...]) -> Any: + value = tree + for part in path: + if isinstance(part, int): + if not isinstance(value, list) or part >= len(value): + raise ValidationError(f"field path list index unavailable: {part}") + value = value[part] + else: + if not isinstance(value, dict) or part not in value: + raise ValidationError(f"field path key unavailable: {part!r}") + value = value[part] + return value + + +def set_field(tree: Any, path: tuple[str | int, ...], replacement: str) -> None: + parent = tree + for part in path[:-1]: + parent = parent[part] + parent[path[-1]] = replacement + + +def serialized_file_name(obj: Any) -> str: + assets_file = getattr(obj, "assets_file", None) + return str(getattr(assets_file, "name", None) or "") + + +def object_name(tree: dict[str, Any], obj: Any) -> str: + for key in ("m_Name", "name", "Name"): + value = tree.get(key) + if isinstance(value, str) and value: + return value + return f"{obj.type.name}_{obj.path_id}" + + +def object_index(env: Any) -> dict[tuple[str, int], list[Any]]: + result: dict[tuple[str, int], list[Any]] = defaultdict(list) + for obj in env.objects: + result[(serialized_file_name(obj), int(obj.path_id))].append(obj) + return result + + +def unique_object(index: dict[tuple[str, int], list[Any]], row: dict[str, Any]) -> Any: + asset_file = row.get("serialized_file") + if not isinstance(asset_file, str) or not asset_file: + raise ValidationError("serialized_file missing from manifest row") + try: + path_id = int(row.get("path_id")) + except (TypeError, ValueError) as exc: + raise ValidationError(f"invalid PathID: {row.get('path_id')!r}") from exc + matches = index.get((asset_file, path_id), []) + if not matches: + raise ValidationError(f"object missing: serialized_file={asset_file!r}, PathID={path_id}") + if len(matches) != 1: + raise ValidationError( + f"object is ambiguous: serialized_file={asset_file!r}, PathID={path_id}, matches={len(matches)}" + ) + return matches[0] + + +def validate_manifest_rows(manifest: dict[str, Any], source_values: list[str], translations: list[str]) -> list[dict[str, Any]]: + if manifest.get("format") != FORMAT: + raise ValidationError(f"unsupported manifest format: {manifest.get('format')!r}") + rows = manifest.get("rows") + if not isinstance(rows, list): + raise ValidationError("manifest rows must be a list") + if not (len(rows) == len(source_values) == len(translations)): + raise ValidationError( + f"row count mismatch: manifest={len(rows)}, source_csv={len(source_values)}, " + f"translation_csv={len(translations)}" + ) + if [row.get("row_index") for row in rows if isinstance(row, dict)] != list(range(len(rows))): + raise ValidationError("manifest row_index values are not contiguous and ordered") + + seen_locations: set[tuple[str, str, int, str]] = set() + occurrence_groups: dict[str, list[tuple[int, int]]] = defaultdict(list) + prepared: list[dict[str, Any]] = [] + for index, (row, source_flat, translation_flat) in enumerate(zip(rows, source_values, translations)): + if not isinstance(row, dict): + raise ValidationError(f"manifest row {index} is not an object") + original = row.get("original") + if not isinstance(original, str): + raise ValidationError(f"row {index}: original must be a string") + if flatten_text(original) != source_flat: + raise ValidationError(f"row {index}: source CSV differs from manifest original") + if row.get("source_hash_sha256") != text_sha256(original): + raise ValidationError(f"row {index}: source field SHA-256 mismatch") + expected_tokens = {"ordered": TOKEN_RE.findall(original), "count": len(TOKEN_RE.findall(original))} + if row.get("tokens") != expected_tokens: + raise ValidationError(f"row {index}: manifest token metadata mismatch") + events = newline_events(original) + if row.get("newlines") != {"count": len(events), "ordered": events}: + raise ValidationError(f"row {index}: manifest newline metadata mismatch") + + field_path = validate_and_tuple_path(row) + relative = safe_relative_path(row.get("bundle_path")) + try: + path_id = int(row.get("path_id")) + occurrence = int(row.get("occurrence")) + occurrence_count = int(row.get("occurrence_count")) + except (TypeError, ValueError) as exc: + raise ValidationError(f"row {index}: invalid PathID or occurrence metadata") from exc + location = ( + relative.as_posix(), + str(row.get("serialized_file")), + path_id, + json.dumps(field_path, ensure_ascii=False, separators=(",", ":")), + ) + if location in seen_locations: + raise ValidationError(f"row {index}: duplicate object field location in manifest") + seen_locations.add(location) + occurrence_groups[original].append((occurrence, occurrence_count)) + + if translation_flat: + translated = decode_translation(translation_flat, original) + translated_tokens = TOKEN_RE.findall(translated) + if translated_tokens != expected_tokens["ordered"]: + raise ValidationError( + f"row {index}: token sequence mismatch: source={expected_tokens['ordered']!r}, " + f"translation={translated_tokens!r}" + ) + if newline_events(translated) != events: + raise ValidationError(f"row {index}: newline style/count mismatch after decoding") + else: + translated = original + + prepared.append( + { + "row": row, + "relative": relative, + "field_path": field_path, + "translated": translated, + "changed": bool(translation_flat) and translated != original, + "blank": not translation_flat, + } + ) + + for original, occurrences in occurrence_groups.items(): + totals = {total for _, total in occurrences} + expected_total = len(occurrences) + if totals != {expected_total} or sorted(number for number, _ in occurrences) != list( + range(1, expected_total + 1) + ): + raise ValidationError( + f"occurrence metadata is inconsistent for source SHA-256 {text_sha256(original)}" + ) + return prepared + + +def manifest_bundle_hashes(manifest: dict[str, Any]) -> dict[str, str]: + source = manifest.get("source") + hashes = source.get("bundle_hashes_sha256") if isinstance(source, dict) else None + if not isinstance(hashes, dict): + raise ValidationError("manifest source.bundle_hashes_sha256 is missing") + return hashes + + +def validate_source_csv_identity(manifest: dict[str, Any], source_csv: Path) -> tuple[str, str]: + name = manifest.get("source_csv_name") + expected_hash = manifest.get("source_csv_sha256") + if not isinstance(name, str) or not name: + raise ValidationError("manifest source_csv_name is missing") + if not isinstance(expected_hash, str) or not re.fullmatch(r"[0-9a-fA-F]{64}", expected_hash): + raise ValidationError("manifest source_csv_sha256 is missing or invalid") + actual_hash = file_sha256(source_csv) + if actual_hash.lower() != expected_hash.lower(): + raise ValidationError( + f"source CSV SHA-256 mismatch: expected={expected_hash}, actual={actual_hash}" + ) + return name, actual_hash + + +def resolve_bundle(root: Path, relative: Path, label: str) -> Path: + candidate = (root / relative).resolve() + if not inside(candidate, root): + raise ValidationError(f"{label} bundle escapes its root: {relative.as_posix()}") + return candidate + + +def verify_bundle_fields(env: Any, items: list[dict[str, Any]], *, expected_translated: bool) -> None: + index = object_index(env) + tree_cache: dict[tuple[str, int], dict[str, Any]] = {} + for item in items: + row = item["row"] + obj = unique_object(index, row) + key = (serialized_file_name(obj), int(obj.path_id)) + if key not in tree_cache: + tree = obj.read_typetree() + if not isinstance(tree, dict): + raise ValidationError(f"object TypeTree is not a dictionary: {key!r}") + if obj.type.name != row.get("object_type"): + raise ValidationError(f"object type drift at {key!r}") + if object_name(tree, obj) != row.get("object_name"): + raise ValidationError(f"object name drift at {key!r}") + tree_cache[key] = tree + expected = item["translated"] if expected_translated else row["original"] + current = get_field(tree_cache[key], item["field_path"]) + if not isinstance(current, str) or current != expected: + phase = "candidate" if expected_translated else "source" + raise ValidationError(f"row {row['row_index']}: {phase} field differs from expected text") + + +def apply_bundle_fields(env: Any, items: list[dict[str, Any]]) -> None: + index = object_index(env) + grouped: dict[tuple[str, int], list[dict[str, Any]]] = defaultdict(list) + objects: dict[tuple[str, int], Any] = {} + for item in items: + obj = unique_object(index, item["row"]) + key = (serialized_file_name(obj), int(obj.path_id)) + objects[key] = obj + grouped[key].append(item) + for key, object_items in grouped.items(): + obj = objects[key] + tree = obj.read_typetree() + if not isinstance(tree, dict): + raise ValidationError(f"object TypeTree is not a dictionary: {key!r}") + changed = False + for item in object_items: + row = item["row"] + current = get_field(tree, item["field_path"]) + if current != row["original"]: + raise ValidationError(f"row {row['row_index']}: source changed between preflight and write") + if item["changed"]: + set_field(tree, item["field_path"], item["translated"]) + changed = True + if changed: + obj.save_typetree(tree) + + +def write_report(report: dict[str, Any], path: Path | None) -> None: + rendered = json.dumps(report, ensure_ascii=False, indent=2) + "\n" + if path is not None: + path = path.resolve() + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(rendered, encoding="utf-8") + print(rendered, end="") + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("game_root", type=Path, help="original Unity game root (never modified)") + parser.add_argument("--manifest", type=Path, required=True, help="v2 TypeTree UI manifest") + parser.add_argument("--source-csv", type=Path, required=True, help="original one-column extraction CSV") + parser.add_argument("--translation-csv", type=Path, required=True, help="matching one-column translation CSV") + parser.add_argument( + "--base-root", + type=Path, + help="external prior-candidate root; matching bundles are used as the write base", + ) + parser.add_argument( + "--out-root", + type=Path, + help="external candidate root (default: sibling _ui_candidate)", + ) + parser.add_argument("--report", type=Path, help="optional JSON report path") + parser.add_argument("--write", action="store_true", help="write verified external candidates") + return parser.parse_args() + + +def main() -> int: + args = parse_args() + game_root = args.game_root.resolve() + if not game_root.is_dir(): + raise SystemExit(f"game_root is not a directory: {game_root}") + base_root = args.base_root.resolve() if args.base_root else None + if base_root is not None: + if inside(base_root, game_root): + raise SystemExit("--base-root must be outside game_root") + if not base_root.is_dir(): + raise SystemExit(f"--base-root is not a directory: {base_root}") + out_root = ( + args.out_root.resolve() + if args.out_root + else game_root.parent / f"{game_root.name}_ui_candidate" + ) + out_root = out_root.resolve() + if inside(out_root, game_root): + raise SystemExit("candidate --out-root must be outside game_root") + + report: dict[str, Any] = { + "format": "unity-typetree-ui-writeback-report-v2", + "mode": "write-external-candidates" if args.write else "dry-run", + "source_game_modified": False, + "candidate_root": str(out_root) if args.write else None, + "base_root": str(base_root) if base_root else None, + "catalog_crc_policy": "not modified; apply bundle and matching external catalog candidate atomically", + "counts": {"rows": 0, "changed": 0, "blank_preserved": 0, "unchanged": 0, "bundles": 0, "failed": 0}, + "bundles": [], + "issues": [], + } + + try: + manifest = json.loads(args.manifest.read_text(encoding="utf-8")) + source_csv_name, source_csv_hash = validate_source_csv_identity(manifest, args.source_csv) + source_column, source_values = read_one_column_csv(args.source_csv) + translation_column, translations = read_one_column_csv(args.translation_csv) + prepared = validate_manifest_rows(manifest, source_values, translations) + bundle_hashes = manifest_bundle_hashes(manifest) + report["source_column"] = source_column + report["source_csv_name"] = source_csv_name + report["source_csv_sha256"] = source_csv_hash + report["translation_column"] = translation_column + report["counts"]["rows"] = len(prepared) + report["counts"]["changed"] = sum(item["changed"] for item in prepared) + report["counts"]["blank_preserved"] = sum(item["blank"] for item in prepared) + report["counts"]["unchanged"] = sum( + not item["changed"] and not item["blank"] for item in prepared + ) + except Exception as exc: + report["counts"]["failed"] = 1 + report["issues"].append({"phase": "manifest-and-csv-preflight", "reason": str(exc)}) + write_report(report, args.report) + return 1 + + by_bundle: dict[str, list[dict[str, Any]]] = defaultdict(list) + for item in prepared: + by_bundle[item["relative"].as_posix()].append(item) + report["counts"]["bundles"] = len(by_bundle) + + # Phase 1: read-only validation of every bundle and every tracked field. + preflight: dict[str, dict[str, Any]] = {} + for relative_text, items in sorted(by_bundle.items()): + relative = items[0]["relative"] + entry: dict[str, Any] = { + "bundle_path": relative_text, + "base": None, + "base_sha256": None, + "changed_fields": sum(item["changed"] for item in items), + "candidate": None, + "candidate_sha256": None, + "staged_sha256": None, + "staged_reopen": None, + "final_destination_reopen": None, + "all_fields_verified": None, + } + report["bundles"].append(entry) + try: + original_path = resolve_bundle(game_root, relative, "game") + if not original_path.is_file(): + raise ValidationError(f"original bundle is missing: {original_path}") + expected_hash = bundle_hashes.get(relative_text) + if not isinstance(expected_hash, str) or not expected_hash: + raise ValidationError("bundle SHA-256 missing from manifest") + actual_original_hash = file_sha256(original_path) + if actual_original_hash.lower() != expected_hash.lower(): + raise ValidationError( + f"original bundle SHA-256 mismatch: expected={expected_hash}, actual={actual_original_hash}" + ) + + selected = original_path + if base_root is not None: + layered = resolve_bundle(base_root, relative, "base") + if layered.is_file(): + selected = layered + selected_bytes = selected.read_bytes() + selected_hash = hashlib.sha256(selected_bytes).hexdigest() + env = UnityPy.load(selected_bytes) + verify_bundle_fields(env, items, expected_translated=False) + entry["base"] = str(selected) + entry["base_sha256"] = selected_hash + preflight[relative_text] = {"path": selected, "sha256": selected_hash, "entry": entry} + except Exception as exc: + report["counts"]["failed"] += 1 + report["issues"].append( + {"phase": "bundle-preflight", "bundle_path": relative_text, "reason": str(exc)} + ) + + if report["counts"]["failed"] or not args.write: + write_report(report, args.report) + return 1 if report["counts"]["failed"] else 0 + + changed_bundles = [key for key, items in by_bundle.items() if any(item["changed"] for item in items)] + if not changed_bundles: + write_report(report, args.report) + return 0 + + # Phase 2: build and reopen all candidates in an external temporary root. + out_root.parent.mkdir(parents=True, exist_ok=True) + scratch = Path(tempfile.mkdtemp(prefix=".unity-ui-candidate-", dir=str(out_root.parent))).resolve() + staged_root = scratch / "staged" + backups_root = scratch / "backups" + staged: dict[str, Path] = {} + try: + for relative_text in sorted(changed_bundles): + items = by_bundle[relative_text] + state = preflight[relative_text] + selected: Path = state["path"] + selected_bytes = selected.read_bytes() + if hashlib.sha256(selected_bytes).hexdigest() != state["sha256"]: + raise ValidationError(f"base bundle changed after preflight: {relative_text}") + env = UnityPy.load(selected_bytes) + verify_bundle_fields(env, items, expected_translated=False) + apply_bundle_fields(env, items) + staged_path = resolve_bundle(staged_root, items[0]["relative"], "staging") + staged_path.parent.mkdir(parents=True, exist_ok=True) + candidate_bytes = env.file.save() + staged_path.write_bytes(candidate_bytes) + + reopened = UnityPy.load(candidate_bytes) + verify_bundle_fields(reopened, items, expected_translated=True) + entry = state["entry"] + entry["staged_reopen"] = True + entry["staged_sha256"] = file_sha256(staged_path) + staged[relative_text] = staged_path + del reopened + del env + + # UnityPy can retain a Windows file handle until its environments are + # collected. Release them before atomically promoting staged bundles. + gc.collect() + + # Promotion has rollback for pre-existing external candidates. No path + # in game_root is ever a source or destination of these moves. + promoted: list[tuple[str, Path, Path | None]] = [] + try: + for ordinal, relative_text in enumerate(sorted(staged)): + relative = by_bundle[relative_text][0]["relative"] + destination = resolve_bundle(out_root, relative, "candidate") + if inside(destination, game_root): + raise ValidationError("candidate path resolved inside game_root") + destination.parent.mkdir(parents=True, exist_ok=True) + backup: Path | None = None + if destination.exists(): + if not destination.is_file(): + raise ValidationError(f"candidate destination is not a file: {destination}") + backup = backups_root / f"{ordinal:06d}.bak" + backup.parent.mkdir(parents=True, exist_ok=True) + os.replace(destination, backup) + try: + os.replace(staged[relative_text], destination) + except Exception: + if backup is not None and backup.exists(): + os.replace(backup, destination) + raise + promoted.append((relative_text, destination, backup)) + + # The transaction is not complete until the exact bytes at every + # final destination reopen and every tracked field verifies. A + # failure here rolls back all destinations promoted above. + for relative_text, destination, _ in promoted: + destination_bytes = destination.read_bytes() + final_env = UnityPy.load(destination_bytes) + verify_bundle_fields(final_env, by_bundle[relative_text], expected_translated=True) + final_hash = hashlib.sha256(destination_bytes).hexdigest() + entry = preflight[relative_text]["entry"] + if final_hash != entry["staged_sha256"]: + raise ValidationError(f"final destination bytes differ from staged candidate: {relative_text}") + entry["final_destination_reopen"] = True + entry["all_fields_verified"] = True + entry["candidate_sha256"] = final_hash + del final_env + + for relative_text, destination, _ in promoted: + preflight[relative_text]["entry"]["candidate"] = str(destination) + except Exception: + for relative_text, destination, backup in reversed(promoted): + if destination.exists(): + destination.unlink() + if backup is not None and backup.exists(): + os.replace(backup, destination) + entry = preflight[relative_text]["entry"] + entry["candidate"] = None + entry["candidate_sha256"] = None + entry["final_destination_reopen"] = False + entry["all_fields_verified"] = False + raise + except Exception as exc: + report["counts"]["failed"] += 1 + report["issues"].append({"phase": "candidate-build-or-promotion", "reason": str(exc)}) + finally: + shutil.rmtree(scratch, ignore_errors=True) + + write_report(report, args.report) + return 1 if report["counts"]["failed"] else 0 + + +if __name__ == "__main__": + exit_code = main() + sys.stdout.flush() + sys.stderr.flush() + os._exit(exit_code)