Fix merged TOC visual lines and add standalone offline deployment packages

[Host003]

Summary

Right-align TOC page numbers and redraw dot leaders in translation

• Detect TOC entries with both dot leaders (....15) and whitespace leaders (spaces + 11), including Roman-numeral page numbers

• Store toc_page_number / toc_page_number_x2 / toc_dot_leader on PdfParagraph; mark single-line entry paragraphs as well as split multi-line ones

• Typesetting: pin translated page numbers to the original right-edge column, absorb leader gaps, redraw dot leaders when the source used them; fall back to normal flow when the title wraps or is too long

• Add split_toc_lines config (default on) with --no-split-toc-lines CLI flag

---

Summary by cubic

Adds standalone offline deployment packages and fixes table-of-contents (TOC) translation layout.

Bug Fixes

• Detects TOC entries with dot or whitespace leaders, including Roman-numeral page numbers, and splits merged multi-line entries into per-entry paragraphs.
• Typesetting pins translated page numbers to the original right-edge column, redraws dot leaders, and restores missing page numbers when the model omits them; falls back to normal flow when the title wraps or is too long.
• Adds split_toc_lines config (default on) with a --no-split-toc-lines CLI flag.

New Features

• Adds scripts/build_offline_packages.ps1 producing fully-offline Windows x64 and Linux x86_64 bundles (portable Python, all dependencies, assets) for air-gapped intranets, documented in the README.

^{Written for commit 8784908. Summary will update on new commits.}

[cubic-dev-ai]

9 issues found across 8 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="babeldoc/format/pdf/translation_config.py">

<violation number="1" location="babeldoc/format/pdf/translation_config.py:208">
P3: `split_toc_lines` was inserted in the middle of the signature, before `skip_translation` and 11 other existing parameters, contrary to the convention comment directly above: "new parameters should be added at the end of the function". Positional callers of `TranslationConfig` (notably external library users) now have every argument from `skip_translation` onward shifted by one, silently changing which option they set. Move the parameter to the end, after `disable_same_text_fallback`.</violation>
</file>

<file name="babeldoc/format/pdf/document_il/il_version_1.py">

<violation number="1" location="babeldoc/format/pdf/document_il/il_version_1.py:1242">
P2: The new pdfParagraph attributes are added only to il_version_1.py and not to the IL schema source. Per docs/CONTRIBUTING.md, il_version_1.rnc is the source of truth and il_version_1.rng/.xsd/.py are auto-generated and must not be manually modified. This PR leaves the RNG/XSD/RNC without tocPageNumber/tocPageNumberX2/tocDotLeader, so the schema no longer matches the emitted documents (any schema-validating consumer would reject them), and regenerating the classes from the XSD with xsdata would silently drop these fields and break the TOC feature. Add the three attributes to the PDFParagraph definition in il_version_1.rnc, then regenerate il_version_1.rng, il_version_1.xsd, and il_version_1.py per the documented workflow.</violation>
</file>

<file name="babeldoc/format/pdf/document_il/midend/typesetting.py">

<violation number="1" location="babeldoc/format/pdf/document_il/midend/typesetting.py:1359">
P2: The Roman-page branch cannot recognize Arabic translations because its scan excludes digits, making `joined.isdigit()` unreachable. A source `iii` translated as `3` therefore falls back to appending `iii` while retaining `3`, producing a duplicate page number; accept and compare both numeral forms before appending a fallback.</violation>

<violation number="2" location="babeldoc/format/pdf/document_il/midend/typesetting.py:1542">
P2: This skips TOC gap units before pinning succeeds, so the documented normal-flow fallback cannot restore them. Gate this `continue` on `toc_can_pin` (or otherwise preserve the units) so oversized or wrapped entries retain their spacing and leaders.</violation>
</file>

<file name="scripts/build_offline_packages.ps1">

<violation number="1" location="scripts/build_offline_packages.ps1:89">
P3: A relative `-OutputDir` breaks later steps: the assets-resolution and win64-verify steps `Push-Location $env:TEMP` and then invoke `$tagPy`, which is relative (`...\win64\python\python.exe`) when `$OutputDir` was passed relative, so Python is looked up under TEMP and the runs fail (and `$genDir` would be created under TEMP). The default (joined from `$RootDir`) is absolute and works.</violation>

<violation number="2" location="scripts/build_offline_packages.ps1:192">
P3: `-Platform linux-x64` can never run: the assets-resolution step always requires the win64 staged interpreter, so a Linux-only build throws here. The parameter docs and README ("-Platform win64" shown as the Windows-only example; Docker "only required when building the Linux package") advertise Linux-only builds, but the script also silently requires a full win64 stage (uv + Windows) on top of Docker.</violation>

<violation number="3" location="scripts/build_offline_packages.ps1:261">
P2: The Windows launchers call `python\Scripts\babeldoc.exe`, but pip's Windows console-script stubs embed the build machine's absolute interpreter path, so they stop working once the folder is relocated — which README explicitly promises ("the folder is relocatable afterwards"). This is the exact problem the script documents and works around for Linux in `install.sh` ("pip-generated console scripts carry a build-time absolute shebang, so the Linux wrapper invokes `python -m babeldoc.main`") — the same fix should be applied on Windows.</violation>

<violation number="4" location="scripts/build_offline_packages.ps1:352">
P2: The Linux verifier hardcodes `python3.13` even though `-PythonVersion` accepts other supported versions, so non-3.13 builds fail unless `-SkipVerify` is used. Verify through the selected interpreter instead of hardcoding the site-packages path.</violation>
</file>

<file name="babeldoc/format/pdf/document_il/midend/paragraph_finder.py">

<violation number="1" location="babeldoc/format/pdf/document_il/midend/paragraph_finder.py:1017">
P2: This treats periods in chapter numbering or title text as dot leaders. Such lines get `toc_dot_leader=True`, causing typesetting to insert spurious dotted leaders; count leader characters only in the title/page-number gap (`ordered[k:j]`).</violation>
</file>

<sub>Reply to a comment to ask cubic a question or push back. It learns from your replies.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: The new pdfParagraph attributes are added only to il_version_1.py and not to the IL schema source. Per docs/CONTRIBUTING.md, il_version_1.rnc is the source of truth and il_version_1.rng/.xsd/.py are auto-generated and must not be manually modified. This PR leaves the RNG/XSD/RNC without tocPageNumber/tocPageNumberX2/tocDotLeader, so the schema no longer matches the emitted documents (any schema-validating consumer would reject them), and regenerating the classes from the XSD with xsdata would silently drop these fields and break the TOC feature. Add the three attributes to the PDFParagraph definition in il_version_1.rnc, then regenerate il_version_1.rng, il_version_1.xsd, and il_version_1.py per the documented workflow.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At babeldoc/format/pdf/document_il/il_version_1.py, line 1242:

<comment>The new pdfParagraph attributes are added only to il_version_1.py and not to the IL schema source. Per docs/CONTRIBUTING.md, il_version_1.rnc is the source of truth and il_version_1.rng/.xsd/.py are auto-generated and must not be manually modified. This PR leaves the RNG/XSD/RNC without tocPageNumber/tocPageNumberX2/tocDotLeader, so the schema no longer matches the emitted documents (any schema-validating consumer would reject them), and regenerating the classes from the XSD with xsdata would silently drop these fields and break the TOC feature. Add the three attributes to the PDFParagraph definition in il_version_1.rnc, then regenerate il_version_1.rng, il_version_1.xsd, and il_version_1.py per the documented workflow.</comment>

<file context>
@@ -1239,6 +1239,27 @@ class Meta:
             "type": "Attribute",
         },
     )
+    toc_page_number: str | None = field(
+        default=None,
+        metadata={
</file context>

[cubic-dev-ai]

P2: This skips TOC gap units before pinning succeeds, so the documented normal-flow fallback cannot restore them. Gate this continue on toc_can_pin (or otherwise preserve the units) so oversized or wrapped entries retain their spacing and leaders.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At babeldoc/format/pdf/document_il/midend/typesetting.py, line 1542:

<comment>This skips TOC gap units before pinning succeeds, so the documented normal-flow fallback cannot restore them. Gate this `continue` on `toc_can_pin` (or otherwise preserve the units) so oversized or wrapped entries retain their spacing and leaders.</comment>

<file context>
@@ -1360,8 +1474,133 @@ def _layout_typesetting_units(
         # 遍历所有排版单元
         for i, unit in enumerate(typesetting_units):
+            # 目录页码/空隙单元由钉入逻辑统一处理
+            if toc_plan is not None and i in toc_plan["gap"]:
+                continue
+
</file context>

Suggested change

	if toc_plan is not None and i in toc_plan["gap"]:
	if toc_plan is not None and toc_can_pin and i in toc_plan["gap"]:

[cubic-dev-ai]

P2: The Roman-page branch cannot recognize Arabic translations because its scan excludes digits, making joined.isdigit() unreachable. A source iii translated as 3 therefore falls back to appending iii while retaining 3, producing a duplicate page number; accept and compare both numeral forms before appending a fallback.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At babeldoc/format/pdf/document_il/midend/typesetting.py, line 1359:

<comment>The Roman-page branch cannot recognize Arabic translations because its scan excludes digits, making `joined.isdigit()` unreachable. A source `iii` translated as `3` therefore falls back to appending `iii` while retaining `3`, producing a duplicate page number; accept and compare both numeral forms before appending a fallback.</comment>

<file context>
@@ -1297,6 +1297,120 @@ def _get_width_before_next_break_point(
+                typesetting_units[k].try_get_unicode() for k in reversed(number_rev)
+            ).lower()
+            # 允许模型把罗马页码转写为阿拉伯数字（如 iii -> 3）
+            matched = joined == page_number_token.lower() or joined.isdigit()
+
+        if not matched or not number_rev:
</file context>

[cubic-dev-ai]

P2: The Linux verifier hardcodes python3.13 even though -PythonVersion accepts other supported versions, so non-3.13 builds fail unless -SkipVerify is used. Verify through the selected interpreter instead of hardcoding the site-packages path.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At scripts/build_offline_packages.ps1, line 352:

<comment>The Linux verifier hardcodes `python3.13` even though `-PythonVersion` accepts other supported versions, so non-3.13 builds fail unless `-SkipVerify` is used. Verify through the selected interpreter instead of hardcoding the site-packages path.</comment>

<file context>
@@ -0,0 +1,400 @@
+tar xzf /out/__TARNAME__ -C /t
+D=/t/__LINDIR__
+$D/bin/python3 -m babeldoc.main --version
+grep -c _split_line_visual_bands $D/lib/python3.13/site-packages/babeldoc/format/pdf/document_il/midend/paragraph_finder.py
+$D/bin/python3 -c 'import onnxruntime,cv2,sklearn,scipy,skimage,freetype,hyperscan,tiktoken,Levenshtein,uharfbuzz,pymupdf; print(42)'
+echo LINUX_VERIFY_OK
</file context>

[cubic-dev-ai]

P2: This treats periods in chapter numbering or title text as dot leaders. Such lines get toc_dot_leader=True, causing typesetting to insert spurious dotted leaders; count leader characters only in the title/page-number gap (ordered[k:j]).

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At babeldoc/format/pdf/document_il/midend/paragraph_finder.py, line 1017:

<comment>This treats periods in chapter numbering or title text as dot leaders. Such lines get `toc_dot_leader=True`, causing typesetting to insert spurious dotted leaders; count leader characters only in the title/page-number gap (`ordered[k:j]`).</comment>

<file context>
@@ -927,6 +960,278 @@ def process_independent_paragraphs(
+            return None
+
+        # 3. 点引导符（连续点或间隔点），不依赖空隙大小
+        has_dot_leader = bool(TOC_DOT_LEADER_REGEX.search(text)) or (
+            text.count(".") >= TOC_SPACED_DOT_MIN_COUNT
+        )
</file context>

Suggested change

	has_dot_leader = bool(TOC_DOT_LEADER_REGEX.search(text)) or (
	text.count(".") >= TOC_SPACED_DOT_MIN_COUNT
	)
	has_dot_leader = (
	sum(is_toc_leader_char(c.char_unicode) for c in ordered[k:j])
	>= TOC_SPACED_DOT_MIN_COUNT
	)

[cubic-dev-ai]

P2: The Windows launchers call python\Scripts\babeldoc.exe, but pip's Windows console-script stubs embed the build machine's absolute interpreter path, so they stop working once the folder is relocated — which README explicitly promises ("the folder is relocatable afterwards"). This is the exact problem the script documents and works around for Linux in install.sh ("pip-generated console scripts carry a build-time absolute shebang, so the Linux wrapper invokes python -m babeldoc.main") — the same fix should be applied on Windows.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At scripts/build_offline_packages.ps1, line 261:

<comment>The Windows launchers call `python\Scripts\babeldoc.exe`, but pip's Windows console-script stubs embed the build machine's absolute interpreter path, so they stop working once the folder is relocated — which README explicitly promises ("the folder is relocatable afterwards"). This is the exact problem the script documents and works around for Linux in `install.sh` ("pip-generated console scripts carry a build-time absolute shebang, so the Linux wrapper invokes `python -m babeldoc.main`") — the same fix should be applied on Windows.</comment>

<file context>
@@ -0,0 +1,400 @@
+'@
+    $wrapper = @'
+@echo off
+"%~dp0python\Scripts\babeldoc.exe" %*
+'@
+    Set-Ascii -Path $initBat -Content $bat -Crlf $true
</file context>

[cubic-dev-ai]

P3: split_toc_lines was inserted in the middle of the signature, before skip_translation and 11 other existing parameters, contrary to the convention comment directly above: "new parameters should be added at the end of the function". Positional callers of TranslationConfig (notably external library users) now have every argument from skip_translation onward shifted by one, silently changing which option they set. Move the parameter to the end, after disable_same_text_fallback.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At babeldoc/format/pdf/translation_config.py, line 208:

<comment>`split_toc_lines` was inserted in the middle of the signature, before `skip_translation` and 11 other existing parameters, contrary to the convention comment directly above: "new parameters should be added at the end of the function". Positional callers of `TranslationConfig` (notably external library users) now have every argument from `skip_translation` onward shifted by one, silently changing which option they set. Move the parameter to the end, after `disable_same_text_fallback`.</comment>

<file context>
@@ -205,6 +205,7 @@ def __init__(
         save_auto_extracted_glossary: bool = True,
         enable_graphic_element_process: bool = True,
         merge_alternating_line_numbers: bool = True,
+        split_toc_lines: bool = True,
         skip_translation: bool = False,
         skip_form_render: bool = False,
</file context>

[cubic-dev-ai]

P3: A relative -OutputDir breaks later steps: the assets-resolution and win64-verify steps Push-Location $env:TEMP and then invoke $tagPy, which is relative (...\win64\python\python.exe) when $OutputDir was passed relative, so Python is looked up under TEMP and the runs fail (and $genDir would be created under TEMP). The default (joined from $RootDir) is absolute and works.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At scripts/build_offline_packages.ps1, line 89:

<comment>A relative `-OutputDir` breaks later steps: the assets-resolution and win64-verify steps `Push-Location $env:TEMP` and then invoke `$tagPy`, which is relative (`...\win64\python\python.exe`) when `$OutputDir` was passed relative, so Python is looked up under TEMP and the runs fail (and `$genDir` would be created under TEMP). The default (joined from `$RootDir`) is absolute and works.</comment>

<file context>
@@ -0,0 +1,400 @@
+$RootDir = Resolve-Path (Join-Path $ScriptDir "..")
+Set-Location $RootDir
+
+if (-not $OutputDir) { $OutputDir = Join-Path $RootDir "_offline_babeldoc" }
+$PyFull = $PythonVersion
+$LinuxPyDir = "cpython-$PyFull-linux-x86_64-gnu"
</file context>

Suggested change

	if (-not $OutputDir) { $OutputDir = Join-Path $RootDir "_offline_babeldoc" }
	if (-not $OutputDir) { $OutputDir = Join-Path $RootDir "_offline_babeldoc" }; $OutputDir = [System.IO.Path]::GetFullPath($OutputDir)

[cubic-dev-ai]

P3: -Platform linux-x64 can never run: the assets-resolution step always requires the win64 staged interpreter, so a Linux-only build throws here. The parameter docs and README ("-Platform win64" shown as the Windows-only example; Docker "only required when building the Linux package") advertise Linux-only builds, but the script also silently requires a full win64 stage (uv + Windows) on top of Docker.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At scripts/build_offline_packages.ps1, line 192:

<comment>`-Platform linux-x64` can never run: the assets-resolution step always requires the win64 staged interpreter, so a Linux-only build throws here. The parameter docs and README ("-Platform win64" shown as the Windows-only example; Docker "only required when building the Linux package") advertise Linux-only builds, but the script also silently requires a full win64 stage (uv + Windows) on top of Docker.</comment>

<file context>
@@ -0,0 +1,400 @@
+    # Compute the expected tag with the freshly built Windows interpreter.
+    $tagPy = Join-Path $WinStage "python\python.exe"
+    if (-not (Test-Path $tagPy)) {
+        throw "win64 stage is required to compute the assets tag; build win64 too"
+    }
+    Push-Location $env:TEMP
</file context>