Skip to content

fix: distinguish NewCM text fonts from math fonts - #622

Open
pentaoa wants to merge 1 commit into
funstory-ai:mainfrom
pentaoa:fix/newcm-text-font-classification
Open

pentaoa wants to merge 1 commit into
funstory-ai:mainfrom
pentaoa:fix/newcm-text-font-classification

Conversation

@pentaoa

@pentaoa pentaoa commented Sep 29, 2026 •

Copy link
Copy Markdown

Related Issue(s)

Fixes #613. Related: #614.

Motivation and Context

The Stanford CS336 assignment PDF uses NewCM10-Regular, NewCM10-Bold, and NewCM10-Italic for ordinary prose, alongside NewCMMath-Book for mathematics. The precise formula-font patterns currently match every NewCM / NewComputerModern font, so the body text is classified as formulas and protected with placeholders rather than sent for translation.

This also explains why --formular-font-pattern ".*Math.*" cannot work around the issue: the precise built-in pattern returns before the custom pattern is evaluated.

Summary of Changes

In babeldoc/format/pdf/document_il/utils/formular_helper.py, narrow the two precise matches to NewCMMath and NewComputerModernMath. Ordinary NewCM fonts can then follow the existing text/custom-pattern classification, while actual math fonts remain protected.

In tests/test_formular_helper.py, cover regular/bold/italic text, PDF subset prefixes, both math-font spellings, existing Computer Modern/Latin Modern behavior, and page/XObject font scopes. Each case runs with the default formula pattern and the custom .*Math.* pattern.

#614 restores the paragraph bounding-box IoU fallback; this change addresses the separate font-classification cause documented in #613.

PR Type

  • 🐛 Bug Fix
  • 🧪 Test

Breaking Changes

None.

Testing Instructions

  • uv run pytest tests/test_formular_helper.py -q: 24 passed, importing the production module and using the real IL dataclasses.
  • uv run ruff check .: passed.
  • uv run ruff format --check babeldoc/format/pdf/document_il/utils/formular_helper.py tests/test_formular_helper.py: passed.
  • git diff --check: passed.

PyMuPDF inspection of the linked PDF found 97,688 characters in NewCM10-Regular, 6,105 in bold, 884 in italic, and 2,674 in NewCMMath-Book, confirming that the affected font names occur in the reported document.

Contributor Checklist

  • Read the contribution guide and kept the patch small.
  • Reviewed the changed code.
  • Linked the existing issue and added regression tests.
  • The changed files follow the project's formatting and lint rules.
  • No new dependency or user-facing configuration is introduced.

Limitations

Verification covers offline font classification, font-resource scopes, and the source PDF's font metadata. End-to-end translation through an external service is unverified.


Summary by cubic

Fixes #613: NewCM text fonts (e.g. NewCM10-Regular) were matched by the precise formula-font pattern and protected as math, preventing translation. The pattern now only matches math fonts (NewCMMath*, NewComputerModernMath*), so ordinary NewCM text fonts classify as text. Adds regression tests for font classification and page/XObject scopes, including the custom .*Math.* pattern.

Written for commit 2c86910. Summary will update on new commits.

Review in cubic

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 2 files

Re-trigger cubic

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] NewCM text fonts are misclassified as formulas, leaving body text untranslated

1 participant