Skip to content

Fix conversion data-loss paths in convert.py - #35

Draft
cursor[bot] wants to merge 1 commit into
mainfrom
cursor/critical-bug-investigation-6f71
Draft

cursor[bot] wants to merge 1 commit into
mainfrom
cursor/critical-bug-investigation-6f71

Conversation

@cursor

@cursor cursor Bot commented Jun 28, 2026

Copy link
Copy Markdown

Bug and impact

  • convert.py wrote Calibre's temporary HTMLZ output next to the input file as <input-stem>.htmlz and deleted that path after extraction. If a user already had a sibling book.htmlz, conversion overwrote and then deleted user-owned data.
  • Default Markdown cleanup also auto-deleted monotonic standalone numeric lines. Books that legitimately contain standalone problem, verse, table, or section numbers such as 1, 2, 3, 4 could lose content before chunking; the manifest then hashed the already-truncated source, so later validation could not detect the loss.

Root cause

  • The HTMLZ intermediate path was derived from input_file instead of a private temp directory.
  • The LNDS page-number heuristic ran by default, even though standalone numeric lines are ambiguous and may be real book content.

Fix

  • Write HTMLZ intermediates under tempfile.TemporaryDirectory(prefix="<book>_htmlz_") and remove the sibling-path cleanup.
  • Preserve standalone numeric lines by default unless adjacent to Calibre noise.
  • Add explicit --auto-strip-page-numbers for the existing monotonic-page heuristic and keep --strip-page-numbers for aggressive legacy cleanup.
  • Update English and Chinese README troubleshooting text.
  • Add regression tests for sibling HTMLZ preservation and default numeric-line preservation.

Validation

  • python3 -m unittest tests.test_convert -v — 19 passed.
  • Post-fix repro: default preserves 1..4; auto_strip_page_numbers=True removes them; conversion uses /tmp/book_htmlz_.../book.htmlz; sibling book.htmlz remains with content USER DATA.
  • python3 -m unittest discover -s tests -p 'test_*.py' -v — 207 passed, 3 skipped.
  • python3 -m compileall scripts tests — passed.
  • Standard Alice mock full pipeline: conversion generated 38 chunks; merge generated HTML/DOCX/PDF. EPUB failed only for the documented Ubuntu Calibre 7.6.0 bytes/str packaging bug.
Open in Web View Automation 

Co-authored-by: Rainman <deusyu@users.noreply.github.com>
@cursor
cursor Bot deleted the cursor/critical-bug-investigation-6f71 branch July 2, 2026 08:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant