Repository navigation
Conversation
Arbortext letter-spaces a heading as letter, space glyph, letter, and moves the pen after every space: back by a fifth of its advance inside a word, forward by one and a half between words. On the page that is tracking — EUR-Lex 02021R2178, Annex I, `1.1.1. KPI bezogen auf den Umsatz (Umsatz-KPI)`: letters 1.7 pt apart, words 5.1 pt — and pdftotext reads the words. With a space glyph at every letter, which docling-parse keeps as a character, the line cell read `K P I b e z o g e n a u f …`. The spaces are past the 30 % mark of drop_overprinted_spaces (docling-project#992), and the tracking rule of docling-project#921 never saw a chain: a space glyph always splits. textparse::drop_letter_spacing_spaces takes those spaces out of the text layer. The rule is narrow, a deliberate divergence like docling-project#992's: - single glyphs and space glyphs alternate on one line (no run of words does that); - three or more consecutive spaces between letters each have part of their advance taken back (the next letter starts before 0.95 of it) and leave one gap in points, within 0.25 pt — and the letters are not one character repeated; - every space of the stretch that leaves that gap goes (the short words, the brackets and the hyphen), the others stay as the word spaces; - those, and the spaces between the stretch and its neighbours, must be at least 1.5 x the letter gap. A line whose letters are no closer than its words — spaces at their own width, a tightened line's `von a b c d bis` — has no letter spacing to tell from its word spacing and is left as drawn; so are `C C C C C` and `1 2 3 4 5`. The letter after a dropped space is marked (Glyph::spaced) and the word contraction keeps that gap inside the word (dp_lines::tracked_gaps). The detection of docling-project#921 alone would not do: it needs three letter gaps in one line run, and a justified heading — the document's 1.1.2, its words 26 pt apart — leaves `KPI` and `auf` alone on their line cells, which read `KP I` and `a u f`. The document's twelve runs all read as pdftotext reads them, among them `(CapEx)(CapEx-KPI)`: the typesetter set that word space no wider than the letter spacing, and it is gone on the page too. Nothing else in its 4,185 line cells changes. pdf_word_gap=legacy knows no tracking and still cuts at each letter gap, as for docling-project#921. PDF snapshots 101/101 exact, groundtruth 9 / 21 strict, 511 diff lines: unchanged. Tests: the reporter's construction through the parser — the heading, the justified heading, a word alone on its line, a fixed factor, legacy — and what must stay as drawn (spaces at or over their width, word spaces not plainly wider, a tightened line, one character repeated, digits). The first fails without either half of the change. Closes docling-project#1139 Signed-off-by: Artem Kustikov <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QNSzBK759SLC5oULeR3YeB
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Arbortext letter-spaces a heading as letter, space glyph, letter, and moves the pen after every space: back by a fifth of its advance inside a word, forward by one and a half between words. On the page that is tracking — EUR-Lex 02021R2178, Annex I,
1.1.1. KPI bezogen auf den Umsatz (Umsatz-KPI): letters 1.7 pt apart, words 5.1 pt — and pdftotext reads the words. With a space glyph at every letter, which docling-parse keeps as a character, the line cell readK P I b e z o g e n a u f …. The spaces are past the 30 % mark of drop_overprinted_spaces (#992), and the tracking rule of #921 never saw a chain: a space glyph always splits.textparse::drop_letter_spacing_spaces takes those spaces out of the text layer. The rule is narrow, a deliberate divergence like #992's:
von a b c d bis— has no letter spacing to tell from its word spacing and is left as drawn; so areC C C C Cand1 2 3 4 5.The letter after a dropped space is marked (Glyph::spaced) and the word contraction keeps that gap inside the word (dp_lines::tracked_gaps). The detection of #921 alone would not do: it needs three letter gaps in one line run, and a justified heading — the document's 1.1.2, its words 26 pt apart — leaves
KPIandaufalone on their line cells, which readKP Ianda u f.The document's twelve runs all read as pdftotext reads them, among them
(CapEx)(CapEx-KPI): the typesetter set that word space no wider than the letter spacing, and it is gone on the page too. Nothing else in its 4,185 line cells changes. pdf_word_gap=legacy knows no tracking and still cuts at each letter gap, as for #921.PDF snapshots 101/101 exact, groundtruth 9 / 21 strict, 511 diff lines: unchanged.
Tests: the reporter's construction through the parser — the heading, the justified heading, a word alone on its line, a fixed factor, legacy — and what must stay as drawn (spaces at or over their width, word spaces not plainly wider, a tightened line, one character repeated, digits). The first fails without either half of the change.
Closes #1139
Claude-Session: https://claude.ai/code/session_01QNSzBK759SLC5oULeR3YeB