How to analyze parsing issues with an existing PDF?

[ronnybremer]

I am currently using this crate to create new PDFs, which works very well. A new task requires me to use an existing PDF and add text to it (not a form, a simple PDF). So I adopted the code to parse the PDF, get the page ops and add to it.
Unfortunately, the entire page was just black after saving it.
So I removed my code to add ops to the page and simply do this:

    let mut warnings: Vec<PdfWarnMsg> = Vec::new();
    let mut pdf = PdfDocument::parse(
        original_pdf,
        &PdfParseOptions {
            fail_on_error: true,
        },
        &mut warnings,
    )
    .map_err(|err| anyhow!(err))?;
    error!("warnings: {:?}", warnings);
    // remove all pages but the first
    pdf.pages = pdf.pages.iter().take(1).map(|page| page.clone()).collect();
    let mut page = pdf.pages[0].clone();
    pdf.pages = vec![];
    // add content to page - currently disabled

    // return the result
    Ok(pdf
        .with_pages(vec![page])
        .save(&PdfSaveOptions::default(), &mut Vec::new()))

expecting the saved PDF to be the first page of the original PDF. However, the page is still fully black:

In the above code the following parser warnings are returned:

warnings: [
PdfWarnMsg { page: 0, op_id: 0, severity: Warning, msg: "Type0 font C2_0 missing DescendantFonts" },
PdfWarnMsg { page: 0, op_id: 0, severity: Warning, msg: "Unknown base font: FKZKQC+UniversLTStd-Bold" },
PdfWarnMsg { page: 0, op_id: 0, severity: Warning, msg: "Unknown base font: FKZKQC+UniversLTStd" },
PdfWarnMsg { page: 0, op_id: 0, severity: Warning, msg: "Unknown base font: FKZKQC+UniversLTStd-Cn" },
PdfWarnMsg { page: 1, op_id: 0, severity: Warning, msg: "Unknown base font: FKZKQC+UniversLTStd-Bold" },
PdfWarnMsg { page: 1, op_id: 0, severity: Warning, msg: "Unknown base font: FKZKQC+UniversLTStd" },
PdfWarnMsg { page: 0, op_id: 0, severity: Error, msg: "Info: unhandled operator 'K'" },
PdfWarnMsg { page: 0, op_id: 978, severity: Error, msg: "Info: unhandled operator 'k'" },
PdfWarnMsg { page: 0, op_id: 986, severity: Error, msg: "Info: unhandled operator 'k'" },
PdfWarnMsg { page: 0, op_id: 1034, severity: Error, msg: "Warning: 'EMC' with no current_layer" },
PdfWarnMsg { page: 0, op_id: 1044, severity: Error, msg: "Warning: 'EMC' with no current_layer" },
PdfWarnMsg { page: 0, op_id: 1054, severity: Error, msg: "Warning: 'EMC' with no current_layer" },
PdfWarnMsg { page: 0, op_id: 1065, severity: Error, msg: "Warning: 'EMC' with no current_layer" },
PdfWarnMsg { page: 0, op_id: 1076, severity: Error, msg: "Warning: 'EMC' with no current_layer" },
PdfWarnMsg { page: 0, op_id: 1093, severity: Error, msg: "Warning: 'EMC' with no current_layer" },
PdfWarnMsg { page: 1, op_id: 0, severity: Error, msg: "Info: unhandled operator 'K'" },
PdfWarnMsg { page: 1, op_id: 28, severity: Error, msg: "Info: unhandled operator 'k'" },
PdfWarnMsg { page: 1, op_id: 41, severity: Error, msg: "Warning: 'EMC' with no current_layer" },
PdfWarnMsg { page: 1, op_id: 43, severity: Error, msg: "Info: unhandled operator 'k'" },
PdfWarnMsg { page: 1, op_id: 52, severity: Error, msg: "Info: unhandled operator 'k'" },
PdfWarnMsg { page: 1, op_id: 60, severity: Error, msg: "Warning: 'EMC' with no current_layer" },
PdfWarnMsg { page: 1, op_id: 62, severity: Error, msg: "Info: unhandled operator 'k'" },
PdfWarnMsg { page: 1, op_id: 70, severity: Error, msg: "Info: unhandled operator 'k'" },
PdfWarnMsg { page: 1, op_id: 78, severity: Error, msg: "Warning: 'EMC' with no current_layer" },
PdfWarnMsg { page: 1, op_id: 80, severity: Error, msg: "Info: unhandled operator 'k'" },
PdfWarnMsg { page: 1, op_id: 88, severity: Error, msg: "Info: unhandled operator 'k'" },
PdfWarnMsg { page: 1, op_id: 102, severity: Error, msg: "Warning: 'EMC' with no current_layer" },
PdfWarnMsg { page: 1, op_id: 104, severity: Error, msg: "Info: unhandled operator 'k'" },
PdfWarnMsg { page: 1, op_id: 112, severity: Error, msg: "Info: unhandled operator 'k'" }
]

My first impression is, that an outline around the page is missing its width, causing it to be infinite and therefore covering the entire page. What would be the best way to troubleshoot this? Is this more a lopdf issue?

I can provide the PDF in question, but not in a public space.

Thank you for your advice.

[fschutt]

PdfWarnMsg { page: 1, op_id: 80, severity: Error, msg: "Info: unhandled operator 'k'" },
PdfWarnMsg { page: 0, op_id: 0, severity: Error, msg: "Info: unhandled operator 'K'" },

This is the PDF CMYK operator:

• c m y k K sets the color for subsequent stroking operations (outlines).
• c m y k k sets the color for subsequent non-stroking operations (fills).

Right now, I think the CMYK operations were ignored. I don't exactly remember the reason as to why.

It is likely that your PDF has some form of background rectangle, which is encoded in CMYK space. printpdf currently ignores CMYK operations, and therefore it falls back to the default fill color, which is not white, but black.

This should be fixed in printpdf, lopdf is not the cause.

[ronnybremer]

Thank you for the info @fschutt. I would not know enough to fix it myself, but I will try to convert the PDF into RGB, if thats even possible. Indeed, there are two little triangles on the first page, which I assume are the once causing the issue.
I'll keep you updated.

[ronnybremer]

Allright. Converting the PDF from CMYK to RGB with the command

gs -sDEVICE=pdfwrite -dBATCH -dNOPAUSE -dCompatibilityLevel=1.4 -dColorConversionStrategy=/sRGB -dProcessColorModel=/DeviceRGB -sOutputFile=test.pdf test.pdf.original

worked. The PDF now shows the first page upon save.

Only issue is, all text is gone. I do assume it's because of the missing fonts. I tried to add the fonts to the PdfDocument asset list after parsing, but that hasn't helped so far. Maybe adding them to the OS will help, I will continue to do some tests.
FYI: I added the fonts directly to the resources.fonts.map by using the same FontId as printed in the warnings. Not sure if that is the correct approach or if the text got already lost during parsing.

I am open to any advice. Thanks!

[ronnybremer]

I forgot to add the warnings I received from parse, which have changed since the conversion to RGB:

warnings: [
PdfWarnMsg { page: 0, op_id: 0, severity: Info, msg: "Found font data for R13 (3464 bytes)" },
PdfWarnMsg { page: 0, op_id: 0, severity: Info, msg: "Successfully read font data" },
PdfWarnMsg { page: 0, op_id: 0, severity: Info, msg: "Successfully loaded font at index 0" },
PdfWarnMsg { page: 0, op_id: 0, severity: Info, msg: "Successfully read HEAD table" },
PdfWarnMsg { page: 0, op_id: 0, severity: Info, msg: "MAXP table: 227 glyphs" },
PdfWarnMsg { page: 0, op_id: 0, severity: Info, msg: "Font has 227 glyphs" },
PdfWarnMsg { page: 0, op_id: 0, severity: Info, msg: "Successfully read LOCA table" },
PdfWarnMsg { page: 0, op_id: 0, severity: Info, msg: "Successfully read GLYF table" },
PdfWarnMsg { page: 0, op_id: 0, severity: Warning, msg: "Failed to create allsorts Font: an expected data value was missing" }, PdfWarnMsg { page: 0, op_id: 0, severity: Error, msg: "Failed to parse font data for R13" },
PdfWarnMsg { page: 0, op_id: 0, severity: Warning, msg: "Unknown base font: ECNDYC+UniversLTStd-Cn" },
PdfWarnMsg { page: 0, op_id: 0, severity: Warning, msg: "Unknown base font: DRPAUO+UniversLTStd" },
PdfWarnMsg { page: 0, op_id: 0, severity: Warning, msg: "Unknown base font: CBRIKF+UniversLTStd-Bold" },
PdfWarnMsg { page: 1, op_id: 0, severity: Warning, msg: "Unknown base font: DRPAUO+UniversLTStd" },
PdfWarnMsg { page: 1, op_id: 0, severity: Warning, msg: "Unknown base font: CBRIKF+UniversLTStd-Bold" }]

[fschutt]

PdfWarnMsg { 
    page: 0, 
    op_id: 0, 
    severity: Warning, 
    msg: "Failed to create allsorts Font: an expected data value was missing" 
}, 
PdfWarnMsg { 
    page: 0, 
    op_id: 0, 
    severity: Error, 
    msg: "Failed to parse font data for R13" 
},

Mmh, that seems to be the problem. In allsorts, this is a ParseError::MissingValue, but I cannot trace where it's actually thrown.

1. Try using Inkscape to import the PDF, I just want to confirm that it was the CMYK that was the "black page problem"
2. Is there now anything on the page (do other Ops get exported properly)? So it's only the fonts missing, right?
3. I would need to see what's up with the "R13" font. printpdf parses it here:

printpdf/src/font.rs

Lines 1005 to 1013 in ba84a83

	impl ParsedFont {
	pub fn from_bytes(
	font_bytes: &[u8],
	font_index: usize,
	warnings: &mut Vec<PdfWarnMsg>,
	) -> Option<Self> {
	use allsorts_subset_browser::tag;
	let scope = ReadScope::new(font_bytes);

Clone the repository locally and just std::fs::write(font_bytes) the font to a file. I need to see why the R13 font can't be parsed.

printpdf needs to have access to the fonts to get things like the reverse glyph mapping (mapping glyphs back to characters), etc. (TODO: technically we could fall back to the PDFs glyph ID table, but still). Decoding the fonts from the PDF back is rather crucial for the functionality. The font data seems to be found, but there's an error parsing the actual font.

Sorry for the bad experience, those parsing code paths aren't very well tested.

[ronnybremer]

I shall do that and give you feedback.
To your questions:

1. Inkscape opens it perfectly, but substitutes the fonts as I don't have them installed on my Mac. According to the document properties, no color profile was initially defined, so I assume the CMYK ops came from the initial export done in InDesign.
2. Yes, all elements but the text are visible and rendered correctly on the saved PDF.
3. Working on it.

And btw, I don't have a bad experience :) PDFs are harder than they originally look like and getting this workflow done is a challenge. But without your crate it would be impossible.

[ronnybremer]

Ok, got the font saved. I'll upload it here.

Interestingly, though, the cloned repo panics:

thread 'tokio-runtime-worker' panicked at /home/me/printpdf/src/font.rs:1056:71:
called `Result::unwrap()` on an `Err` value: MissingValue

r13-font.zip

[ronnybremer]

Hmm. Looks like the GIT version has an issue with my previous code (generating a new PDF from scratch). Haven't changed a bit, PDF comes out at 111k, but opening it gives "document contains only empty pages". Thought I let you know, I'll dig through the changes from 8.2 to main.

Please disregard. The debugging I did caused this.

[ronnybremer]

I took the saved font file and tried to parse it like this:

    let mut warnings: Vec<PdfWarnMsg> = Vec::new();
    let tst_font_bytes = include_bytes!("/tmp/r13-font");
    let tst_font =
        printpdf::ParsedFont::from_bytes(tst_font_bytes, 0, &mut warnings).unwrap();
    error!("warnings: {:?}", warnings);

strangely, it worked, here are the warnings:

warnings: [
PdfWarnMsg { page: 0, op_id: 0, severity: Info, msg: "Successfully read font data" },
PdfWarnMsg { page: 0, op_id: 0, severity: Info, msg: "Successfully loaded font at index 0" },
PdfWarnMsg { page: 0, op_id: 0, severity: Info, msg: "Successfully read HEAD table" },
PdfWarnMsg { page: 0, op_id: 0, severity: Info, msg: "MAXP table: 44 glyphs" },
PdfWarnMsg { page: 0, op_id: 0, severity: Info, msg: "Font has 44 glyphs" },
PdfWarnMsg { page: 0, op_id: 0, severity: Info, msg: "Successfully read LOCA table" },
PdfWarnMsg { page: 0, op_id: 0, severity: Info, msg: "Successfully created allsorts Font" },
PdfWarnMsg { page: 0, op_id: 0, severity: Info, msg: "Successfully parsed cmap subtable" },
PdfWarnMsg { page: 0, op_id: 0, severity: Info, msg: "Successfully read HMTX data" },
PdfWarnMsg { page: 0, op_id: 0, severity: Info, msg: "Successfully read HHEA table" },
PdfWarnMsg { page: 0, op_id: 0, severity: Info, msg: "Font metrics: units_per_em=2048" },
PdfWarnMsg { page: 0, op_id: 0, severity: Info, msg: "Successfully read GLYF table" },
PdfWarnMsg { page: 0, op_id: 0, severity: Warning, msg: "Failed to convert glyph 1 to OwnedGlyph from GLYF font" }, PdfWarnMsg { page: 0, op_id: 0, severity: Info, msg: "Successfully decoded 43 glyphs" },
PdfWarnMsg { page: 0, op_id: 0, severity: Info, msg: "Font space width: 569" },
PdfWarnMsg { page: 0, op_id: 0, severity: Info, msg: "Font parsing completed successfully" }]

A single conversion error for a glyph but otherwise the font gets parsed. I'll dig a bit more into the code to see why this parameter missing is thrown.

[ronnybremer]

Nope, that must have been a fluke in my testing. Could have hit a code path which overrode the /tmp/r13-font file. Digging further.

[ronnybremer]

Added those debug statements to src/fonts.rs just before line 1056:

use allsorts_subset_browser::font_data::FontData::OpenType;
use allsorts_subset_browser::font_data::FontData::Woff;
use allsorts_subset_browser::font_data::FontData::Woff2;
match font_file {
  OpenType(_) => println!("font_file: OpenType"),
  Woff(_) => println!("font_file: Woff"),
  Woff2(_) => println!("font_file: Woff2"),
}
println!("provider: {:?}", provider.table_tags());

When parsing my own TTF it prints:

font_file: OpenType
provider: Some([1330851634, 1668112752, 1668707360, 1718642541, 1735162214, 1751474532, 1751672161, 1752003704, 1819239265, 1835104368, 1851878757, 1886352244, 1886545264])

During the PDF parsing process it prints:

font_file: OpenType
provider: Some([1668707360, 1718642541, 1735162214, 1751474532, 1751672161, 1752003704, 1819239265, 1835104368, 1886545264])

So obviously a few numbers are missing there in the table tags, don't know what they mean yet.

[fschutt]

(AI slop)

So obviously a few numbers are missing there in the table tags, don't know what they mean yet.

Integer	4-Byte Hex	ASCII Tag	Table Name	Status in PDF
1330851634	4F532F32	OS/2	OS/2 and Windows Metrics	Missing
1668112752	636d6170	cmap	Character to Glyph Mapping	Missing
1668707360	63767420	cvt	Control Value Table	Present
1718642541	6670676d	fpgm	Font Program	Present
1735162214	676c7966	glyf	Glyph Data	Missing
1751474532	68656164	head	Font Header	Missing
1751672161	68686561	hhea	Horizontal Header	Present
1752003704	686d7478	hmtx	Horizontal Metrics	Present
1819239265	6d617870	maxp	Maximum Profile	Present
1835104368	6e616d65	name	Naming Table	Missing
1851878757	706f7374	post	PostScript	Missing
1886352244	70726570	prep	Control Value Program	Present

The most critical missing tables for the allsorts parser are:

• head (Font Header): This is absolutely fundamental. It contains global information about the font, like the version number, creation date, and crucially, the unitsPerEm value which is essential for scaling glyphs.
• maxp (Maximum Profile): Contains the number of glyphs in the font. Without this, the parser doesn't know how many glyphs to expect.
• cmap (Character to Glyph Mapping): This table maps character codes (like the letter 'A') to glyph indices. It's essential for figuring out which glyph to draw for a given character.
• glyf (Glyph Data): This table contains the actual vector outlines for each glyph. Without this, there is no visual representation of the characters.

The parser cannot "create" the font object in memory because the foundational data it needs is absent from the byte stream it received from printpdf.

However: The fact that parsing the saved file works, but parsing the stream from the PDF fails, points to one of two possibilities:

1. Incomplete Font Subsetting: The PDF creator did not embed the full font. It performed "subsetting," which is common to reduce file size. However, it may have created an incomplete or non-compliant subset of the font, stripping out tables that printpdf / allsorts considers mandatory. A more lenient parser (like the one in Inkscape or Adobe Reader) might be able to reconstruct the missing data or work around it.

2. A Bug in printpdf's Font Extraction: This is also a strong possibility. The code in printpdf that is responsible for finding and extracting the font data stream from the PDF's internal object structure might be flawed. It could be misinterpreting the stream's length or location, resulting in a truncated byte stream being passed to ParsedFont::from_bytes. This truncated stream would lack the tables located towards the end of the original font file, which perfectly explains your observations.

Given that a standalone test with the full font file works, a bug in how printpdf is extracting the font stream from the PDF is the most likely culprit. It is not reading the entire font program from the PDF into the font_bytes slice.

[ronnybremer]

Definitely subsetting. I ran the PDF through an analyzer:

Font count: 4
Byte size: 11,766 (39.24% of the file)

object	name	type	encoding	embedded	subset	unicode	bytes
17 0	FKZKQC+UniversLTStd-Bold	Type 1C	WinAnsi	yes	yes	yes	3,913
18 0	FKZKQC+UniversLTStd	Type 1C	WinAnsi	yes	yes	yes	3,642
24 0	FKZKQC+Webdings	CID TrueType	Identity-H	yes	yes	yes	2,377
25 0	FKZKQC+UniversLTStd-Cn	Type 1C	WinAnsi	yes	yes	yes	1,834

[ronnybremer]

Comparing the hexdump of the saved font /tmp/r13-font with the output of the embedded font object in the analyzer, it appears to be the Webdings font having the issue. As you mentioned above, a few parts are missing indeed, for example the font object in the PDF starts with:

74 72 75 65 00 0b 00 80 00 03 00 30 4f 53 2f 32 true.... ...0OS/2
00 00 00 00 00 00 00 bc 00 00 00 56 63 76 74 20 ........ ...Vcvt

while the parsed font starts with:

00 01 00 00 00 09 00 80 00 03 00 10 63 76 74 20 ............cvt

but looking further down I would say ~80% of the hex data is identical.