Add quality-aware server-side PDF OCR ingestion

[dovinmu]

Summary

Adds end-to-end server-side PDF ingestion to Antfly. One artifact configuration accepts inline PDF bytes or a URL, fetches and parses the PDF inside Antfly, emits page units, selectively OCRs, chunks, embeds, and populates lexical plus semantic indexes—without external preprocessing or intermediate JSON.

What changed

• Quality-aware OCR: configurable fallback heuristics and forced OCR, embedded-vs-OCR quality selection, table/layout-aware prompts, Florence's <OCR> default, and prompt-echo/trivial-output rejection. Inline JPEG captioning is also supported.
• Bounded rendering and transport: 150-DPI target rendering adapts to configurable dimension/pixel caps (never below a safe 72-DPI floor), emits compressed PNGs, and applies inference-specific limits to Antfly-generated data images while retaining remote-content limits for external URLs.
• Failure handling and provenance: records per-page extraction/OCR method, selection, requested/effective DPI, dimensions, encoded size, timing, concrete failure stage, and retryability. Failed OCR keeps usable embedded text and remains observable/reprocessable; malformed OCR options are rejected and producer credentials remain write-only in status responses.
• Indexing correctness and performance: atomically provisions artifact-backed indexes, feeds chunks into full-text search, batches synchronous OCR pages through the shared Reader/Florence path, and batches document embeddings across page boundaries with bounded item/byte caps and persisted telemetry. Exact source counts propagate through runtime status so strict coverage stays correct across sync barriers, restore, and restart.
• PDF compatibility and safety: improves multi-page rendering/caching and common fonts, CMaps, forms, images, predictors, and encrypted PDFs while bounding decode filters, traversal, decompression, canvas, pattern, and transparency work. Unit and public /merge E2Es cover inline/URL, born-digital, scanned, garbled, table, paged OCR, hybrid retrieval, and restart coverage paths.

Notes

Replaces #360 after #335 merged and includes current main, including its Florence optimizations. DocsAF is a client/configurator that uses Antfly; Antfly does not depend on DocsAF.

[ajroetker]

unit-test
+- run test 181 pass, 1 fail (182 total)
error: 'src.reader.test.apply explicit mask alpha uses mask alpha or grayscale channel' failed:
       expected 11, found 255
       /home/runner/_work/_tool/zig/0.16.0/x64/lib/std/testing.zig:118:17: 0x16d1514 in expectEqualInner__anon_73259 (std.zig)
                       return error.TestExpectedEqual;
                       ^
       /home/runner/_work/_tool/zig/0.16.0/x64/lib/std/testing.zig:83:5: 0x17a098e in expectEqual (pdf_test_root.zig)
           return expectEqualInner(T, expected, actual);
           ^
       /home/runner/_work/antfly/antfly/zig/lib/pdf/src/reader.zig:11163:5: 0x17a09d5 in test.apply explicit mask alpha uses mask alpha or grayscale channel (pdf_test_root.zig)
           try std.testing.expectEqual(@as(u8, 11), rgba[3]);
           ^
failed command: /mnt/cache/antfly/zig/zig-base/29382115987-1/zig-local/o/fd31d3bbc7004a0bc67f3b278967f961/test --cache-dir=/mnt/cache/antfly/zig/zig-base/29382115987-1/zig-local --seed=0x8a1a06e4 --listen=-```

[ajroetker]

I don't think this should be "ocr" specific (for instance it should also apply to auto image embedding of pages OR ocr) same for all these fields, a lot of them apply to chunks too

[ajroetker]

[codex] Concrete proposal: replace the top-level ocr_* fields with processing_stages: []DocumentArtifactStageSummary. Each stage should identify an operation (extract, render, OCR, caption, transcribe, chunk, or embed) and expose common counts such as input, attempted, completed, selected, retained_source, failed, and output. Failures should identify a generic unit_id plus an optional locator, rather than only PDF page numbers. OCR-specific quality metrics can remain nested in the operation details. This gives chunking and direct page/image embedding the same observability contract.

[ajroetker]

@timkaye11 can you review the inference changes

[ajroetker]

[codex] @timkaye11 Could you review the inference portion, particularly the new Florence batch path and serial fallback, batch byte/image limits, the lifetime and nesting of the thread-local profiling context, and whether local-only diagnostic propagation is intentional?

[ajroetker]

I don't think this file should reference std.posix? Seems like maybe this should be in lib/platform and the resource_manager abstraction

[ajroetker]

[codex] Please move the OS mechanism behind antfly_platform.process, for example ensureFileDescriptorSoftLimitAtLeast(target: u64), and keep only the 4096 policy in startup code. The existing storage ResourceManager does not seem like the right owner unless it is deliberately expanded beyond memory and storage budgets. Also, isServerSubcommand already diverges from the dispatch block: it omits lite and ha, so those processes never receive the check. Ideally server command classification should have one source of truth.

[ajroetker]

This is typically called a "trace_id" and we should probably name it that

[ajroetker]

[codex] Looking further upstream, this value is a stable prefix of the source SHA-256, so trace_id would overstate its semantics. Please either name it profile_source_id/source_fingerprint throughout and log it explicitly as such, or introduce a real per-execution trace ID and propagate that through the extraction and inference calls. correlation_id is too vague, but a stable content fingerprint should not be presented as a distributed trace identifier.

[ajroetker]

See my other comment but I don't think these fields should be ocr specific but should be more generic, maybe have a little convo with fable and codex about making this more adaptable to satisfy embedding multipage directly, ocr, document extraction, captioning, chunking for text/audio/images

[ajroetker]

[codex] I would normalize this as provenance.source plus provenance.derivations[], instead of continuing to add sibling ocr_* fields. Each derivation can have a common operation, status, selected flag, parent references, output descriptor, confidence, and failure. Render dimensions and DPI belong to a render derivation, while OCR quality belongs to an OCR derivation. Chunks and embeddings should point to their parent unit/artifact through the same mechanism. The current payload emits the OCR values in three places, which makes future captioning, transcription, and multimodal embedding awkward and prone to drift. A pragmatic boundary for this PR would be to keep the internal OCR-specific Zig fields temporarily but map them into the generic public schema before merging.

[ajroetker]

[codex] e2e-base is still red: https://github.com/antflydb/antfly/actions/runs/29602361560/job/87957590465. The artifact restart test reaches produced/skipped/terminal-failed = 1/1/1, but strict coverage remains incomplete because the group is reported stale/config-mismatched. Please root-cause that state transition rather than widening the wait. The autoscaling split failure may be independent, but it also needs a clean rerun or diagnosis before merge.

[ajroetker]

[codex] Re-review of the current head (01c28ba): there have been no code commits since the previous review, and the existing review threads remain unresolved.

1. [P1] The decoded-stream limit is still bypassable during ASCII85 allocation.

The fix checks ASCII85 output only after ascii85DecodeAlloc returns. Inside the decoder, every z expands to four bytes with no budget check:

antfly/zig/lib/pdf/src/reader.zig

Lines 4878 to 4880 in 01c28ba

	if (std.mem.eql(u8, name, "ASCII85Decode")) {
	const decoded = try ascii85DecodeAlloc(alloc, input);
	return try enforceDecodedStreamLimit(alloc, decoded, max_decoded_stream_bytes);

and

antfly/zig/lib/pdf/src/reader.zig

Lines 4925 to 4955 in 01c28ba

	fn ascii85DecodeAlloc(alloc: Allocator, input: []const u8) ![]u8 {
	var digits = std.ArrayList(u8).empty;
	defer digits.deinit(alloc);
	var out = std.ArrayList(u8).empty;
	defer out.deinit(alloc);
	var i: usize = 0;
	while (i < input.len) : (i += 1) {
	const ch = input[i];
	if (isPdfWhitespace(ch)) continue;
	if (ch == '~') break;
	if (ch == 'z') {
	if (digits.items.len != 0) return error.MalformedAscii85;
	try out.appendSlice(alloc, &.{ 0, 0, 0, 0 });
	continue;
	}
	if (ch < '!' or ch > 'u') return error.MalformedAscii85;
	try digits.append(alloc, ch - '!');
	if (digits.items.len == 5) {
	try appendAscii85Group(alloc, &out, digits.items, 4);
	digits.clearRetainingCapacity();
	}
	}
	if (digits.items.len > 0) {
	const original_len = digits.items.len;
	while (digits.items.len < 5) try digits.append(alloc, 'u' - '!');
	try appendAscii85Group(alloc, &out, digits.items, original_len - 1);
	}
	return try out.toOwnedSlice(alloc);

. A filter chain can RunLength-decode roughly 4 MiB into 256 MiB of z, then make ASCII85 attempt a 1 GiB allocation before DecodedStreamTooLarge is returned. Pass the cap into ascii85DecodeAlloc and check it before every append.

2. [P1] Predictor parameters use unchecked arithmetic from an untrusted PDF.

Columns, Colors, and BitsPerComponent are multiplied directly before casting to usize:

antfly/zig/lib/pdf/src/reader.zig

Lines 5114 to 5120 in 01c28ba

	const columns_i = if (param.?.get("Columns")) |obj| obj.asInteger() orelse 1 else 1;
	const colors_i = if (param.?.get("Colors")) |obj| obj.asInteger() orelse 1 else 1;
	const bits_i = if (param.?.get("BitsPerComponent")) |obj| obj.asInteger() orelse 8 else 8;
	if (columns_i <= 0 or colors_i <= 0 or bits_i <= 0) return error.UnsupportedPredictor;
	const bytes_per_pixel: usize = @intCast(@divTrunc(colors_i * bits_i + 7, 8));
	const row_len: usize = @intCast(@divTrunc(columns_i * colors_i * bits_i + 7, 8));

. Large integers can overflow or trap instead of returning a parser error. Use checked multiplication/addition, reject unsupported component counts and bit depths, and ensure row_len is bounded by decoded.len and the decoded-stream budget before allocation.

3. [P2] Credential redaction can silently delete legitimate table schema fields.

redactInlineEnrichmentProducerConfigs recursively removes every object key named producer_json from the entire serialized TableStatus, rather than only enrichment configurations:

antfly/zig/pkg/antfly/src/api/tables.zig

Lines 1840 to 1871 in 01c28ba

	fn projectInlineEnrichmentConfigsInTableStatusJson(alloc: std.mem.Allocator, encoded: []const u8) ![]u8 {
	var arena_impl = std.heap.ArenaAllocator.init(alloc);
	defer arena_impl.deinit();
	const arena = arena_impl.allocator();
	var parsed = try std.json.parseFromSlice(std.json.Value, arena, encoded, .{});
	redactInlineEnrichmentProducerConfigs(&parsed.value);
	return try std.json.Stringify.valueAlloc(alloc, parsed.value, .{ .emit_null_optional_fields = false });
	}
	fn projectSingleTableStatusJson(alloc: std.mem.Allocator, encoded: []const u8, indexes_json: []const u8) ![]u8 {
	var arena_impl = std.heap.ArenaAllocator.init(alloc);
	defer arena_impl.deinit();
	const arena = arena_impl.allocator();
	var parsed = try std.json.parseFromSlice(std.json.Value, arena, encoded, .{});
	try attachArtifactEnrichmentsToTableStatus(arena, &parsed.value, indexes_json);
	redactInlineEnrichmentProducerConfigs(&parsed.value);
	return try std.json.Stringify.valueAlloc(alloc, parsed.value, .{ .emit_null_optional_fields = false });
	}
	/// Producer configuration is accepted on writes but is deliberately omitted
	/// from table-status responses. Besides provider credentials, producer_json
	/// can contain arbitrary nested reader configuration supplied by a client.
	fn redactInlineEnrichmentProducerConfigs(value: *std.json.Value) void {
	switch (value.*) {
	.array => |*array| {
	for (array.items) |*item| redactInlineEnrichmentProducerConfigs(item);
	},
	.object => |*object| {
	_ = object.swapRemove("producer_json");
	var it = object.iterator();
	while (it.next()) |entry| redactInlineEnrichmentProducerConfigs(entry.value_ptr);
	},

. Document schemas permit arbitrary JSON Schema properties, so a legitimate field named producer_json disappears from status responses:

antfly/specs/openapi/antfly/schema.yaml

Lines 35 to 62 in 01c28ba

	schema:
	type: object
	additionalProperties: true
	description: |
	A valid JSON Schema defining the document's structure.
	This is used to infer indexing rules and field types.
	TableSchema:
	type: object
	description: Schema definition for a table with multiple document types
	properties:
	version:
	type: integer
	format: uint32
	description: Version of the schema. Used for migrations.
	default_type:
	type: string
	description: Default type to use from the document_types.
	enforce_types:
	type: boolean
	description: |
	Whether to enforce that documents must match one of the provided document types.
	If false, documents not matching any type will be accepted but not indexed.
	document_schemas:
	type: object
	additionalProperties:
	$ref: "#/components/schemas/DocumentSchema"
	description: A map of type names to their document json schemas.

. Scope redaction to indexes..enrichments[] and table-level artifact enrichments, with a regression test for a document schema property named producer_json.

Fix assessment:

• Malformed OCR option types: fixed correctly, although unknown keys remain accepted.
• Producer credential leak: the leak is fixed, but the redaction is overly broad as described above.
• Decoded stream bounds: incomplete; RunLength and LZW are bounded during growth, but ASCII85 is not.
• Generic processing-stage API, derivation-based provenance, POSIX/platform abstraction, and correlation identifier semantics: not addressed.
• CI remains red for e2e-base and Playwright, with no newer run at this head.

[ajroetker]

[codex] I opened #392 with fixes for the concrete review blockers plus the platform/source-fingerprint cleanups. It is based on an exact mirror of this PR's reported head (01c28ba), because the visible fork branch currently points to a partial diverged history. The generic processing-stage and derivation-provenance schema redesign remains separate because it needs a coordinated public API/SDK migration.