Support dense CAD content streams

[yzxcj797]

Summary

• Raise the page content-stream safety budget from 1,000,000 to 2,000,000 operators.
• Preserve the hard fail-closed boundary for pathological streams above the new limit.
• Propagate a machine-readable content_operation_limit OCR reason when a page is skipped by that budget, instead of making it indistinguishable from a genuinely empty page.
• Add regression coverage for the reported 1.1M-operator CAD shape, text survival below the limit, fail-closed behavior above it, and public per-page diagnostics.

Fixes #307.

Rationale

The existing bounded decoder protects against unbounded operation allocation, but the one-million threshold also rejects real CAD/Revit pages containing native text. The issue reports valid pages at approximately 1.07M and 1.31M operators. A two-million ceiling covers those documents while retaining a finite allocation guard.

The new reason is also deliberately distinct from no_text: callers can tell that extraction was stopped by the safety budget and choose an appropriate fallback.

Testing

• cargo test
  • 954 library tests pass
  • 163 integration tests pass
  • 3 CLI tests pass
  • 2 documentation tests pass
• cargo clippy -- -D warnings — passes
• cargo fmt --all -- --check — passes
• git diff --check — passes

The optional OCR runtime suites were not run locally; this change does not alter OCR runtime code.

---

Summary by cubic

Supports dense CAD/Revit PDFs by doubling the content-stream operator budget and emitting a dedicated OCR reason when the limit is hit. Previously, pages over 1,000,000 operators looked like empty text and could imply encoding issues; now the cap is 2,000,000 and over-cap pages are skipped with reason "content_operation_limit" without flagging encoding corruption.

• Adds OCR reason "content_operation_limit" and surfaces it in per-page diagnostics and ocrReasonsByPage; extractPagesMarkdown sets the page’s ocr_reason accordingly.
• Tracks pages skipped by the operator budget and includes them in pages_needing_ocr; does not set has_encoding_issues when this is the only reason.
• Introduces extract_page_text_items_with_operation_limit; makes content_exceeds_operation_limit pub(crate); adds regression tests for under/over-cap behavior.
• Migration: if you surface OCR reasons, accept "content_operation_limit"; no API shape changes to @firecrawl/pdf-inspector.
• Addresses Dense CAD/DWG PDF pages can return empty text when content streams exceed 1M operations #307.

^{Written for commit 4229cf6. Summary will update on new commits.}

[cubic-dev-ai]

All reported issues were addressed across 8 files

<sub>Shadow auto-approve: would not auto-approve because issues were found.

Fix all with cubic | Re-trigger cubic</sub>

[yzxcj797]

Addressed in 4229cf6:

• process_pdf_mem now includes budget-skipped pages in pages_needing_ocr.
• content_operation_limit no longer by itself sets has_encoding_issues; actual text-quality or encoding signals still do.
• Added a process-level regression covering both invariants.
• Corrected the Node documentation: the budget reason is page-level (ocrReasonsByPage), not a region-level ocrReason.

Validation: full cargo test, cargo clippy -- -D warnings, cargo fmt --all -- --check, and git diff --check pass.

[cubic-dev-ai]

1 issue found across 3 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="napi/README.md">

<violation number="1" location="napi/README.md:97">
P3: This statement sits in the `extractTextInRegions` section, but that function returns `PageRegionTexts[]`, which has no `ocrReasonsByPage` field — `ocrReasonsByPage` exists only on the `OcrPdfResult` returned by `processPdfWithOcr`. A reader of the region-extraction docs cannot find the reason in that result. Move the `content_operation_limit` note to the `processPdfWithOcr` section (next to `ocrReasonsByPage`) or state explicitly that the page-level reason is exposed only through `processPdfWithOcr`/`processPdf`, not `extractTextInRegions`.</violation>
</file>

<sub>Shadow auto-approve: would not auto-approve because issues were found.

Fix all with cubic | Re-trigger cubic</sub>

[cubic-dev-ai]

P3: This statement sits in the extractTextInRegions section, but that function returns PageRegionTexts[], which has no ocrReasonsByPage field — ocrReasonsByPage exists only on the OcrPdfResult returned by processPdfWithOcr. A reader of the region-extraction docs cannot find the reason in that result. Move the content_operation_limit note to the processPdfWithOcr section (next to ocrReasonsByPage) or state explicitly that the page-level reason is exposed only through processPdfWithOcr/processPdf, not extractTextInRegions.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At napi/README.md, line 97:

<comment>This statement sits in the `extractTextInRegions` section, but that function returns `PageRegionTexts[]`, which has no `ocrReasonsByPage` field — `ocrReasonsByPage` exists only on the `OcrPdfResult` returned by `processPdfWithOcr`. A reader of the region-extraction docs cannot find the reason in that result. Move the `content_operation_limit` note to the `processPdfWithOcr` section (next to `ocrReasonsByPage`) or state explicitly that the page-level reason is exposed only through `processPdfWithOcr`/`processPdf`, not `extractTextInRegions`.</comment>

<file context>
@@ -94,7 +94,7 @@ console.log(result.confidence)     // 0.875
 Extract text within bounding-box regions from a PDF. Designed for hybrid OCR pipelines where a layout model detects regions in rendered page images, and this function extracts text from the PDF structure for text-based pages — skipping GPU OCR.
 
-Each region result includes a `needsOcr` flag that signals unreliable extraction (empty text, GID-encoded fonts, garbage text, encoding issues). Known causes are exposed in `ocrReason`; for example, a suspected garbled text layer reports `"suspected_garbled_text"`, and a page skipped by the content-stream safety budget reports `"content_operation_limit"`.
+Each region result includes a `needsOcr` flag that signals unreliable extraction (empty text, GID-encoded fonts, garbage text, encoding issues). Known region causes are exposed in `ocrReason`; for example, a suspected garbled text layer reports `"suspected_garbled_text"`. A page skipped by the content-stream safety budget reports `"content_operation_limit"` in the page-level `ocrReasonsByPage` result.
 
 ```typescript
</file context>

Suggested change

Each region result includes a `needsOcr` flag that signals unreliable extraction (empty text, GID-encoded fonts, garbage text, encoding issues). Known region causes are exposed in `ocrReason`; for example, a suspected garbled text layer reports `"suspected_garbled_text"`. A page skipped by the content-stream safety budget reports `"content_operation_limit"` in the page-level `ocrReasonsByPage` result.

Each region result includes a `needsOcr` flag that signals unreliable extraction (empty text, GID-encoded fonts, garbage text, encoding issues). Known region causes are exposed in `ocrReason`; for example, a suspected garbled text layer reports `"suspected_garbled_text"`. A page skipped by the content-stream safety budget is reported via `processPdfWithOcr`/`processPdf`, which exposes `"content_operation_limit"` in the page-level `ocrReasonsByPage` result.