Add CC-scale MinerU-HTML layout-clustering + propagation pipeline (91% fewer LLM calls, F1=0.91)

[VibhuJawa]

Dripper Common Crawl Clustering Pipeline

Full clustering+propagation pipeline for HTML extraction at CC scale using NeMo Curator best practices.

Architecture

Stage	Implementation	Result
Stage 1a	DOMFeatureExtractionStage(ProcessingStage), RayActorPoolExecutor	86k pages in 102s
Stage 1b	HostDBSCANStage(ProcessingStage), Resources(gpus=1), cuML	92.9% LLM call reduction
Stage 1c+2+2b	GPU vLLM kv-fp8 via Pipeline.run()	164.9 p/s/node
Stage 3	_Stage3PropagationStage(ProcessingStage), PPT=16, LPT+HTML-size sort	F1=0.8450
Stage 3b	GPU fallback for over-extracted siblings (pred>2.5x ref)	F1=0.9175

Validated Results

• F1 = 0.9175 > 0.90 target (+0.181 vs original pipeline)
• GPU throughput = 164.9 p/s/node > 163 target
• Curator best practices: ProcessingStage, RayActorPoolExecutor, Resources, dripper_cached_venv

Key design decisions

• Single execution backend: RayActorPoolExecutor (ProcessPool fallback removed)
• PPT=16 siblings per task with HTML-size descending sort for optimal load balancing
• html_element_dict pre-parsed on driver once per cluster (avoids repeated JSON+eval per sibling)
• Sim-gate: use LBP body even when structural similarity < 0.75 threshold
• Stage 3b GPU fallback routes 14% of over-extracted siblings back to LLM

LOC summary (tutorials/text/dripper-common-crawl/)

• stage3_cpu_propagation.py: 1107 → 870 lines (-237) — removed ProcessPool path, collapsed argparse
• dashboard_server.py: removed from PR (NVIDIA-internal hostnames/paths, dev tool only)

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[greptile-apps]

Greptile Summary

This PR adds a 7-stage CC-scale MinerU-HTML pipeline that replaces per-page LLM inference with GPU-accelerated DOM clustering (cuML DBSCAN) + template propagation, achieving ~91% fewer LLM calls and F1=0.91 vs the standalone baseline. The tutorial scripts under dripper-common-crawl/ are well-validated and benchmark-proven; however, the library-level integration stages in nemo_curator/stages/text/experimental/dripper/ have three defects that prevent the deferred-propagation path from working correctly.

• Tutorial pipeline (stage1a–stage3b, run_mineru_pipeline.sh): correct pickle+base64 serialization, robust _parse_mapping_json, two-tier static/dynamic LBP with per-cluster F1 validation, and offline-batched vLLM with chat-template application — all four bugs from development are fixed here.
• Library stages (DripperHTMLLayoutTemplateStage, DripperHTMLLayoutPropagationStage): bug Make NeMo-Curator installable in non GPU environments #4 (tuple-key destruction) is not ported — stage.py still uses json.dumps(default=str) while propagation_stage.py uses json.loads; additionally propagation_stage.py has an iloc/loc mismatch that silently reads wrong rows on non-default-indexed DataFrames.
• Infra changes (llm_client.py, openai_client.py, core/client.py, dynamo/ray-serve configs): all look correct; the query_model_with_usage extension and _run_with_retry_and_concurrency refactor are clean improvements.

Confidence Score: 3/5

Safe to merge for teams using only the tutorial scripts; the library integration stages for deferred propagation are broken and should not be used until the serialization and indexing defects are fixed.

The tutorial pipeline and benchmarks are solid and correctly implement the four bug fixes. The library-level DripperHTMLLayoutTemplateStage+DripperHTMLLayoutPropagationStage deferred-propagation path has two independent defects: json.dumps(default=str) in stage.py destroys the tuple keys that LayoutBatchParser requires (the same bug #4 the PR fixes in the tutorial scripts), and propagation_stage.py's df.iloc[idx] reads positional rows when idx is a label, silently producing wrong or out-of-bounds results. Both would cause the integrated library path to produce incorrect output with no error surfaced to the caller.

nemo_curator/stages/text/experimental/dripper/propagation_stage.py and nemo_curator/stages/text/experimental/dripper/stage.py (lines around the mapping_json serialization and iloc/loc indexing).

Important Files Changed

Filename	Overview
nemo_curator/stages/text/experimental/dripper/propagation_stage.py	New library stage for deferred template propagation; has two P1 bugs: iloc/loc indexing mismatch that silently reads wrong rows on non-default-indexed DataFrames, and json.loads cannot decode pickle+base64 mapping blobs from stage2b output.
nemo_curator/stages/text/experimental/dripper/stage.py	Large new file implementing the full Dripper pipeline stages; the deferred-propagation serialization path uses json.dumps(default=str) which stringifies tuple keys in html_element_dict, leaving bug #4 unfixed for the library integration path even though it was fixed in the tutorial scripts.
nemo_curator/stages/text/experimental/dripper/gpu_layout_clustering.py	New GPU-accelerated DBSCAN clustering module using cuML/cupy with sklearn fallback; logic is sound but accesses llm-webkit private internals via fragile name-mangling guesses.
nemo_curator/stages/text/experimental/dripper/init.py	New package init; exports 7 stages from stage.py but omits DripperHTMLLayoutPropagationStage from propagation_stage.py.
tutorials/text/dripper-common-crawl/stage2b_cpu_postprocess.py	Tutorial postprocess stage correctly uses pickle+base64 for lossless serialization of mapping templates with tuple keys; aligns with stage3 deserialization.
tutorials/text/dripper-common-crawl/stage3_cpu_propagation.py	Tutorial propagation stage with robust _parse_mapping_json that handles both pickle+base64 and legacy JSON; two-tier static/dynamic LBP with per-cluster validation is well-implemented.
nemo_curator/models/client/llm_client.py	Refactors retry/concurrency logic into a reusable _run_with_retry_and_concurrency helper and adds _coerce_generation_config; behavioral change from returning [] to raising RuntimeError on the unexpected-path is intentional and safer.
nemo_curator/models/client/openai_client.py	Adds OpenAIChatCompletionResult dataclass and query_model_with_usage methods to expose token usage counters; _usage_int correctly handles bool subclass-of-int edge case.
tests/stages/text/experimental/dripper/test_pipeline_correctness.py	39 pure-Python regression tests covering pickle+base64 round-trip, xpath parsing, HTML coercion, F1 metrics, and grep-based wiring guards; correctly locks in the four bug fixes.
tutorials/text/dripper-common-crawl/stage2_gpu_inference_offline.py	Offline-batched vLLM inference with per-GPU subprocess isolation, LPT slice balancing, and correct chat-template application; replaces the per-request Ray-Serve bottleneck.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["Stage 1a: CPU\nDOM feature extraction\n(get_feature, 64 workers)"] --> B["Stage 1b: GPU\ncuML DBSCAN clustering\n(pick cluster reps)"]
    B --> C["Stage 1c: CPU\nsimplify + build_prompt\n(reps + singletons only)"]
    C --> D["Stage 2: GPU\nOffline-batched vLLM\n(1 LLM.generate per GPU)"]
    D --> E["Stage 2b: CPU\npickle+base64 serialization ✓"]
    E --> F["Stage 3: CPU\nLayoutBatchParser propagation\n_parse_mapping_json ✓"]
    F --> G{propagation success?}
    G -->|yes| H["Output: dripper_content"]
    G -->|fallback| I["Stage 3b: LLM re-inference"]
    I --> H
    J["Library: DripperHTMLLayoutTemplateStage"] -->|"json.dumps default=str ❌"| K["DripperHTMLLayoutPropagationStage\niloc bug ❌"]
    K --> L["LayoutBatchParser fails"]
    style E fill:#d4edda
    style F fill:#d4edda
    style J fill:#f8d7da
    style K fill:#f8d7da
    style L fill:#f8d7da

Loading

_{Reviews (1): Last reviewed commit: "Fix stage1b GPU OOM: chunk oversized hos..." | Re-trigger Greptile}

[greptile-apps]

iloc vs loc indexing mismatch

df.index[pending_mask] yields label values, but the row is fetched via df.iloc[idx], which uses a positional integer offset, not a label. pd.Series always has a get method, so hasattr(df.iloc[idx], "get") is always True — the fallback branch (df.loc[idx]) is dead code. On any DataFrame whose index is not the default RangeIndex(0, N) (e.g. after filtering, concatenation, or re-indexing), df.iloc[idx] will silently return the wrong row or raise IndexError, while the label-based writes at lines 140–146 (df.at[idx, ...]) will correctly target the right row — producing corrupted output. The read should always use df.loc[idx].

Suggested change

	for idx in df.index[pending_mask]:
	row = df.iloc[idx] if hasattr(df.iloc[idx], "get") else df.loc[idx]
	for idx in df.index[pending_mask]:
	row = df.loc[idx]

[greptile-apps]

Bug #4 (tuple-key serialization) not fixed in the library integration path

The PR description identifies pickle+base64 as the fix for tuple keys being destroyed by json.dumps (Bug #4), and the tutorial scripts (stage2b_cpu_postprocess.py) correctly use base64.b64encode(pickle.dumps(template)). However, DripperHTMLLayoutTemplateStage still serializes via json.dumps(mapping_data, default=str), which stringifies every tuple key in html_element_dict (e.g. ("div", "class", "content") → "('div', 'class', 'content')"). The companion DripperHTMLLayoutPropagationStage.process() then deserializes with json.loads, so those keys arrive as strings rather than tuples, silently breaking LayoutBatchParser when the integrated library pipeline is used with defer_propagation=True. The fix applied to the tutorial scripts was not ported to these library-level stages.

[greptile-apps]

json.loads cannot decode pickle+base64 mapping blobs from stage2b

When DripperHTMLLayoutPropagationStage is used as a downstream stage of DripperHTMLLayoutTemplateStage with the current json.dumps serialization (see the sibling comment on stage.py), the mapping data is JSON and json.loads works. But if a user feeds the stage2b tutorial output (pickle+base64 format) into this library stage, json.loads will raise on every row and the bare except: pass will silently discard every mapping, causing no_mapping_data_for_cluster errors for 100% of rows. The library stage should use the same _parse_mapping_json helper from stage3_cpu_propagation.py that handles both formats (or at least document the accepted format explicitly).

[greptile-apps]

assert statements are disabled by the -O interpreter flag

Using bare assert for runtime precondition checks in library code is unsafe: Python's -O (optimize) mode removes all assert statements at compile time. If a caller constructs the stage without calling setup() first (e.g., in a worker process where setup() was expected via worker_metadata), these guards silently disappear and the code proceeds with None bindings, yielding opaque AttributeErrors later. Replace with explicit RuntimeError guards.

[greptile-apps]

DripperHTMLLayoutPropagationStage missing from public exports

propagation_stage.py introduces DripperHTMLLayoutPropagationStage as the CPU companion to DripperHTMLLayoutTemplateStage, but it is not imported or listed in __all__ here. Consumers of nemo_curator.stages.text.experimental.dripper cannot discover or import the new stage through the package's public surface without going directly to the submodule.

[greptile-apps]

Private name-mangling access is fragile across llm-webkit versions

_get_simp_features tries to locate the feature-vectorization helper by guessing multiple mangled names (_html_layout_cosin__simp_features, __simp_features, simp_features) via getattr. Python's name-mangling rule applies to class bodies only — a module-level __simp_features is not automatically mangled, so the string "_html_layout_cosin__simp_features" relies on an implementation detail of how the module was authored. Any refactoring or rename in llm_web_kit silently falls through to the third fallback name (simp_features) and then raises a RuntimeError, breaking all GPU clustering without a clear error message until runtime. Consider pinning the llm-webkit version or exposing this as a public helper in llm-webkit upstream.