[BUG] Parquet reader reads nearly the entire file when the page index has offset indexes but no column indexes

[a-hirota]

Describe the bug

cudf.read_parquet / cudf::io::read_parquet can read nearly the entire file when it contains at least one BYTE_ARRAY column and its page index contains offset indexes but no column indexes. This happens even when only one row group is requested. PyArrow produces this layout with write_page_index=True, write_statistics=False; parquet-rs can also produce offset-index-only files.

In the reproducer below, requesting one of 16 row groups from a 121.73 MB file produces a 129.46 MB increase in process read-syscall bytes, compared with 7.75 MB when both index types are present. The extra read occurs while loading the page index during reader construction.

Steps/Code to reproduce bug

import os, tempfile
import numpy as np, pyarrow as pa, pyarrow.parquet as pq
import cudf

def rchar():  # bytes read by this process through read syscalls
    with open("/proc/self/io") as f:
        return int(next(l for l in f if l.startswith("rchar:")).split()[1])

n = 8_000_000
tbl = pa.table({
    "k": np.arange(n, dtype="int64"),
    "v": np.random.default_rng(0).integers(0, 1_000_000, n),
    "s": pa.array(np.char.mod("str-%08d", np.arange(n) % 500_000)),  # a string column
})
d = tempfile.mkdtemp()
variants = {
    "no_page_index":       dict(write_page_index=False, write_statistics=True),
    "column+offset_index": dict(write_page_index=True,  write_statistics=True),
    "offset_index_only":   dict(write_page_index=True,  write_statistics=False),
}
for name, kw in variants.items():
    path = os.path.join(d, name + ".parquet")
    pq.write_table(tbl, path, row_group_size=n // 16, **kw)
    c0 = pq.read_metadata(path).row_group(0).column(0)
    before = rchar()
    df = cudf.read_parquet(path, row_groups=[[0]])
    delta = rchar() - before
    print(f"{name:20s} file={os.path.getsize(path)/1e6:7.2f} MB  rg0.col0 has_column_index={c0.has_column_index}"
          f" has_offset_index={c0.has_offset_index}  rows={len(df)}  rchar delta={delta/1e6:7.2f} MB")

Observed measurements (cuDF 26.08.00, PyArrow 21.0.0):

no_page_index        file= 121.73 MB  rg0.col0 has_column_index=False has_offset_index=False  rows=500000  rchar delta=   9.99 MB
column+offset_index  file= 121.74 MB  rg0.col0 has_column_index=True has_offset_index=True  rows=500000  rchar delta=   7.75 MB
offset_index_only    file= 121.73 MB  rg0.col0 has_column_index=False has_offset_index=True  rows=500000  rchar delta= 129.46 MB

rchar delta is the change in Linux /proc/self/io rchar: bytes returned by read-family system calls for the process, including reads served from the page cache. It is not a measurement of physical storage reads and can include other reads made by the process.

The same behavior was observed using libcudf directly (C++ read_parquet with row_groups({{0}}) on a 67.8 MB pyarrow file): 1.36 MB read without page index vs 69.66 MB with offset-index-only; strace shows 4 MiB preads from offset 0 through nearly the entire file before the requested column chunk is read. On a 303 GB TPC-H orders.parquet written this way, each reader construction reads ~303 GB to return one row group.

Expected behavior

Loading the page index should read only its byte range. Reading a selected row group should require the file metadata, page indexes, and selected column chunks, without also reading unrelated data from the beginning of the file.

Environment overview

• Environment location: Bare-metal (NVIDIA GB10 / DGX Spark, aarch64)
• Method of cuDF install: conda (cudf/libcudf 26.08.00 nightly, cuda-version 13.3, driver 580.95.05)
• PyArrow version: 21.0.0

Additional context

Root cause

metadata::metadata() loads the page index with one host_read, starting at the first row group's first column's column_index_offset (reader_impl_helpers.cpp at 60436a8):

  if (read_page_indexes and has_strings and not row_groups.empty() and
      not row_groups.front().columns.empty()) {
    // column index and offset index are encoded back to back.
    // the first column of the first row group will have the first column index, the last
    // column of the last row group will have the final offset index.
    int64_t const min_offset = row_groups.front().columns.front().column_index_offset;
    auto const& last_col     = row_groups.back().columns.back();
    int64_t const max_offset = last_col.offset_index_offset + last_col.offset_index_length;

    if (max_offset > min_offset) {
      size_t const length     = max_offset - min_offset;
      auto const page_idx_buf = source->host_read(min_offset, length);
      setup_page_index({page_idx_buf->data(), length}, min_offset);
    }
  }

When no column index was written, column_index_offset is unset (0), so min_offset == 0. The read spans from byte 0 to the end of the last offset index, which covers nearly the entire file in the reproduced layout.

setup_page_index() already checks that each index's offset and length are positive before parsing it. The missing check is in the byte-range calculation above. When metadata is loaded from the source, this extra read occurs once per reader construction, including repeated read_parquet calls and new chunked_parquet_reader instances.

Related changes and release history

• Use page statistics in Parquet reader #14973, included in v24.04.00, replaced guarded per-index reads with a single range starting at the first column chunk's column_index_offset. Source history points to this change as the introduction of the problematic range calculation. v24.02.00 read each existing index separately.
• Remove unnecessary work from read_parquet_metadata #20180, included in v25.12.00, skips page-index reads for read_parquet_metadata. It leaves the range calculation unchanged for regular reads.
• Relax parquet page index requirements in hybrid scan #23386 adds an offset-index fallback to the experimental hybrid scan reader on main. Its page_index_byte_range() helper checks the first and last column chunks when computing the range. The standard reader still lacks this fallback. The hybrid scan fallback is also absent from the v26.08.00 and v26.08.01 release tags.

The problematic standard-reader calculation is present in v26.08.01 and in main at 60436a8 (2026-09-05). These historical version comparisons are based on source inspection, not execution of the reproducer on older releases.

Suggested fix

Use the same validated fallback as the hybrid scan helper: for the first column chunk, use its column-index offset if present, otherwise its offset-index offset; for the last column chunk, use the end of its offset index if present, otherwise the end of its column index. Index presence should require positive offset and length, and the resulting range should require min_offset > 0 and max_offset > min_offset. A shared helper could keep the two readers consistent.