Skip to content

[FEA] Add LZO decompression support to the ORC reader #24051

Description

@wjxiz1992

Is your feature request related to a problem? Please describe.

The libcudf ORC reader recognizes the ORC LZO compression kind in the file footer, but rejects it because is_supported_read_orc does not include compression_type::LZO in its supported decompression codecs.

On the current main branch:

  • orc_decompressor maps the ORC footer's LZO value to compression_type::LZO, then fails with Unsupported compression type for ORC reader:

    cudf/cpp/src/io/orc/orc.cpp

    Lines 397 to 413 in e44e3cf

    case ZLIB:
    _compression = compression_type::ZLIB;
    m_log2MaxRatio = 11; // < 2048:1
    break;
    case SNAPPY:
    _compression = compression_type::SNAPPY;
    m_log2MaxRatio = 5; // < 32:1
    break;
    case LZO: _compression = compression_type::LZO; break;
    case LZ4: _compression = compression_type::LZ4; break;
    case ZSTD:
    m_log2MaxRatio = 15;
    _compression = compression_type::ZSTD;
    break;
    default: CUDF_FAIL("Invalid compression type");
    }
    CUDF_EXPECTS(is_supported_read_orc(_compression), "Unsupported compression type for ORC reader");
  • is_supported_read_orc currently allows ZLIB, SNAPPY, ZSTD, and LZ4, but not LZO:
    bool is_supported_read_orc(compression_type compression)
    {
    if (compression == compression_type::AUTO or compression == compression_type::NONE) {
    return true;
    }
    return ((compression == compression_type::ZLIB or compression == compression_type::SNAPPY or
    compression == compression_type::ZSTD or compression == compression_type::LZ4) and
    detail::is_decompression_supported(compression));

This is observable through cudf-spark. Spark 3.3 can write and read an LZO-compressed ORC dataset on CPU. With RAPIDS enabled, the read is planned as a GPU ORC scan and fails deterministically in libcudf.

The previously executed cudf-spark reproduction captured the following GPU/CPU result from the same Spark session:

RAPIDS Enabled = YES
Plugins Loaded = YES
GPU Operators: YES

SUMMARY_CASE_ID = ORC_LZO
GPU_STATUS = EXCEPTION
GPU_EXCEPTION_CLASS = org.apache.spark.SparkException
GPU_ROOT_CAUSE_CLASS = ai.rapids.cudf.CudfException
GPU_ROOT_CAUSE_MESSAGE = CUDF failure at: ../../../thirdparty/cudf/cpp/src/io/orc/orc.cpp:413: Unsupported compression type for ORC reader
CPU_STATUS = SUCCESS
CPU_EXCEPTION_CLASS = <none>
CPU_ROOT_CAUSE_CLASS = <none>
CPU_ROOT_CAUSE_MESSAGE = <none>

The executed GPU stack contains OrcTableReader -> MakeOrcTableProducer -> MultiFileOrcPartitionReader. The full reproduction command, Spark code, environment, and output are recorded in NVIDIA/cudf-spark#15551.

Describe the solution you'd like

Add LZO decompression support to the libcudf ORC reader and include ORC reader tests using LZO-compressed inputs. Once implemented, is_supported_read_orc(compression_type::LZO) should report support and LZO-compressed ORC data should produce the same results as a CPU ORC reader.

Describe alternatives you've considered

cudf-spark could inspect the ORC compression kind before selecting the GPU reader and fall back to Spark's CPU ORC reader for LZO files. That would avoid the runtime failure, but it would not provide GPU-accelerated reads and would leave LZO as a libcudf ORC compatibility gap.

Additional context

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ? - Needs TriageSparkFunctionality that helps Spark RAPIDScuIOcuIO issuefeature requestNew feature or requestlibcudfAffects libcudf (C++/CUDA) code.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions