Is your feature request related to a problem? Please describe.
The libcudf ORC reader recognizes the ORC LZO compression kind in the file footer, but rejects it because is_supported_read_orc does not include compression_type::LZO in its supported decompression codecs.
On the current main branch:
orc_decompressor maps the ORC footer's LZO value to compression_type::LZO, then fails with Unsupported compression type for ORC reader:
|
case ZLIB: |
|
_compression = compression_type::ZLIB; |
|
m_log2MaxRatio = 11; // < 2048:1 |
|
break; |
|
case SNAPPY: |
|
_compression = compression_type::SNAPPY; |
|
m_log2MaxRatio = 5; // < 32:1 |
|
break; |
|
case LZO: _compression = compression_type::LZO; break; |
|
case LZ4: _compression = compression_type::LZ4; break; |
|
case ZSTD: |
|
m_log2MaxRatio = 15; |
|
_compression = compression_type::ZSTD; |
|
break; |
|
default: CUDF_FAIL("Invalid compression type"); |
|
} |
|
CUDF_EXPECTS(is_supported_read_orc(_compression), "Unsupported compression type for ORC reader"); |
is_supported_read_orc currently allows ZLIB, SNAPPY, ZSTD, and LZ4, but not LZO:
|
bool is_supported_read_orc(compression_type compression) |
|
{ |
|
if (compression == compression_type::AUTO or compression == compression_type::NONE) { |
|
return true; |
|
} |
|
|
|
return ((compression == compression_type::ZLIB or compression == compression_type::SNAPPY or |
|
compression == compression_type::ZSTD or compression == compression_type::LZ4) and |
|
detail::is_decompression_supported(compression)); |
This is observable through cudf-spark. Spark 3.3 can write and read an LZO-compressed ORC dataset on CPU. With RAPIDS enabled, the read is planned as a GPU ORC scan and fails deterministically in libcudf.
The previously executed cudf-spark reproduction captured the following GPU/CPU result from the same Spark session:
RAPIDS Enabled = YES
Plugins Loaded = YES
GPU Operators: YES
SUMMARY_CASE_ID = ORC_LZO
GPU_STATUS = EXCEPTION
GPU_EXCEPTION_CLASS = org.apache.spark.SparkException
GPU_ROOT_CAUSE_CLASS = ai.rapids.cudf.CudfException
GPU_ROOT_CAUSE_MESSAGE = CUDF failure at: ../../../thirdparty/cudf/cpp/src/io/orc/orc.cpp:413: Unsupported compression type for ORC reader
CPU_STATUS = SUCCESS
CPU_EXCEPTION_CLASS = <none>
CPU_ROOT_CAUSE_CLASS = <none>
CPU_ROOT_CAUSE_MESSAGE = <none>
The executed GPU stack contains OrcTableReader -> MakeOrcTableProducer -> MultiFileOrcPartitionReader. The full reproduction command, Spark code, environment, and output are recorded in NVIDIA/cudf-spark#15551.
Describe the solution you'd like
Add LZO decompression support to the libcudf ORC reader and include ORC reader tests using LZO-compressed inputs. Once implemented, is_supported_read_orc(compression_type::LZO) should report support and LZO-compressed ORC data should produce the same results as a CPU ORC reader.
Describe alternatives you've considered
cudf-spark could inspect the ORC compression kind before selecting the GPU reader and fall back to Spark's CPU ORC reader for LZO files. That would avoid the runtime failure, but it would not provide GPU-accelerated reads and would leave LZO as a libcudf ORC compatibility gap.
Additional context
Is your feature request related to a problem? Please describe.
The libcudf ORC reader recognizes the ORC
LZOcompression kind in the file footer, but rejects it becauseis_supported_read_orcdoes not includecompression_type::LZOin its supported decompression codecs.On the current
mainbranch:orc_decompressormaps the ORC footer'sLZOvalue tocompression_type::LZO, then fails withUnsupported compression type for ORC reader:cudf/cpp/src/io/orc/orc.cpp
Lines 397 to 413 in e44e3cf
is_supported_read_orccurrently allows ZLIB, SNAPPY, ZSTD, and LZ4, but not LZO:cudf/cpp/src/io/functions.cpp
Lines 326 to 334 in e44e3cf
This is observable through cudf-spark. Spark 3.3 can write and read an LZO-compressed ORC dataset on CPU. With RAPIDS enabled, the read is planned as a GPU ORC scan and fails deterministically in libcudf.
The previously executed cudf-spark reproduction captured the following GPU/CPU result from the same Spark session:
The executed GPU stack contains
OrcTableReader -> MakeOrcTableProducer -> MultiFileOrcPartitionReader. The full reproduction command, Spark code, environment, and output are recorded in NVIDIA/cudf-spark#15551.Describe the solution you'd like
Add LZO decompression support to the libcudf ORC reader and include ORC reader tests using LZO-compressed inputs. Once implemented,
is_supported_read_orc(compression_type::LZO)should report support and LZO-compressed ORC data should produce the same results as a CPU ORC reader.Describe alternatives you've considered
cudf-spark could inspect the ORC compression kind before selecting the GPU reader and fall back to Spark's CPU ORC reader for LZO files. That would avoid the runtime failure, but it would not provide GPU-accelerated reads and would leave LZO as a libcudf ORC compatibility gap.
Additional context
write and read - file source orc - codec: lzoinFileSourceCodecSuite