[Optimize] Accelerate zlib crc32_z with PCLMULQDQ and VPCLMULQDQ on x86_64 platform

[liuweilw1024]

Add PCLMULQDQ and VPCLMULQDQ based CRC32 SIMD implementations for extra/zlib 1.2.13 on x86_64 platform, providing significant speedup over the default table-based computation.

• Add arch_x86.c/h for CPU feature detection (AVX, AVX512/VL)
• Add crc32_x86.h with SIMD CRC32 kernels using PCLMULQDQ
• Add crc32_x86_pclmul.c for AVX (128-bit) PCLMULQDQ path
• Add crc32_x86_vpclmulqdq.c for AVX512/VL VPCLMULQDQ path
• Modify crc32.c to dispatch to SIMD implementations at runtime
• Update CMakeLists.txt to include new x86_64 source files
• Add unit tests verifying SIMD CRC32 matches table-based reference

[sunjianhua1990s]

@liuweilw1024 Thanks for the optimization — the direction is right, the implementation is clean, and the tests are well covered. Before merging, could you add two things?

1. Evidence of performance necessity

• Micro-benchmarks: throughput of the new vs. the original table-based implementation across input sizes (e.g. 64B / 1KB / 16KB / 1MB), covering both the PCLMUL (128-bit) and VPCLMULQDQ (512-bit) paths, with the test CPU model noted.
• End-to-end gains (most important): under representative workloads (e.g. redo/binlog write-heavy, network compression enabled), use perf to show CRC32's share of overall CPU time, plus before/after TPS/QPS or latency. The function-level speedup is significant, but if CRC32 is only a small fraction of total CPU time the end-to-end benefit may be limited — data would help justify bringing this in.

2. Suggestion on maintenance

This change modifies the bundled extra/zlib copy directly, which means re-merging it on every future zlib upgrade — a long-term cost. I'd suggest first trying to upstream it to zlib: once accepted, we'd get it simply by following zlib releases, with zero extra maintenance. If upstream won't take it soon, please organize it as a minimal, well-isolated patch (separate files + a centralized dispatch hook) with its origin documented, to keep future zlib upgrades easy to merge.

[liuweilw1024]

@sunjianhua1990s Thanks for your comment. Here are some performance statistics data and the progress pushing code to the upstream community of zlib.

@liuweilw1024 Thanks for the optimization — the direction is right, the implementation is clean, and the tests are well covered. Before merging, could you add two things?

1. Evidence of performance necessity

• Micro-benchmarks: throughput of the new vs. the original table-based implementation across input sizes (e.g. 64B / 1KB / 16KB / 1MB), covering both the PCLMUL (128-bit) and VPCLMULQDQ (512-bit) paths, with the test CPU model noted.
• End-to-end gains (most important): under representative workloads (e.g. redo/binlog write-heavy, network compression enabled), use perf to show CRC32's share of overall CPU time, plus before/after TPS/QPS or latency. The function-level speedup is significant, but if CRC32 is only a small fraction of total CPU time the end-to-end benefit may be limited — data would help justify bringing this in.

[Response]
1、Comparison of average cost of crc32_z and the throughput.
On hygon 7490 platform which supports PCLMUL instructions but not VPCLMULQDQ, I have compare the original implementation(table-based) of crc32_z in zlib/crc32.c and the new PCLMUL-based implementation. Below is the comparison result. We can see PCLMUL can accelerate the THP of crc32 calculation by maximum 10 times.

By testing on hygon 7447v platform which support VPCLMULQDQ, we found it can accelerate the THP of crc32 calculation by maximum 20 times.

2、Performance gain in polardbx-engine
In flush stage of ordered commit, crc32_z is called in the backtrace of binlog_cache_mngr::flush() to calculate checksum for binlog data.
(1) The average cost of binlog_cache_mngr::flush() decreases by about 13% in sysbench oltp_read_write and oltp_write_only test items on hygon 7490 CPU. Benefiting from this, the cost of whole flush stage of ordered commit decreases to varying degrees. The maximum TPS will increase by 4% in oltp_read_write and 5% in oltp_write_only under running one polardbx-engine instance on 7490 single socket.


(2) The average cost of binlog_cache_mngr::flush() decreases by about 16% in sysbench oltp_read_write and oltp_write_only test items on hygon 7447v CPU. The maximum TPS will increase by 7% in oltp_read_write and 14.5% in oltp_write_only under running one polardbx-engine instance on 7447v single socket.

2. Suggestion on maintenance

This change modifies the bundled extra/zlib copy directly, which means re-merging it on every future zlib upgrade — a long-term cost. I'd suggest first trying to upstream it to zlib: once accepted, we'd get it simply by following zlib releases, with zero extra maintenance. If upstream won't take it soon, please organize it as a minimal, well-isolated patch (separate files + a centralized dispatch hook) with its origin documented, to keep future zlib upgrades easy to merge.

[Response]
Yes, it's a good suggestion to push to code to zlib upstream community for better maintenance and widely promote. Now we have just submitted a PR which is still under reviewing to the upstream zlib community: madler/zlib#1279. I will keep in track with the progress of merging and notify here.