WIP: Upgrade DataFusion to arrow-rs/parquet 58.0.0 / `object_store` 13.0.0 #19728

alamb · 2026-01-10T12:19:14Z

Which issue does this PR close?

Follow on to Upgrade DataFusion to arrow-rs/parquet 57.2.0 #19355
related to Release arrow-rs / parquet Major version 58.0.0 (January 2026) arrow-rs#8466
Closes [datafusion-spark] [SQL] [TEST] IntervalMonthDayNano(0,0,0) give line blank #17455

Oustanding issues

Regression on main (58): Parquet argument error: Parquet error: Required field type_ is missing arrow-rs#9315

Rationale for this change

Keep datafusion up to date (and test Arrow using DataFusion tests)

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

….2.0

datafusion/sqllogictest/test_files/spark/datetime/make_interval.slt

alamb · 2026-01-10T13:42:26Z

datafusion-cli/src/main.rs

        | alltypes_plain.parquet            | 1851            | 8882                | 2    | page_index=false |
-        | alltypes_tiny_pages.parquet       | 454233          | 269266              | 2    | page_index=true  |
-        | lz4_raw_compressed_larger.parquet | 380836          | 1347                | 2    | page_index=false |
+        | alltypes_tiny_pages.parquet       | 454233          | 269074              | 2    | page_index=true  |


I think this reduction in metadata size is a direct consequence of @WaterWhisperer's PR to improve PageEncoding representation

feat: change default behavior for Parquet PageEncodingStats to bitmask arrow-rs#9051

Dandandan · 2026-01-10T14:18:49Z

Run benchmarks

alamb-ghbot · 2026-01-10T17:46:31Z

🤖 ./gh_compare_branch.sh gh_compare_branch.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/update_arrow_58 (35b97fa) to b9a3b9f diff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

alamb-ghbot · 2026-01-10T18:26:55Z

🤖: Benchmark completed

Details

Comparing HEAD and alamb_update_arrow_58
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query    ┃        HEAD ┃ alamb_update_arrow_58 ┃    Change ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 0 │  2354.54 ms │            2406.77 ms │ no change │
│ QQuery 1 │   939.82 ms │             929.76 ms │ no change │
│ QQuery 2 │  1882.93 ms │            1891.80 ms │ no change │
│ QQuery 3 │  1163.47 ms │            1165.13 ms │ no change │
│ QQuery 4 │  2306.07 ms │            2241.55 ms │ no change │
│ QQuery 5 │ 28452.06 ms │           28080.09 ms │ no change │
│ QQuery 6 │  4018.71 ms │            4055.12 ms │ no change │
│ QQuery 7 │  3790.29 ms │            3670.14 ms │ no change │
└──────────┴─────────────┴───────────────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                    ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                    │ 44907.89ms │
│ Total Time (alamb_update_arrow_58)   │ 44440.36ms │
│ Average Time (HEAD)                  │  5613.49ms │
│ Average Time (alamb_update_arrow_58) │  5555.05ms │
│ Queries Faster                       │          0 │
│ Queries Slower                       │          0 │
│ Queries with No Change               │          8 │
│ Queries with Failure                 │          0 │
└──────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃        HEAD ┃ alamb_update_arrow_58 ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │     1.86 ms │               1.94 ms │     no change │
│ QQuery 1  │    50.56 ms │              50.42 ms │     no change │
│ QQuery 2  │   134.60 ms │             134.00 ms │     no change │
│ QQuery 3  │   156.94 ms │             152.15 ms │     no change │
│ QQuery 4  │  1074.92 ms │            1227.05 ms │  1.14x slower │
│ QQuery 5  │  1349.11 ms │            1512.35 ms │  1.12x slower │
│ QQuery 6  │     1.82 ms │               1.86 ms │     no change │
│ QQuery 7  │    56.21 ms │              53.80 ms │     no change │
│ QQuery 8  │  1404.44 ms │            1546.63 ms │  1.10x slower │
│ QQuery 9  │  1744.32 ms │            1884.58 ms │  1.08x slower │
│ QQuery 10 │   344.43 ms │             351.00 ms │     no change │
│ QQuery 11 │   393.88 ms │             399.97 ms │     no change │
│ QQuery 12 │  1259.31 ms │            1442.13 ms │  1.15x slower │
│ QQuery 13 │  1916.02 ms │            2053.79 ms │  1.07x slower │
│ QQuery 14 │  1228.80 ms │            1371.03 ms │  1.12x slower │
│ QQuery 15 │  1223.04 ms │            1359.64 ms │  1.11x slower │
│ QQuery 16 │  2568.94 ms │            2675.51 ms │     no change │
│ QQuery 17 │  2521.47 ms │            2647.03 ms │     no change │
│ QQuery 18 │  6120.19 ms │            5000.68 ms │ +1.22x faster │
│ QQuery 19 │   120.13 ms │             119.44 ms │     no change │
│ QQuery 20 │  1948.33 ms │            1896.06 ms │     no change │
│ QQuery 21 │  2245.14 ms │            2178.01 ms │     no change │
│ QQuery 22 │  3847.87 ms │            3802.23 ms │     no change │
│ QQuery 23 │ 21572.08 ms │           12373.92 ms │ +1.74x faster │
│ QQuery 24 │   225.72 ms │             216.64 ms │     no change │
│ QQuery 25 │   471.41 ms │             478.99 ms │     no change │
│ QQuery 26 │   218.83 ms │             235.13 ms │  1.07x slower │
│ QQuery 27 │  2823.98 ms │            2691.15 ms │     no change │
│ QQuery 28 │ 23860.88 ms │           23263.75 ms │     no change │
│ QQuery 29 │   982.83 ms │             966.29 ms │     no change │
│ QQuery 30 │  1343.19 ms │            1378.01 ms │     no change │
│ QQuery 31 │  1521.78 ms │            1379.20 ms │ +1.10x faster │
│ QQuery 32 │  5104.40 ms │            4890.24 ms │     no change │
│ QQuery 33 │  5837.64 ms │            5506.92 ms │ +1.06x faster │
│ QQuery 34 │  5933.39 ms │            5605.95 ms │ +1.06x faster │
│ QQuery 35 │  1921.56 ms │            1960.63 ms │     no change │
│ QQuery 36 │    65.96 ms │              68.18 ms │     no change │
│ QQuery 37 │    45.85 ms │              45.16 ms │     no change │
│ QQuery 38 │    66.53 ms │              65.89 ms │     no change │
│ QQuery 39 │   102.71 ms │             106.18 ms │     no change │
│ QQuery 40 │    28.23 ms │              25.52 ms │ +1.11x faster │
│ QQuery 41 │    24.10 ms │              23.51 ms │     no change │
│ QQuery 42 │    19.09 ms │              18.80 ms │     no change │
└───────────┴─────────────┴───────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Benchmark Summary                    ┃             ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ Total Time (HEAD)                    │ 103882.45ms │
│ Total Time (alamb_update_arrow_58)   │  93161.37ms │
│ Average Time (HEAD)                  │   2415.87ms │
│ Average Time (alamb_update_arrow_58) │   2166.54ms │
│ Queries Faster                       │           6 │
│ Queries Slower                       │           9 │
│ Queries with No Change               │          28 │
│ Queries with Failure                 │           0 │
└──────────────────────────────────────┴─────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃      HEAD ┃ alamb_update_arrow_58 ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1  │ 119.70 ms │             118.08 ms │     no change │
│ QQuery 2  │  35.82 ms │              36.60 ms │     no change │
│ QQuery 3  │  40.70 ms │              36.99 ms │ +1.10x faster │
│ QQuery 4  │  30.37 ms │              30.72 ms │     no change │
│ QQuery 5  │  91.59 ms │              90.47 ms │     no change │
│ QQuery 6  │  21.09 ms │              20.76 ms │     no change │
│ QQuery 7  │ 222.34 ms │             225.43 ms │     no change │
│ QQuery 8  │  44.85 ms │              40.29 ms │ +1.11x faster │
│ QQuery 9  │ 122.80 ms │             104.76 ms │ +1.17x faster │
│ QQuery 10 │  81.06 ms │              66.33 ms │ +1.22x faster │
│ QQuery 11 │  23.80 ms │              24.89 ms │     no change │
│ QQuery 12 │  52.22 ms │              52.78 ms │     no change │
│ QQuery 13 │  48.94 ms │              49.75 ms │     no change │
│ QQuery 14 │  14.95 ms │              14.96 ms │     no change │
│ QQuery 15 │  30.50 ms │              30.48 ms │     no change │
│ QQuery 16 │  28.47 ms │              29.06 ms │     no change │
│ QQuery 17 │ 151.61 ms │             154.03 ms │     no change │
│ QQuery 18 │ 287.03 ms │             288.27 ms │     no change │
│ QQuery 19 │  39.57 ms │              39.77 ms │     no change │
│ QQuery 20 │  57.69 ms │              57.36 ms │     no change │
│ QQuery 21 │ 322.81 ms │             318.54 ms │     no change │
│ QQuery 22 │  21.98 ms │              21.61 ms │     no change │
└───────────┴───────────┴───────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                    ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                    │ 1889.89ms │
│ Total Time (alamb_update_arrow_58)   │ 1851.95ms │
│ Average Time (HEAD)                  │   85.90ms │
│ Average Time (alamb_update_arrow_58) │   84.18ms │
│ Queries Faster                       │         4 │
│ Queries Slower                       │         0 │
│ Queries with No Change               │        18 │
│ Queries with Failure                 │         0 │
└──────────────────────────────────────┴───────────┘

Dandandan · 2026-01-10T18:32:08Z

run benchmarks

Dandandan · 2026-01-10T18:32:18Z

run benchmark tpch

alamb-ghbot · 2026-01-10T18:40:16Z

🤖 ./gh_compare_branch.sh gh_compare_branch.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/update_arrow_58 (35b97fa) to b9a3b9f diff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

alamb-ghbot · 2026-01-10T19:18:22Z

🤖: Benchmark completed

Details

Comparing HEAD and alamb_update_arrow_58
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query    ┃        HEAD ┃ alamb_update_arrow_58 ┃    Change ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 0 │  2389.61 ms │            2355.66 ms │ no change │
│ QQuery 1 │   975.65 ms │             969.78 ms │ no change │
│ QQuery 2 │  1875.49 ms │            1871.78 ms │ no change │
│ QQuery 3 │  1174.32 ms │            1145.53 ms │ no change │
│ QQuery 4 │  2258.94 ms │            2223.01 ms │ no change │
│ QQuery 5 │ 27824.57 ms │           28406.72 ms │ no change │
│ QQuery 6 │  4031.59 ms │            4002.29 ms │ no change │
│ QQuery 7 │  3463.56 ms │            3533.17 ms │ no change │
└──────────┴─────────────┴───────────────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                    ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                    │ 43993.73ms │
│ Total Time (alamb_update_arrow_58)   │ 44507.93ms │
│ Average Time (HEAD)                  │  5499.22ms │
│ Average Time (alamb_update_arrow_58) │  5563.49ms │
│ Queries Faster                       │          0 │
│ Queries Slower                       │          0 │
│ Queries with No Change               │          8 │
│ Queries with Failure                 │          0 │
└──────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃        HEAD ┃ alamb_update_arrow_58 ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │     1.90 ms │               1.92 ms │     no change │
│ QQuery 1  │    52.12 ms │              50.35 ms │     no change │
│ QQuery 2  │   142.92 ms │             133.95 ms │ +1.07x faster │
│ QQuery 3  │   154.00 ms │             152.77 ms │     no change │
│ QQuery 4  │  1072.51 ms │            1063.15 ms │     no change │
│ QQuery 5  │  1369.60 ms │            1320.07 ms │     no change │
│ QQuery 6  │     1.82 ms │               1.86 ms │     no change │
│ QQuery 7  │    54.39 ms │              52.98 ms │     no change │
│ QQuery 8  │  1446.73 ms │            1417.75 ms │     no change │
│ QQuery 9  │  1835.01 ms │            1706.51 ms │ +1.08x faster │
│ QQuery 10 │   344.52 ms │             354.29 ms │     no change │
│ QQuery 11 │   392.19 ms │             397.62 ms │     no change │
│ QQuery 12 │  1286.14 ms │            1275.41 ms │     no change │
│ QQuery 13 │  1947.40 ms │            1962.96 ms │     no change │
│ QQuery 14 │  1259.13 ms │            1231.11 ms │     no change │
│ QQuery 15 │  1247.90 ms │            1222.46 ms │     no change │
│ QQuery 16 │  2534.25 ms │            2515.84 ms │     no change │
│ QQuery 17 │  2514.25 ms │            2527.22 ms │     no change │
│ QQuery 18 │  5440.35 ms │            4806.73 ms │ +1.13x faster │
│ QQuery 19 │   119.26 ms │             118.76 ms │     no change │
│ QQuery 20 │  1929.28 ms │            1872.84 ms │     no change │
│ QQuery 21 │  2213.86 ms │            2178.92 ms │     no change │
│ QQuery 22 │  4264.32 ms │            3784.45 ms │ +1.13x faster │
│ QQuery 23 │ 20258.80 ms │           12236.28 ms │ +1.66x faster │
│ QQuery 24 │   215.95 ms │             219.20 ms │     no change │
│ QQuery 25 │   488.91 ms │             481.13 ms │     no change │
│ QQuery 26 │   226.35 ms │             216.76 ms │     no change │
│ QQuery 27 │  2799.99 ms │            2713.67 ms │     no change │
│ QQuery 28 │ 23840.67 ms │           23084.98 ms │     no change │
│ QQuery 29 │   974.43 ms │             971.18 ms │     no change │
│ QQuery 30 │  1351.52 ms │            1299.11 ms │     no change │
│ QQuery 31 │  1399.75 ms │            1421.76 ms │     no change │
│ QQuery 32 │  4761.26 ms │            4490.64 ms │ +1.06x faster │
│ QQuery 33 │  5521.96 ms │            5313.98 ms │     no change │
│ QQuery 34 │  5548.81 ms │            5584.56 ms │     no change │
│ QQuery 35 │  1979.74 ms │            1851.73 ms │ +1.07x faster │
│ QQuery 36 │    66.91 ms │              67.24 ms │     no change │
│ QQuery 37 │    46.16 ms │              47.24 ms │     no change │
│ QQuery 38 │    66.08 ms │              67.97 ms │     no change │
│ QQuery 39 │   102.12 ms │             100.78 ms │     no change │
│ QQuery 40 │    28.61 ms │              27.21 ms │     no change │
│ QQuery 41 │    23.45 ms │              22.02 ms │ +1.06x faster │
│ QQuery 42 │    19.24 ms │              19.38 ms │     no change │
└───────────┴─────────────┴───────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Benchmark Summary                    ┃             ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ Total Time (HEAD)                    │ 101344.52ms │
│ Total Time (alamb_update_arrow_58)   │  90386.74ms │
│ Average Time (HEAD)                  │   2356.85ms │
│ Average Time (alamb_update_arrow_58) │   2102.02ms │
│ Queries Faster                       │           8 │
│ Queries Slower                       │           0 │
│ Queries with No Change               │          35 │
│ Queries with Failure                 │           0 │
└──────────────────────────────────────┴─────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃      HEAD ┃ alamb_update_arrow_58 ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1  │ 118.62 ms │             117.02 ms │     no change │
│ QQuery 2  │  37.93 ms │              38.57 ms │     no change │
│ QQuery 3  │  41.97 ms │              37.52 ms │ +1.12x faster │
│ QQuery 4  │  31.39 ms │              30.22 ms │     no change │
│ QQuery 5  │  91.63 ms │              91.98 ms │     no change │
│ QQuery 6  │  21.13 ms │              20.89 ms │     no change │
│ QQuery 7  │ 229.95 ms │             228.06 ms │     no change │
│ QQuery 8  │  39.25 ms │              37.93 ms │     no change │
│ QQuery 9  │ 107.60 ms │             107.66 ms │     no change │
│ QQuery 10 │  69.45 ms │              69.75 ms │     no change │
│ QQuery 11 │  23.16 ms │              24.35 ms │  1.05x slower │
│ QQuery 12 │  53.07 ms │              52.75 ms │     no change │
│ QQuery 13 │  51.17 ms │              48.55 ms │ +1.05x faster │
│ QQuery 14 │  15.28 ms │              15.11 ms │     no change │
│ QQuery 15 │  30.14 ms │              30.73 ms │     no change │
│ QQuery 16 │  31.26 ms │              29.13 ms │ +1.07x faster │
│ QQuery 17 │ 159.46 ms │             154.29 ms │     no change │
│ QQuery 18 │ 287.59 ms │             284.81 ms │     no change │
│ QQuery 19 │  39.20 ms │              39.99 ms │     no change │
│ QQuery 20 │  56.56 ms │              57.03 ms │     no change │
│ QQuery 21 │ 300.14 ms │             325.49 ms │  1.08x slower │
│ QQuery 22 │  22.64 ms │              22.18 ms │     no change │
└───────────┴───────────┴───────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                    ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                    │ 1858.59ms │
│ Total Time (alamb_update_arrow_58)   │ 1864.00ms │
│ Average Time (HEAD)                  │   84.48ms │
│ Average Time (alamb_update_arrow_58) │   84.73ms │
│ Queries Faster                       │         3 │
│ Queries Slower                       │         2 │
│ Queries with No Change               │        17 │
│ Queries with Failure                 │         0 │
└──────────────────────────────────────┴───────────┘

alamb-ghbot · 2026-01-23T17:46:31Z

🤖: Benchmark completed

Details

Comparing HEAD and alamb_update_arrow_58
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃      HEAD ┃ alamb_update_arrow_58 ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1  │ 103.86 ms │             102.37 ms │     no change │
│ QQuery 2  │  34.24 ms │              31.66 ms │ +1.08x faster │
│ QQuery 3  │  41.84 ms │              37.83 ms │ +1.11x faster │
│ QQuery 4  │  31.74 ms │              30.69 ms │     no change │
│ QQuery 5  │  93.78 ms │              81.15 ms │ +1.16x faster │
│ QQuery 6  │  20.98 ms │              20.41 ms │     no change │
│ QQuery 7  │ 164.38 ms │             151.24 ms │ +1.09x faster │
│ QQuery 8  │  41.61 ms │              40.04 ms │     no change │
│ QQuery 9  │ 110.11 ms │             105.14 ms │     no change │
│ QQuery 10 │  68.03 ms │              64.96 ms │     no change │
│ QQuery 11 │  18.85 ms │              19.20 ms │     no change │
│ QQuery 12 │  52.25 ms │              56.12 ms │  1.07x slower │
│ QQuery 13 │  51.50 ms │              52.96 ms │     no change │
│ QQuery 14 │  15.48 ms │              15.11 ms │     no change │
│ QQuery 15 │  31.13 ms │              30.64 ms │     no change │
│ QQuery 16 │  29.45 ms │              27.64 ms │ +1.07x faster │
│ QQuery 17 │ 146.17 ms │             149.32 ms │     no change │
│ QQuery 18 │ 293.06 ms │             277.06 ms │ +1.06x faster │
│ QQuery 19 │  41.97 ms │              43.65 ms │     no change │
│ QQuery 20 │  56.57 ms │              56.52 ms │     no change │
│ QQuery 21 │ 191.41 ms │             194.18 ms │     no change │
│ QQuery 22 │  22.69 ms │              22.41 ms │     no change │
└───────────┴───────────┴───────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                    ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                    │ 1661.10ms │
│ Total Time (alamb_update_arrow_58)   │ 1610.31ms │
│ Average Time (HEAD)                  │   75.50ms │
│ Average Time (alamb_update_arrow_58) │   73.20ms │
│ Queries Faster                       │         6 │
│ Queries Slower                       │         1 │
│ Queries with No Change               │        15 │
│ Queries with Failure                 │         0 │
└──────────────────────────────────────┴───────────┘

alamb · 2026-01-30T21:32:27Z

datafusion-cli/src/object_storage/instrumented.rs

        let timestamp = Utc::now();
        let range = options.range.clone();

+        let head = options.head;


A substantial amount of the changes in tis PR are due to the upgrade to object_store 0.13 where several of the trait methods are consolidated (e.g. get, get_opts, head, etc) have been consolidated.

You can see the upgrade guide here

https://github.com/apache/arrow-rs-object-store/blob/8ef1aaa9cad5699971de1173d16637d01795da6e/src/lib.rs#L698-L736

Sadly, the docs.rs page is broken, and I have filed a ticket for that:

No doc build for latest object_store release ( 0.13.1) rust-lang/docs.rs#3173

alamb · 2026-01-30T21:53:38Z

datafusion-cli/src/object_storage/instrumented.rs

        self.inner.list_with_delimiter(prefix).await
    }

-    async fn copy(&self, from: &Path, to: &Path) -> Result<()> {


copy and copy_if_not_exists were consolidated

alamb · 2026-01-30T22:03:08Z

datafusion/core/tests/dataframe/mod.rs

    // Testing case1:
    // Inserting query schema mismatch: Expected table field 'a' with type Float16, but got 'a' with type Utf8.
-    // And the cast is not supported from Utf8 to Float16.
+    // And the cast is not supported from Binary to Float16.


@Jefffrey added support for Utf8->Float casting in apache/arrow-rs#9262 so this test started failing it expected this not to work 😆

alamb · 2026-01-30T22:13:44Z

run benchmarks

alamb-ghbot · 2026-01-30T22:13:51Z

🤖 ./gh_compare_branch.sh gh_compare_branch.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/update_arrow_58 (9276a8a) to b50e1f7 diff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

alamb-ghbot · 2026-01-30T22:54:24Z

🤖: Benchmark completed

Details

Comparing HEAD and alamb_update_arrow_58
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query    ┃        HEAD ┃ alamb_update_arrow_58 ┃        Change ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │  2300.55 ms │            2285.79 ms │     no change │
│ QQuery 1 │   888.82 ms │             848.38 ms │     no change │
│ QQuery 2 │  1838.33 ms │            1668.46 ms │ +1.10x faster │
│ QQuery 3 │  1062.70 ms │            1016.53 ms │     no change │
│ QQuery 4 │  2207.01 ms │            2149.28 ms │     no change │
│ QQuery 5 │ 28391.81 ms │           26709.42 ms │ +1.06x faster │
│ QQuery 6 │  4015.66 ms │            3798.09 ms │ +1.06x faster │
│ QQuery 7 │  2764.10 ms │            3129.12 ms │  1.13x slower │
└──────────┴─────────────┴───────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                    ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                    │ 43468.98ms │
│ Total Time (alamb_update_arrow_58)   │ 41605.09ms │
│ Average Time (HEAD)                  │  5433.62ms │
│ Average Time (alamb_update_arrow_58) │  5200.64ms │
│ Queries Faster                       │          3 │
│ Queries Slower                       │          1 │
│ Queries with No Change               │          4 │
│ Queries with Failure                 │          0 │
└──────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃        HEAD ┃ alamb_update_arrow_58 ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │     2.57 ms │               2.62 ms │     no change │
│ QQuery 1  │    52.29 ms │              51.35 ms │     no change │
│ QQuery 2  │   131.21 ms │             157.22 ms │  1.20x slower │
│ QQuery 3  │   158.00 ms │             160.65 ms │     no change │
│ QQuery 4  │  1003.75 ms │            1100.67 ms │  1.10x slower │
│ QQuery 5  │  1274.89 ms │            1292.35 ms │     no change │
│ QQuery 6  │     8.90 ms │               8.68 ms │     no change │
│ QQuery 7  │    57.40 ms │              54.76 ms │     no change │
│ QQuery 8  │  1404.34 ms │            1450.20 ms │     no change │
│ QQuery 9  │  1779.84 ms │            1810.15 ms │     no change │
│ QQuery 10 │   348.93 ms │             343.77 ms │     no change │
│ QQuery 11 │   402.87 ms │             389.47 ms │     no change │
│ QQuery 12 │  1217.33 ms │            1236.32 ms │     no change │
│ QQuery 13 │  1884.21 ms │            1897.57 ms │     no change │
│ QQuery 14 │  1227.78 ms │            1186.45 ms │     no change │
│ QQuery 15 │  1194.65 ms │            1209.51 ms │     no change │
│ QQuery 16 │  2474.11 ms │            2475.47 ms │     no change │
│ QQuery 17 │  2495.10 ms │            2460.91 ms │     no change │
│ QQuery 18 │  5788.51 ms │            4774.53 ms │ +1.21x faster │
│ QQuery 19 │   122.74 ms │             119.04 ms │     no change │
│ QQuery 20 │  1951.17 ms │            1869.84 ms │     no change │
│ QQuery 21 │  2180.84 ms │            2164.30 ms │     no change │
│ QQuery 22 │  3769.84 ms │            3670.68 ms │     no change │
│ QQuery 23 │ 21956.02 ms │           11740.22 ms │ +1.87x faster │
│ QQuery 24 │   225.99 ms │             205.29 ms │ +1.10x faster │
│ QQuery 25 │   492.66 ms │             440.34 ms │ +1.12x faster │
│ QQuery 26 │   242.76 ms │             214.04 ms │ +1.13x faster │
│ QQuery 27 │  2648.17 ms │            2645.02 ms │     no change │
│ QQuery 28 │ 24806.18 ms │           23211.80 ms │ +1.07x faster │
│ QQuery 29 │   975.12 ms │             996.28 ms │     no change │
│ QQuery 30 │  1285.64 ms │            1226.34 ms │     no change │
│ QQuery 31 │  1353.60 ms │            1302.63 ms │     no change │
│ QQuery 32 │  4604.07 ms │            4118.16 ms │ +1.12x faster │
│ QQuery 33 │  5724.34 ms │            5290.62 ms │ +1.08x faster │
│ QQuery 34 │  6090.23 ms │            5758.74 ms │ +1.06x faster │
│ QQuery 35 │  1916.42 ms │            1903.83 ms │     no change │
│ QQuery 36 │   195.79 ms │             200.93 ms │     no change │
│ QQuery 37 │    80.83 ms │              76.20 ms │ +1.06x faster │
│ QQuery 38 │   121.70 ms │             120.20 ms │     no change │
│ QQuery 39 │   363.07 ms │             346.26 ms │     no change │
│ QQuery 40 │    47.72 ms │              45.95 ms │     no change │
│ QQuery 41 │    41.43 ms │              39.62 ms │     no change │
│ QQuery 42 │    35.37 ms │              35.41 ms │     no change │
└───────────┴─────────────┴───────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Benchmark Summary                    ┃             ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ Total Time (HEAD)                    │ 104138.38ms │
│ Total Time (alamb_update_arrow_58)   │  89804.36ms │
│ Average Time (HEAD)                  │   2421.82ms │
│ Average Time (alamb_update_arrow_58) │   2088.47ms │
│ Queries Faster                       │          10 │
│ Queries Slower                       │           2 │
│ Queries with No Change               │          31 │
│ Queries with Failure                 │           0 │
└──────────────────────────────────────┴─────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃      HEAD ┃ alamb_update_arrow_58 ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1  │ 127.36 ms │             101.81 ms │ +1.25x faster │
│ QQuery 2  │  34.57 ms │              30.76 ms │ +1.12x faster │
│ QQuery 3  │  43.88 ms │              34.21 ms │ +1.28x faster │
│ QQuery 4  │  30.62 ms │              30.60 ms │     no change │
│ QQuery 5  │  89.42 ms │              82.88 ms │ +1.08x faster │
│ QQuery 6  │  20.55 ms │              20.22 ms │     no change │
│ QQuery 7  │ 162.38 ms │             143.91 ms │ +1.13x faster │
│ QQuery 8  │  38.16 ms │              37.29 ms │     no change │
│ QQuery 9  │ 101.72 ms │             100.50 ms │     no change │
│ QQuery 10 │  67.45 ms │              62.15 ms │ +1.09x faster │
│ QQuery 11 │  19.52 ms │              17.23 ms │ +1.13x faster │
│ QQuery 12 │  51.35 ms │              54.04 ms │  1.05x slower │
│ QQuery 13 │  48.96 ms │              46.76 ms │     no change │
│ QQuery 14 │  15.54 ms │              14.75 ms │ +1.05x faster │
│ QQuery 15 │  30.39 ms │              29.28 ms │     no change │
│ QQuery 16 │  27.91 ms │              27.69 ms │     no change │
│ QQuery 17 │ 143.51 ms │             145.64 ms │     no change │
│ QQuery 18 │ 288.04 ms │             267.65 ms │ +1.08x faster │
│ QQuery 19 │  39.53 ms │              40.14 ms │     no change │
│ QQuery 20 │  56.22 ms │              55.93 ms │     no change │
│ QQuery 21 │ 184.76 ms │             193.64 ms │     no change │
│ QQuery 22 │  22.88 ms │              21.85 ms │     no change │
└───────────┴───────────┴───────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                    ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                    │ 1644.73ms │
│ Total Time (alamb_update_arrow_58)   │ 1558.92ms │
│ Average Time (HEAD)                  │   74.76ms │
│ Average Time (alamb_update_arrow_58) │   70.86ms │
│ Queries Faster                       │         9 │
│ Queries Slower                       │         1 │
│ Queries with No Change               │        12 │
│ Queries with Failure                 │         0 │
└──────────────────────────────────────┴───────────┘

alamb added 10 commits January 6, 2026 17:55

Update to pre-release arrow

120d2b0

Fix clippy due to UnionFields::new is deprecated

40a6a71

Fix real bug in test

f50b9c3

Update test for fixed issue upstream

87563d8

Update tests for new feature

4afd46f

Update tests for struct casting

70e2841

Update to latest

b29b662

Merge remote-tracking branch 'apache/main' into alamb/update_arrow_57…

6a2e631

….2.0

Merge branch 'main' into alamb/update_arrow_57.2.0

ca11966

Update to pre-release main

63a85eb

github-actions bot added sqllogictest SQL Logic Tests (.slt) common Related to common crate proto Related to proto crate functions Changes to functions implementation datasource Changes to the datasource crate physical-plan Changes to the physical-plan crate labels Jan 10, 2026

Update expected value of make_interval

277aac4

alamb commented Jan 10, 2026

View reviewed changes

datafusion/sqllogictest/test_files/spark/datetime/make_interval.slt Show resolved Hide resolved

Update expected output

bb29b0c

alamb commented Jan 10, 2026

View reviewed changes

alamb mentioned this pull request Jan 10, 2026

feat: change default behavior for Parquet PageEncodingStats to bitmask apache/arrow-rs#9051

Merged

Update to latest arrow

35b97fa

alamb mentioned this pull request Jan 10, 2026

fix: liberal parsing of zero scale decimals apache/arrow-rs#8700

Merged

alamb mentioned this pull request Jan 27, 2026

Add ScalarValue::RunEndEncoded variant #19895

Open

alamb added 3 commits January 30, 2026 14:04

Merge remote-tracking branch 'apache/main' into alamb/update_arrow_58

d79a8a0

Update to latest

21fabb5

Update for deprecated

36cfd5e

alamb changed the title ~~WIP: Upgrade DataFusion to arrow-rs/parquet 58.0.0~~ WIP: Upgrade DataFusion to arrow-rs/parquet 58.0.0 / object_store 13.0.0 Jan 30, 2026

alamb added 8 commits January 30, 2026 15:03

Update to object_store 0.13.0

c273703

Update more

855b51b

More updates

bdd19a6

updates

0d0cc96

cleanups

ccf7a14

remove uneeded code

1a65bf7

Merge remote-tracking branch 'apache/main' into alamb/update_arrow_58

206de80

update

778ac16

github-actions bot added core Core DataFusion crate datasource Changes to the datasource crate spark labels Jan 30, 2026

alamb added 4 commits January 30, 2026 15:56

Remove old comments

ddd778d

fixup request counting

afb5ba3

fix delete

0622248

clippy

e4449fd

alamb mentioned this pull request Jan 30, 2026

Regression on main (58): Parquet argument error: Parquet error: Required field type_ is missing apache/arrow-rs#9315

Open

alamb added 3 commits January 30, 2026 16:49

Upate tests and for clippy

987f879

more clipy

70a958a

Update test as now utf8->float is supported

5452d50

alamb commented Jan 30, 2026

View reviewed changes

fmt

9276a8a

WIP: Upgrade DataFusion to arrow-rs/parquet 58.0.0 / object_store 13.0.0 #19728

Are you sure you want to change the base?

WIP: Upgrade DataFusion to arrow-rs/parquet 58.0.0 / object_store 13.0.0 #19728

Conversation

alamb commented Jan 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Uh oh!

alamb Jan 10, 2026

Choose a reason for hiding this comment

Uh oh!

Dandandan commented Jan 10, 2026

Uh oh!

alamb-ghbot commented Jan 10, 2026

Uh oh!

alamb-ghbot commented Jan 10, 2026

Uh oh!

Dandandan commented Jan 10, 2026

Uh oh!

Dandandan commented Jan 10, 2026

Uh oh!

alamb-ghbot commented Jan 10, 2026

Uh oh!

alamb-ghbot commented Jan 10, 2026

Uh oh!

alamb-ghbot commented Jan 23, 2026

Uh oh!

alamb Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

alamb commented Jan 30, 2026

Uh oh!

alamb-ghbot commented Jan 30, 2026

Uh oh!

alamb-ghbot commented Jan 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

WIP: Upgrade DataFusion to arrow-rs/parquet 58.0.0 / `object_store` 13.0.0 #19728

WIP: Upgrade DataFusion to arrow-rs/parquet 58.0.0 / `object_store` 13.0.0 #19728

alamb commented Jan 10, 2026 •

edited

Loading