Perf optimizations + polars 0.54 upgrade (with benchmarks)

[ptournaris-ai]

Summary

Performance optimizations for the anonymization hot path, plus the polars 0.48 → 0.54 upgrade, with a reproducible benchmark harness.

Four commits, reviewable in order:

1. perf: config cache + vectorized sanitize + drop per-column clone
2. perf: scan parquet lazily from S3 with predicate/slice pushdown
3. deps: upgrade polars 0.48.1 → 0.54.4
4. bench: benchmark harnesses + metrics

Performance changes

• Memoize anonymization config — load_config_for did a disk read + TOML parse on every Parquet file; now parsed once and shared as an Arc (process-wide cache).
• Vectorize sanitize_null_bytes — replaced the row-by-row Vec<Option<String>> rebuild with a single columnar when/then/otherwise expression.
• Drop per-column Series clone — the transform loop now moves the produced Series into the DataFrame instead of cloning it.
• Lazy S3 scan with pushdown — replaced "download whole object into memory → ParquetReader::finish() → filter afterwards" with LazyFrame::scan_parquet(s3://…) that pushes the configured filter and keep_num_of_records limit into the read, so only matching row groups/columns are fetched and decoded. collect() runs in spawn_blocking. Credentials are resolved via the default aws-config chain (IRSA/IMDS/env/SSO) and handed to polars, matching get_object auth.

Measured (medians, 2M rows — see docs/BENCHMARKS.md)

Hot path	OLD	NEW	Speedup
sanitize_null_bytes	179 ms	19 ms	~9.4×
read + selective filter	17.0 ms	12.4 ms	~1.4×
read + keep_num_of_records	16.1 ms	4.0 ms	~4.0×
config load (per 64 files)	7.46 ms	1.53 µs	~4,900×

Each benchmark compares the exact OLD vs NEW function body; equivalence tests assert NEW is byte-identical to OLD before timings are trusted. The S3 scan path is additionally validated end-to-end against a local Floci/LocalStack S3 endpoint (benchmarks/s3-integration).

polars 0.48 → 0.54

Required API migration (mechanical): get_columns()→columns(), get_column_names_str()→get_column_names(), scan_parquet now takes PlRefPath, ChunkedArray .into_iter()→.iter(), with_column takes Column not Series, DataFrame::new(height, cols), AnyValue::Decimal(value, precision, scale), drop new_streaming/streaming features (add regex), MSRV → 1.88.

⚠️ Dependency note: the polars bump depends on dms-cdc-operator being on polars 0.54. Until nikoshet/rust-dms-cdc-operator#41 is merged & released, this branch pins it via [patch.crates-io] to a fork branch. Do not merge the polars bump commit until that release lands; the first two perf commits are safe on polars 0.48 today and can be split off if you prefer to ship them first.

The private submodules (rustic-bg-whole-table-transformator, rustic-local-data-importer-cli) were not reachable from this environment and still need the same mechanical polars-0.54 fixes applied.

Verification

cargo check --workspace --all-targets is clean on 0.54 (with submodules trimmed locally); 14 transform-crate unit tests and the microbench equivalence tests pass; the S3 integration test passes against Floci. DB/S3-backed integration tests that need the docker-compose fixtures were not run.

[ptournaris-ai]

Superseded by upstream PR bluegroundltd#43.