Skip to content

Add backend abstraction for sketch implementations#207

Merged
milindsrivastava1997 merged 4 commits intomainfrom
sketchlib-backend-abstraction
Mar 20, 2026
Merged

Add backend abstraction for sketch implementations#207
milindsrivastava1997 merged 4 commits intomainfrom
sketchlib-backend-abstraction

Conversation

@GnaneshGnani
Copy link
Copy Markdown
Contributor

Summary

Introduces the backend abstraction layer (ImplMode) that enables switching between legacy hand-written sketch implementations and future sketchlib-rust backends. This PR adds the configuration infrastructure without yet integrating sketchlib-rust.

Changes

New Files

asap-common/sketch-core/src/config.rs

  • Defines ImplMode enum with Legacy and Sketchlib variants
  • Provides OnceLock-based global configuration for three sketch types:
    • Count-Min Sketch
    • KLL (quantile estimation)
    • Count-Min-With-Heap (top-k tracking)
  • configure() function to set backends once at process startup
  • Helper functions: use_sketchlib_for_count_min(), use_sketchlib_for_kll(), use_sketchlib_for_count_min_with_heap()
  • force_legacy_mode_for_tests() helper for test environments

asap-query-engine/tests/test_both_backends.rs

  • Integration test that automatically runs lib tests with both Legacy and Sketchlib backends
  • When running cargo test -p query_engine_rust, tests run with Legacy by default
  • This test spawns a second run with --features sketchlib-tests to exercise Sketchlib mode
  • Prevents regression by ensuring both backends work correctly

asap-common/sketch-core/src/bin/sketchlib_fidelity.rs

  • Skeleton fidelity testing binary with CLI argument parsing
  • Helper functions for metrics: Pearson correlation, MAPE, RMSE, rank fraction
  • RNG implementation for reproducible benchmarks
  • Will be expanded in subsequent PRs as sketch types are integrated

asap-common/sketch-core/report.md

  • Documentation for running fidelity tests
  • Instructions for backend selection via CLI flags
  • Placeholder for benchmark results (to be added in subsequent PRs)

Modified Files

asap-common/sketch-core/src/lib.rs

  • Added pub mod config;
  • Added test constructor to force legacy mode during tests (ensures existing tests continue to pass)

asap-common/sketch-core/Cargo.toml

  • Added clap for CLI parsing in the fidelity binary
  • Added ctor dev dependency for test initialization

asap-query-engine/src/main.rs

  • Added three new CLI arguments:
    • --sketch-cms-impl: Backend for Count-Min Sketch (default: sketchlib)
    • --sketch-kll-impl: Backend for KLL Sketch (default: sketchlib)
    • --sketch-cmwh-impl: Backend for Count-Min-With-Heap (default: sketchlib)
  • Calls config::configure() at startup before any sketch operations

asap-query-engine/src/lib.rs

  • Added test constructor that configures backends based on sketchlib-tests feature
  • Without feature: uses Legacy backend
  • With feature: uses Sketchlib backend

asap-query-engine/Cargo.toml

  • Added sketchlib-tests feature flag
  • Added ctor dev dependency for test initialization

Technical Approach

  • OnceLock Pattern: Thread-safe, one-time configuration prevents race conditions
  • Feature-Based Testing: sketchlib-tests feature allows testing both backends in a single cargo test invocation
  • No Runtime Overhead: Backend selection happens once at startup, zero overhead during operation
  • Fail-Fast: Configuration errors detected immediately at startup

Testing

# Build succeeds
cargo build

# Tests pass with legacy backend (default)
cargo test -p sketch-core
cargo test -p query_engine_rust

# Fidelity binary runs (no benchmarks yet)
cargo run -p sketch-core --bin sketchlib_fidelity

Copy link
Copy Markdown
Contributor

@milindsrivastava1997 milindsrivastava1997 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI is failing. Apart from this, and the one comment, LGTM

@GnaneshGnani GnaneshGnani force-pushed the sketchlib-backend-abstraction branch from ef13416 to c8f3b9e Compare March 20, 2026 13:37
@GnaneshGnani GnaneshGnani linked an issue Mar 20, 2026 that may be closed by this pull request
@milindsrivastava1997 milindsrivastava1997 merged commit e6e1858 into main Mar 20, 2026
4 checks passed
@milindsrivastava1997 milindsrivastava1997 deleted the sketchlib-backend-abstraction branch March 20, 2026 17:11
GnaneshGnani added a commit that referenced this pull request Mar 20, 2026
- Merged latest main (includes backend abstraction from PR #207)
- Set Count-Min Sketch to use sketchlib backend by default
- Keep KLL and Count-Min-With-Heap in legacy mode (not yet implemented)
- UDFs already correctly configured: CMS uses sketchlib
GnaneshGnani added a commit that referenced this pull request Mar 20, 2026
- Merged latest main (includes backend abstraction from PR #207)
- Set Count-Min Sketch to use sketchlib backend by default
- Set Count-Min-With-Heap to use sketchlib backend by default
- Keep KLL in legacy mode (not yet implemented)
- UDFs correctly configured: CMS and CMWH use sketchlib
GnaneshGnani added a commit that referenced this pull request Mar 20, 2026
- Merged latest main (includes backend abstraction from PR #207)
- Set all sketch backends to use sketchlib by default
- Count-Min, Count-Min-With-Heap, and KLL all use sketchlib
- UDFs correctly configured: all use sketchlib
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use sketchlib-rust sketches

2 participants