Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
4b3563d
Add standalone precompute engine to replace Arroyo streaming pipeline
zzylol Feb 21, 2026
898f1dd
Restore standalone precompute engine binary with fixed imports
zzylol Feb 23, 2026
c7cd8af
Wire up DatasketchesKLL in accumulator factory and add E2E test
zzylol Feb 23, 2026
8804d03
Add raw-only mode to PrecomputeEngine that skips aggregation
zzylol Feb 23, 2026
3ee048f
Add debug-level tracing spans for e2e latency measurement in precompu…
zzylol Feb 23, 2026
9e19d84
Add 1000-sample batch latency test to E2E precompute test
zzylol Feb 23, 2026
bdfc2ea
Add 10M-sample throughput test to E2E precompute test
zzylol Feb 23, 2026
d8f4a6c
update
zzylol Feb 24, 2026
b5c2eb8
Fix sliding window aggregation: feed samples into all overlapping win…
zzylol Feb 24, 2026
0045f98
Add late data handling for closed windows and precompute engine desig…
zzylol Feb 26, 2026
3feb7a0
Minor wording fix in precompute engine design doc
zzylol Feb 26, 2026
2218997
Clarify single-machine multi-threaded architecture in design doc
zzylol Feb 26, 2026
2ff938f
Add multi-connector ingest support (Prometheus + VictoriaMetrics)
zzylol Feb 26, 2026
222582b
Run cargo fmt
zzylol Feb 26, 2026
2b120e5
Add pane-based incremental sliding window computation
zzylol Feb 26, 2026
9301017
Add windowed aggregation benchmarks and fix build issues
zzylol Feb 26, 2026
5832572
Add scalability benchmark varying worker count (1-16)
zzylol Feb 26, 2026
a43f925
Make throughput test multi-threaded for data generation and sending
zzylol Feb 27, 2026
2ee9284
Increase throughput test drain timeout from 60s to 120s
zzylol Feb 27, 2026
8a9f83d
Fix PR405 reorg integration for ingest decode and config APIs
zzylol Mar 6, 2026
896304a
Fix precompute engine clippy warnings
zzylol Mar 6, 2026
c9090bf
Fix clippy lints in test_e2e_precompute
zzylol Mar 6, 2026
369836c
update
zzylol Mar 9, 2026
f43ee29
Add CapturingOutputSink and worker correctness unit tests
zzylol Mar 9, 2026
858b5f6
Update precompute engine design doc
zzylol Mar 9, 2026
e2f5f46
doc: explain sliding window behaviour with cross-worker GROUP BY
zzylol Mar 9, 2026
3353560
doc: document accumulator lifecycle and ownership in worker section
zzylol Mar 9, 2026
d7c7837
doc: document load balancing trade-offs and alternatives for SeriesRo…
zzylol Mar 9, 2026
d9c65d8
Fix misleading WindowManager struct comment
zzylol Mar 9, 2026
e8e9317
doc: add fault tolerance TODO section; fix rustfmt formatting in work…
zzylol Mar 9, 2026
f545e1c
Fix Clippy len_without_is_empty warning
zzylol Mar 9, 2026
194d6eb
test: add worker test that loads aggregation config from streaming_co…
zzylol Mar 9, 2026
88eb1a8
add unit test comparing precompute engine vs arroyosketch accumulator…
zzylol Mar 11, 2026
ea737ec
Format precompute worker tests
zzylol Mar 11, 2026
0104f8a
refactor: eliminate code and data duplication in precompute engine
zzylol Mar 11, 2026
6d1e70e
doc: fix second-tier merge worker design — correctness issues
zzylol Mar 11, 2026
08281dd
feat: add bench_precompute_sketch binary for precompute engine benchm…
zzylol Mar 11, 2026
e17b2ad
add test
zzylol Mar 11, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
233 changes: 147 additions & 86 deletions Cargo.lock

Large diffs are not rendered by default.

14 changes: 13 additions & 1 deletion asap-query-engine/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ flate2 = "1.0"
async-trait = "0.1"
xxhash-rust = { version = "0.8", features = ["xxh32", "xxh64"] }
dsrs = { git = "https://github.com/ProjectASAP/datasketches-rs" }
sketchlib-rust = { git = "https://github.com/ProjectASAP/sketchlib-rust" }
sketchlib-rust = { git = "https://github.com/ProjectASAP/sketchlib-rust", rev = "663a1df" }
base64 = "0.21"
hex = "0.4"
sqlparser = "0.59.0"
Expand All @@ -55,6 +55,18 @@ zstd = "0.13"
reqwest = { version = "0.11", features = ["json"] }
tracing-appender = "0.2"

[[bin]]
name = "precompute_engine"
path = "src/bin/precompute_engine.rs"

[[bin]]
name = "test_e2e_precompute"
path = "src/bin/test_e2e_precompute.rs"

[[bin]]
name = "bench_precompute_sketch"
path = "src/bin/bench_precompute_sketch.rs"

[dev-dependencies]
tempfile = "3.20.0"

Expand Down
8 changes: 6 additions & 2 deletions asap-query-engine/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,12 @@ COPY Cargo.toml ./
COPY Cargo.lock ./
COPY asap-query-engine/Cargo.toml ./asap-query-engine/

# Create a dummy main.rs to build dependencies
RUN mkdir -p asap-query-engine/src && echo "fn main() {}" > asap-query-engine/src/main.rs
# Create dummy Rust targets to build dependencies.
# Cargo.toml declares explicit bin paths, so they must exist in this cache layer.
RUN mkdir -p asap-query-engine/src/bin \
&& echo "fn main() {}" > asap-query-engine/src/main.rs \
&& echo "fn main() {}" > asap-query-engine/src/bin/precompute_engine.rs \
&& echo "fn main() {}" > asap-query-engine/src/bin/test_e2e_precompute.rs

# Build dependencies (this layer will be cached)
WORKDIR /code/asap-query-engine
Expand Down
Loading
Loading