From 9385736ad48c689d87d4d620c93ecc611a9656ea Mon Sep 17 00:00:00 2001 From: Milind Srivastava Date: Tue, 24 Mar 2026 13:57:25 -0400 Subject: [PATCH] Updated docs to rename asap-sketch-ingest to asap-summary-ingest --- README.md | 8 ++++---- .../asap_query_latency/README.md | 8 ++++---- docs/01-getting-started/architecture.md | 6 +++--- docs/02-components/README.md | 18 +++++++++--------- .../development/adapt-pr-after-reorg.md | 6 +++--- .../development/add-new-sketch.md | 14 +++++++------- .../operations/manual-stack-run-clickhouse.md | 6 +++--- .../operations/manual-stack-run-prometheus.md | 6 +++--- .../operations/publishing-docker-images.md | 2 +- docs/README.md | 4 ++-- 10 files changed, 39 insertions(+), 39 deletions(-) diff --git a/README.md b/README.md index ec3aa34..163d76f 100644 --- a/README.md +++ b/README.md @@ -44,12 +44,12 @@ ASAPQuery uses **streaming sketches** to: ## Architecture -ASAPQuery has four main components: the **asap-planner-rs** generates sketch configurations from your query workload, **asap-sketch-ingest** deploys streaming pipelines in **Arroyo** that continuously build sketches from live Prometheus metrics, and **asap-query-engine** intercepts PromQL queries and serves them from those pre-computed sketches. +ASAPQuery has four main components: the **asap-planner-rs** generates sketch configurations from your query workload, **asap-summary-ingest** deploys streaming pipelines in **Arroyo** that continuously build sketches from live Prometheus metrics, and **asap-query-engine** intercepts PromQL queries and serves them from those pre-computed sketches. ### Components -- **[asap-planner-rs](asap-planner-rs/)** - Analyzes a PromQL query workload and auto-generates sketch configurations for asap-sketch-ingest and asap-query-engine -- **[asap-sketch-ingest](asap-sketch-ingest/)** - Deploys Arroyo streaming pipelines that continuously compute and publish sketches from live metrics +- **[asap-planner-rs](asap-planner-rs/)** - Analyzes a PromQL query workload and auto-generates sketch configurations for asap-summary-ingest and asap-query-engine +- **[asap-summary-ingest](asap-summary-ingest/)** - Deploys Arroyo streaming pipelines that continuously compute and publish sketches from live metrics - **[arroyo](https://github.com/ProjectASAP/arroyo)** - Fork of the [Arroyo](https://github.com/ArroyoSystems/arroyo) stream processing engine that runs the sketch-building SQL pipelines - **[asap-query-engine](asap-query-engine/)** - Intercepts incoming PromQL queries and serves them from pre-computed sketches, falling back to Prometheus for unsupported queries @@ -58,7 +58,7 @@ ASAPQuery has four main components: the **asap-planner-rs** generates sketch con ``` ├── asap-quickstart/ # Self-contained demo (start here!) ├── asap-planner-rs/ # Auto-configuration service -├── asap-sketch-ingest/ # Arroyo pipeline deployer +├── asap-summary-ingest/ # Arroyo pipeline deployer └── asap-query-engine/ # Query serving engine # Note: Arroyo fork lives at https://github.com/ProjectASAP/arroyo ``` diff --git a/asap-tools/execution-utilities/asap_query_latency/README.md b/asap-tools/execution-utilities/asap_query_latency/README.md index 7fc17bd..a10adda 100644 --- a/asap-tools/execution-utilities/asap_query_latency/README.md +++ b/asap-tools/execution-utilities/asap_query_latency/README.md @@ -96,15 +96,15 @@ CLICKHOUSE_BIN=$INSTALL_DIR/clickhouse bash scripts/init_clickhouse.sh ### Step 5 — Start Arroyo cluster ```bash -~/ASAPQuery/asap-sketch-ingest/target/release/arroyo \ - --config ~/ASAPQuery/asap-sketch-ingest/config.yaml cluster \ +~/ASAPQuery/asap-summary-ingest/target/release/arroyo \ + --config ~/ASAPQuery/asap-summary-ingest/config.yaml cluster \ > /tmp/arroyo.log 2>&1 & ``` ### Step 6 — Create ArroyoSketch pipeline ```bash -cd ~/ASAPQuery/asap-sketch-ingest +cd ~/ASAPQuery/asap-summary-ingest python3 run_arroyosketch.py \ --source_type kafka \ --kafka_input_format json \ @@ -195,7 +195,7 @@ $KAFKA/kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic hits $KAFKA/kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic sketch_topic # Delete old Arroyo pipeline -cd ~/ASAPQuery/asap-sketch-ingest +cd ~/ASAPQuery/asap-summary-ingest python3 delete_pipeline.py --all_pipelines # Clear ClickHouse table diff --git a/docs/01-getting-started/architecture.md b/docs/01-getting-started/architecture.md index 5afa5e0..1fb7f9d 100644 --- a/docs/01-getting-started/architecture.md +++ b/docs/01-getting-started/architecture.md @@ -185,7 +185,7 @@ graph LR |-----------|---------|------------|----------| | **asap-query-engine** | Answers PromQL queries using sketches | Rust | `asap-query-engine/` | | **Arroyo** | Stream processing for building sketches | Rust (forked) | [github.com/ProjectASAP/arroyo](https://github.com/ProjectASAP/arroyo) | -| **asap-sketch-ingest** | Configures Arroyo pipelines from config | Python | `asap-sketch-ingest/` | +| **asap-summary-ingest** | Configures Arroyo pipelines from config | Python | `asap-summary-ingest/` | | **asap-planner-rs** | Auto-determines sketch parameters | Rust | `asap-planner-rs/` | | **Kafka** | Message broker for sketch distribution | Apache Kafka | (external) | | **Prometheus** | Time-series database (existing) | Go | (external) | @@ -228,7 +228,7 @@ graph LR - Serde for serialization - DataSketches (dsrs) for sketch algorithms -- **Python** - asap-sketch-ingest, experiment framework +- **Python** - asap-summary-ingest, experiment framework - PyYAML for config parsing - Jinja2 for SQL templates - Requests for HTTP clients @@ -259,7 +259,7 @@ ASAPQuery/ │ │ └── tests/ # Integration tests │ └── docs/ # QueryEngine dev docs │ -├── asap-sketch-ingest/ # Pipeline configurator +├── asap-summary-ingest/ # Pipeline configurator │ ├── run_arroyosketch.py # Main script │ ├── templates/ # Jinja2 SQL templates │ └── utils/ # Arroyo API client diff --git a/docs/02-components/README.md b/docs/02-components/README.md index a4762e4..dc3fd79 100644 --- a/docs/02-components/README.md +++ b/docs/02-components/README.md @@ -8,7 +8,7 @@ This document provides an overview of all ASAP components and links to detailed |-----------|---------|------------|-------| | **asap-query-engine** | Answers PromQL queries using sketches | Rust | [Details](query-engine.md) · [Code](../../asap-query-engine/) · [Dev Docs](../../asap-query-engine/docs/README.md) | | **Arroyo** | Stream processing for building sketches | Rust (forked) | [Details](arroyo.md) · [Code](https://github.com/ProjectASAP/arroyo) | -| **asap-sketch-ingest** | Configures Arroyo pipelines from config | Python | [Details](arroyosketch.md) · [Code](../../asap-sketch-ingest/) · [README](../../asap-sketch-ingest/README.md) | +| **asap-summary-ingest** | Configures Arroyo pipelines from config | Python | [Details](arroyosketch.md) · [Code](../../asap-summary-ingest/) · [README](../../asap-summary-ingest/README.md) | | **asap-planner-rs-rs** | Auto-determines sketch parameters | Rust | [Details](controller.md) · [Code](../../asap-planner-rs-rs/) | | **Exporters** | Generate synthetic metrics for testing | Rust/Python | [Details](exporters.md) · [Code](../../asap-tools/data-sources/prometheus-exporters/) · [README](../../asap-tools/data-sources/prometheus-exporters/README.md) | | **asap-tools** | Experiment framework for CloudLab | Python | [Details](utilities.md) · [Code](../../asap-tools/) · [Docs](../../asap-tools/docs/architecture.md) | @@ -20,7 +20,7 @@ graph TB subgraph "Configuration (Offline)" U[User] -->|edits| CC[controller-config.yaml] CC --> C[asap-planner-rs] - C -->|streaming_config.yaml| AS[asap-sketch-ingest] + C -->|streaming_config.yaml| AS[asap-summary-ingest] C -->|inference_config.yaml| Q AS -->|create pipelines| A end @@ -78,7 +78,7 @@ These run once to set up the system: - Selects sketch algorithms - Generates configs for Arroyo and QueryEngine -- **[asap-sketch-ingest](arroyosketch.md)** - Creates Arroyo pipelines +- **[asap-summary-ingest](arroyosketch.md)** - Creates Arroyo pipelines - Reads streaming_config.yaml - Renders SQL templates - Creates pipelines via Arroyo API @@ -112,7 +112,7 @@ Performance-critical components written in Rust: Configuration and orchestration in Python: - **asap-planner-rs** - Query analysis and config generation -- **asap-sketch-ingest** - Pipeline configuration +- **asap-summary-ingest** - Pipeline configuration - **asap-tools** - Experiment framework - **Python Exporters** - Simpler metric generators @@ -127,15 +127,15 @@ asap-query-engine Arroyo ├── Prometheus (runtime) - Remote write source ├── Kafka (runtime) - Sketch output -└── SQL pipelines (config) - From asap-sketch-ingest +└── SQL pipelines (config) - From asap-summary-ingest -asap-sketch-ingest +asap-summary-ingest ├── Arroyo (runtime) - Creates pipelines via API └── streaming_config.yaml (config) - From asap-planner-rs asap-planner-rs ├── controller-config.yaml (input) - User-provided -├── streaming_config.yaml (output) - For asap-sketch-ingest +├── streaming_config.yaml (output) - For asap-summary-ingest └── inference_config.yaml (output) - For asap-query-engine Exporters @@ -152,7 +152,7 @@ asap-tools - [asap-query-engine](query-engine.md) - Query processor deep dive - [Arroyo](arroyo.md) - Streaming engine + ASAP customizations -- [asap-sketch-ingest](arroyosketch.md) - Pipeline configurator +- [asap-summary-ingest](arroyosketch.md) - Pipeline configurator - [asap-planner-rs](controller.md) - Auto-configuration service - [Exporters](exporters.md) - Metric generators - [asap-tools](utilities.md) - Experiment framework @@ -163,6 +163,6 @@ For implementation details, see READMEs co-located with code: - [asap-query-engine/docs/](../../asap-query-engine/docs/README.md) - Extensibility guides - [asap-planner-rs/README.md](../../asap-planner-rs/README.md) - asap-planner-rs internals -- [asap-sketch-ingest/README.md](../../asap-sketch-ingest/README.md) - Pipeline config internals +- [asap-summary-ingest/README.md](../../asap-summary-ingest/README.md) - Pipeline config internals - [asap-tools/data-sources/prometheus-exporters/README.md](../../asap-tools/data-sources/prometheus-exporters/README.md) - Exporter implementations - [asap-tools/docs/](../../asap-tools/docs/architecture.md) - Experiment framework architecture diff --git a/docs/03-how-to-guides/development/adapt-pr-after-reorg.md b/docs/03-how-to-guides/development/adapt-pr-after-reorg.md index 1d07009..04c7f5e 100644 --- a/docs/03-how-to-guides/development/adapt-pr-after-reorg.md +++ b/docs/03-how-to-guides/development/adapt-pr-after-reorg.md @@ -12,7 +12,7 @@ This guide walks you through rebasing onto the new structure. |---|---| | `QueryEngineRust/` | `asap-query-engine/` | | `CommonDependencies/` | `asap-common/` | -| `ArroyoSketch/` | `asap-sketch-ingest/` | +| `ArroyoSketch/` | `asap-summary-ingest/` | | `Controller/` | `asap-planner/` | | `Utilities/` | `asap-tools/` | | `PrometheusExporters/` | `asap-tools/data-sources/prometheus-exporters/` | @@ -123,7 +123,7 @@ git push --force-with-lease origin your-branch-name Key CI jobs to watch: - **Rust** — Cargo build and tests for `asap-query-engine` and `asap-common` - **Python** — Linting/type-checking for `asap-tools/queriers/prometheus-client` -- **Docker** — Image builds for `asap-query-engine`, `asap-planner-rs`, `asap-sketch-ingest` +- **Docker** — Image builds for `asap-query-engine`, `asap-planner-rs`, `asap-summary-ingest` --- @@ -142,7 +142,7 @@ sed -i 's|CommonDependencies|asap-common|g' your_file.toml # Rename in Python files sed -i 's|"Utilities"|"asap-tools"|g' your_file.py sed -i 's|"Controller"|"asap-planner"|g' your_file.py -sed -i 's|"ArroyoSketch"|"asap-sketch-ingest"|g' your_file.py +sed -i 's|"ArroyoSketch"|"asap-summary-ingest"|g' your_file.py sed -i 's|"PrometheusClient"|"asap-tools/queriers/prometheus-client"|g' your_file.py sed -i 's|"PrometheusExporters"|"asap-tools/data-sources/prometheus-exporters"|g' your_file.py diff --git a/docs/03-how-to-guides/development/add-new-sketch.md b/docs/03-how-to-guides/development/add-new-sketch.md index 13b1652..75cf698 100644 --- a/docs/03-how-to-guides/development/add-new-sketch.md +++ b/docs/03-how-to-guides/development/add-new-sketch.md @@ -1,6 +1,6 @@ # How to Add a New Sketch Algorithm -Adding a new sketch requires changes to 3 components: asap-common (sketch selection logic), asap-sketch-ingest (UDF for building sketches), and asap-query-engine (deserialization and query logic). +Adding a new sketch requires changes to 3 components: asap-common (sketch selection logic), asap-summary-ingest (UDF for building sketches), and asap-query-engine (deserialization and query logic). ## Step 1: asap-common - Define Sketch Mapping @@ -14,9 +14,9 @@ Adding a new sketch requires changes to 3 components: asap-common (sketch select --- -## Step 2: asap-sketch-ingest - Create Sketch UDF +## Step 2: asap-summary-ingest - Create Sketch UDF -**File to create**: `asap-sketch-ingest/templates/udfs/yoursketchname_[subop].rs.j2` (or `.rs` if no template vars) +**File to create**: `asap-summary-ingest/templates/udfs/yoursketchname_[subop].rs.j2` (or `.rs` if no template vars) **What to implement**: - Rust UDF function using `#[udf]` macro @@ -29,8 +29,8 @@ Adding a new sketch requires changes to 3 components: asap-common (sketch select **Validate**: Run `python validate_udfs.py` to check UDF compiles. **Reference examples**: -- `asap-sketch-ingest/templates/udfs/datasketcheskll_.rs.j2` -- `asap-sketch-ingest/templates/udfs/countminsketch_sum.rs.j2` +- `asap-summary-ingest/templates/udfs/datasketcheskll_.rs.j2` +- `asap-summary-ingest/templates/udfs/countminsketch_sum.rs.j2` --- @@ -86,7 +86,7 @@ pub use your_sketch_accumulator::*; - [ ] `validate_udfs.py` passes (ArroyoSketch) - [ ] `cargo build --release` succeeds (asap-query-engine) - [ ] `cargo test` passes (asap-query-engine) -- [ ] End-to-end: asap-planner-rs → asap-sketch-ingest → Arroyo → Kafka → QueryEngine → Query result +- [ ] End-to-end: asap-planner-rs → asap-summary-ingest → Arroyo → Kafka → QueryEngine → Query result --- @@ -95,7 +95,7 @@ pub use your_sketch_accumulator::*; | Component | Format | Example | |-----------|--------|---------| | asap-common mapping | PascalCase | `DatasketchesKLL` | -| asap-sketch-ingest UDF filename | lowercase_subop | `datasketcheskll_.rs.j2` | +| asap-summary-ingest UDF filename | lowercase_subop | `datasketcheskll_.rs.j2` | | QueryEngine accumulator | PascalCase + Accumulator | `DatasketchesKLLAccumulator` | | `get_accumulator_type()` return | Must match mapping | `"DatasketchesKLL"` | diff --git a/docs/03-how-to-guides/operations/manual-stack-run-clickhouse.md b/docs/03-how-to-guides/operations/manual-stack-run-clickhouse.md index 439b3ca..4bb2960 100644 --- a/docs/03-how-to-guides/operations/manual-stack-run-clickhouse.md +++ b/docs/03-how-to-guides/operations/manual-stack-run-clickhouse.md @@ -14,7 +14,7 @@ This guide covers running the ASAP stack manually with Clickhouse for developmen ``` ~/code/ -├── asap-sketch-ingest/ # Pipeline configuration scripts +├── asap-summary-ingest/ # Pipeline configuration scripts │ ├── config.yaml # Arroyo cluster config │ └── run_arroyosketch.py # Creates sources, sinks, and pipelines ├── asap-query-engine/ # Query interception layer @@ -230,7 +230,7 @@ Then create the tables as shown in the Baseline section (step 4). ```bash cd ~/code/arroyo -./target/release/arroyo --config ~/code/asap-sketch-ingest/config.yaml cluster +./target/release/arroyo --config ~/code/asap-summary-ingest/config.yaml cluster ``` Arroyo API runs at `http://localhost:5115`. Verify with: @@ -243,7 +243,7 @@ curl http://localhost:5115/api/v1/pipelines Run `run_arroyosketch.py` to create Arroyo sources, sinks, and pipeline. For Clickhouse, always use Kafka source: ```bash -cd ~/code/asap-sketch-ingest +cd ~/code/asap-summary-ingest python run_arroyosketch.py \ --source_type kafka \ --kafka_input_format json \ diff --git a/docs/03-how-to-guides/operations/manual-stack-run-prometheus.md b/docs/03-how-to-guides/operations/manual-stack-run-prometheus.md index 8d08403..c09bf44 100644 --- a/docs/03-how-to-guides/operations/manual-stack-run-prometheus.md +++ b/docs/03-how-to-guides/operations/manual-stack-run-prometheus.md @@ -14,7 +14,7 @@ This guide covers running the ASAP stack manually with Prometheus for developmen ``` ~/code/ -├── asap-sketch-ingest/ # Pipeline configuration scripts +├── asap-summary-ingest/ # Pipeline configuration scripts │ ├── config.yaml # Arroyo cluster config │ └── run_arroyosketch.py # Creates sources, sinks, and pipelines ├── asap-query-engine/ # Query interception layer @@ -153,7 +153,7 @@ cd ~/code/asap-tools/data-sources/prometheus-exporters/fake_exporter/fake_export ```bash cd ~/code/arroyo -./target/release/arroyo --config ~/code/asap-sketch-ingest/config.yaml cluster +./target/release/arroyo --config ~/code/asap-summary-ingest/config.yaml cluster ``` Arroyo API runs at `http://localhost:5115`. Verify with: @@ -166,7 +166,7 @@ curl http://localhost:5115/api/v1/pipelines Run `run_arroyosketch.py` to create Arroyo sources, sinks, and pipeline: ```bash -cd ~/code/asap-sketch-ingest +cd ~/code/asap-summary-ingest python run_arroyosketch.py \ --source_type prometheus_remote_write \ --prometheus_bind_ip 0.0.0.0 \ diff --git a/docs/03-how-to-guides/operations/publishing-docker-images.md b/docs/03-how-to-guides/operations/publishing-docker-images.md index 800255a..c0bc316 100644 --- a/docs/03-how-to-guides/operations/publishing-docker-images.md +++ b/docs/03-how-to-guides/operations/publishing-docker-images.md @@ -17,7 +17,7 @@ That's it. GitHub Actions will build and push all images tagged as both `v0.2.0` |---|---| | `ghcr.io/projectasap/asap-base` | `asap-common/installation/` | | `ghcr.io/projectasap/asap-planner-rs` | `asap-planner-rs/` | -| `ghcr.io/projectasap/asap-sketch-ingest` | `asap-sketch-ingest/` | +| `ghcr.io/projectasap/asap-summary-ingest` | `asap-summary-ingest/` | | `ghcr.io/projectasap/asap-query-engine` | `asap-query-engine/` | | `ghcr.io/projectasap/asap-prometheus-client` | `asap-tools/queriers/prometheus-client/` | diff --git a/docs/README.md b/docs/README.md index fb380b2..6cb5a7b 100644 --- a/docs/README.md +++ b/docs/README.md @@ -13,7 +13,7 @@ Deep dives into each component: - [Component Index](02-components/README.md) - Component overview & 1-liners - [Query Engine](02-components/query-engine.md) - Rust query processor - [Arroyo](02-components/arroyo.md) - Streaming engine (fork + customizations) -- [asap-sketch-ingest](02-components/arroyosketch.md) - Pipeline configurator +- [asap-summary-ingest](02-components/arroyosketch.md) - Pipeline configurator - [asap-planner-rs](02-components/controller.md) - Auto-configuration service - [Exporters](02-components/exporters.md) - Metric generators - [asap-tools](02-components/utilities.md) - Experiment framework @@ -55,5 +55,5 @@ Technical details co-located with code: - [asap-query-engine](../asap-query-engine/docs/README.md) - Extensibility guides - [asap-tools/Experiments](../asap-tools/docs/architecture.md) - Experiment framework architecture - [asap-planner-rs](../asap-planner-rs/README.md) - Controller internals -- [asap-sketch-ingest](../asap-sketch-ingest/README.md) - Pipeline configuration +- [asap-summary-ingest](../asap-summary-ingest/README.md) - Pipeline configuration - [asap-tools/data-sources/prometheus-exporters](../asap-tools/data-sources/prometheus-exporters/README.md) - Exporter implementations