DataCollector Integration

[GnaneshGnani]

Changes

1. Code changes

   • Updated OTLP ingest processing in asap-query-engine/src/drivers/ingest/otel.rs.
   • Introduced explicit handling for two ingest outcomes:
     • Raw Metrics Flow
     • Sketch Payload Flow

2. HTTP OTLP request handling

   • Added handling for Content-Encoding: gzip on OTLP HTTP requests.
   • If gzip is set, body is decompressed before protobuf decode.
   • Unsupported/other encodings currently fall back to existing body handling.
   • Added transport-specific debug logs for HTTP receive/response.

3. gRPC OTLP request handling

   • Added transport-specific debug logs for gRPC receive/response.
   • Processing path now passes transport context (HTTP or gRPC) into request processing logs.

4. Raw Metrics Flow behavior

   • Groups parsed points by series key using format_series_key(name, labels).
   • Emits summary and per-series logs:
     • OTLP Raw Metrics Flow: received N raw metric series
     • OTLP Raw Metrics Flow: series <series_key> count=<n>

5. Sketch Payload Flow behavior

   • Detects sketch payload attributes in point attributes.
   • Currently recognized keys:
     • kll.sketch_payload
     • cms.sketch_payload
     • countsketch.sketch_payload
   • Sketch payload points are separated from raw point conversion and tracked as sketch payload entries.
   • Sketch payload bytes are deserialized using SketchEnvelope::decode(...) (from sketchlib-rust).
   • Logs include sketch type for known envelope variants (KLL, CountMin, CountSketch) and payload metadata.
   • Unknown envelope variants are logged as Other; decode failures are logged with decode_error for debugging.
   • Logs payload metadata (metric, attribute name, payload size in bytes):
     • OTLP Sketches: metric='<metric>' attr='<attr>' payload_bytes=<n> sketch_type=<type>
   • Emits sketch-flow summary:
     • OTLP Sketch Payload Flow: received N sketch payload(s), decoded successfully

OTLP message handling summary

Common wire format (both flows):

• OTLP HTTP endpoint: POST /v1/metrics
• OTLP gRPC service/method: opentelemetry.proto.collector.metrics.v1.MetricsService/Export
• Body: Protobuf ExportMetricsServiceRequest
• Optional HTTP compression accepted by this receiver path: gzip (or uncompressed)

Routing behavior:

• If a recognized sketch payload attribute is present on a data point, that point is treated as Sketch Payload Flow.
• Otherwise, point is parsed and included in Raw Metrics Flow.

Verification

1) Start ASAPQuery OTLP receiver

From ASAPQuery/:

cd ASAPQuery
RUST_LOG=query_engine_rust=debug ./target/release/query_engine_rust \
  --enable-otel-ingest \
  --otel-grpc-port 4317 \
  --otel-http-port 4318 \
  --kafka-topic dummy \
  --input-format json \
  --config asap-query-engine/examples/promql/inference_config.yaml \
  --streaming-config asap-query-engine/examples/promql/streaming_config.yaml \
  --streaming-engine arroyo \
  --prometheus-scrape-interval 15 \
  --output-dir /tmp/asapquery \
  --query-language promql \
  --lock-strategy global

Expected startup logs include OTLP listeners on :4317 (gRPC) and :4318 (HTTP).

2) Run DataCollector processors and load

From DataCollector/opentelemetry-collector-contrib-patch/:

Raw Metrics Flow example (KLL):

cd /home/gnanesh/ProjectASAP/DataCollector/opentelemetry-collector-contrib-patch
./KLL --config cmd/kll/asap_query_config/config-otlp-pathway1.yaml

Sketch Payload Flow example (KLL, batch):

cd /home/gnanesh/ProjectASAP/DataCollector/opentelemetry-collector-contrib-patch
./KLL --config cmd/kll/asap_query_config/config-otlp-pathway2-batch.yaml

Other processor binaries/configs:

./cmd/countminsketchcol/dist/countminsketchcol --config cmd/countminsketchcol/asap_query_config/config-otlp-pathway1.yaml

./cmd/countminsketchcol/dist/countminsketchcol --config cmd/countminsketchcol/asap_query_config/config-otlp-pathway2-batch.yaml

./cmd/countsketchcol/dist/countsketchcol --config cmd/countsketchcol/asap_query_config/config-otlp-pathway1.yaml

./cmd/countsketchcol/dist/countsketchcol --config cmd/countsketchcol/asap_query_config/config-otlp-pathway2-batch.yaml

From DataCollector/otel_collector_benchmark/:

cd /home/gnanesh/ProjectASAP/DataCollector/otel_collector_benchmark
go run main.go --endpoint localhost:53217 --workers 2 --hosts 5 --metrics 10 --duration 15s

3) Validate receiver logs

Look for representative logs in ASAPQuery output:

• OTLP received request via HTTP
• OTLP received request via gRPC
• OTLP ingest: received ... (transport=HTTP|gRPC)
• OTLP Raw Metrics Flow: received ...
• OTLP Sketches: metric=... attr=... payload_bytes=...
• OTLP Sketch Payload Flow: received ...

[GnaneshGnani]

@zzylol, I updated the sketchlib-rust dependency to the main branch since the corresponding branch changes have been merged. Please check whenever you have the time