Skip to content

feat(compute-plane): add request-trace-uploader image #1004

Description

@kristinapathak

Why

Dynamo request tracing writes closed, compressed RequestTraceRecord segments to a shared volume. Input and output payload capture uses the request_payload record type. These records are neither normal container logs nor distributed traces.

This umbrella tracks replacing the existing shell sidecar with the focused request-trace-uploader image. The image owns only the closed Dynamo request-trace segment to S3 lifecycle.

Scope

The uploader is not a good fit for an OTel Collector deployment. The optimal solution is a focused binary that does this one job of exporting Dynamo Request Trace Records to S3. It reads the configured trace and audit directories, preserves a source until durable terminal S3 success, and never logs credentials or request payloads.

Delivery plan

  1. #1041: bare-bones scaffold. Build and test the image with configuration validation, closed-segment discovery, and health only. Do not publish, read, upload, delete, or expose metrics in this increment.
  2. feat(request-trace-uploader): add initial S3 segment upload #1047: read one closed segment, upload it to S3, and delete it only after success.
  3. feat(request-trace-uploader): persist S3 upload lifecycle state #1050: add durable lifecycle state, restart recovery, and failure quarantine.
  4. feat(request-trace-uploader): configure S3 upload policy #1051: add configurable retry, timeout, backoff, size, concurrency, outbox, and asynchronous-status policy.
  5. feat(request-trace-uploader): add NCA audit payload drop #1048: add configurable NCA-scoped audit payload drop that follows all records for a matching request ID.
  6. feat(observability): export uploader telemetry via BYOO OTLP #1046: add the uploader own OTLP logs, traces, and metrics.
  7. docs(request-trace-uploader): document deployment and operations #1052: document deployment, operations, alerts, and migration.

Future first-class deployment is tracked separately in #1049. NVCA must inject the sidecar image, shared and state volumes, credentials, probes, configuration, and BYOO signal configuration. Environment injection alone is not enough.

Telemetry contract

When telemetry is added, the uploader exports its own structured logs, traces, and metrics directly to the workload-scoped BYOO Collector using signal-specific OTLP endpoint and protocol configuration. It must not expose or rely on a Prometheus scrape route, use a generic OTLP endpoint fallback, or send request-record payloads through OTLP logs.

Definition of done

  • A versioned, tested image can replace the existing sidecar after the upload lifecycle is complete.
  • Source segments are deleted only after durable terminal S3 success. Terminal failures are retained or quarantined.
  • NCA audit payload drop protects matching request records without removing correlation evidence.
  • Uploader logs, traces, and metrics reach BYOO through OTLP.
  • The deployment and operational contract is documented.

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions