Skip to content

Add opt-in pipeline runtime diagnostics#13

Open
Toti330 wants to merge 3 commits into
mainfrom
feat/runtime-diagnostics
Open

Add opt-in pipeline runtime diagnostics#13
Toti330 wants to merge 3 commits into
mainfrom
feat/runtime-diagnostics

Conversation

@Toti330

@Toti330 Toti330 commented Jul 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds DIAGNOSTICS=1, an opt-in end-of-run report for the generation-to-PostgreSQL pipeline.

Why this is useful

It turns a single final throughput number into measurable evidence: generated CSV payload volume, worker generation cost, worker channel backpressure, writer connection and receive time, star/planet COPY time, commit time, transaction count, and actual batch sizes. This makes it possible to distinguish CPU generation pressure, queue pressure, transfer/COPY time, and commit cost.

Implementation

  • Adds inserter/src/diagnostics.rs with lock-free aggregate counters.
  • Records worker generation and channel-send wait timings only when DIAGNOSTICS=1.
  • Records writer connection, receive, transaction setup, star COPY, planet COPY, and commit stages.
  • Uses a drop guard to report even if run() returns early after diagnostics begins.
  • Documents metrics and explains why aggregate thread time can exceed wall-clock time.

Independence

This PR is based directly on main and is independently usable. It does not require precise timing, tuning, checkpoint, preflight, or progress logging changes.

Verification

  • cargo test --all-targets passed in the Rust Docker toolchain.
  • BENCHMARK=1 DIAGNOSTICS=1 smoke run emitted payload, generation, channel, writer, and batch diagnostics.

Generated by GPT-5.6 Terra & Sol. Ideas from the author; implementation by AI.

@SuperB3333

Copy link
Copy Markdown
Owner

Partially implemented in 44e7c91

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants