Add timed pipeline progress logging

docs/progress-logging.md

@@ -0,0 +1,16 @@
+# Progress Logging
+
+Progress logging is enabled by default and runs in a dedicated sleeping thread. Set `PROGRESS_LOG=0` to disable the logger and all additional stage timers. Set `PROGRESS_LOG_INTERVAL_SECS` to a positive integer number of seconds; the default is `30`.
+
+The logger prints `generation started` before workers launch, then emits time-based reports containing:
+
+- Generated, committed, and overall progress as exact counts and percentages.
+- Interval and whole-run generation and commit rates in seeds/second.
+- Elapsed time and channel occupancy.
+- Aggregate worker and writer-pipeline durations. Writer-pipeline time includes connection, channel receive, COPY, and commit time; use the independent runtime-diagnostics branch when those database stages must be separated. While threads remain active, time is estimated as elapsed time per active thread; final values are measured.
+- Active and completed worker/writer counts, dominant aggregate thread-time stage, and a queue-based likely bottleneck.
+- Linux process RSS, peak RSS, bytes read, and bytes written from `/proc`; unsupported platforms report `n/a`.
+
+Aggregate stage durations are thread time, so overlapping workers or writers can produce totals larger than wall-clock elapsed time. `likely_bottleneck` is a queue-based diagnostic: at least `80%` occupancy indicates database pressure, at most `20%` indicates generation pressure, and an empty generation phase with outstanding commits indicates database drain.
+
+Instrumentation does not add a timer or branch to each generation pass. Timing occurs once per worker or writer lifetime, and process/resource collection runs only in the monitor thread at the configured interval.

inserter/src/main.rs

@@ -5,12 +5,14 @@ mod logging;
 mod metrics;
 mod misc;
 mod threads;
+mod progress;
 
 use crate::{
     checkpoint::{load_workloads, write_checkpoints},
     misc::{check_db_connection, get_env_interval, Timer},
     metrics::write_metrics,
-    threads::*
+    progress::{commit_thread, worker_thread},
+    threads::*,
 };
 use anyhow::{anyhow, Context, Result};
 use crossbeam_channel::{bounded, Receiver, Sender};
@@ -163,6 +165,8 @@ fn run() -> Result<()> {
         }
     }
 
+    let planned_seeds = workloads.iter().map(|workload| workload.len() as u64).sum();
+    let progress_logger = progress::ProgressLogger::start(planned_seeds, entry_reciever.clone())?;
     log_info!("Starting main thread loop");
     // Main thread takes checkpoints and displays metrics to the terminal
     if *TUI_MODE != TUIMode::Off {
@@ -232,6 +236,9 @@ fn run() -> Result<()> {
             .map_err(|_| anyhow!("writer thread panicked"))?
             .context("writer thread failed")?;
     }
+    if let Some(logger) = progress_logger {
+        logger.finish()?;
+    }
 
     let elapsed = start.elapsed();
     let per_second = (*END_SEED - *START_SEED) as f64 / elapsed.as_secs_f64();