Add timed pipeline progress logging#16
Open
Toti330 wants to merge 5 commits into
Open
Conversation
Owner
|
I'll make sense of this implement some consistency/readability changes and commit it |
Owner
|
Implemented 06b6fc3 as a skeleton. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a time-based progress logger for long-running generation jobs. It reports immediately when generation begins, then reports every 30 seconds by default.
Why this is useful
A long seed-generation run should not be silent. The monitor provides exact generated/committed counts and percentages, interval and average rates, elapsed time, queue occupancy, aggregate worker and writer-pipeline time, likely bottleneck, Linux RSS/peak RSS, and process I/O. This gives a lightweight trace-like view without polling every generation pass.
Implementation
PROGRESS_LOG=0to disable the monitor and its timing hooks.PROGRESS_LOG_INTERVAL_SECS=N; default is30.writer_pipelinebecause it includes connection, receive, COPY, and commit work./proc; unsupported systems shown/a.Independence
This PR is based directly on
mainand is independently usable. It is compatible with the other feature branches but does not require them.Verification
cargo test --all-targets: 3 progress tests passed.36/60 (60.00%)at1.002s, then emitted a final complete report.PROGRESS_LOG=0smoke run emitted 0 progress lines.Generated by GPT-5.6 Terra & Sol. Ideas from the author; implementation by AI.