Finish v1: countless mode, documented env vars, generated message table - #2
Merged
Merged
Conversation
A dump arriving on stdin can't be preflighted, so there are no denominators - and until now that sent the whole run to passthrough, handing the user back the firehose the tool exists to swallow. But denominators are the only thing missing: every completion event carries its own object description, and toc.SectionOf maps that to a phase exactly the way the plan would have. So a nil plan now means countless, not passthrough. The aggregator counts phases from the event's description, skips the cap it has no total for, and flags the snapshot; the frame and the summary swap the three bars for three object counts and leave the cleanup row, working line and error panel exactly as they were. Plain mode needed its own rule - its milestones are deciles, and with zero totals it would have printed one line and then gone quiet for the rest of the restore - so it paces on the object count instead. `--replay -` feeds the same view a plan-less log, which is how countless mode can be seen and tested without a stdin restore and a database. Two documented env vars were missing while I was in here. ELE_LOG names the raw log (unset keeps the timestamped default), and ELE_PASSTHROUGH is the one way back to raw pg_restore: argv verbatim, no log, its own exit code. Both live in engine/env.go with ELE_STRICT_EXIT, which was already implemented but read inline.
The wordings the parser matched on were hand-copied, which made every new PostgreSQL major a silent downgrade: a reworded message stops matching, the line becomes Unknown, and the bars quietly stall until somebody notices. tools/pgmsg closes that. It pulls the msgid set of pg_dump's message catalog at REL_13 through REL_18, snapshots each one under tools/pgmsg/testdata, and emits internal/parser/messages_gen.go. Regenerating is deterministic - sorted, no timestamps - so rerunning against the same tags is a no-op, and `just gen-messages--check` proves it offline from the snapshots. The source is a committed .po rather than pg_dump.pot: the .pot template is a build artifact and is 404 at every release tag on the mirror, while the .po translations carry the identical msgid set. Two languages are unioned per tag, since one translation can lag upstream but two are unlikely to lag the same message. What the generator owns is wording and version coverage; what stays hand-written is meaning. gettext carries no semantics, so the msgid -> event bindings live in the generator's own source and are checked against upstream on every run: absent from every tag is an error, absent from some is recorded as a narrower range. `Command was: %s` is already that case - standalone in 17 and 18, folded into the query-error message in 13. The envelope stays hand-written too, because it isn't pg_dump's: the `pg_restore:` prefix, the error:/warning:/detail: tags from common/logging.c, and the from-TOC-entry context line. The golden suite is the equivalence proof - it replays all three fixtures through the rewritten classify unchanged.
Documents restoring from stdin, ELE_LOG and ELE_PASSTHROUGH (and that plain output is still aggregated - passthrough is the only way to the firehose), the `--replay -` form, and where the parser's message wordings come from.
The countless pacing constants had landed between PlainProgress's doc comment and the type, so the type had no doc comment at all. Constants moved above it. In the README, the stdin paragraph sat between the mode list and the sentence explaining those modes. It is now its own subsection at the end of Modes, and the sentence describing what `<plan>` accepts names `-` alongside a dump and a listing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1.
Four gaps between the stated contract and the code. Ordered so the biggest
user-visible one lands first.
Countless mode
A dump on stdin can't be preflighted, so there are no denominators - and that
sent the entire run to passthrough, handing back the firehose the tool exists
to swallow. Denominators turn out to be the only thing missing: every
completion event carries its object description, and
toc.SectionOfmaps thatto a phase exactly the way the plan would have.
A nil plan now means countless, not passthrough. The aggregator counts phases
from the description and skips the cap it has no total for; the frame and
summary swap the three bars for three object counts and leave the cleanup row,
working line and error panel untouched. Plain mode needed its own pacing rule -
its milestones are deciles, so with zero totals it would have printed one line
and gone quiet for the whole restore.
ele --replay - <log>feeds the same view a plan-less log, which is how themode is tested and how you can see it without a stdin restore and a database.
ELE_LOGandELE_PASSTHROUGHBoth were documented and neither existed.
ELE_LOGnames the raw log (unsetkeeps the timestamped default);
ELE_PASSTHROUGH=1is the one way back to rawpg_restore- argv verbatim, no log, its own exit code. They joinELE_STRICT_EXITininternal/engine/env.go.Generated message table
Hand-copied wordings made every new PostgreSQL major a silent downgrade: a
reworded message stops matching, the line becomes
Unknown, and the barsquietly stall.
tools/pgmsgpulls pg_dump's msgid set at REL_13 throughREL_18, snapshots each tag, and emits
internal/parser/messages_gen.go.Deterministic, so rerunning against the same tags is a no-op -
just gen-messages--checkproves it offline.Two things worth flagging for review:
.po, notpg_dump.pot. The.pottemplate isa build artifact and is 404 at every release tag on the mirror; the
.potranslations carry the identical msgid set. Two languages are unioned per tag,
since one translation can lag upstream but two are unlikely to lag the same
message.
hand-written. gettext carries no semantics, so the msgid to event bindings
live in
tools/pgmsg/bindings.goand are checked against upstream on everyrun: absent from every tag is an error, absent from some is recorded as a
narrower range.
Command was: %sis already that case - standalone in 17 and18, folded into the query-error message in 13. The envelope stays hand-written
too, since it isn't pg_dump's: the
pg_restore:prefix, theerror:/warning:/detail:tags fromcommon/logging.c, and thefrom-TOC-entry context line.
Verification
just lintandjust testare green, and the untouched golden suite replayingall three fixtures through the rewritten
classifyis the equivalence proof forthe parser rewrite. New tests cover countless aggregation (serial and
-j), thecountless frame, summary and plain line, the env accessors, the countless replay
and the generated table's ordering invariant.
Smoke-tested against a real pg_restore 17: countless engages on stdin,
ELE_LOGredirects the log with no timestamped file created,
ELE_PASSTHROUGHwrites nolog and returns pg_restore's code, and a real error keeps a nonzero exit.
Not included
Recapturing golden fixtures from a production-sized dump, per the issue. Nothing
here needed new fixtures.
PLAN.mdwas refreshed too - its v1 milestone listed shipped work, section 10'sopen decision was already settled in code, and section 4 named
pg_dump.potasthe catalog source. That file is untracked here (a global gitignore covers it),
so those edits are local only and are not in this PR.