feat: have the weekly sync re-derive the published figures it is designed to break - #165
Merged
Merged
Conversation
…gned to break
The published-figure gate fails on purpose whenever upstream moves an
operation, which is what makes a sync PR worth reviewing. The cost was that
the fix was a person transcribing nine numbers out of four failure messages
into three files, and a reviewer who cannot cheaply tell 19,201 from 19,247
ends up re-deriving the whole page or trusting it.
A step now does that arithmetic before the PR is opened and commits the
result. TestUpdatePublishedFigures rewrites every derivable figure across
docs/coverage.md, README.md and docs/README.md, and prints a before/after
table the PR body carries. It lives beside the gates in cmd/devcloud so the
figures have exactly one derivation, and it is switched by
DEVCLOUD_UPDATE_DOCS rather than a test flag because `go test ./...
-update-docs` fails in every other package. The gate is not relaxed: the
PR's own ci asserts the corrected page against the binary in both
directions.
It writes counts and refuses to write sentences. A service that has newly
stopped serving anything must be named and explained on the page, and a
moved registered count is restated in two places no gate reads; both exit
non-zero with a PROSE REQUIRED block naming what is owed. That refusal is
the line between automating the arithmetic and automating the judgement.
Each service's boto3 exclusions moved out of tests/compatibility/_coverage.py
into exclusions.json, because the published Compatibility-tested figure is
that subtraction and Go cannot import botocore to find it out. One file, two
readers, no second list someone remembers to update.
Review follow-ups, each with a failing test first:
- Nothing read the Compatibility-tested figure back. The updater wrote it
and no gate asserted it, so an exclusion added on the Python side would
have moved the fleet and left the page green.
- quotedFigurePattern is loose because it reads prose, and everyMatch
handed that same looseness to a writer. Its unbounded gap rewrote "Only
3 of the 431 registered services need 2 serving tiers" into "Only 431 of
the 431 ... need 426 serving tiers". Every gap now excludes digits and
is bounded; both live phrasings separate the figures by ", " or " / ",
and a rewording that stops matching still fails loudly.
- The job's failure signal read only the test step. A PROSE REQUIRED
result is continue-on-error, so the cron ended green on the one failure
mode this automation cannot fix for itself.
- TestUpdaterRestoresAMangledFigure compared against the page as
committed, so a merely stale page — the normal state of the sync it was
written for — produced six messages, three naming causes that were not
true. Its baseline is now the page the binary says it should read, and
the coverage gate keeps sole ownership of "are the figures current".
- proseRequired, renderFigureTable and formatFigure ran only under the env
var. formatFigure rendered -123 as "-,123".
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The published-figure gate fails on purpose whenever upstream moves an operation — that failure is the review. The cost was that fixing it meant a person transcribing nine numbers out of four failure messages into three files. The sync now does that arithmetic itself and commits it, so reviewing a sync PR is confirming which operations moved rather than checking whether 19,201 is the right total.
Related Issue
Refs #163
Changes
The updater (
cmd/devcloud/figures_update_test.go, new)TestUpdatePublishedFiguresrewrites all 14 derivable figures acrossdocs/coverage.md,README.mdanddocs/README.md, and prints a before/after table the PR body carries.cmd/devcloudso each figure has exactly one derivation (derivedFigures), and shares the gates' own regexes — a restructured page cannot leave the gate reading one cell and the tool writing another.DEVCLOUD_UPDATE_DOCS=1, not a test flag:go test ./... -update-docsfails in every other package.DEVCLOUD_STARTUP_BUDGETsets the precedent.PROSE REQUIREDblock naming what is owed. That refusal is the line between automating the arithmetic and automating the judgement.The workflow (
.github/workflows/smithy-sync.yml)Re-derive the published figuresstep runs afterRun tests(its ownruncontainsgo test, so moving it earlier would silently repoint three existing sync gates) and before the PR is opened.Published figures: <outcome>and carries the table.ciasserts the corrected page against the binary in both directions.Shared exclusions (
tests/compatibility/exclusions.json, new)_coverage.py. The publishedCompatibility-testedfigure is that subtraction, and Go cannot import botocore to find it out. One file, two readers, no second list someone remembers to update.Review follow-ups — five findings from a local review of the above, each fixed test-first:
Compatibility-testedfigure back — the updater wrote it and no gate asserted itTestPublishedCoverageMatchesTheBinaryeveryMatchgave a deliberately loose prose pattern write access; its unbounded gap rewroteOnly 3 of the 431 registered services need 2 serving tiersintoOnly 431 of the 431 … need 426 serving tiersPROSE REQUIREDresult ended the cron greenifreads bothcontinue-on-erroroutcomesTestUpdaterRestoresAMangledFigurecompared against the page as committed, so a merely stale page — the normal state of the sync — produced six messages, three naming causes that were not trueproseRequired,renderFigureTable,formatFigureran only under the env var;formatFigure(-123)rendered-,123Test Plan
Each review finding was proven RED before its fix:
999; before the assertion existed the gate said nothing, after it:docs/coverage.md publishes 999 compatibility-tested services; 431 registered minus the 5 pinned in exclusions.json is 426.TestUpdaterLeavesUnrelatedSentencesAloneshowed3→431and2→426in a sentence the updater had no business touching."steps.tests.outcome == 'failure'" does not contain "steps.figures.outcome".Registeredcell produceda mangled auto-crud tier row was not restored, so the thousands separator does not survive the rewriteabout a row nothing had touched, anda clean docs/coverage.md was rewrittenabout a page that was not clean. After the fix the same stale page yields one failure from one gate, naming the one true cause.formatFigure(-123, true) = "-,123", want "-123".Two of the added tests (
proseRequired,renderFigureTable) passed on first run — they close a coverage gap rather than fix a defect, and are reported as such rather than as RED→GREEN.End-to-end, on a tree whose figures are current:
git statusstays clean afterwards — an updater that rewrites an unchanged tree would open an empty PR every week.go test -coverreports 0.0% of statements forcmd/devcloud, and that is the honest number rather than a shortfall: every test in the package asserts ondocs/coverage.md,README.mdandsmithy-sync.yml— assets that are not Go — plus pure helpers that live in_test.gofiles and are outside the statement count.Checklist
docs/contributing.mdanddocs/coverage.mdnow describe re-derivation as done-by-the-sync