Skip to content

feat: have the weekly sync re-derive the published figures it is designed to break - #165

Merged
skyoo2003 merged 1 commit into
mainfrom
feat/phase-6-sync-cost-mitigation
Sep 13, 2026
Merged

skyoo2003 merged 1 commit into
mainfrom
feat/phase-6-sync-cost-mitigation

Conversation

@skyoo2003

Copy link
Copy Markdown
Owner

Summary

The published-figure gate fails on purpose whenever upstream moves an operation — that failure is the review. The cost was that fixing it meant a person transcribing nine numbers out of four failure messages into three files. The sync now does that arithmetic itself and commits it, so reviewing a sync PR is confirming which operations moved rather than checking whether 19,201 is the right total.

Related Issue

Refs #163

Changes

The updater (cmd/devcloud/figures_update_test.go, new)

  • TestUpdatePublishedFigures rewrites all 14 derivable figures across docs/coverage.md, README.md and docs/README.md, and prints a before/after table the PR body carries.
  • Lives beside the gates in cmd/devcloud so each figure has exactly one derivation (derivedFigures), and shares the gates' own regexes — a restructured page cannot leave the gate reading one cell and the tool writing another.
  • Switched by DEVCLOUD_UPDATE_DOCS=1, not a test flag: go test ./... -update-docs fails in every other package. DEVCLOUD_STARTUP_BUDGET sets the precedent.
  • Writes counts, refuses sentences. A service that newly serves nothing, or a moved registered count restated in prose no gate reads, exits non-zero with a PROSE REQUIRED block naming what is owed. That refusal is the line between automating the arithmetic and automating the judgement.

The workflow (.github/workflows/smithy-sync.yml)

  • A Re-derive the published figures step runs after Run tests (its own run contains go test, so moving it earlier would silently repoint three existing sync gates) and before the PR is opened.
  • The PR body states Published figures: <outcome> and carries the table.
  • The gate is not relaxed — the PR's own ci asserts the corrected page against the binary in both directions.

Shared exclusions (tests/compatibility/exclusions.json, new)

  • The boto3 exclusion sets moved out of _coverage.py. The published Compatibility-tested figure is that subtraction, and Go cannot import botocore to find it out. One file, two readers, no second list someone remembers to update.

Review follow-ups — five findings from a local review of the above, each fixed test-first:

Finding Fix
Nothing read the Compatibility-tested figure back — the updater wrote it and no gate asserted it Assertion added to TestPublishedCoverageMatchesTheBinary
everyMatch gave a deliberately loose prose pattern write access; its unbounded gap rewrote Only 3 of the 431 registered services need 2 serving tiers into Only 431 of the 431 … need 426 serving tiers Every gap now excludes digits and is bounded
The job's failure signal read only the test step, so a PROSE REQUIRED result ended the cron green if reads both continue-on-error outcomes
TestUpdaterRestoresAMangledFigure compared against the page as committed, so a merely stale page — the normal state of the sync — produced six messages, three naming causes that were not true Baseline is now the page the binary says it should read; the coverage gate keeps sole ownership of "are the figures current"
proseRequired, renderFigureTable, formatFigure ran only under the env var; formatFigure(-123) rendered -,123 Three tests added; the sign is held aside before separators are spliced

Test Plan

CGO_ENABLED=0 go vet ./...      # clean
CGO_ENABLED=0 go test ./...     # 865 packages, no FAIL
gofmt -l cmd/ internal/         # clean

Each review finding was proven RED before its fix:

  • Missing gate — set the published cell to 999; before the assertion existed the gate said nothing, after it: docs/coverage.md publishes 999 compatibility-tested services; 431 registered minus the 5 pinned in exclusions.json is 426.
  • Loose write — TestUpdaterLeavesUnrelatedSentencesAlone showed 3→431 and 2→426 in a sentence the updater had no business touching.
  • Failure signal — "steps.tests.outcome == 'failure'" does not contain "steps.figures.outcome".
  • Stale-page misdiagnosis — a single mangled Registered cell produced a mangled auto-crud tier row was not restored, so the thousands separator does not survive the rewrite about a row nothing had touched, and a clean docs/coverage.md was rewritten about a page that was not clean. After the fix the same stale page yields one failure from one gate, naming the one true cause.
  • Formatting — formatFigure(-123, true) = "-,123", want "-123".

Two of the added tests (proseRequired, renderFigureTable) passed on first run — they close a coverage gap rather than fix a defect, and are reported as such rather than as RED→GREEN.

End-to-end, on a tree whose figures are current:

$ DEVCLOUD_UPDATE_DOCS=1 CGO_ENABLED=0 go test ./cmd/devcloud/ -run TestUpdatePublishedFigures
**0 of 14 published figures moved.**

git status stays clean afterwards — an updater that rewrites an unchanged tree would open an empty PR every week.

go test -cover reports 0.0% of statements for cmd/devcloud, and that is the honest number rather than a shortfall: every test in the package asserts on docs/coverage.md, README.md and smithy-sync.yml — assets that are not Go — plus pure helpers that live in _test.go files and are outside the statement count.

Checklist

  • Self-reviewed the code
  • Added/updated tests
  • Lint/format passes — pre-commit ran gofmt, go vet, go build, ruff, ruff format; all passed
  • Updated documentation (if applicable) — docs/contributing.md and docs/coverage.md now describe re-derivation as done-by-the-sync
  • Added a Changie changelog fragment for user-facing changes

…gned to break

The published-figure gate fails on purpose whenever upstream moves an
operation, which is what makes a sync PR worth reviewing. The cost was that
the fix was a person transcribing nine numbers out of four failure messages
into three files, and a reviewer who cannot cheaply tell 19,201 from 19,247
ends up re-deriving the whole page or trusting it.

A step now does that arithmetic before the PR is opened and commits the
result. TestUpdatePublishedFigures rewrites every derivable figure across
docs/coverage.md, README.md and docs/README.md, and prints a before/after
table the PR body carries. It lives beside the gates in cmd/devcloud so the
figures have exactly one derivation, and it is switched by
DEVCLOUD_UPDATE_DOCS rather than a test flag because `go test ./...
-update-docs` fails in every other package. The gate is not relaxed: the
PR's own ci asserts the corrected page against the binary in both
directions.

It writes counts and refuses to write sentences. A service that has newly
stopped serving anything must be named and explained on the page, and a
moved registered count is restated in two places no gate reads; both exit
non-zero with a PROSE REQUIRED block naming what is owed. That refusal is
the line between automating the arithmetic and automating the judgement.

Each service's boto3 exclusions moved out of tests/compatibility/_coverage.py
into exclusions.json, because the published Compatibility-tested figure is
that subtraction and Go cannot import botocore to find it out. One file, two
readers, no second list someone remembers to update.

Review follow-ups, each with a failing test first:

  - Nothing read the Compatibility-tested figure back. The updater wrote it
    and no gate asserted it, so an exclusion added on the Python side would
    have moved the fleet and left the page green.
  - quotedFigurePattern is loose because it reads prose, and everyMatch
    handed that same looseness to a writer. Its unbounded gap rewrote "Only
    3 of the 431 registered services need 2 serving tiers" into "Only 431 of
    the 431 ... need 426 serving tiers". Every gap now excludes digits and
    is bounded; both live phrasings separate the figures by ", " or " / ",
    and a rewording that stops matching still fails loudly.
  - The job's failure signal read only the test step. A PROSE REQUIRED
    result is continue-on-error, so the cron ended green on the one failure
    mode this automation cannot fix for itself.
  - TestUpdaterRestoresAMangledFigure compared against the page as
    committed, so a merely stale page — the normal state of the sync it was
    written for — produced six messages, three naming causes that were not
    true. Its baseline is now the page the binary says it should read, and
    the coverage gate keeps sole ownership of "are the figures current".
  - proseRequired, renderFigureTable and formatFigure ran only under the env
    var. formatFigure rendered -123 as "-,123".
@skyoo2003
skyoo2003 merged commit f9c471b into main Sep 13, 2026
10 of 11 checks passed
@skyoo2003
skyoo2003 deleted the feat/phase-6-sync-cost-mitigation branch September 13, 2026 09:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant