feat: state coverage as two targets and gate the three figures nothing read - #163
Merged
Merged
Conversation
…g read docs/coverage.md published 431 registered at the top and, further down, called the target "205 services, not 431" while describing those 431 as not targeted. Both halves were written honestly at different times and the page could not be read as one claim. The summary table has been gated against the binary since Milestone 6; the target table under #the-target never was, so every number there could be wrong with CI green, and for a while every number there was. The fix is to stop pretending there is one target. Routing is a safety property: a registered service is answered at localhost:4747, an unregistered one leaves the machine and bills a real account, so the only number that satisfies it is all of them — 431 of 431, met. Depth is the separate and smaller promise, still 205, still where the 2026-09-05 demand study put it, because what that study refused was the cost of hand-building 283 services and the codegen scaffold removed that cost without producing any evidence of demand. A service count can no longer be read as a promise of fidelity because the page no longer states one number that could carry both meanings. TestPublishedTargetTableMatchesTheBinary now reads the two-axis table the way tierRow reads the tiers: the routing target is checked against plugin.DefaultRegistry.RegisteredServices(), and because the serving target is a decision rather than a measurement, it is read from the page and used as the arithmetic the third row must satisfy. The table cannot be internally inconsistent and cannot drift from the binary. CI now fails the build past 45 MiB. That is a regression ceiling and not the published figure — coverage.md measures 36.8 MiB on Apple Silicon and CI builds for linux, so a tight budget would fail on the platform difference rather than on a regression. The ceiling is named in coverage.md so the failure message points somewhere the number exists. cmd/devcloud/budget_test.go gates startup, and its first version did not. It timed Registry.Construct, which only calls the factory, and every factory in the tree is a struct literal — 84 microseconds against a 150 ms budget. The cost is in Init: S3Provider.Init does an os.MkdirAll and opens a SQLite database, and 431 of those is how a 42 ms startup becomes a second. Verified rather than assumed: injecting 8 ms per service into Registry.Init left the old gate at 79 microseconds and passing, and fails the new one at 4.0 s. It now brings every service up the way main.go does, measures 141 ms for all 431, and fails past 2 s. The ceiling is loose on purpose. The gate runs on a shared runner where a 2x reading is indistinguishable from a noisy neighbour, so it catches the regression that is real — a provider doing per-call work at startup, which arrives as 10x. It also fails if any registered service stops initializing, which main.go only warns about. The 194-model figures in sync_test.go and contributing.md predate vendoring the long tail and are now 420. The 2026-09-06 churn measurement is unchanged but says what it was taken at, since the diff a reviewer now faces is larger than anything measured there by roughly the same factor. RED/GREEN evidence: docs/testing/phase-4-review-followup.tdd.md
The 2 s ceiling this branch added was chosen as ~14x the 141 ms the fleet takes to come up locally, on the reasoning that 14x was room enough for a shared runner. CI measured the same path at 2.048 s on arm64. The runner is itself 14x slower, so the whole margin was the machine and none of it was headroom. Raising the number does not rescue the design. What is worth catching is a provider that starts opening a file or a database per service at startup, and that costs 1.2x to 2x, because that is what 431 extra file opens are worth. A ceiling loose enough to clear a 14x difference between machines plus run-to-run variance cannot also fail at 2x. Even the 8 ms-per-service injection used to prove the gate works — far larger than a realistic regression — only reaches 5.5 s against a 2.05 s baseline, so it would pass under any ceiling that does not flake. docs/coverage.md raised this objection before the gate was written; the gate overrode it and CI settled it. So the ceiling is gone and the test keeps the half that holds. Every one of the 431 registered services must initialize, which is new and was never covered: main.go brings the long tail up with fatal=false, so a service that is registered but can no longer initialize degrades to a warning in a log nobody reads. The elapsed time is logged beside it and becomes an assertion only when DEVCLOUD_STARTUP_BUDGET names one, which is how the published figure is re-taken on a machine whose speed is known. The binary size gate is untouched. Size does not vary with how busy a runner is.
Two figures in the runtime section were written before CI had produced the numbers they describe, and one of them was wrong. The binary ceiling was justified as absorbing a platform difference: 45 MiB against a published 36.8 MiB measured on Apple Silicon, with CI building for linux. CI measures 35 MiB on arm64 and 37 MiB on amd64, so the platform accounts for under 2 MiB and the remaining 8 MiB is slack. Saying so is the point of publishing it. The startup retreat was argued from one reading — 2.048 s on an arm64 runner against 141 ms locally. The run after it measured 1.414 s on the same runner type and 1.375 s on amd64, on identical code. A 45% swing between runs is a better argument than a slow machine: the regression worth catching costs 1.2x to 2x, so it is smaller than the noise a ceiling would have to clear. The docs and the TDD evidence now argue from the variance rather than from the one failure.
This was referenced Sep 13, 2026
skyoo2003
added a commit
that referenced
this pull request
Sep 13, 2026
…e fragments The 400-character ceiling was wide enough to hold two full sentences plus a subordinate clause, so it never pushed against the one-sentence rule it was meant to back — the #163 fragment reached 484 and still read as being in the spirit of the limit. At 200 the ceiling and the rule push the same way. Refits the eight unreleased fragments that were over. What came out is detail the linked issue already carries: the eight service names in #161, the CI-runner timing in #163, the DynamoDB and Lambda operation counts in #167. Also trims RELEASE.md's "right length" example, which was 203 characters and would have failed the limit the paragraph above it states, and labels both examples with their length.
skyoo2003
added a commit
that referenced
this pull request
Sep 13, 2026
…g to 200 (#168) * fix: give six changelog fragments the issue link the release tag checks Six fragments wrote `Issue:` at the top level instead of under `custom:`, where `.changie.yaml` declares it. `changie batch` rendered them as `[#<no value>](.../issues/<no value>)`, which release.yml:234 rejects — the tag would have failed at the notes-validation step, after the guardrails had already passed. The pre-flight grep in RELEASE.md (`grep -L 'Issue: "[0-9]'`) matches both spellings, so it did not catch this. Also trims the #163 fragment from 484 to 337 characters, under the 400-char ceiling `.changie.yaml` sets and `changie new` enforces on fragments it writes itself. The CI-runner timing detail it dropped is in the issue. * docs: correct the two protocol-table rows that miscount EC2 as having no model The protocol table said 12 services have no in-tree Smithy model and exactly one speaks a protocol the parser cannot read. Counted against internal/generated/fidelity/manifest_gen.go, it is 11 and two: `ec2` is model-backed but speaks `aws.protocols#ec2Query`, which parser.go:441-453 does not recognise alongside the five it does. That put the page at odds with fidelity-manifest.md:106 and compatibility-policy.md:78, which both say 11, and left a core service's missing engine coverage unexplained. Rows now sum to 431 either way, so the arithmetic did not expose it, and no test gates this table. Adds a paragraph separating what EC2 loses from what a registered-only service loses: EC2 is served by a hand-written provider and is not in the registered-only five, but its model-declared tail stays `unimplemented` rather than falling back to `auto-crud`. * docs: halve the changelog body ceiling to 200 characters and refit the fragments The 400-character ceiling was wide enough to hold two full sentences plus a subordinate clause, so it never pushed against the one-sentence rule it was meant to back — the #163 fragment reached 484 and still read as being in the spirit of the limit. At 200 the ceiling and the rule push the same way. Refits the eight unreleased fragments that were over. What came out is detail the linked issue already carries: the eight service names in #161, the CI-runner timing in #163, the DynamoDB and Lambda operation counts in #167. Also trims RELEASE.md's "right length" example, which was 203 characters and would have failed the limit the paragraph above it states, and labels both examples with their length.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
docs/coverage.mdpublished 431 registered at the top and, further down, called the target "205 services, not 431" while describing those 431 as not targeted. Both halves were written honestly at different times, and the page could not be read as one claim. The summary table has been gated against the binary since Milestone 6; the target table under#the-targetnever was — so every number there could be wrong with CI green, and for a while every number there was.This PR stops pretending there is one target, and gates the published figures that can be gated. One of them turned out not to be, and that story is below.
Changes
The two axes
Routing is a safety property: a registered service is answered at
localhost:4747, an unregistered one leaves the machine and bills a real AWS account. The only number that satisfies it is all of them. Depth is the separate, smaller promise — still 205, still where the demand study put it, because what that study refused was the cost of hand-building 283 services, and the codegen scaffold removed that cost without producing any evidence of demand.A service count can no longer be read as a promise of fidelity, because the page no longer states one number that could carry both meanings.
New gates
TestPublishedTargetTableMatchesTheBinary— reads the two-axis table the waytierRowreads the tiers. The routing target is checked againstplugin.DefaultRegistry.RegisteredServices(); the serving target is a decision rather than a measurement, so it is read from the page and used as the arithmetic the third row must satisfy. The table can neither drift from the binary nor contradict itself.Binary size ceiling in CI — fails the build past 45 MiB. Measured: 35 MiB arm64, 37 MiB amd64, against a published 36.8 MiB on Apple Silicon.
Every registered service must initialize —
main.gobrings the long tail up withfatal=false, so a service that is registered but can no longer start degrades to a warning in a log nobody reads.budget_test.gofails on it. This is narrower than the startup gate originally intended here, for the reason below.Stale figures
sync_test.goandcontributing.mdstill said 194 vendored models; now 420. The 2026-09-06 churn measurement is unchanged but now says what it was taken at, since the diff a reviewer faces today is larger by roughly the same factor.The startup gate: two wrong answers before the right one
Worth reading, because both mistakes were caught by something other than my own confidence.
First version — blind. It timed
Registry.Construct, which only calls the factory, and every factory in the tree is a struct literal (return &S3Provider{}). It measured 84 µs against a 150 ms budget. The real cost is inInit, whereS3Provider.Initdoes anos.MkdirAlland opens a SQLite database. Caught in local review; proven by injectingtime.Sleep(8 * time.Millisecond)intoRegistry.Init:Second version — flaky. Rewritten to time
Initthe waymain.godoes, with a 2 s ceiling picked as ~14x the 141 ms it takes locally. CI failed it at 2.048 s on arm64: the runner is itself ~10x slower, so the entire margin was the machine and none of it was headroom.Why raising the number does not fix it. The next run, on identical code, measured 1.414 s arm64 and 1.375 s amd64 — a 45% swing:
8e95b5217d2415The regression worth catching — a provider that starts opening a file or a database per service — costs 1.2x to 2x, which is smaller than the runner's own variance. A ceiling that survives 45% noise cannot fail on a 2x regression.
docs/coverage.mdraised this objection before the gate was written; the gate overrode it and CI settled it.Resolution. The correctness half is asserted on every run (all 431 must initialize). The timing is logged, and becomes an assertion only when a budget is named on a machine whose speed is known:
That is how the published figure is re-taken per release. The docs now say startup is measured and not gated, rather than claiming a gate that cannot fire.
Files Changed
docs/coverage.mddocs/demand.md,docs/README.md,README.mddocs/contributing.mdcmd/devcloud/coverage_test.gocmd/devcloud/budget_test.gocmd/devcloud/sync_test.go.github/workflows/ci.ymldocs/testing/phase-4-review-followup.tdd.mdchanges/unreleased/*.yamlTesting
CI green on all 9 checks.
cmd/devcloud's 0.0% statement coverage is expected and unchanged — that package asserts the binary's published claims against the registry it ships, notmain.go's statements.Known gaps, stated deliberately in
docs/testing/phase-4-review-followup.tdd.md: startup is not gated in CI at all; the 45 MiB ceiling has ~8 MiB of slack;Initis exercised with default options only.Related Issues
None. Follows #160, #161, #162.