Skip to content

feat: state coverage as two targets and gate the three figures nothing read - #163

Merged
skyoo2003 merged 4 commits into
mainfrom
feat/phase-4-two-axis-docs-and-gates
Sep 13, 2026
Merged

skyoo2003 merged 4 commits into
mainfrom
feat/phase-4-two-axis-docs-and-gates

Conversation

@skyoo2003

@skyoo2003 skyoo2003 commented Sep 12, 2026 •

Copy link
Copy Markdown
Owner

Summary

docs/coverage.md published 431 registered at the top and, further down, called the target "205 services, not 431" while describing those 431 as not targeted. Both halves were written honestly at different times, and the page could not be read as one claim. The summary table has been gated against the binary since Milestone 6; the target table under #the-target never was — so every number there could be wrong with CI green, and for a while every number there was.

This PR stops pretending there is one target, and gates the published figures that can be gated. One of them turned out not to be, and that story is below.

Changes

The two axes

Axis Services Governed by
Routing target 431 / 431 — met leak-zero; every published model is registered
Serving target 205 — met the demand study, sampled 2026-09-05
Registered and engine-served, outside the serving target 226 no depth promise

Routing is a safety property: a registered service is answered at localhost:4747, an unregistered one leaves the machine and bills a real AWS account. The only number that satisfies it is all of them. Depth is the separate, smaller promise — still 205, still where the demand study put it, because what that study refused was the cost of hand-building 283 services, and the codegen scaffold removed that cost without producing any evidence of demand.

A service count can no longer be read as a promise of fidelity, because the page no longer states one number that could carry both meanings.

New gates

  1. TestPublishedTargetTableMatchesTheBinary — reads the two-axis table the way tierRow reads the tiers. The routing target is checked against plugin.DefaultRegistry.RegisteredServices(); the serving target is a decision rather than a measurement, so it is read from the page and used as the arithmetic the third row must satisfy. The table can neither drift from the binary nor contradict itself.

  2. Binary size ceiling in CI — fails the build past 45 MiB. Measured: 35 MiB arm64, 37 MiB amd64, against a published 36.8 MiB on Apple Silicon.

  3. Every registered service must initialize — main.go brings the long tail up with fatal=false, so a service that is registered but can no longer start degrades to a warning in a log nobody reads. budget_test.go fails on it. This is narrower than the startup gate originally intended here, for the reason below.

Stale figures

sync_test.go and contributing.md still said 194 vendored models; now 420. The 2026-09-06 churn measurement is unchanged but now says what it was taken at, since the diff a reviewer faces today is larger by roughly the same factor.

The startup gate: two wrong answers before the right one

Worth reading, because both mistakes were caught by something other than my own confidence.

First version — blind. It timed Registry.Construct, which only calls the factory, and every factory in the tree is a struct literal (return &S3Provider{}). It measured 84 µs against a 150 ms budget. The real cost is in Init, where S3Provider.Init does an os.MkdirAll and opens a SQLite database. Caught in local review; proven by injecting time.Sleep(8 * time.Millisecond) into Registry.Init:

--- PASS: TestOldGateConstructOnly          79µs        <-- unmoved by the injection
--- FAIL: TestRegisteredFleetComesUpWithinItsBudget      4.018s over a 2s budget

Second version — flaky. Rewritten to time Init the way main.go does, with a 2 s ceiling picked as ~14x the 141 ms it takes locally. CI failed it at 2.048 s on arm64: the runner is itself ~10x slower, so the entire margin was the machine and none of it was headroom.

Why raising the number does not fix it. The next run, on identical code, measured 1.414 s arm64 and 1.375 s amd64 — a 45% swing:

Run arm64 amd64
8e95b52 2.048 s (failed) canceled by fail-fast
17d2415 1.414 s 1.375 s
local 0.141–0.168 s —

The regression worth catching — a provider that starts opening a file or a database per service — costs 1.2x to 2x, which is smaller than the runner's own variance. A ceiling that survives 45% noise cannot fail on a 2x regression. docs/coverage.md raised this objection before the gate was written; the gate overrode it and CI settled it.

Resolution. The correctness half is asserted on every run (all 431 must initialize). The timing is logged, and becomes an assertion only when a budget is named on a machine whose speed is known:

DEVCLOUD_STARTUP_BUDGET=300ms go test ./cmd/devcloud/

That is how the published figure is re-taken per release. The docs now say startup is measured and not gated, rather than claiming a gate that cannot fire.

Files Changed

File Change
docs/coverage.md Modified — two-axis target, runtime section rewritten
docs/demand.md, docs/README.md, README.md Modified — depth vs routing wording
docs/contributing.md Modified — 194 → 420 models, rewrap
cmd/devcloud/coverage_test.go Modified — target-table gate
cmd/devcloud/budget_test.go Added — fleet initialization gate + startup measurement
cmd/devcloud/sync_test.go Modified — 194 → 420 models
.github/workflows/ci.yml Modified — binary size ceiling
docs/testing/phase-4-review-followup.tdd.md Added — full RED/GREEN record including both wrong answers
changes/unreleased/*.yaml Added — changelog

Testing

go vet ./...                  # clean
gofmt -l ./cmd ./internal     # clean
go test ./... -count=1        # all packages ok

CI green on all 9 checks. cmd/devcloud's 0.0% statement coverage is expected and unchanged — that package asserts the binary's published claims against the registry it ships, not main.go's statements.

Known gaps, stated deliberately in docs/testing/phase-4-review-followup.tdd.md: startup is not gated in CI at all; the 45 MiB ceiling has ~8 MiB of slack; Init is exercised with default options only.

Related Issues

None. Follows #160, #161, #162.

…g read

docs/coverage.md published 431 registered at the top and, further down, called
the target "205 services, not 431" while describing those 431 as not targeted.
Both halves were written honestly at different times and the page could not be
read as one claim. The summary table has been gated against the binary since
Milestone 6; the target table under #the-target never was, so every number there
could be wrong with CI green, and for a while every number there was.

The fix is to stop pretending there is one target. Routing is a safety property:
a registered service is answered at localhost:4747, an unregistered one leaves
the machine and bills a real account, so the only number that satisfies it is
all of them — 431 of 431, met. Depth is the separate and smaller promise, still
205, still where the 2026-09-05 demand study put it, because what that study
refused was the cost of hand-building 283 services and the codegen scaffold
removed that cost without producing any evidence of demand. A service count can
no longer be read as a promise of fidelity because the page no longer states one
number that could carry both meanings.

TestPublishedTargetTableMatchesTheBinary now reads the two-axis table the way
tierRow reads the tiers: the routing target is checked against
plugin.DefaultRegistry.RegisteredServices(), and because the serving target is a
decision rather than a measurement, it is read from the page and used as the
arithmetic the third row must satisfy. The table cannot be internally
inconsistent and cannot drift from the binary.

CI now fails the build past 45 MiB. That is a regression ceiling and not the
published figure — coverage.md measures 36.8 MiB on Apple Silicon and CI builds
for linux, so a tight budget would fail on the platform difference rather than
on a regression. The ceiling is named in coverage.md so the failure message
points somewhere the number exists.

cmd/devcloud/budget_test.go gates startup, and its first version did not. It
timed Registry.Construct, which only calls the factory, and every factory in the
tree is a struct literal — 84 microseconds against a 150 ms budget. The cost is
in Init: S3Provider.Init does an os.MkdirAll and opens a SQLite database, and
431 of those is how a 42 ms startup becomes a second. Verified rather than
assumed: injecting 8 ms per service into Registry.Init left the old gate at
79 microseconds and passing, and fails the new one at 4.0 s. It now brings every
service up the way main.go does, measures 141 ms for all 431, and fails past 2 s.
The ceiling is loose on purpose. The gate runs on a shared runner where a 2x
reading is indistinguishable from a noisy neighbour, so it catches the
regression that is real — a provider doing per-call work at startup, which
arrives as 10x. It also fails if any registered service stops initializing,
which main.go only warns about.

The 194-model figures in sync_test.go and contributing.md predate vendoring the
long tail and are now 420. The 2026-09-06 churn measurement is unchanged but
says what it was taken at, since the diff a reviewer now faces is larger than
anything measured there by roughly the same factor.

RED/GREEN evidence: docs/testing/phase-4-review-followup.tdd.md
@github-actions github-actions Bot added documentation Improvements or additions to documentation ci CI/CD workflows and scripts tests Test code and test infrastructure labels Sep 12, 2026
The 2 s ceiling this branch added was chosen as ~14x the 141 ms the fleet takes
to come up locally, on the reasoning that 14x was room enough for a shared
runner. CI measured the same path at 2.048 s on arm64. The runner is itself 14x
slower, so the whole margin was the machine and none of it was headroom.

Raising the number does not rescue the design. What is worth catching is a
provider that starts opening a file or a database per service at startup, and
that costs 1.2x to 2x, because that is what 431 extra file opens are worth. A
ceiling loose enough to clear a 14x difference between machines plus run-to-run
variance cannot also fail at 2x. Even the 8 ms-per-service injection used to
prove the gate works — far larger than a realistic regression — only reaches
5.5 s against a 2.05 s baseline, so it would pass under any ceiling that does
not flake. docs/coverage.md raised this objection before the gate was written;
the gate overrode it and CI settled it.

So the ceiling is gone and the test keeps the half that holds. Every one of the
431 registered services must initialize, which is new and was never covered:
main.go brings the long tail up with fatal=false, so a service that is
registered but can no longer initialize degrades to a warning in a log nobody
reads. The elapsed time is logged beside it and becomes an assertion only when
DEVCLOUD_STARTUP_BUDGET names one, which is how the published figure is re-taken
on a machine whose speed is known.

The binary size gate is untouched. Size does not vary with how busy a runner is.
Two figures in the runtime section were written before CI had produced the
numbers they describe, and one of them was wrong.

The binary ceiling was justified as absorbing a platform difference: 45 MiB
against a published 36.8 MiB measured on Apple Silicon, with CI building for
linux. CI measures 35 MiB on arm64 and 37 MiB on amd64, so the platform accounts
for under 2 MiB and the remaining 8 MiB is slack. Saying so is the point of
publishing it.

The startup retreat was argued from one reading — 2.048 s on an arm64 runner
against 141 ms locally. The run after it measured 1.414 s on the same runner type
and 1.375 s on amd64, on identical code. A 45% swing between runs is a better
argument than a slow machine: the regression worth catching costs 1.2x to 2x, so
it is smaller than the noise a ceiling would have to clear. The docs and the TDD
evidence now argue from the variance rather than from the one failure.
@skyoo2003
skyoo2003 merged commit 75efc51 into main Sep 13, 2026
8 checks passed
@skyoo2003
skyoo2003 deleted the feat/phase-4-two-axis-docs-and-gates branch September 13, 2026 02:47
skyoo2003 added a commit that referenced this pull request Sep 13, 2026
…e fragments

The 400-character ceiling was wide enough to hold two full sentences plus a
subordinate clause, so it never pushed against the one-sentence rule it was
meant to back — the #163 fragment reached 484 and still read as being in
the spirit of the limit. At 200 the ceiling and the rule push the same way.

Refits the eight unreleased fragments that were over. What came out is
detail the linked issue already carries: the eight service names in #161,
the CI-runner timing in #163, the DynamoDB and Lambda operation counts in
#167.

Also trims RELEASE.md's "right length" example, which was 203 characters
and would have failed the limit the paragraph above it states, and labels
both examples with their length.
skyoo2003 added a commit that referenced this pull request Sep 13, 2026
…g to 200 (#168)

* fix: give six changelog fragments the issue link the release tag checks

Six fragments wrote `Issue:` at the top level instead of under `custom:`,
where `.changie.yaml` declares it. `changie batch` rendered them as
`[#<no value>](.../issues/<no value>)`, which release.yml:234 rejects — the
tag would have failed at the notes-validation step, after the guardrails
had already passed.

The pre-flight grep in RELEASE.md (`grep -L 'Issue: "[0-9]'`) matches both
spellings, so it did not catch this.

Also trims the #163 fragment from 484 to 337 characters, under the 400-char
ceiling `.changie.yaml` sets and `changie new` enforces on fragments it
writes itself. The CI-runner timing detail it dropped is in the issue.

* docs: correct the two protocol-table rows that miscount EC2 as having no model

The protocol table said 12 services have no in-tree Smithy model and exactly
one speaks a protocol the parser cannot read. Counted against
internal/generated/fidelity/manifest_gen.go, it is 11 and two: `ec2` is
model-backed but speaks `aws.protocols#ec2Query`, which parser.go:441-453
does not recognise alongside the five it does.

That put the page at odds with fidelity-manifest.md:106 and
compatibility-policy.md:78, which both say 11, and left a core service's
missing engine coverage unexplained. Rows now sum to 431 either way, so the
arithmetic did not expose it, and no test gates this table.

Adds a paragraph separating what EC2 loses from what a registered-only
service loses: EC2 is served by a hand-written provider and is not in the
registered-only five, but its model-declared tail stays `unimplemented`
rather than falling back to `auto-crud`.

* docs: halve the changelog body ceiling to 200 characters and refit the fragments

The 400-character ceiling was wide enough to hold two full sentences plus a
subordinate clause, so it never pushed against the one-sentence rule it was
meant to back — the #163 fragment reached 484 and still read as being in
the spirit of the limit. At 200 the ceiling and the rule push the same way.

Refits the eight unreleased fragments that were over. What came out is
detail the linked issue already carries: the eight service names in #161,
the CI-runner timing in #163, the DynamoDB and Lambda operation counts in
#167.

Also trims RELEASE.md's "right length" example, which was 203 characters
and would have failed the limit the paragraph above it states, and labels
both examples with their length.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci CI/CD workflows and scripts documentation Improvements or additions to documentation tests Test code and test infrastructure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant