Repository navigation
fix: repair AWS service onboarding and measure its cost - #136
Merged
Merged
Conversation
…port block TestGenerateImportsReproducesCommittedFile runs scripts/generate-imports.sh against a scratch tree and compares the result with the committed cmd/devcloud/imports.go. RED: the generator emits 'import ()' because it skips every service directory lacking register.go, and no service has one — registration lives in an init() inside provider.go. The weekly Smithy sync runs this generator, so the job has been gutting service registration. TestImportsCoverEveryRegisteredService locks imports.go against drift in either direction. It passes today; it is a regression lock, not a RED case.
Gate on the package actually calling DefaultRegistry.Register instead of on a register.go filename that no service has ever had. Registration lives in an init() inside provider.go, so the old gate skipped all 103 packages and wrote 'import ()' — which empties the binary rather than degrading it, since an unimported package never runs its init(). Three supporting changes, each with a failure it prevents: - build into a temp file and mv on success, so a refusal cannot truncate a good imports.go (mirrors download-smithy-models.sh) - refuse and exit non-zero below MIN_SERVICES (default 50), so the weekly sync cannot regenerate an empty file and report success - delegate the SPDX line to add-spdx-header.sh rather than restating the rule GREEN: TestGenerateImportsReproducesCommittedFile passes; a real run reproduces the committed imports.go byte for byte (103 services).
…m auto-crud
Scaffold (internal/codegen/templates/scaffold_provider.go.tmpl):
HandleRequest returned generated.ErrNotImplemented, a plain refusal the
gateway never routes to the CRUD engine (internal/gateway/router.go:61), so a
freshly scaffolded service was registered, routed, and served zero
operations. It now returns plugin.ErrUnhandledOp, and Init keeps cfg.DataDir
instead of dropping it.
Manifest (gen_fidelity.go, scan_handverified.go):
tierFor labelled an operation auto-crud from CRUD-registry membership alone.
Registry membership means classifiable, not reachable — so the manifest would
have advertised every scaffolded service's CRUD operations as served while the
binary refused them. ProviderScan now reports EngineWired, detected by a
receiver-scoped AST walk for plugin.ErrUnhandledOp, and only a wired provider
can produce an auto-crud label. BuildFidelityData takes the ProviderScan map
directly, which also drops an intermediate map in cmd/codegen.
RED (executed, before the fix):
--- FAIL: TestGenerateScaffoldOptsIntoCRUDEngine
rendered scaffold does not contain "plugin.ErrUnhandledOp"
--- FAIL: TestGenerateScaffoldKeepsDataDir
rendered scaffold does not contain "cfg.DataDir"
build failed: unknown field EngineWired in struct literal of type ProviderScan
build failed: cannot use providers (map[string]ProviderScan) as
map[string][]string in argument to BuildFidelityData
GREEN: all five tests pass; go build and the full go test ./... suite are
green across 110 packages.
Key validation — 'make codegen' produces NO diff under internal/generated/.
The manifest is byte-identical, which proves the wiring detection agrees with
today's 46 engine-wired JSON services and demoted no live operation.
No separate RED checkpoint commit: the repo's go-vet pre-commit hook refuses a
tree that does not compile, and the fidelity RED was a compile failure. The
RED output is preserved above and in the evidence report.
Task 5 — one parameterized boto3 fixture (service_client) plus a smoke test that picks a parameterless List*/Describe* operation from the service model and asserts it returns 200. Adding a service now means adding its name to a list, not writing a fixture; the 104 existing named fixtures are untouched because they back real tests rather than scaffolding. Verified in both directions: - comprehend, scaffolded with zero hand-written code, PASSES - reverting the provider to generated.ErrNotImplemented makes it FAIL with 'routed but not served', and the response is HTTP 500 InternalError — so before the template fix a scaffolded service also broke the project's 'unimplemented operations return a clean AWS error' guarantee Task 6 — measured baseline. comprehend, end to end: download 0s | codegen 2s | imports 0s | build 1s = 3s total 53 generated lines in internal/services/comprehend/provider.go 0 hand-written lines Manifest reports it honestly: 49 auto-crud, 36 unimplemented, 0 hand-verified. Two existing invariants required updating, both encoding 'every service is hand-written': TestScanProviders and TestFidelityManifestCoverage asserted every service has at least one hand-verified operation. A scaffolded service legitimately has none. The invariant is now 'every service serves something — hand-verified or engine-served', which is stronger: zero of both is the real defect and is what these checks now catch. go test ./... green across 110 packages; make test-compat 776 passed (775 baseline + 1 smoke); codegen verified idempotent by hash.
Four Changie fragments for #136, plus the RED/GREEN evidence report at .claude/tdd/aws-service-coverage-100.tdd.md so the proof survives a squash merge. Also corrects the plan's protocol-scope claim: crud.JSONProtocol matches only protocols starting with 'json', so rest-json cannot be engine-served either — only 39 of 93 in-tree models qualify, not the 66 the plan implied.
skyoo2003
added a commit
that referenced
this pull request
Sep 5, 2026
* test: add reproducer for imports.go regeneration emitting an empty import block
TestGenerateImportsReproducesCommittedFile runs scripts/generate-imports.sh
against a scratch tree and compares the result with the committed
cmd/devcloud/imports.go. RED: the generator emits 'import ()' because it
skips every service directory lacking register.go, and no service has one —
registration lives in an init() inside provider.go. The weekly Smithy sync
runs this generator, so the job has been gutting service registration.
TestImportsCoverEveryRegisteredService locks imports.go against drift in
either direction. It passes today; it is a regression lock, not a RED case.
* fix: generate-imports.sh emitted an empty import block
Gate on the package actually calling DefaultRegistry.Register instead of on
a register.go filename that no service has ever had. Registration lives in an
init() inside provider.go, so the old gate skipped all 103 packages and wrote
'import ()' — which empties the binary rather than degrading it, since an
unimported package never runs its init().
Three supporting changes, each with a failure it prevents:
- build into a temp file and mv on success, so a refusal cannot truncate a
good imports.go (mirrors download-smithy-models.sh)
- refuse and exit non-zero below MIN_SERVICES (default 50), so the weekly
sync cannot regenerate an empty file and report success
- delegate the SPDX line to add-spdx-header.sh rather than restating the rule
GREEN: TestGenerateImportsReproducesCommittedFile passes; a real run
reproduces the committed imports.go byte for byte (103 services).
* fix: scaffolded services served nothing while the manifest called them auto-crud
Scaffold (internal/codegen/templates/scaffold_provider.go.tmpl):
HandleRequest returned generated.ErrNotImplemented, a plain refusal the
gateway never routes to the CRUD engine (internal/gateway/router.go:61), so a
freshly scaffolded service was registered, routed, and served zero
operations. It now returns plugin.ErrUnhandledOp, and Init keeps cfg.DataDir
instead of dropping it.
Manifest (gen_fidelity.go, scan_handverified.go):
tierFor labelled an operation auto-crud from CRUD-registry membership alone.
Registry membership means classifiable, not reachable — so the manifest would
have advertised every scaffolded service's CRUD operations as served while the
binary refused them. ProviderScan now reports EngineWired, detected by a
receiver-scoped AST walk for plugin.ErrUnhandledOp, and only a wired provider
can produce an auto-crud label. BuildFidelityData takes the ProviderScan map
directly, which also drops an intermediate map in cmd/codegen.
RED (executed, before the fix):
--- FAIL: TestGenerateScaffoldOptsIntoCRUDEngine
rendered scaffold does not contain "plugin.ErrUnhandledOp"
--- FAIL: TestGenerateScaffoldKeepsDataDir
rendered scaffold does not contain "cfg.DataDir"
build failed: unknown field EngineWired in struct literal of type ProviderScan
build failed: cannot use providers (map[string]ProviderScan) as
map[string][]string in argument to BuildFidelityData
GREEN: all five tests pass; go build and the full go test ./... suite are
green across 110 packages.
Key validation — 'make codegen' produces NO diff under internal/generated/.
The manifest is byte-identical, which proves the wiring detection agrees with
today's 46 engine-wired JSON services and demoted no live operation.
No separate RED checkpoint commit: the repo's go-vet pre-commit hook refuses a
tree that does not compile, and the fidelity RED was a compile failure. The
RED output is preserved above and in the evidence report.
* test: add generic service smoke coverage and measure onboarding cost
Task 5 — one parameterized boto3 fixture (service_client) plus a smoke test
that picks a parameterless List*/Describe* operation from the service model
and asserts it returns 200. Adding a service now means adding its name to a
list, not writing a fixture; the 104 existing named fixtures are untouched
because they back real tests rather than scaffolding.
Verified in both directions:
- comprehend, scaffolded with zero hand-written code, PASSES
- reverting the provider to generated.ErrNotImplemented makes it FAIL with
'routed but not served', and the response is HTTP 500 InternalError — so
before the template fix a scaffolded service also broke the project's
'unimplemented operations return a clean AWS error' guarantee
Task 6 — measured baseline. comprehend, end to end:
download 0s | codegen 2s | imports 0s | build 1s = 3s total
53 generated lines in internal/services/comprehend/provider.go
0 hand-written lines
Manifest reports it honestly: 49 auto-crud, 36 unimplemented, 0 hand-verified.
Two existing invariants required updating, both encoding 'every service is
hand-written': TestScanProviders and TestFidelityManifestCoverage asserted
every service has at least one hand-verified operation. A scaffolded service
legitimately has none. The invariant is now 'every service serves something —
hand-verified or engine-served', which is stronger: zero of both is the real
defect and is what these checks now catch.
go test ./... green across 110 packages; make test-compat 776 passed
(775 baseline + 1 smoke); codegen verified idempotent by hash.
* docs: add changelog fragments and TDD evidence report
Four Changie fragments for #136, plus the RED/GREEN evidence report at
.claude/tdd/aws-service-coverage-100.tdd.md so the proof survives a squash
merge. Also corrects the plan's protocol-scope claim: crud.JSONProtocol
matches only protocols starting with 'json', so rest-json cannot be
engine-served either — only 39 of 93 in-tree models qualify, not the 66 the
plan implied.
This was referenced Sep 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The write-only-mutator sweep decided by name arithmetic (
occurrences == declaration lines), so a live namesake covered a dead declaration's zero call sites. Measured on this tree:sendhas 9 declarations and 120 occurrences,record6,delete/tick/clamped3 — if every call site of onesenddisappeared, the other eight kept the gate green on exactly the defect class it exists to catch.The verdict now counts per declaration site. SwiftPM already writes an index store on every build (
$(swift build --show-bin-path)/index/store), where each declaration carries a USR and each reference carries that USR, so the data was there for free.Tools/MutatorGatereads it;scripts/mutator-sweep.shkeeps the policy (allowlist,debugprefix rule).Note on the issue's proposed fix: swift-syntax alone cannot close this. It makes declaration sites precise but cannot resolve which
sendastore.send(...)call binds to — that needs type inference, so name pooling would survive. Periphery was rejected too: its upstream repository is archived and the brew formula is deprecated (disabled 2027-08-12), so CI cannot depend on it. The index store is the semantic superset of both.Closes #233
Type of change
UI changes
N/A — no files under
Sources/PokeTokenBar/UI/changed.What the gate now catches
scripts/verify-mutator-gate.shplants six defects and compares exit codes, so the verdict lives in an exit code rather than a memory of having checked once:send, beside 9 live namesakesdebugprefix with a Tests callerdebugprefix without a Tests callerExactly one case flipped, which is both the RED evidence for this fix and proof the harness discriminates instead of painting everything red. Case 5 stays green because the tool follows
.overrideOfrelations — without that, a protocol-dispatched mutator would stop CI for a defect that does not exist.What deleting the awk extractor exposed
A fourth counting asymmetry of the class #235 fixed:
save()inside a non-funcmember was attributed to the precedingfunc. Three candidates changed, all measured:clampedgone — a ghost.init(fileURL:)atPokemonChat.swift:675callssave()at:687, and nofuncappears between:394and:687, so a different type'sclamped(x:y:)was carrying that call.mergegone — a ghost. It is a local function nested insidebackfillMissingOwnedNames().backfillMissingOwnedNames()added — a real non-private mutator the old extractor structurally could not see, because its own nestedfunc mergestole the name before thesave()line. It has callers, so nothing was dead; the gate simply had never looked at it.Findings on the current tree are exactly the existing allowlist (five
debug*hooks with Tests callers, plussetHiddenandsetTheme), so no real code regressed.New failure modes, each guarded and injection-tested
Moving to an index means the index itself can lie. All three guards were verified by injecting the failure:
exit 2with a message. Measured on both a nonexistent path and an empty directory. A silent "0 findings" here would mean a gate that sees nothing while reporting green.Remaining ceilings (recorded in
defect-log.md)didSet, setters) callingsave()are not candidates — the class stayed "functions", as before.Cost
indexstore-dbcold buildtest-gate.shThe sweep builds with
--enable-code-coverageto match the gate's laterswift test; mismatched flags made SwiftPM recompile the whole module (19s measured, 0.18s once aligned).Tools/MutatorGateis a separate package on purpose — puttingindexstore-dbin the root manifest would drag the dependency through the release build and signing path.The dependency is pinned by revision to
swift-6.1.2-RELEASE(54212fc), matching CI's toolchain (Swift 6.1.2 onmacos-15); indexstore-db has no semver tags for SwiftPM to read.Files Changed
Tools/MutatorGate/{Package.swift,Package.resolved,Sources/main.swift}scripts/mutator-sweep.shscripts/verify-mutator-gate.shscripts/test-gate.shdocs/reference/defect-log.mddocs/reference/memory-home-plan.mdtest-gate.showns the sweepTesting
./scripts/test-gate.sh— passes (coverage 90.84% ≥ 69%)./scripts/verify-mutator-gate.sh— 6/6 as expected; the same harness fails 1/6 on the pre-fix implementationThe harness is deliberately outside CI: it rebuilds per case and would make the gate verify itself inside itself. It is the thing to run when the sweep's verdict is touched.
Checklist
swift buildandswift testpass locally