Skip to content

fix: repair AWS service onboarding and measure its cost - #136

Merged
skyoo2003 merged 5 commits into
mainfrom
feat/repeatable-service-onboarding
Sep 5, 2026
Merged

skyoo2003 merged 5 commits into
mainfrom
feat/repeatable-service-onboarding

Conversation

@skyoo2003

Copy link
Copy Markdown
Owner

Summary

The write-only-mutator sweep decided by name arithmetic (occurrences == declaration lines), so a live namesake covered a dead declaration's zero call sites. Measured on this tree: send has 9 declarations and 120 occurrences, record 6, delete/tick/clamped 3 — if every call site of one send disappeared, the other eight kept the gate green on exactly the defect class it exists to catch.

The verdict now counts per declaration site. SwiftPM already writes an index store on every build ($(swift build --show-bin-path)/index/store), where each declaration carries a USR and each reference carries that USR, so the data was there for free. Tools/MutatorGate reads it; scripts/mutator-sweep.sh keeps the policy (allowlist, debug prefix rule).

Note on the issue's proposed fix: swift-syntax alone cannot close this. It makes declaration sites precise but cannot resolve which send a store.send(...) call binds to — that needs type inference, so name pooling would survive. Periphery was rejected too: its upstream repository is archived and the brew formula is deprecated (disabled 2027-08-12), so CI cannot depend on it. The index store is the semantic superset of both.

Closes #233

Type of change

  • Bug fix
  • New feature
  • Refactor / cleanup
  • Documentation
  • Other:

UI changes

N/A — no files under Sources/PokeTokenBar/UI/ changed.

What the gate now catches

scripts/verify-mutator-gate.sh plants six defects and compares exit codes, so the verdict lives in an exit code rather than a memory of having checked once:

# Injected Expected Name arithmetic Declaration sites
1 dead mutator named send, beside 9 live namesakes red green red
2 dead mutator with a unique name red red red
3 mutator with a caller green green green
4a debug prefix with a Tests caller green green green
4b debug prefix without a Tests caller red red red
5 reachable only through a protocol requirement green green green

Exactly one case flipped, which is both the RED evidence for this fix and proof the harness discriminates instead of painting everything red. Case 5 stays green because the tool follows .overrideOf relations — without that, a protocol-dispatched mutator would stop CI for a defect that does not exist.

What deleting the awk extractor exposed

A fourth counting asymmetry of the class #235 fixed: save() inside a non-func member was attributed to the preceding func. Three candidates changed, all measured:

  • clamped gone — a ghost. init(fileURL:) at PokemonChat.swift:675 calls save() at :687, and no func appears between :394 and :687, so a different type's clamped(x:y:) was carrying that call.
  • merge gone — a ghost. It is a local function nested inside backfillMissingOwnedNames().
  • backfillMissingOwnedNames() added — a real non-private mutator the old extractor structurally could not see, because its own nested func merge stole the name before the save() line. It has callers, so nothing was dead; the gate simply had never looked at it.

Findings on the current tree are exactly the existing allowlist (five debug* hooks with Tests callers, plus setHidden and setTheme), so no real code regressed.

New failure modes, each guarded and injection-tested

Moving to an index means the index itself can lie. All three guards were verified by injecting the failure:

  • Stale index → the sweep builds before it reads, so it cannot judge from an old store.
  • Missing or empty store → exit 2 with a message. Measured on both a nonexistent path and an empty directory. A silent "0 findings" here would mean a gate that sees nothing while reporting green.
  • Stale unit for a deleted file → skipped when the declaration line cannot be read (1 finding with the file present → 0 when only the file moves), so probe files leave no ghosts.

Remaining ceilings (recorded in defect-log.md)

  • Calls through a protocol or superclass are recorded against the requirement's USR, so with two or more conformers one live call keeps the others looking alive (dynamic dispatch; periphery has the same limit).
  • A dead cluster that only calls itself (A→B→A) passes.
  • Accessors (didSet, setters) calling save() are not candidates — the class stayed "functions", as before.

Cost

Measured
indexstore-db cold build 21s (no CI cache needed)
Sweep, warm ~5s
Full test-gate.sh 1m43s, coverage 90.84%

The sweep builds with --enable-code-coverage to match the gate's later swift test; mismatched flags made SwiftPM recompile the whole module (19s measured, 0.18s once aligned). Tools/MutatorGate is a separate package on purpose — putting indexstore-db in the root manifest would drag the dependency through the release build and signing path.

The dependency is pinned by revision to swift-6.1.2-RELEASE (54212fc), matching CI's toolchain (Swift 6.1.2 on macos-15); indexstore-db has no semver tags for SwiftPM to read.

Files Changed

File Change Why
Tools/MutatorGate/{Package.swift,Package.resolved,Sources/main.swift} Added Reads the index store, reports dead declarations per site
scripts/mutator-sweep.sh Added The sweep, extracted so the gate and the harness run the same code
scripts/verify-mutator-gate.sh Added Six-case defect-injection harness (stand-alone, not in CI)
scripts/test-gate.sh Modified −110 lines: the awk extractor and its approximation comments are gone
docs/reference/defect-log.md Modified Ceiling marked resolved; new asymmetry, new ceilings, injection evidence
docs/reference/memory-home-plan.md Modified No longer claims test-gate.sh owns the sweep

Testing

  • ./scripts/test-gate.sh — passes (coverage 90.84% ≥ 69%)
  • ./scripts/verify-mutator-gate.sh — 6/6 as expected; the same harness fails 1/6 on the pre-fix implementation
  • Guard injections for the empty store, the missing store, and the stale unit, as described above

The harness is deliberately outside CI: it rebuilds per case and would make the gate verify itself inside itself. It is the thing to run when the sweep's verdict is touched.

Checklist

  • swift build and swift test pass locally
  • PR title and description are written in English
  • UI changes are described above (before/after — images optional)
  • No copyrighted assets, secrets, or private tooling references are committed
  • Tests were added or updated for this change

…port block

TestGenerateImportsReproducesCommittedFile runs scripts/generate-imports.sh
against a scratch tree and compares the result with the committed
cmd/devcloud/imports.go. RED: the generator emits 'import ()' because it
skips every service directory lacking register.go, and no service has one —
registration lives in an init() inside provider.go. The weekly Smithy sync
runs this generator, so the job has been gutting service registration.

TestImportsCoverEveryRegisteredService locks imports.go against drift in
either direction. It passes today; it is a regression lock, not a RED case.
Gate on the package actually calling DefaultRegistry.Register instead of on
a register.go filename that no service has ever had. Registration lives in an
init() inside provider.go, so the old gate skipped all 103 packages and wrote
'import ()' — which empties the binary rather than degrading it, since an
unimported package never runs its init().

Three supporting changes, each with a failure it prevents:
- build into a temp file and mv on success, so a refusal cannot truncate a
  good imports.go (mirrors download-smithy-models.sh)
- refuse and exit non-zero below MIN_SERVICES (default 50), so the weekly
  sync cannot regenerate an empty file and report success
- delegate the SPDX line to add-spdx-header.sh rather than restating the rule

GREEN: TestGenerateImportsReproducesCommittedFile passes; a real run
reproduces the committed imports.go byte for byte (103 services).
…m auto-crud

Scaffold (internal/codegen/templates/scaffold_provider.go.tmpl):
HandleRequest returned generated.ErrNotImplemented, a plain refusal the
gateway never routes to the CRUD engine (internal/gateway/router.go:61), so a
freshly scaffolded service was registered, routed, and served zero
operations. It now returns plugin.ErrUnhandledOp, and Init keeps cfg.DataDir
instead of dropping it.

Manifest (gen_fidelity.go, scan_handverified.go):
tierFor labelled an operation auto-crud from CRUD-registry membership alone.
Registry membership means classifiable, not reachable — so the manifest would
have advertised every scaffolded service's CRUD operations as served while the
binary refused them. ProviderScan now reports EngineWired, detected by a
receiver-scoped AST walk for plugin.ErrUnhandledOp, and only a wired provider
can produce an auto-crud label. BuildFidelityData takes the ProviderScan map
directly, which also drops an intermediate map in cmd/codegen.

RED (executed, before the fix):
  --- FAIL: TestGenerateScaffoldOptsIntoCRUDEngine
      rendered scaffold does not contain "plugin.ErrUnhandledOp"
  --- FAIL: TestGenerateScaffoldKeepsDataDir
      rendered scaffold does not contain "cfg.DataDir"
  build failed: unknown field EngineWired in struct literal of type ProviderScan
  build failed: cannot use providers (map[string]ProviderScan) as
      map[string][]string in argument to BuildFidelityData

GREEN: all five tests pass; go build and the full go test ./... suite are
green across 110 packages.

Key validation — 'make codegen' produces NO diff under internal/generated/.
The manifest is byte-identical, which proves the wiring detection agrees with
today's 46 engine-wired JSON services and demoted no live operation.

No separate RED checkpoint commit: the repo's go-vet pre-commit hook refuses a
tree that does not compile, and the fidelity RED was a compile failure. The
RED output is preserved above and in the evidence report.
Task 5 — one parameterized boto3 fixture (service_client) plus a smoke test
that picks a parameterless List*/Describe* operation from the service model
and asserts it returns 200. Adding a service now means adding its name to a
list, not writing a fixture; the 104 existing named fixtures are untouched
because they back real tests rather than scaffolding.

Verified in both directions:
- comprehend, scaffolded with zero hand-written code, PASSES
- reverting the provider to generated.ErrNotImplemented makes it FAIL with
  'routed but not served', and the response is HTTP 500 InternalError — so
  before the template fix a scaffolded service also broke the project's
  'unimplemented operations return a clean AWS error' guarantee

Task 6 — measured baseline. comprehend, end to end:
  download 0s | codegen 2s | imports 0s | build 1s  = 3s total
  53 generated lines in internal/services/comprehend/provider.go
  0 hand-written lines
Manifest reports it honestly: 49 auto-crud, 36 unimplemented, 0 hand-verified.

Two existing invariants required updating, both encoding 'every service is
hand-written': TestScanProviders and TestFidelityManifestCoverage asserted
every service has at least one hand-verified operation. A scaffolded service
legitimately has none. The invariant is now 'every service serves something —
hand-verified or engine-served', which is stronger: zero of both is the real
defect and is what these checks now catch.

go test ./... green across 110 packages; make test-compat 776 passed
(775 baseline + 1 smoke); codegen verified idempotent by hash.
@github-actions github-actions Bot added tests Test code and test infrastructure codegen Smithy codegen and generated code services AWS service implementations labels Sep 5, 2026
Four Changie fragments for #136, plus the RED/GREEN evidence report at
.claude/tdd/aws-service-coverage-100.tdd.md so the proof survives a squash
merge. Also corrects the plan's protocol-scope claim: crud.JSONProtocol
matches only protocols starting with 'json', so rest-json cannot be
engine-served either — only 39 of 93 in-tree models qualify, not the 66 the
plan implied.
@skyoo2003
skyoo2003 merged commit 1fff608 into main Sep 5, 2026
8 checks passed
@skyoo2003
skyoo2003 deleted the feat/repeatable-service-onboarding branch September 5, 2026 07:23
skyoo2003 added a commit that referenced this pull request Sep 5, 2026
* test: add reproducer for imports.go regeneration emitting an empty import block

TestGenerateImportsReproducesCommittedFile runs scripts/generate-imports.sh
against a scratch tree and compares the result with the committed
cmd/devcloud/imports.go. RED: the generator emits 'import ()' because it
skips every service directory lacking register.go, and no service has one —
registration lives in an init() inside provider.go. The weekly Smithy sync
runs this generator, so the job has been gutting service registration.

TestImportsCoverEveryRegisteredService locks imports.go against drift in
either direction. It passes today; it is a regression lock, not a RED case.

* fix: generate-imports.sh emitted an empty import block

Gate on the package actually calling DefaultRegistry.Register instead of on
a register.go filename that no service has ever had. Registration lives in an
init() inside provider.go, so the old gate skipped all 103 packages and wrote
'import ()' — which empties the binary rather than degrading it, since an
unimported package never runs its init().

Three supporting changes, each with a failure it prevents:
- build into a temp file and mv on success, so a refusal cannot truncate a
  good imports.go (mirrors download-smithy-models.sh)
- refuse and exit non-zero below MIN_SERVICES (default 50), so the weekly
  sync cannot regenerate an empty file and report success
- delegate the SPDX line to add-spdx-header.sh rather than restating the rule

GREEN: TestGenerateImportsReproducesCommittedFile passes; a real run
reproduces the committed imports.go byte for byte (103 services).

* fix: scaffolded services served nothing while the manifest called them auto-crud

Scaffold (internal/codegen/templates/scaffold_provider.go.tmpl):
HandleRequest returned generated.ErrNotImplemented, a plain refusal the
gateway never routes to the CRUD engine (internal/gateway/router.go:61), so a
freshly scaffolded service was registered, routed, and served zero
operations. It now returns plugin.ErrUnhandledOp, and Init keeps cfg.DataDir
instead of dropping it.

Manifest (gen_fidelity.go, scan_handverified.go):
tierFor labelled an operation auto-crud from CRUD-registry membership alone.
Registry membership means classifiable, not reachable — so the manifest would
have advertised every scaffolded service's CRUD operations as served while the
binary refused them. ProviderScan now reports EngineWired, detected by a
receiver-scoped AST walk for plugin.ErrUnhandledOp, and only a wired provider
can produce an auto-crud label. BuildFidelityData takes the ProviderScan map
directly, which also drops an intermediate map in cmd/codegen.

RED (executed, before the fix):
  --- FAIL: TestGenerateScaffoldOptsIntoCRUDEngine
      rendered scaffold does not contain "plugin.ErrUnhandledOp"
  --- FAIL: TestGenerateScaffoldKeepsDataDir
      rendered scaffold does not contain "cfg.DataDir"
  build failed: unknown field EngineWired in struct literal of type ProviderScan
  build failed: cannot use providers (map[string]ProviderScan) as
      map[string][]string in argument to BuildFidelityData

GREEN: all five tests pass; go build and the full go test ./... suite are
green across 110 packages.

Key validation — 'make codegen' produces NO diff under internal/generated/.
The manifest is byte-identical, which proves the wiring detection agrees with
today's 46 engine-wired JSON services and demoted no live operation.

No separate RED checkpoint commit: the repo's go-vet pre-commit hook refuses a
tree that does not compile, and the fidelity RED was a compile failure. The
RED output is preserved above and in the evidence report.

* test: add generic service smoke coverage and measure onboarding cost

Task 5 — one parameterized boto3 fixture (service_client) plus a smoke test
that picks a parameterless List*/Describe* operation from the service model
and asserts it returns 200. Adding a service now means adding its name to a
list, not writing a fixture; the 104 existing named fixtures are untouched
because they back real tests rather than scaffolding.

Verified in both directions:
- comprehend, scaffolded with zero hand-written code, PASSES
- reverting the provider to generated.ErrNotImplemented makes it FAIL with
  'routed but not served', and the response is HTTP 500 InternalError — so
  before the template fix a scaffolded service also broke the project's
  'unimplemented operations return a clean AWS error' guarantee

Task 6 — measured baseline. comprehend, end to end:
  download 0s | codegen 2s | imports 0s | build 1s  = 3s total
  53 generated lines in internal/services/comprehend/provider.go
  0 hand-written lines
Manifest reports it honestly: 49 auto-crud, 36 unimplemented, 0 hand-verified.

Two existing invariants required updating, both encoding 'every service is
hand-written': TestScanProviders and TestFidelityManifestCoverage asserted
every service has at least one hand-verified operation. A scaffolded service
legitimately has none. The invariant is now 'every service serves something —
hand-verified or engine-served', which is stronger: zero of both is the real
defect and is what these checks now catch.

go test ./... green across 110 packages; make test-compat 776 passed
(775 baseline + 1 smoke); codegen verified idempotent by hash.

* docs: add changelog fragments and TDD evidence report

Four Changie fragments for #136, plus the RED/GREEN evidence report at
.claude/tdd/aws-service-coverage-100.tdd.md so the proof survives a squash
merge. Also corrects the plan's protocol-scope claim: crud.JSONProtocol
matches only protocols starting with 'json', so rest-json cannot be
engine-served either — only 39 of 93 in-tree models qualify, not the 66 the
plan implied.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

codegen Smithy codegen and generated code services AWS service implementations tests Test code and test infrastructure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant