Skip to content

feat: derive service aliases from models and onboard the AI/ML category - #137

Merged
skyoo2003 merged 4 commits into
mainfrom
feat/derive-service-aliases
Sep 5, 2026
Merged

skyoo2003 merged 4 commits into
mainfrom
feat/derive-service-aliases

Conversation

@skyoo2003

Copy link
Copy Markdown
Owner

Summary

Derives the gateway's service-alias table from the Smithy models instead of a hand-maintained 63-clause switch, then uses that to onboard the entire AI/Machine Learning category — all 48 upstream models, 42 of them new, with zero hand-written code. Registered services go 105 → 148.

This is Milestone 2 of .claude/prds/aws-service-coverage-100.prd.md ("One category proven end to end"), and it closes that PRD's Open Question 2 (service ID aliasing) and Open Question 3 (is 431 the right denominator).

Related Issue

Refs #136 — this is the follow-on milestone to "repair AWS service onboarding and measure its cost", which measured 3s/service but only for services whose X-Amz-Target prefix already resolved. rekognition failed outright there; it works here.

Fixes #

Changes

Alias derivation (the blocker)

  • ir.Model carries the six identifiers a caller can address a service by: shape name, signing name, endpoint prefix, ARN namespace, CloudFormation name, SDK id. Named provider-neutrally — a non-AWS source leaves them empty.
  • New codegen.BuildAliases derives alias → service ID and emits internal/generated/aliases/aliases_gen.go. It reports contested aliases rather than resolving them: picking one claimant silently sends the other's traffic to the wrong provider.
  • normalizeServiceID is now a 36-entry override map (contested / legacy / substitution, each group documented) plus the generated table. It no longer grows per service.
  • selfNamedClaimant settles the common false collision: every runtime split-out signs with its parent's name, so sagemaker is claimed by 8 services — but exactly one of them is sagemaker. That's a fact in the models, not a preference. It resolves 10 of 18 collisions and removed 3 overrides that had been written by hand.

AI/ML category

  • 42 new models, generated packages, and scaffolded providers. No hand-written Go.
  • 48/48 registered; 17 engine-served or hand-written; 31 registered-only.

Honesty about what "registered" means

  • The fidelity manifest now carries each service's Protocol, so registered and covered are separable numbers.
  • New docs/coverage.md publishes registered / serving ≥1 / registered-only and never the first alone.
  • TestFidelityManifestCoverage used to require every registered service to serve something. That is false by design once rest-json services are registered, so it is now the sharper check: the CRUD engine holds operations for this service and the manifest serves none — which is wiring, and nothing else.

Three defects the category surfaced, each fixed at its root

  • smithy.api#Primitive* was unmapped in smithyToGoType, so bedrockagent and omics emitted undefined identifiers and the build failed. All seven prelude primitives are mapped, not only the two that broke.
  • make codegen was not idempotent: healthlake.json bundles two namespaces with 8 colliding short names, and model.Shapes is keyed by short name while iterating a Go map. Fixed with sorted iteration plus service-namespace precedence — sorting alone is deterministic and still wrong, because the foreign namespace sorts first.
  • One routing regression, caught by the compat suite and missed by unit tests: mapping ses → sesv2 sent every SES v1 Query call to the v2 provider. SES v1 and SESv2 publish identical identifiers and only the protocol separates them, which DetectProtocol already handles.

Test Plan

  • CGO_ENABLED=0 go test ./... — all packages pass.
  • pytest tests/compatibility — 790 passed, 2 skipped (776 baseline + 14 new).
  • make codegen run three times, hashed — byte-identical (eca3ebfd…). This is a new guarantee; it was broken before this PR.
  • bash scripts/generate-imports.sh — idempotent.
  • All 93 aliases that routed before still route to the same service, pinned by a golden test extracted verbatim from the old switch before it was touched. This is the load-bearing check for replacing hand-written routing with generated routing.
  • rekognition end to end: download → codegen → scaffold → build → boto3 ListCollections returns 200, with zero hand-written gateway lines.
  • A registered-only service is asserted to answer with an AWS error code, never a fabricated 200 (test_registered_only_service_declines_cleanly).

Measured (105 → 147 services):

105 147 Extrapolated to 431
Binary 30.8 MiB 31.3 MiB ~35 MiB
Cold start ~460 ms ~460 ms flat
Idle RSS 57.3 MiB 57.7 MiB ~61 MiB

CI cost at 147: go test ./... 22s, compat suite 9s, codegen 2s, build 1s. The single-binary, zero-config property survives the full 431 target.

Reviewing this PR: 277 files, but 261 are generated packages, downloaded models, or scaffolds. The reviewable surface is 16 files — everything outside internal/generated/, smithy-models/, and internal/services/.

Checklist

  • Self-reviewed the code
  • Added/updated tests
  • Lint/format passes (pre-commit: gofmt, go vet, go build, ruff — all green)
  • Updated documentation (docs/coverage.md)
  • Added a Changie changelog fragment for user-facing changes

Known gaps

  • lex routes nowhere. All four Lex services sign as lex and none is named it; all are rest-json, so none can be engine-served. Each still routes under its own unambiguous name. Distinguishing them needs URL-path routing, which is only worth writing once one of them has a provider.
  • amazonrdsv19 and cognito are likewise deliberately unrouted — 2 of 3 and 1 of 2 claimants would be wrong. An empty override value means "decided, and the decision is not to guess".
  • The 29 legacy aliases are preserved because deleting them is a silent break for older SDKs, not because a client was observed sending them.

Evidence, with RED/GREEN output verbatim for all 23 guarantees: .claude/tdd/aws-service-coverage-100-milestone-2.tdd.md (local, .claude/ is gitignored).

Onboarding a service needed a hand-written entry in a 63-clause switch in
the gateway whenever its X-Amz-Target prefix did not already resolve.
rekognition could not be onboarded at all: its prefix is
RekognitionService, and normalizeServiceID had no case for it.

RED
  normalizeServiceID("rekognitionservice") = "rekognitionservice", want "rekognition"
  serviceFromTarget("RekognitionService.ListCollections") = "rekognitionservice", want "rekognition"
  plus compile-time: ir.Model has no ShapeName/SigningName/EndpointPrefix/
  ARNNamespace/CloudFormationName/SDKID; BuildAliases undefined;
  internal/generated/aliases does not exist.

GREEN
  ParseSmithyJSON carries the six identifiers a caller can address a service
  by. BuildAliases derives alias -> service ID from them and reports, rather
  than resolves, the 9 aliases more than one service claims. The gateway
  consults a documented override map (contested / legacy / substitution)
  then the generated table.

  93 aliases that routed before still route to the same service.
  rekognition onboards with 0 hand-written lines and answers a boto3 call.
  go test ./... green; 777 boto3 compat tests pass (776 + rekognition smoke).
  make codegen is idempotent (hashed across two runs).

One regression was caught by the compat suite and not by the unit tests:
mapping "ses" to sesv2 in the override map sent every SES v1 Query call to
the v2 provider. SES v1 and SESv2 publish identical identifiers and only
the protocol separates them, which DetectProtocol already does. The four
aliases involved are now pinned in the golden list.

Refs: .claude/plans/aws-service-coverage-100-milestone-2.plan.md (Tasks 1-3)
All 48 upstream AI/ML models are registered: 42 downloaded, generated and
scaffolded with no hand-written code. 105 -> 147 registered services.

Three defects the category surfaced, each fixed at its root:

  smithy.api#Primitive* was unmapped in smithyToGoType, so bedrockagent and
  omics emitted undefined identifiers and the build failed. All seven prelude
  primitives are now mapped, not only the two that happened to appear.

  Every runtime split-out signs with its parent's name, so sagemaker was
  claimed by 8 services and bedrock by 6, and BuildAliases reported them all
  as unresolvable. They are not: when exactly one claimant IS the alias, that
  service names itself and the rest borrow. selfNamedClaimant settles 10 of
  the 18 collisions from the models, and removes 3 hand-written overrides.
  The 8 that remain have no self-named claimant and stay a human decision.

  TestFidelityManifestCoverage required every registered service to serve
  something, which the 30 new rest-json services cannot. The manifest now
  carries each service's Protocol, and the invariant is the sharper one:
  the CRUD engine holds operations for this service but the manifest serves
  none. forecastquery is json-1.1 and legitimately serves nothing — its whole
  API is QueryForecast/QueryWhatIfForecast, neither CRUD-shaped.

Registered is not covered, and docs/coverage.md never prints the first
number without the other two: 148 registered, 117 serving >=1 operation,
31 registered-only. test_registered_only_service_declines_cleanly asserts a
registered-only service answers with an AWS error code, never a fabricated
200.

Measured, 105 -> 147 services: binary 30.8 -> 31.3 MiB, cold start ~460 ms
unchanged, idle RSS 57.3 -> 57.7 MiB. Extrapolated to 431: ~35 MiB binary,
~61 MiB RSS, startup flat. CI: go test ./... 22s, codegen 2s, build 1s,
790 boto3 compat tests in 9s.

Refs: .claude/plans/aws-service-coverage-100-milestone-2.plan.md (Tasks 4-7)
`make codegen` emitted a different healthlake/types.go and
crudregistry/registry_gen.go on every run.

healthlake.json bundles two namespaces — com.amazonaws.healthlake and
com.amazon.awshealthlakedatatransformationfrontendservice — with 8 colliding
short names. model.Shapes is keyed by short name and the loop iterated a Go
map, so the last writer won at random: ResourceArn or ResourceARN, TagList or
TagMap, differently each time. It is the only model of 137 with this property,
which is why it survived until a whole category was onboarded at once.

Sorting alone would be deterministic and still wrong: com.amazon.awshealthlake…
sorts before com.amazonaws.healthlake, so it would pick the foreign shape every
time. Sorted iteration plus service-namespace precedence — the model's own
service defines its API; a second bundled namespace is a dependency.

RED: 50 iterations, all "expected Mine, actual Theirs".
GREEN: three consecutive `make codegen` runs hash identical.

Refs: .claude/plans/aws-service-coverage-100-milestone-2.plan.md (Task 4)
@github-actions github-actions Bot added documentation Improvements or additions to documentation tests Test code and test infrastructure codegen Smithy codegen and generated code services AWS service implementations labels Sep 5, 2026
@skyoo2003
skyoo2003 merged commit ce6e622 into main Sep 5, 2026
8 checks passed
@skyoo2003
skyoo2003 deleted the feat/derive-service-aliases branch September 5, 2026 08:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

codegen Smithy codegen and generated code documentation Improvements or additions to documentation services AWS service implementations tests Test code and test infrastructure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant