Skip to content
5 changes: 5 additions & 0 deletions changes/unreleased/Fixed-20260906-153000.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
kind: Fixed
body: 'The four Amazon Lex services are reachable from boto3. `lex-models`, `lex-runtime`, `lexv2-models` and `lexv2-runtime` all sign as `lex`, and no service is called `lex`, so the alias stayed contested and every Lex call died as `UnknownService` — the services were registered, counted as serving operations, and reachable by nobody. The shared-signing-name split that already separates `mediastore-data`, `sagemaker-runtime` and Timestream now recognises a signing name that is a group key rather than a member service ID, so each request is answered by the sibling whose route table models its method and path. `DeleteBot` is claimed by two of them at `DELETE /bots/{id}` and is still refused rather than guessed at. Compatibility-tested services: 199 to 203'
time: 2026-09-06T15:30:00.000000+09:00
custom:
Issue: "146"
5 changes: 5 additions & 0 deletions changes/unreleased/Fixed-20260906-160000.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
kind: Fixed
body: 'Three operations the fidelity manifest calls `hand-verified` are now reachable. `appsync.ListApis`, `eks.ListAccessPolicies` and `opensearch.ListApplications` were each implemented by a `case` clause in their provider, but the provider''s hand-written path resolver did not know the route boto3 sends — `GET /v2/apis`, `GET /access-policies` and `GET /2021-01-01/opensearch/list-applications` — so the operation arrived empty and the call answered `NotImplemented`. Each provider now falls back to its own model''s route table when its resolver recognises nothing, which recovers the operation instead of losing it. No published figure moves: these operations were already labelled served, and the label is now true'
time: 2026-09-06T16:00:00.000000+09:00
custom:
Issue: "146"
5 changes: 5 additions & 0 deletions changes/unreleased/Fixed-20260906-170000.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
kind: Fixed
body: 'The fidelity manifest no longer drops operations with short names. The scan that decides which operations are hand-verified required a name of four characters or more, so `resourcegroups.Untag` was recorded and `Tag` — implemented beside it in the same dispatch switch — was reported as unimplemented. A probe then asked for `Tag`, got the answer the provider has always given, and reported a fabricated success that was never fabricated: the code was real and the manifest was wrong about it. Short dispatch literals are now collected separately and promoted only where the service''s own model declares them, so `Tag` counts and the `GET`/`PUT` literals every path resolver switches on still cannot become operations. Hand-verified operations: 4,496 to 4,497'
time: 2026-09-06T17:00:00.000000+09:00
custom:
Issue: "146"
5 changes: 5 additions & 0 deletions changes/unreleased/Fixed-20260906-180000.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
kind: Fixed
body: 'S3 no longer answers an unimplemented bucket sub-resource with a bucket listing. `GET /{Bucket}?analytics` matched no sub-resource branch and fell through to `ListObjects`, so botocore read the resulting 200 `<ListBucketResult>` as a successful `ListBucketAnalyticsConfigurations`. `?inventory`, `?metrics`, `?replication`, `?lifecycle`, `?encryption`, `?versions` and `?object-lock` all took the same path. A bucket-level GET carrying a query parameter that is neither a sub-resource this provider serves nor a parameter a listing carries now returns a clean `NotImplemented`. The check is an allow-list of listing parameters rather than a deny-list of sub-resource names, so a sub-resource AWS adds later declines instead of being served as a listing'
time: 2026-09-06T18:00:00.000000+09:00
custom:
Issue: "146"
55 changes: 32 additions & 23 deletions docs/coverage.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ things:
| **Registered** | The gateway routes the service. A call reaches DevCloud instead of falling through to real AWS. | **205** |
| **Serving ≥1 operation** | At least one operation returns a real, store-backed answer — hand-written or served by the generic CRUD engine. | **201** |
| **Registered-only** | Routed, but every operation declines with a clean AWS error. Nothing is served. | **4** |
| **Compatibility-tested** | A boto3 test exercises the service in CI and passes — either serving an operation or declining cleanly. | **199** |
| **Compatibility-tested** | A boto3 test exercises the service in CI and passes — either serving an operation or declining cleanly. | **203** |

Every number on this page is asserted against the binary by
`go test ./cmd/devcloud/`. Editing one here without the code moving fails CI, and
Expand All @@ -22,9 +22,9 @@ Per operation, from the [fidelity manifest](fidelity-manifest.md):

| Tier | Operations |
|---|---|
| `hand-verified` | 4,496 |
| `hand-verified` | 4,497 |
| `auto-crud` | 5,193 |
| `unimplemented` | 2,718 |
| `unimplemented` | 2,717 |
| **total known** | **12,407** |

## Why a registered service can serve nothing
Expand Down Expand Up @@ -63,36 +63,39 @@ and `sagemakerruntimehttp2` (`InvokeEndpoint*`), and `rds-data`
(`ExecuteStatement`, `BeginTransaction`, `CommitTransaction`,
`RollbackTransaction`). No protocol change reaches them.

## Why the compatibility-tested number is 199, not 205
## Why the compatibility-tested number is 203, not 205

Every registered service is exercised by
`tests/compatibility/test_service_smoke.py`, which parametrises over the
generated service list rather than a hand-written one — so a service cannot be
registered and quietly go untested, which is what 31 of them were until this was
measured. Six are excluded, in two groups, and neither group is a backlog.
measured. Two are excluded, and they are not a backlog.

**Two have no boto3 client at all.** botocore publishes 431 clients and neither
`sagemakerruntimehttp2` nor `transcribestreaming` is among them —
`sagemaker-runtime` and `transcribe` are different clients with different APIs.
No boto3 test can exist for a client that does not exist. This is a property of
the AWS SDK, not of DevCloud, and it is the ceiling on this number.

**Four are registered and unreachable.** All four Lex clients — `lex-models`,
`lex-runtime`, `lexv2-models`, `lexv2-runtime` — sign as `lex`, and no service
is named `lex`, so the alias stays contested and routes nowhere rather than
sending one service's traffic to another. The consequence, which this page did
not state until it was measured: the "reached by its own unambiguous name"
escape below does not exist for a boto3 caller, because boto3 signs with the
contested name and offers nothing else. The four are counted in *serving ≥1
operation*, because their operations are classified and would be served; they
are not counted here, because nothing can ask for them.

Splitting them on the URL is what resolved `opensearch` / `elasticsearchservice`
and API Gateway v1 / v2, and it does not work here: `/bots` is the first path
segment for three of the four. A correct split needs full path matching across
all four, which is routing work rather than coverage work.
`test_lex_services_are_unreachable_from_boto3` pins the current state, so
fixing the routing fails that test and moves this number up with it.
**The four Lex services used to be a third exclusion, and are not any more.**
All four Lex clients — `lex-models`, `lex-runtime`, `lexv2-models`,
`lexv2-runtime` — sign as `lex`, and no service is named `lex`, so the alias is
contested and still routes nowhere on its own. What this page got wrong was the
escape it offered: "reached by its own unambiguous name" does not exist for a
boto3 caller, because boto3 signs with the contested name and offers nothing
else. Measuring it found four services registered, counted as serving, and
reachable by nobody.

The split that resolved `opensearch` / `elasticsearchservice` and API Gateway
v1 / v2 does work here, and it needed no new routing. Services that share a
signing name are already separated by asking which one's route table models the
request; the group was simply never reached, because the lookup only recognised
a *member* service ID and `lex` is the group's *key*. Each Lex request now
reaches the sibling that models its method and path — `GET /bots` is
`lex-models`, `POST /bots` is `lexv2-models`, `/bot/…/session` is `lex-runtime`,
`/bots/…/botAliases/…/sessions/…` is `lexv2-runtime`. One operation is claimed by
two of them, `DeleteBot` at `DELETE /bots/{id}`, and it is still refused rather
than guessed at: deleting the wrong bot is worse than an honest error.

Being engine-servable is not the same as being served. Thirteen of the fifteen
`query` services and three of the four `rest-xml` ones have hand-written
Expand Down Expand Up @@ -182,8 +185,14 @@ so `sagemaker` is claimed by eight services at once. Where exactly one claimant
carries the name as its own service ID, that service wins and the rest are
treated as borrowers — see `codegen.BuildAliases`. Where none does, the alias is
left unrouted rather than guessed: all four Lex services sign as `lex` and none
is called `lex`, so `lex` routes nowhere and each Lex service is reached by its
own unambiguous name.
is called `lex`, so `lex` resolves to nothing on its own.

An unrouted alias is not the same as an unreachable service, and this page once
conflated them — it claimed each Lex service was "reached by its own unambiguous
name", which no boto3 caller has, since boto3 sends the contested name and
nothing else. The name is still not guessed; the request is instead handed to
the group of services that share it, and answered by the one whose route table
models its method and path. Two candidates, or none, and it is refused.

## The target, and the rule that set it

Expand Down
14 changes: 14 additions & 0 deletions internal/codegen/gen_fidelity.go
Original file line number Diff line number Diff line change
Expand Up @@ -91,9 +91,23 @@ func BuildFidelityData(
hand[op] = true
universe[op] = true
}
declared := make(map[string]bool, len(modelOps[serviceID]))
for _, op := range modelOps[serviceID] {
declared[op] = true
universe[op] = true
}
// A dispatch literal too short to be told from an HTTP verb by shape is
// an operation when, and only when, the model declares it.
// resourcegroups implements Tag and Untag in one switch and only Untag
// was long enough to be scanned, so the manifest called implemented code
// unimplemented. The model is asked here rather than in the scanner
// because this is where the model already is — the scan reads Go source,
// not Smithy.
for _, op := range provider.ShortOperations {
if declared[op] {
hand[op] = true
}
}
for op := range crudOps[serviceID] {
universe[op] = true
}
Expand Down
43 changes: 43 additions & 0 deletions internal/codegen/gen_fidelity_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,49 @@ func TestBuildFidelityDataRequiresEngineWiring(t *testing.T) {
"a provider that never reaches the engine serves nothing, whatever the CRUD registry says")
}

// TestBuildFidelityDataPromotesShortDeclaredOperations covers the operation name
// too short for the scanner's shape test.
//
// opNamePattern requires four characters, so `Untag` is scanned and `Tag` is
// not, even though resourcegroups implements both in the same switch. The
// manifest called Tag unimplemented, a probe asked for it, got the 200 the
// provider has always returned, and reported a fabricated success that was
// never fabricated — the manifest was simply wrong about served code.
//
// Loosening the pattern alone is not the fix: at two characters it admits the
// HTTP-verb literals that path-resolver switches are full of. The model is what
// separates them. `Tag` is an operation because resourcegroups.json says so;
// `GET` is not an operation in any model AWS publishes.
func TestBuildFidelityDataPromotesShortDeclaredOperations(t *testing.T) {
modelOps := map[string][]string{"svc": {"Tag", "Untag", "ListGroups"}}
providers := map[string]ProviderScan{
"svc": {
Operations: []string{"Untag", "ListGroups"},
// Below the strict shape test. Only the model decides which of
// these is an operation.
ShortOperations: []string{"Tag", "GET", "PUT"},
EngineWired: false,
},
}

data := BuildFidelityData(modelOps, map[string]string{"svc": "rest-json"}, providers, nil)

assert.Equal(t, "HandVerified", tierOf(t, data, "svc", "Tag"),
"the model declares Tag and the provider dispatches it, so it is served")
assert.Equal(t, "HandVerified", tierOf(t, data, "svc", "Untag"),
"the long name was never in doubt and must not regress")

for _, svc := range data.Services {
if svc.ServiceID != "svc" {
continue
}
for _, op := range svc.Ops {
assert.NotContains(t, []string{"GET", "PUT"}, op.Name,
"a short literal no model declares must not become an operation")
}
}
}

// TestBuildFidelityDataHandVerifiedBeatsWiring keeps the documented precedence:
// a hand-written implementation always wins, and it does not depend on engine
// wiring because such an operation never falls through to the engine at all.
Expand Down
60 changes: 47 additions & 13 deletions internal/codegen/scan_handverified.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,18 @@ import (
// switch with real operations (HTTP verbs, content types, protocol tokens).
var opNamePattern = regexp.MustCompile(`^[A-Z][A-Za-z0-9]{3,}$`)

// shortOpNamePattern is the same shape at two or three characters — the length
// where a real operation name and an HTTP verb stop being distinguishable by
// shape. `Tag` is an operation of resourcegroups; `GET` and `PUT` are what a
// path resolver switches on, and every provider that routes by path has them.
//
// Literals matching this are collected into ProviderScan.ShortOperations and
// are not operations yet. BuildFidelityData promotes the ones the service's
// model declares, which is the only source that can tell the two apart. Without
// the split, resourcegroups.Tag was dropped while Untag beside it in the same
// switch was kept, and the manifest reported implemented code as unimplemented.
var shortOpNamePattern = regexp.MustCompile(`^[A-Z][A-Za-z0-9]{1,2}$`)

// pathRoutedOps lists the operations of providers that route on HTTP method and
// path instead of an operation name, so no operation-name literal appears in
// their HandleRequest for the scan to find. Keep in sync by hand when these
Expand Down Expand Up @@ -70,6 +82,10 @@ var pathRoutedOps = map[string][]string{
type ProviderScan struct {
// Operations are the operation names the provider dispatches by hand.
Operations []string
// ShortOperations are dispatch literals too short to be told apart from an
// HTTP verb by shape. They become operations only where the service's model
// declares them; see shortOpNamePattern.
ShortOperations []string
// EngineWired reports whether the provider hands the operations it does not
// implement to the generic CRUD engine, by returning plugin.ErrUnhandledOp.
// Only a wired provider's CRUD-shaped operations are actually served, so
Expand Down Expand Up @@ -110,9 +126,11 @@ func ScanProviders(servicesDir string) (map[string]ProviderScan, error) {
return nil, err
}
for id, recvType := range pkg.serviceTypes {
ops, short := pkg.dispatchedOps(recvType)
result[id] = ProviderScan{
Operations: opsFor(id, pkg.dispatchedOps(recvType)),
EngineWired: pkg.wiresCRUDEngine(recvType),
Operations: opsFor(id, ops),
ShortOperations: sortedSet(short),
EngineWired: pkg.wiresCRUDEngine(recvType),
}
}
}
Expand All @@ -125,12 +143,18 @@ func opsFor(serviceID string, scanned map[string]bool) []string {
if declared, ok := pathRoutedOps[serviceID]; ok {
return append([]string(nil), declared...)
}
ops := make([]string, 0, len(scanned))
for op := range scanned {
ops = append(ops, op)
return sortedSet(scanned)
}

// sortedSet flattens a literal set into a stable slice, so a regeneration diff
// shows provider changes and nothing else.
func sortedSet(set map[string]bool) []string {
out := make([]string, 0, len(set))
for k := range set {
out = append(out, k)
}
sort.Strings(ops)
return ops
sort.Strings(out)
return out
}

// packageScan is one service package's providers, keyed by receiver type name.
Expand Down Expand Up @@ -191,8 +215,11 @@ func scanServiceDir(dir string) (packageScan, error) {
// provider switches in HandleRequest itself: sqs branches on protocol into
// handleQueryRequest/handleJSONRequest, and kafka wraps handleRequest to
// camelCase the response.
func (p packageScan) dispatchedOps(recvType string) map[string]bool {
ops := make(map[string]bool)
// It returns two sets: the literals that carry an operation's shape on their
// own, and the short ones only the model can confirm.
func (p packageScan) dispatchedOps(recvType string) (ops, short map[string]bool) {
ops = make(map[string]bool)
short = make(map[string]bool)
visited := make(map[string]bool)

var walk func(name string)
Expand All @@ -203,13 +230,13 @@ func (p packageScan) dispatchedOps(recvType string) map[string]bool {
}
visited[name] = true

collectCaseLiterals(fn.Body, ops)
collectCaseLiterals(fn.Body, ops, short)
for _, delegate := range delegateCalls(fn) {
walk(delegate)
}
}
walk("HandleRequest")
return ops
return ops, short
}

// wiresCRUDEngine reports whether a provider type opts into the generic CRUD
Expand Down Expand Up @@ -252,7 +279,7 @@ func (p packageScan) wiresCRUDEngine(recvType string) bool {
// Keyed is the point: `switch op { case "GetItem": }` dispatches, whereas a
// tagless `switch { case remaining == 0: }` is control flow whose cases are
// expressions, not operation names.
func collectCaseLiterals(body *ast.BlockStmt, ops map[string]bool) {
func collectCaseLiterals(body *ast.BlockStmt, ops, short map[string]bool) {
ast.Inspect(body, func(n ast.Node) bool {
sw, ok := n.(*ast.SwitchStmt)
if !ok || sw.Tag == nil {
Expand All @@ -264,8 +291,15 @@ func collectCaseLiterals(body *ast.BlockStmt, ops map[string]bool) {
continue
}
for _, expr := range clause.List {
if lit, ok := stringLiteral(expr); ok && opNamePattern.MatchString(lit) {
lit, ok := stringLiteral(expr)
if !ok {
continue
}
switch {
case opNamePattern.MatchString(lit):
ops[lit] = true
case shortOpNamePattern.MatchString(lit):
short[lit] = true
}
}
}
Expand Down
21 changes: 21 additions & 0 deletions internal/codegen/scan_handverified_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,27 @@ func TestScanProviders(t *testing.T) {
}
}

// TestScanProvidersCollectsShortLiteralsSeparately covers the names the shape
// test is too strict for.
//
// opNamePattern needs four characters. `Tag` has three, so resourcegroups —
// which implements Tag and Untag next to each other in one switch — had one of
// them scanned and the other silently dropped. They are collected here rather
// than admitted into Operations, because at this length the scan cannot tell an
// operation from the HTTP verbs a path resolver switches on. BuildFidelityData
// asks the model, which can.
func TestScanProvidersCollectsShortLiteralsSeparately(t *testing.T) {
scans, err := ScanProviders("../services")
require.NoError(t, err)

assert.Contains(t, scans["resourcegroups"].ShortOperations, "Tag",
"Tag is dispatched by the provider and is three characters long")
assert.NotContains(t, scans["resourcegroups"].Operations, "Tag",
"the strict set keeps its shape test; promotion happens against the model")
assert.Contains(t, scans["resourcegroups"].Operations, "Untag",
"the four-character sibling was never in doubt")
}

// TestScanProvidersKeepsOperationsTheModelOmits is the regression that made the
// scan drop real coverage: it used to intersect with the Smithy model, so every
// operation a provider serves beyond its in-tree model vanished from the
Expand Down
11 changes: 11 additions & 0 deletions internal/gateway/protocol.go
Original file line number Diff line number Diff line change
Expand Up @@ -318,7 +318,18 @@ func resolveSharedSigningName(service string, match func(candidate string) bool)

// signingNameOf finds the signing name whose group contains service. The groups
// are small and few, and this runs once per request that reaches a shared name.
//
// A service may also arrive *as* a signing name. Most shared names are the
// parent service's own ID, so the group is found by membership — but four Lex
// services sign as "lex" and none of them is called that, so the alias stays
// contested and normalizeServiceID hands the unresolved name straight through.
// Recognising the group key is what lets those requests reach the split at all;
// the split then answers with the sibling that models the request, or refuses
// when two do.
func signingNameOf(service string) string {
if _, ok := aliases.SigningSiblings[service]; ok {
return service
}
for name, group := range aliases.SigningSiblings {
for _, id := range group {
if id == service {
Expand Down
Loading