Repository navigation
fix: close the four recorded coverage defects - #146
Merged
Merged
Conversation
RED: CGO_ENABLED=0 go test ./internal/gateway/ -run TestDetectProtocol_Lex
--- FAIL: TestDetectProtocol_Lex/models_v1_get_bots
--- FAIL: TestDetectProtocol_Lex/models_v1_get_intents
--- FAIL: TestDetectProtocol_Lex/models_v1_builtin_intents
--- FAIL: TestDetectProtocol_Lex/models_v2_list_bots
--- FAIL: TestDetectProtocol_Lex/models_v2_create_bot
--- FAIL: TestDetectProtocol_Lex/models_v2_describe_bot
--- FAIL: TestDetectProtocol_Lex/runtime_v1_get_session
--- FAIL: TestDetectProtocol_Lex/runtime_v1_put_session
--- FAIL: TestDetectProtocol_Lex/runtime_v2_get_session
--- FAIL: TestDetectProtocol_Lex/runtime_v2_delete_session
--- PASS: TestDetectProtocol_Lex/delete_bot_is_contested
Every failure reads `expected: <a lex service>, actual: "lex"` — the request
never leaves the contested alias. The one PASS is the genuine collision
(DELETE /bots/{id}, claimed by both model services), which must stay refused.
Recorded in .claude/tdd/aws-service-coverage-100-milestone-6.tdd.md as
"Coverage and known gaps", item 3.
GREEN: CGO_ENABLED=0 go test ./internal/gateway/ -run TestDetectProtocol_Lex → ok
CGO_ENABLED=0 go test ./... → all pass
DEVCLOUD_BIN=dist/devcloud pytest tests/compatibility/
→ 1123 passed, 17 skipped, 2 xfailed (was 1119 passed)
All four Lex clients sign as "lex" and no service is called "lex", so the alias
stays contested. normalizeServiceID hands the unresolved name through, the
registry has nothing under it, and every Lex call died as UnknownService — four
services registered, counted in "serving >=1 operation", and reachable by nobody.
The fix needed no new routing. resolveSharedSigningName already picks the
sibling whose route table models the request; it was never reached, because
signingNameOf only recognised a *member* service ID and "lex" is the group's
key. One map lookup.
The four route tables separate cleanly on method and literal segment:
GET /bots lexmodelbuildingservice
POST /bots lexmodelsv2 (ListBots)
PUT /bots lexmodelsv2 (CreateBot)
GET /bot/{n}/alias/{a}/user/{u}/session lexruntimeservice
GET /bots/{b}/botAliases/.../sessions/{s} lexruntimev2
One genuine collision, DeleteBot at DELETE /bots/{id}, claimed by both model
services. It stays refused: deleting the wrong bot is worse than an honest
UnknownService. Asserted, not left to be found.
The pin test told whoever fixed this to delete it, so it is deleted and replaced
by a check on the direction that still matters — UNREACHABLE_FROM_BOTO3 must
stay empty. docs/coverage.md's compatibility-tested figure moves 199 -> 203, the
Lex exclusion section is rewritten, and the stale claim that each Lex service is
"reached by its own unambiguous name" is corrected: no boto3 caller has one.
Plan: .claude/plans/aws-service-coverage-100-milestone-7.plan.md, Task 1
RED: DEVCLOUD_BIN=dist/devcloud pytest tests/compatibility/test_handverified_is_reachable.py
FAILED [appsync.ListApis]
FAILED [eks.ListAccessPolicies]
FAILED [opensearch.ListApplications]
3 failed in 1.56s
Each answers NotImplemented. All three are labelled hand-verified in the
fidelity manifest, and all three are implemented — codegen reads the case
clause, and the case clause is there. What is missing is the route: the
gateway passes no operation name for rest-json, so the provider recovers it
from method and path with a hand-written resolver, and none of these three
resolvers knows the path its own model publishes.
appsync/ListApis GET /v2/apis resolveOp strips only /v1
eks/ListAccessPolicies GET /access-policies no case at all
opensearch/ListApplications
GET /2021-01-01/opensearch/list-applications
resolveOp knows only GET /application
Recorded in .claude/tdd/aws-service-coverage-100-milestone-6.tdd.md as
"Coverage and known gaps", item 2.
GREEN: DEVCLOUD_BIN=dist/devcloud pytest tests/compatibility/test_handverified_is_reachable.py
→ 3 passed
CGO_ENABLED=0 go test ./... → all pass
DEVCLOUD_BIN=dist/devcloud pytest tests/compatibility/
→ 1126 passed, 17 skipped, 2 xfailed (was 1123)
make codegen; git status --porcelain internal/generated → clean
golangci-lint run → 0 issues
appsync/ListApis, eks/ListAccessPolicies and opensearch/ListApplications are
each implemented by a case clause the provider already has. Codegen reads that
clause and labels the operation hand-verified — which proves the code exists,
not that any request arrives at it. For rest-json those are different
questions: the gateway passes no operation name, so the provider recovers it
from method and path with a hand-written resolver, and none of these three
resolvers knew the route its own model publishes.
The resolvers are partial copies of a table codegen already emits. Each
provider now ends with crud.Route(service, method, uri) instead of losing the
request. The fallback fires only where the resolver produced nothing, so it
cannot displace a decision a provider made — it fills gaps.
crud.Route is HasRoute's implementation, exported; HasRoute becomes its
one-line caller. Nothing else in the gateway changes.
No published figure moves. These operations were already counted as served;
what changes is that the count is now true about them.
27 other providers hand-roll the same kind of resolver. Whether any of them has
the same gap is not measured here — the compat manifest carries served
operations but not their tier, so the fleet-wide version of this test needs a
codegen change first. Recorded, not guessed at.
Plan: .claude/plans/aws-service-coverage-100-milestone-7.plan.md, Task 2
RED (compile-time, the intended missing field):
CGO_ENABLED=0 go test ./internal/codegen/
internal/codegen/gen_fidelity_test.go:83:4: unknown field ShortOperations
in struct literal of type ProviderScan
internal/codegen/scan_handverified_test.go:93:45:
scans["resourcegroups"].ShortOperations undefined
FAIL github.com/skyoo2003/devcloud/internal/codegen [build failed]
The RED could not be committed on its own: .pre-commit-config.yaml runs go vet,
which rejects a non-compiling tree, so RED and GREEN share this commit and the
output above is the evidence. Same constraint as #144.
GREEN: CGO_ENABLED=0 go test ./internal/codegen/ → ok
CGO_ENABLED=0 go test ./... → all pass
make codegen → exactly one tier change
DEVCLOUD_BIN=dist/devcloud pytest tests/compatibility/
→ 1126 passed, 17 skipped, 1 xfailed (was 2 xfailed)
golangci-lint run → 0 issues
opNamePattern requires four characters. resourcegroups implements Tag and Untag
in the same switch; Untag was scanned, Tag was not, and the manifest called
implemented code unimplemented. test_no_fabricated_success then asked for Tag,
got the 200 the provider has always returned, and reported a fabricated success.
It was never fabricated. The answer was real and the manifest was wrong, so the
probe was right to complain and wrong about what it had found — which is why
that entry recorded only an observation and refused to name a mechanism.
Loosening the pattern is not the fix. At two characters it admits the HTTP verbs
every path resolver switches on. The model separates them: Tag is an operation
because resourcegroups.json says so, and no model AWS publishes declares GET.
Short literals are collected into ProviderScan.ShortOperations and promoted by
BuildFidelityData only where the service's model declares them.
Fleet-wide, exactly one operation is short enough to be affected:
rg -o '"[A-Z][A-Za-z0-9]{2}":\s+Tier[A-Za-z]+' \
internal/generated/fidelity/manifest_gen.go → 1 hit, "Tag"
The gate caught the consequence, as designed:
coverage_test.go:162: docs/coverage.md publishes 4496 hand-verified
operations, the manifest holds 4497
coverage_test.go:162: docs/coverage.md publishes 2718 unimplemented
operations, the manifest holds 2717
Plan: .claude/plans/aws-service-coverage-100-milestone-7.plan.md, Task 3
RED: CGO_ENABLED=0 go test ./internal/services/s3/ -run UnhandledBucketSubresource
--- FAIL: .../analytics --- FAIL: .../lifecycle
--- FAIL: .../inventory --- FAIL: .../encryption
--- FAIL: .../metrics --- FAIL: .../versions
--- FAIL: .../replication --- FAIL: .../object-lock
"<ListBucketResult><Name>sub-bucket</Name>..." should not contain
"ListBucketResult"
"200" is not greater than or equal to "400"
GET /{bucket}?analytics misses every bucket sub-resource check and falls through
to listObjects, so botocore reads a 200 <ListBucketResult> as a successful
ListBucketAnalyticsConfigurations. Seven more sub-resources take the same path.
The seven legitimate listing shapes in the same test already pass, and are there
so the fix cannot be a guard that declines a real ListObjects.
Recorded in .claude/tdd/query-providers-fabricate-success.tdd.md as one of the
two remaining xfail(strict) violations.
GREEN: CGO_ENABLED=0 go test ./internal/services/s3/ → ok (8 sub-resources
decline, 7 listing shapes still serve)
CGO_ENABLED=0 go test ./... → all pass
DEVCLOUD_BIN=dist/devcloud pytest tests/compatibility/
→ 1127 passed, 17 skipped, 0 xfailed
make codegen; git status --porcelain internal/generated → clean
bash scripts/generate-imports.sh; git diff --exit-code → clean
golangci-lint run → 0 issues
GET /{bucket}?analytics matched no sub-resource branch and fell through to
listObjects, which answers 200 with <ListBucketResult>. botocore reads that as
a successful ListBucketAnalyticsConfigurations — a fabricated success, the one
thing docs/coverage.md calls absolute. The recorded report named one
sub-resource; the fall-through covered eight.
The guard is an allow-list of the parameters a listing carries, not a deny-list
of sub-resource names, because the two fail in opposite directions. A missing
allow-list entry declines a real listing and a test says so at once; a missing
deny-list entry serves a sub-resource as a listing, silently, which is the
defect. A sub-resource AWS adds next year now declines on its own.
The xfail(strict) marker did its job: it turned XPASS the moment the guard
landed, so KNOWN_UNFIXED could not quietly keep an entry for something already
fixed. Both entries are now gone and the dict is empty.
Plan: .claude/plans/aws-service-coverage-100-milestone-7.plan.md, Task 4
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the three defects
#144recorded and did not fix, plus the twoxfail(strict)leftovers#145pinned. Every exception the coverage claim carried is now removed rather than documented:UNREACHABLE_FROM_BOTO3andKNOWN_UNFIXEDare both empty, and noxfailremains in the compatibility suite.No service is added. Registered stays 205; compatibility-tested rises 199 → 203.
Related Issue
Refs #144, #145
Changes
Two of the recorded defects were not what their reports said, and measuring before implementing is what caught it.
The four Lex services were reachable by no boto3 caller. All four clients sign as
lex, no service is calledlex, so the alias stays contested and every call died asUnknownService. The report said this needed a four-way path split; it needed a map lookup.resolveSharedSigningNamealready picks the sibling whose route table models the request —signingNameOfonly recognised a member service ID, andlexis the group's key. The four route tables separate on method and literal segment (GET /bots→lex-models,POST /bots→lexv2-models,/bot/…/session→lex-runtime,/bots/…/botAliases/…→lexv2-runtime). Exactly one operation collides —DeleteBotatDELETE /bots/{id}— and it stays refused, because deleting the wrong bot is worse than an honest error.Three operations labelled
hand-verifiedwere reached by no route.appsync/ListApis,eks/ListAccessPoliciesandopensearch/ListApplicationsare each implemented by acaseclause; codegen reads that clause, which proves the code exists, not that a request arrives at it. Their hand-written path resolvers did not know the route boto3 sends.crud.Route(HasRoute's implementation, exported) is now the fall-through, recovering the operation from the service's own model. It fires only where the resolver produced nothing, so it fills gaps and cannot displace a decision a provider made.resourcegroups.Tagwas never a fabricated success.opNamePatternrequires four characters, soUntagwas scanned andTag— implemented beside it in the same switch — was dropped, and the manifest reported implemented code as unimplemented. The 200 was real code answering; the probe believed the manifest and the report inherited the error. Relaxing the regex to two characters would admit theGET/PUTliterals every path resolver switches on, fleet-wide, to fix one operation — a census found exactly one operation short enough to be affected. Short literals are now promoted only where the service's model declares them.S3 answered an unimplemented bucket sub-resource with a bucket listing.
GET /{Bucket}?analyticsmatched no sub-resource branch and fell through tolistObjects, so botocore read the 200<ListBucketResult>as a successfulListBucketAnalyticsConfigurations. The report named one sub-resource; the fall-through covered eight (?inventory,?metrics,?replication,?lifecycle,?encryption,?versions,?object-lock). The guard is an allow-list of the parameters a listing carries, not a deny-list of sub-resource names: a missing allow-list entry declines a real listing and a test says so at once, while a missing deny-list entry serves a sub-resource as a listing, silently — which is the defect.Docs republished by the gate, not by hand.
TestPublishedOperationTiersMatchTheManifestand the Python figure gate both fired and had to be satisfied: compatibility-tested 199 → 203, hand-verified 4,496 → 4,497, unimplemented 2,718 → 2,717. The stale claim that each Lex service is "reached by its own unambiguous name" is corrected — no boto3 caller has one.Test Plan
Every fix has a reproducer that failed first. RED output is quoted in each commit message.
The suite grew 1,119 → 1,127: four Lex services joined the floor parametrisation, three reachability probes were added, one
xfailbecame a pass, and one probe stopped being offered because its operation is now correctly labelled served.New tests:
internal/gateway/protocol_test.go:TestDetectProtocol_Lex— 11 cases including the collision that must stay refusedtests/compatibility/test_handverified_is_reachable.py— 3 operations, asserted individually because the per-service floor test tolerates a single overstated operationinternal/codegen/gen_fidelity_test.go:TestBuildFidelityDataPromotesShortDeclaredOperations— and that a short literal no model declares never becomes an operationinternal/codegen/scan_handverified_test.go:TestScanProvidersCollectsShortLiteralsSeparatelyinternal/services/s3/provider_test.go:TestS3Provider_UnhandledBucketSubresourceDeclines— 8 sub-resources decline, 7 real listing shapes still servetest_lex_services_are_unreachable_from_boto3is deleted, as its own docstring instructed whoever fixed the routing to do, and replaced by a check on the direction that still matters.Not in this PR, recorded rather than folded in
internal/generated/compat/services.json, which currently carries onlyservedOps. That is a codegen change and its own piece of work.appsync,eks,opensearchandresourcegroupsanswerNotImplementedrather than returningplugin.ErrUnhandledOp, so the CRUD engine is never reached for anything they do not name — the same class as the eight query providers in fix: eight query providers fabricated a success for unimplemented actions #145. It re-tiers hundreds of operations and moves the published figures, so it deserves the same standalone treatment. A fleet-wide count of providers with this shape should come first: fix: eight query providers fabricated a success for unimplemented actions #145 found eight where the report named one.Checklist
golangci-lint run)docs/coverage.md)Fixedentries)