feat: serve query and rest-xml from the generic CRUD engine

changes/unreleased/Added-20260905-201000.yaml

@@ -0,0 +1,5 @@
+kind: Added
+body: 'The generic CRUD engine now serves every protocol DevCloud registers. `rest-xml` and `query` were the last two: `rest-xml` recovers its operation from the request method and path against the model''s URI templates, like `rest-json`, and `query` reads it from the `Action` field of the form body. 112 operations move from `unimplemented` to `auto-crud` — 94 in `s3-control` and 18 in `elastic-load-balancing` — and protocol is no longer a reason any registered service serves nothing'
+time: 2026-09-05T20:10:00.000000+09:00
+custom:
+    Issue: "143"

changes/unreleased/Changed-20260905-201001.yaml

@@ -0,0 +1,5 @@
+kind: Changed
+body: '`rest-xml` request bodies are still not buffered by the gateway, now by an explicit rule rather than as a side effect of the protocol being unservable. The engine serves `rest-xml` from the path and query alone, so S3''s large binary uploads keep streaming; `crud.NeedsBody` is the predicate and a test fails if the body is read'
+time: 2026-09-05T20:10:01.000000+09:00
+custom:
+    Issue: "143"

docs/coverage.md

@@ -10,36 +10,35 @@ things:
 | Number | What it means | Today |
 |---|---|---|
 | **Registered** | The gateway routes the service. A call reaches DevCloud instead of falling through to real AWS. | **205** |
-| **Serving ≥1 operation** | At least one operation returns a real, store-backed answer — hand-written or served by the generic CRUD engine. | **199** |
-| **Registered-only** | Routed, but every operation declines with a clean AWS error. Nothing is served. | **6** |
+| **Serving ≥1 operation** | At least one operation returns a real, store-backed answer — hand-written or served by the generic CRUD engine. | **201** |
+| **Registered-only** | Routed, but every operation declines with a clean AWS error. Nothing is served. | **4** |
 
 Per operation, from the [fidelity manifest](fidelity-manifest.md):
 
 | Tier | Operations |
 |---|---|
 | `hand-verified` | 4,496 |
-| `auto-crud` | 4,858 |
-| `unimplemented` | 3,053 |
+| `auto-crud` | 4,970 |
+| `unimplemented` | 2,941 |
 | **total known** | **12,407** |
 
 ## Why a registered service can serve nothing
 
 The generic CRUD engine has to know which operation a request is for, and it has
-to recognise that operation as CRUD-shaped. Two things can stop it.
+to recognise that operation as CRUD-shaped. Only the second one still stops it.
 
-**The protocol does not say which operation.** The `X-Amz-Target` JSON protocols
-name the operation in a header. `rest-json` does not, but every one of its
-operations is bound to an HTTP method and a URI template in the model, and
-`internal/shared/httproute` matches a request back to the operation from that
-pair — so the engine serves it too. `query` and `rest-xml` have neither a target
-header nor a modelled path the engine can match, so they are unreachable by it
-and serve nothing until somebody writes the provider by hand.
+**The protocol always says which operation.** Each one says it somewhere
+different, and the engine reads all of them: the `X-Amz-Target` JSON protocols
+put it in a header; `rest-json` and `rest-xml` bind every operation to an HTTP
+method and a URI template, which `internal/shared/httproute` matches a request
+back to; `query` puts it in the `Action` field of the form body. This used to be
+the main reason a service served nothing. It no longer is.
 
 **The operation is not CRUD-shaped.** `GetThing`, `ListThings`, `CreateThing`
 and their siblings map onto a generic store. `ExecuteStatement`, `InvokeEndpoint`
 and `QueryForecast` do not, and the engine refuses them rather than inventing an
 answer. A service whose entire API is that shape serves nothing whatever its
-protocol.
+protocol. This is now the *only* reason.
 
 Registered services by protocol:
 
@@ -48,19 +47,21 @@ Registered services by protocol:
 | `json-1.1` | 64 | yes — operation from `X-Amz-Target` |
 | `json-1.0` | 17 | yes — operation from `X-Amz-Target` |
 | `rest-json` | 93 | yes — operation from method + path |
-| `query` | 15 | no |
-| `rest-xml` | 4 | no |
+| `query` | 15 | yes — operation from the `Action` form field |
+| `rest-xml` | 4 | yes — operation from method + path |
 | no in-tree model | 12 | n/a — hand-written providers |
 
-Six services are registered and serve nothing. Four are the second case — no
-CRUD-shaped operation anywhere in their API: `forecastquery` (`QueryForecast`,
+Four services are registered and serve nothing, and all four are the same case —
+no CRUD-shaped operation anywhere in their API: `forecastquery` (`QueryForecast`,
 `QueryWhatIfForecast`), the two SageMaker Runtime variants (`InvokeEndpoint*`),
 and `rds-data` (`ExecuteStatement`, `BeginTransaction`, `CommitTransaction`,
 `RollbackTransaction`). No protocol change reaches them.
 
-The other two are the first case, and they are the whole of what is left:
-`elastic-load-balancing` (`query`) and `s3-control` (`rest-xml`). Both are in
-the demand set, and both are what PRD Milestone 5 now covers.
+Being engine-servable is not the same as being served. Thirteen of the fifteen
+`query` services and three of the four `rest-xml` ones have hand-written
+providers that answer their own unknown operations, so they never enter the
+engine and their manifest did not move when the protocols were admitted — the
+`EngineWired` flag records that per service.
 
 A registered operation is not automatically a reachable one. The engine is
 entered only when a provider returns `plugin.ErrUnhandledOp`, so a hand-written
@@ -103,21 +104,28 @@ stopped on the best surface available.
 | | Count |
 |---|---|
 | Registered | 57 |
-| Serving ≥1 operation | 54 |
-| Registered-only | 3 |
+| Serving ≥1 operation | 56 |
+| Registered-only | 1 |
 
 The 57 split by protocol as 34 `rest-json`, 15 `json-1.1`, 6 `json-1.0`, 1
 `query`, 1 `rest-xml`. Two thirds of the set is `rest-json`, which is why the
 engine gaining that protocol is what made this milestone possible rather than a
 matter of arithmetic: without it, 34 of the 57 would have registered and served
 nothing.
 
-The three that do not meet the floor are named above, each for a different
-reason. `rds-data` is the one worth calling out: it is supported by all three
-projects in the demand survey — the strongest signal in the whole set — and it
-still cannot be served generically, because not one of its six operations is
-CRUD-shaped. Breadth does not reach every service, and saying so is cheaper than
-a fabricated success.
+The one that does not meet the floor is `rds-data`, and it is the one worth
+calling out: it is supported by all three projects in the demand survey — the
+strongest signal in the whole set — and it still cannot be served generically,
+because not one of its six operations is CRUD-shaped. Breadth does not reach
+every service, and saying so is cheaper than a fabricated success.
+
+`elastic-load-balancing` and `s3-control` were the other two. They were the last
+of the demand set because of their protocols, not their APIs, and both are
+served now: 18 of ELB's 29 operations and 94 of S3 Control's 97. The eleven and
+the three that remain are the not-CRUD-shaped case again — `ConfigureHealthCheck`
+and `ApplySecurityGroupsToLoadBalancer` among them, which a Terraform `aws_elb`
+resource does call. Breadth is what this milestone bought; depth is still not
+claimed.
 
 ## What counts as a service
 

docs/crud-engine.md

@@ -35,24 +35,62 @@ it. Where that name lives depends on the protocol:
 |---|---|---|
 | `json-1.0`, `json-1.1` | the `X-Amz-Target` header | yes |
 | `rest-json` | the request method and path, matched against the model's URI templates (`internal/shared/httproute`) | yes |
-| `query` | — | no |
-| `rest-xml` | — | no |
-
-For `rest-json` the engine reads parameters from three places, least
-authoritative first: values the model binds with `httpQuery`, then the JSON
-body, then the path labels. The URI is what addresses the resource, so a path
-label wins. A `restJson1` model never binds one member to two of these, so real
-SDK traffic never exercises the precedence.
-
-It does **not** read `httpHeader` members, `httpPayload` blobs, or streaming
-bodies. An operation whose identifier arrives only in a header is therefore
-served with a generated id rather than the caller's — plausible, not faithful,
-which is the engine's stated contract.
-
-A request whose method and path match no route in the service's table is
-**declined** with `InvalidAction`, never served from the store. That is what
-keeps a registered `rest-json` service from answering for paths it does not
-model.
+| `rest-xml` | the same — every `restXml` operation is bound to a method and URI too | yes |
+| `query` | the `Action` field of the form body | yes |
+| `ec2-query` | — | no |
+
+Every protocol DevCloud registers is now readable, so a service that serves
+nothing does so because none of its operations is CRUD-shaped, not because of
+how it talks. `ec2-query` is the one exception and it is not a gap in practice:
+the only service that speaks it is EC2, whose provider is hand-written and never
+reaches the engine.
+
+### Where parameters come from
+
+For the two REST protocols, three places, least authoritative first: values the
+model binds with `httpQuery`, then the request body, then the path labels. The
+URI is what addresses the resource, so a path label wins. A REST model never
+binds one member to two of these, so real SDK traffic never exercises the
+precedence; it is defined so a hand-rolled request cannot redirect a lookup.
+
+For `query`, the form body, flat keys only. `Action` and `Version` are dropped:
+they describe the request rather than the resource, and storing them would echo
+`<Action>CreateLoadBalancer</Action>` back inside a result element. A structured
+member arrives flattened as `Listeners.member.1.Protocol`; the engine has no
+nested shape to put it in, so it is not emitted in the response.
+
+**`rest-xml` request bodies are never read at all.** The gateway does not buffer
+them — S3 speaks `rest-xml` and its bodies are multi-gigabyte uploads that must
+keep streaming — so a `rest-xml` operation is served from its path and query
+alone. Every CRUD-shaped S3 Control operation addresses its resource that way,
+so nothing is lost; an operation that carried its identifier only in a body
+would get a generated id.
+
+The engine does **not** read `httpHeader` members, `httpPayload` blobs, or
+streaming bodies. An operation whose identifier arrives only in a header is
+therefore served with a generated id rather than the caller's — plausible, not
+faithful, which is the engine's stated contract.
+
+### Responses
+
+The JSON protocols get a JSON body. `query` and `rest-xml` get XML, and they do
+not share an envelope: botocore's query parser looks for `<OperationResult>`
+nested inside `<OperationResponse>` and, given anything else, returns a result
+with nothing in it rather than an error, while its `rest-xml` parser maps the
+root element's children straight onto the output shape. List entries are wrapped
+in `<member>`, which is AWS's default for both dialects; a model that flattens a
+list gets the unflattened form, because the engine has no flattening
+information. No `xmlns` is emitted — botocore strips namespaces before matching
+element names.
+
+### Declining
+
+A request whose method and path match no route in the service's table, or whose
+`Action` names an operation the service did not register, is **declined** with
+`InvalidAction` and never served from the store. That is what keeps a registered
+service from answering for operations it does not model — and it matters most
+for `rest-xml`, because `DetectProtocol` routes anything it cannot classify to
+S3.
 
 ## Fidelity tiers
 

internal/codegen/gen_crud_meta.go

@@ -132,18 +132,27 @@ func classifyOps(model *ir.Model) []crudOpData {
 
 // engineServable reports whether the engine can serve a service's protocol.
 //
-// The JSON protocols carry the operation name in X-Amz-Target. rest-json does
-// not, but every one of its operations is bound to a method and URI template,
-// and internal/shared/httproute turns that pair back into an operation name —
-// so the engine can classify it too. query and rest-xml have neither, which is
-// what PRD Milestone 5 owns; admitting them here would mean guessing.
+// The JSON protocols carry the operation name in X-Amz-Target. rest-json and
+// rest-xml do not, but every one of their operations is bound to a method and
+// URI template, and internal/shared/httproute turns that pair back into an
+// operation name — so the engine can classify them too. rest-xml was excluded
+// on the assumption that it had no modelled path; it has one, and all 97 of
+// s3-control's operations carry it.
+//
+// query has no modelled path at all, so it registers no route; the engine
+// matches it by the Action field of the form body instead. It is admitted here
+// because that is a place the operation name genuinely is, not a guess.
+//
+// ec2-query stays out. It is form-encoded like query but not interchangeable
+// with it, and the only service that speaks it has a hand-written provider.
 //
 // This is deliberately the same question crud.Servable answers at runtime. The
 // two must agree: a protocol classified here but refused there registers
 // operations nothing can reach, and the fidelity manifest would call them
 // auto-crud.
 func engineServable(protocol string) bool {
-	return strings.HasPrefix(protocol, "json") || protocol == "rest-json"
+	return strings.HasPrefix(protocol, "json") ||
+		protocol == "rest-json" || protocol == "rest-xml" || protocol == "query"
 }
 
 // ServiceCRUDData classifies an engine-servable model's CRUD operations. It

internal/codegen/gen_crud_meta_test.go

@@ -92,22 +92,62 @@ func TestServiceCRUDDataJSONUnchanged(t *testing.T) {
 	}
 }
 
-// TestServiceCRUDDataRejectsUnservableProtocols holds the boundary Milestone 5
-// owns. query and rest-xml put the operation neither in a header nor in a
-// modelled path the engine can match, so admitting them would serve fabricated
-// answers to callers the engine cannot actually understand.
-func TestServiceCRUDDataRejectsUnservableProtocols(t *testing.T) {
-	for _, protocol := range []string{"query", "rest-xml"} {
-		t.Run(protocol, func(t *testing.T) {
-			model := crudModel(protocol,
-				ir.Operation{Name: "ListGraphs", OutputName: "ListGraphsOutput",
-					HTTPMethod: "GET", HTTPUri: "/v1/graphs"},
-			)
+// TestServiceCRUDDataAdmitsRESTXML is the codegen half of the rest-xml change.
+// This gate and crud.Servable answer the same question and must agree: a
+// protocol admitted here but refused there registers operations nothing can
+// reach, and the fidelity manifest would publish them as auto-crud.
+func TestServiceCRUDDataAdmitsRESTXML(t *testing.T) {
+	model := crudModel("rest-xml",
+		ir.Operation{Name: "ListAccessPoints", OutputName: "ListAccessPointsOutput",
+			HTTPMethod: "GET", HTTPUri: "/v20180820/accesspoint"},
+	)
 
-			_, ok := ServiceCRUDData(model)
-			assert.False(t, ok, "%s must not be engine-servable", protocol)
-		})
+	data, ok := ServiceCRUDData(model)
+	require.True(t, ok, "rest-xml service must be engine-servable")
+
+	ops := map[string]crudOpData{}
+	for _, op := range data.Ops {
+		ops[op.Op] = op
 	}
+	// The route is the whole classification story for rest-xml, so it has to
+	// survive into the registry the same way rest-json's does.
+	assert.Equal(t, "GET", ops["ListAccessPoints"].Method)
+	assert.Equal(t, "/v20180820/accesspoint", ops["ListAccessPoints"].URI)
+}
+
+// TestServiceCRUDDataAdmitsQuery closes the last protocol gap. query has no
+// modelled path at all — its operations carry no http trait — so unlike the
+// REST protocols it registers no route, and the engine matches it by the
+// Action field of the form body instead.
+func TestServiceCRUDDataAdmitsQuery(t *testing.T) {
+	model := crudModel("query",
+		ir.Operation{Name: "DescribeLoadBalancers", OutputName: "DescribeLoadBalancersOutput"},
+	)
+
+	data, ok := ServiceCRUDData(model)
+	require.True(t, ok, "query service must be engine-servable")
+
+	ops := map[string]crudOpData{}
+	for _, op := range data.Ops {
+		ops[op.Op] = op
+	}
+	// No route, and that is correct rather than a gap: a query operation has
+	// no method or URI to register, and Register skips an empty URI so the
+	// service contributes nothing to the route table.
+	assert.Empty(t, ops["DescribeLoadBalancers"].Method)
+	assert.Empty(t, ops["DescribeLoadBalancers"].URI)
+}
+
+// TestServiceCRUDDataRejectsEC2Query holds the remaining boundary. ec2Query is
+// form-encoded like query but not interchangeable with it, and the only service
+// that speaks it has a hand-written provider that never reaches the engine.
+func TestServiceCRUDDataRejectsEC2Query(t *testing.T) {
+	model := crudModel("ec2-query",
+		ir.Operation{Name: "DescribeInstances", OutputName: "DescribeInstancesOutput"},
+	)
+
+	_, ok := ServiceCRUDData(model)
+	assert.False(t, ok, "ec2-query must not be engine-servable")
 }
 
 // TestServiceCRUDDataSkipsUnclassifiableService is the rds-data case: a

internal/gateway/router.go

@@ -54,12 +54,19 @@ func (sr *ServiceRouter) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 
 	op := extractOperationName(r, protocol)
 
-	// Buffer the body for the protocols the CRUD fallback engine can serve, so
-	// it can re-read it if the provider does not handle the operation. Cheap for
-	// JSON and rest-json payloads; skipped for REST-XML (S3), where bodies may
-	// be large binary uploads and the engine would refuse them anyway.
+	// Buffer the body for the protocols whose parameters live in it, so the
+	// engine can re-read it if the provider does not handle the operation. Cheap
+	// for JSON, rest-json and query payloads.
+	//
+	// This asks crud.NeedsBody, not crud.Servable, and the difference is
+	// rest-xml. The engine serves rest-xml — it classifies the operation from
+	// the method and path like rest-json — but it must do so without the body:
+	// S3 speaks rest-xml, its bodies are large binary uploads, and every
+	// CRUD-shaped S3 Control operation addresses its resource with a path label
+	// or a query term anyway. Asking the servable question here would buffer
+	// every PutObject to serve a provider that never reaches the engine.
 	var body []byte
-	if crud.Servable(protocol) {
+	if crud.NeedsBody(protocol) {
 		var rerr error
 		if body, rerr = io.ReadAll(r.Body); rerr != nil {
 			writeAWSError(w, protocol, http.StatusBadRequest, "SerializationException", "failed to read request body")