feat: measure demand for unregistered services, and re-scope the coverage target

[skyoo2003]

Summary

DevCloud's roadmap aimed at every AWS service AWS publishes — 431 model files. That target rested on an assumption nobody had tested: that the 283 services it does not register are services anyone wants. This PR tests the assumption before any bulk work is funded, ships the instrument that keeps measuring it, and applies the pre-registered decision rule. The assumption did not hold: the 100% target is dropped and the published target becomes 205 services.

Related Issue

Refs #

Milestone 3 of the AWS service coverage PRD (.claude/prds/aws-service-coverage-100.prd.md, gitignored). Follows #136 (Milestone 1 — onboarding cost) and #137 (Milestone 2 — one category end to end).

Changes

The instrument

• internal/admin/unrouted.go — UnroutedCollector counts requests for services the gateway does not register. A map rather than LogCollector's ring buffer: a 100-entry ring discards exactly the accumulation that makes this evidence. Nil-receiver safe, so the miss branch stays one unconditional call and the admin-disabled path costs nothing.
• internal/gateway/router.go — one line on the !ok branch that already existed. Nothing needed re-plumbing.
• GET /devcloud/api/unrouted — per-service count, first seen, last seen. Requires admin.enabled: true.
• .github/ISSUE_TEMPLATE/service_request.yml — "Service Not Supported", asks which operations you actually call and takes the endpoint output.

Service IDs come from caller-controlled headers, so keys are truncated at 64 bytes and capped, and the report carries droppedServiceIds. An evidence structure that silently stopped recording would read as complete.

Two ceilings, documented at the collector, in docs/coverage.md, and in the issue form: DetectProtocol falls through to ("rest-xml", "s3") for anything it cannot classify, and s3 is registered — so unclassifiable traffic is never counted. The endpoint reports a floor, not a census. And a non-zero droppedServiceIds means the list is incomplete.

The verdict

Four outcomes were fixed before the first fetch, including one for "the method itself failed", so the result could not be argued into whichever answer was most convenient once it arrived.

Reading	Value
R1 — missing set M	283
R2 — M built by all three of moto / LocalStack / terraform-provider-aws	8
R2 — M built by at least two	57 (20.1%)
R2 — M built by none	115
R3 — DevCloud's own service asks, all time	0
R4 — instrument liveness	live, verified with boto3

Source sizes: moto 163, LocalStack 119, terraform-provider-aws 273.

Outcome 3 fired (fewer than 100 services with support ≥ 2). Per the rule written in advance: the 100% claim is dropped from published docs before any bulk work, and the published target becomes the demand set.

Published target: 205 services = 148 registered + 57 with demonstrated demand. The other 226 are not refused — any can be onboarded on request, and one real service request outranks all three proxies — but they are no longer promised.

• scripts/demand_rank.py — stdlib only, run by hand and never in CI, like download-smithy-models.sh and for the same reason: the sources are third-party and change under us. Exits non-zero on an empty source rather than writing a file that silently under-counts.
• docs/demand.md — the committed evidence: 283 ranked rows with per-source hits. Byte-identical on re-run from the same inputs.
• docs/coverage.md — the target and the rule that set it, on the same page.

Two join bugs were found and fixed before the rule was read, because the verdict deletes a published claim and a deflated count would have produced it dishonestly:

1. moto's ## Unimplemented: section headings were counted as a service.
2. Separator-stripping alone does not resolve abbreviations. ds, cur, elb, sdb and meteringmarketplace name services in M; ce, logs, events, config and nine others name services already registered.

Unmatched names went 15/14/17 → 0/4/4. The residue is exactly the four services these projects support that AWS publishes no model for (elastictranscoder, qldb, iotanalytics, evidently) — deliberately left unmatched, since mapping them would invent demand for a service that is not in the denominator.

Stale counts

README.md advertised 104 services and docs/faq.md 101, against 148 registered; docs/fidelity-manifest.md also carried stale per-tier operation totals. Per the PRD's top risk — "431 services is read as a capability claim, then a user finds most operations are stubs" — no count now appears without its depth split: 148 registered, 117 serving at least one operation.

Zero services onboarded. git diff --stat internal/services smithy-models is empty. That is a check, not a description.

Files Changed

File	Type
internal/admin/unrouted.go	Added	The collector
internal/admin/unrouted_test.go	Added	8 tests incl. concurrency, cap, truncation, nil
scripts/demand_rank.py	Added	Ranking, run by hand
docs/demand.md	Added	Committed evidence, 283 rows
.github/ISSUE_TEMPLATE/service_request.yml	Added	The ask channel
changes/unreleased/*.yaml	Added	4 Changie fragments
internal/gateway/router.go	Modified	Record on the existing miss branch
internal/gateway/gateway.go	Modified	New takes the collector, nil-able
internal/admin/api.go	Modified	GET /devcloud/api/unrouted
cmd/devcloud/main.go	Modified	Construct it when admin is enabled
internal/{admin,gateway}/*_test.go	Modified	6 new tests, call sites updated
docs/coverage.md	Modified	The verdict, the target, the rule, the ceilings
README.md, docs/{README,faq,fidelity-manifest,services-matrix,compatibility-policy}.md	Modified	Stale counts → count + depth split

Test Plan

CGO_ENABLED=0 go test ./...                                        exit 0, 110 packages
CGO_ENABLED=1 go test -race ./internal/admin/ ./internal/gateway/  ok
golangci-lint run                                                  0 issues
make test-compat                                                   790 passed, 2 skipped, 11.6s
git diff --stat internal/services smithy-models                    empty
python3 scripts/demand_rank.py --out docs/demand.md                byte-identical on re-run

14 new tests. The two that carry the behaviour: TestServiceRouter_RecordsUnroutedService (a miss is counted) and TestServiceRouter_DoesNotRecordRoutedService (a served call is not — otherwise every DevCloud looks like it is missing everything it has).

R4, against the built binary with admin.enabled: true:

POST / X-Amz-Target: AmazonAppFlow_20200101.ListThings       -> 400  (x2)
POST / X-Amz-Target: AWSDirectoryService_20200101.ListThings -> 400
POST / X-Amz-Target: DynamoDB_20120810.ListTables            -> 200   (registered)

GET /devcloud/api/unrouted
{"services":[{"serviceId":"amazonappflow","count":2,...},
             {"serviceId":"awsdirectoryservice","count":1,...}],
 "maxServiceIds":1000,"droppedServiceIds":0}

The registered DynamoDB call is absent, as required. Then with a real boto3 client: boto3.client("appflow").list_connector_entities() records as appflow — the canonical service ID, via the SigV4 signing name. That is the population that matters.

Known limitation

The verdict rests on a proxy. moto, LocalStack and terraform-provider-aws measure emulator and provider effort, not DevCloud's users. This is stated everywhere the number appears — docs/demand.md, docs/coverage.md, the PRD — and it was the deliberate trade for not blocking Milestones 4–6 on an unbounded wait for organic signal. The real measurement now accrues behind it, and either channel can reopen the target.

R3 = 0 is recorded as a finding, not a missing measurement. It is the expected value for a project with no telemetry and two issues, and it is exactly why the proxies carried the verdict.

Checklist

• Self-reviewed the code
• Added/updated tests
• Lint/format passes (golangci-lint run — 0 issues)
• Updated documentation
• Added a Changie changelog fragment (4 fragments: 2 Added, 1 Changed, 1 Fixed)