fix(#856): the compute and edge family of identifier draw sites

CHANGELOG.md

@@ -274,6 +274,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   out now replays with **zero** differences and `StateValid` true. That last step is what makes the
   claim load-bearing rather than cosmetic: a re-minted ETag turns the recorded delete into a
   `PreconditionFailed` against a control nothing had changed. 28 draw sites remain on `crypto/rand`.
+- **The compute and edge family — API Gateway, AppSync, Batch, EMR Serverless, ECR, ELB and
+  Route 53 — mints derived identifiers** (#856). Eleven generators across nine plugin files moved,
+  so a recorded stream that creates a REST API and a resource under its root, an AppSync API with a
+  key and a function, a hosted zone and a record-set change inside it, a load balancer with a target
+  group and a listener naming both ARNs, a Batch job, an EMR Serverless application and job run, and
+  an ECR repository with an image read back by the digest the push minted now replays with **zero**
+  differences and `StateValid` true. Every one of those later requests names what an earlier one
+  minted, which is what makes the stream an assertion rather than a smoke test.
+- **The shared UUID-shaped generator is now `IDMint.HexUUID`** (#856). Batch job IDs and EMR
+  Serverless job-run IDs publish the same `8-4-4-4-12` rendering the six services in the previous
+  tier do, and the helper they would have called was declared in the Lambda plugin and named for a
+  Lambda revision. Promoting it to a method on the mint and rewriting its thirteen call sites is what
+  keeps a Batch job ID from being minted by `generateLambdaRevisionID`; the rendering is unchanged.
+- **An API Gateway ID can now contain a digit** (#856). The `crypto/rand` version read five bytes and
+  mapped each *nibble* through the service's 36-character alphabet, so only `a` through `p` could
+  ever appear and a digit never did. Deriving it draws a byte per character, which reaches the whole
+  published set, so an API, resource, deployment, authorizer or usage-plan ID — and an API Gateway v2
+  route, integration or mapping ID — now looks like one AWS would issue. This is the one identifier
+  whose *alphabet* changed when it was derived, and it widened rather than narrowed. 23 draw sites
+  remain on `crypto/rand`.
 - **A stream recorded under a seed replays under the same seed** (#1140). Every seedable outcome in
   substrate is written through a control-plane endpoint, and only the AWS path recorded anything — so
   a seed never entered the event stream. A replay opens by resetting the whole `StateManager`, and a

docs/services.md

@@ -2207,19 +2207,33 @@ Three kinds of value stay random, and one more is still migrating:
   CloudFormation's [stack and change-set ARNs](#stack-and-change-set-arns-are-deterministic),
   which predate this rule and are what generalising it was modelled on.
 - EC2, IAM, STS, SQS, SNS, Lambda, EFS, FSx, Transfer, ECS, Step Functions, EventBridge,
-  CloudWatch Logs, CloudFront and Service Quotas identifiers are derived today. A CloudFront
-  distribution, invalidation and origin access control all draw from one generator, so the three
-  moved together with the origin access control family (#1277). The remaining services are
-  migrating one family at a time, tracked on #856; until a service moves, its identifiers are
-  still drawn from `crypto/rand` and a replay of a stream creating one of its resources still
-  diverges.
-
-Six of those services publish an identifier from one shared generator rather than declaring their
+  CloudWatch Logs, CloudFront, Service Quotas, API Gateway (v1 and v2), AppSync, Batch, EMR
+  Serverless, ECR, ELB and Route 53 identifiers are derived today. A CloudFront distribution,
+  invalidation and origin access control all draw from one generator, so the three moved together
+  with the origin access control family (#1277). The remaining services are migrating one family
+  at a time, tracked on #856; until a service moves, its identifiers are still drawn from
+  `crypto/rand` and a replay of a stream creating one of its resources still diverges.
+
+Nine of those services publish an identifier from one shared generator rather than declaring their
 own, so they moved together: an ECS task ID, a Step Functions execution name, an SQS message ID,
-an EventBridge event ID, a CloudWatch Logs upload sequence token and a Service Quotas request ID
-are all the same sixteen derived bytes rendered in UUID *shape* — `8-4-4-4-12` lowercase hex
-without the RFC 4122 version and variant bits, which is the form substrate published before it
-derived them and is unchanged by deriving them.
+an EventBridge event ID, a CloudWatch Logs upload sequence token, a Service Quotas request ID, a
+Lambda revision ID, a Batch job ID and an EMR Serverless job-run ID are all the same sixteen
+derived bytes rendered in UUID *shape* — `8-4-4-4-12` lowercase hex without the RFC 4122 version
+and variant bits, which is the form substrate published before it derived them and is unchanged by
+deriving them.
+
+One identifier's **alphabet** changed when it was derived, and widened rather than narrowed. An API
+Gateway ID is ten lowercase alphanumeric characters; the `crypto/rand` version read five bytes and
+mapped each *nibble* through that 36-character alphabet, so only `a` through `p` could appear and a
+digit never did. Drawing a byte per character reaches the whole published set, so an API ID,
+resource ID, deployment ID, authorizer ID, usage-plan ID or API Gateway v2 route, integration and
+mapping ID now looks like one AWS would issue.
+
+An ECR image digest is minted rather than computed from the manifest, so it is reproducible across
+a replay but is not the SHA-256 of the image it names, and two pushes of identical manifest bytes
+store two images where AWS stores one. [#1283](https://github.com/scttfrdmn/substrate/issues/1283)
+tracks deriving it from the manifest, which changes what the digest *means* rather than where its
+bytes come from.
 
 An SQS send mints a message's initial receipt handle and each receive replaces it, matching real
 SQS: two `ReceiveMessage` calls that return the same message hand back different handles and only

docs/testing-guide.md

@@ -336,8 +336,9 @@ all; before #856 every such stream diverged on its first create, which is why th
 tests above are built on caller-chosen bucket and key names instead.
 
 Two caveats. **Not every service's identifiers are derived yet.** EC2, IAM, STS, SQS, SNS,
-Lambda, EFS, FSx, Transfer, ECS, Step Functions, EventBridge, CloudWatch Logs, CloudFront and
-Service Quotas are; the rest are migrating one family at a time, and until a service moves, a
+Lambda, EFS, FSx, Transfer, ECS, Step Functions, EventBridge, CloudWatch Logs, CloudFront,
+Service Quotas, API Gateway (v1 and v2), AppSync, Batch, EMR Serverless, ECR, ELB and Route 53
+are; the rest are migrating one family at a time, and until a service moves, a
 replay of a stream creating one of its resources still diverges. And a recording made against an
 **unfrozen** clock can still diverge on a `state_hash_after` even when every identifier
 matches, because a handler reading the live clock stamps its record a few hundred

emulator/apigateway_plugin.go

@@ -2,7 +2,6 @@ package emulator
 
 import (
 	"context"
-	"crypto/rand"
 	"encoding/base64"
 	"encoding/json"
 	"fmt"
@@ -115,7 +114,7 @@ func (p *APIGatewayPlugin) HandleRequest(ctx *RequestContext, req *AWSRequest) (
 	case "DeleteUsagePlan":
 		return p.deleteUsagePlan(ctx, params["planId"])
 	case "CreateUsagePlanKey":
-		return apigwJSONResponse(http.StatusCreated, map[string]string{"id": generateAPIGatewayID(), "type": "API_KEY"})
+		return apigwJSONResponse(http.StatusCreated, map[string]string{"id": generateAPIGatewayID(ctx.IDs), "type": "API_KEY"})
 	case "CreateDomainName":
 		return p.createDomainName(ctx, req)
 	case "GetDomainName":
@@ -391,8 +390,8 @@ func (p *APIGatewayPlugin) createRestAPI(ctx *RequestContext, req *AWSRequest) (
 		return nil, &AWSError{Code: "BadRequestException", Message: "name is required", HTTPStatus: http.StatusBadRequest}
 	}
 
-	apiID := generateAPIGatewayID()
-	rootResID := generateAPIGatewayID()
+	apiID := generateAPIGatewayID(ctx.IDs)
+	rootResID := generateAPIGatewayID(ctx.IDs)
 	now := p.tc.Now()
 
 	api := RestAPIState{
@@ -567,7 +566,7 @@ func (p *APIGatewayPlugin) createResource(ctx *RequestContext, req *AWSRequest,
 	}
 	fullPath += body.PathPart
 
-	resID := generateAPIGatewayID()
+	resID := generateAPIGatewayID(ctx.IDs)
 	res := ResourceState{
 		ID:        resID,
 		ParentID:  parentID,
@@ -837,7 +836,7 @@ func (p *APIGatewayPlugin) createDeployment(ctx *RequestContext, req *AWSRequest
 	}
 
 	dep := DeploymentState{
-		ID:          generateAPIGatewayID(),
+		ID:          generateAPIGatewayID(ctx.IDs),
 		Description: body.Description,
 		CreatedDate: p.tc.Now(),
 		APIId:       apiID,
@@ -1061,7 +1060,7 @@ func (p *APIGatewayPlugin) createAuthorizer(ctx *RequestContext, req *AWSRequest
 	}
 
 	auth := AuthorizerState{
-		ID:             generateAPIGatewayID(),
+		ID:             generateAPIGatewayID(ctx.IDs),
 		Name:           body.Name,
 		Type:           body.Type,
 		ProviderARNs:   body.ProviderARNs,
@@ -1282,7 +1281,7 @@ func (p *APIGatewayPlugin) createUsagePlan(ctx *RequestContext, req *AWSRequest)
 	}
 
 	plan := UsagePlanState{
-		ID:          generateAPIGatewayID(),
+		ID:          generateAPIGatewayID(ctx.IDs),
 		Name:        body.Name,
 		Description: body.Description,
 		Tags:        body.Tags,
@@ -1498,18 +1497,21 @@ func (p *APIGatewayPlugin) getBasePathMappings(ctx *RequestContext, req *AWSRequ
 
 // --- ID generation -----------------------------------------------------------
 
-// generateAPIGatewayID generates a 10-character lowercase alphanumeric ID
-// suitable for use as an API Gateway resource identifier.
-func generateAPIGatewayID() string {
-	b := make([]byte, 5)
-	_, _ = rand.Read(b)
-	const chars = "abcdefghijklmnopqrstuvwxyz0123456789"
-	out := make([]byte, 10)
-	for i, by := range b {
-		out[i*2] = chars[by>>4%36]
-		out[i*2+1] = chars[by&0xf%36]
-	}
-	return string(out)
+// apigwIDAlphabet is the alphabet API Gateway publishes its 10-character identifiers in —
+// an API id, a resource id, a deployment id, an authorizer id and a usage-plan id are all
+// drawn from it.
+const apigwIDAlphabet = "abcdefghijklmnopqrstuvwxyz0123456789"
+
+// generateAPIGatewayID mints a 10-character lowercase alphanumeric ID from m, suitable for
+// use as an API Gateway (v1 or v2) resource identifier.
+//
+// The derived form draws from the whole alphabet, which the crypto/rand form it replaces did
+// not: that one read five bytes and mapped each *nibble* through the 36-character alphabet,
+// so only its first sixteen characters — `a` through `p` — could ever appear and a digit
+// never did. Widening it moves the identifier toward what the service publishes rather than
+// away from it, so the rendering change is a fix and not a cost of #856.
+func generateAPIGatewayID(m *IDMint) string {
+	return m.Chars(10, apigwIDAlphabet)
 }
 
 // --- Response helper ---------------------------------------------------------

emulator/apigatewayv2_plugin.go

@@ -274,7 +274,7 @@ func (p *APIGatewayV2Plugin) createAPI(ctx *RequestContext, req *AWSRequest) (*A
 		body.ProtocolType = "HTTP"
 	}
 
-	apiID := generateAPIGatewayID()
+	apiID := generateAPIGatewayID(ctx.IDs)
 	api := V2ApiState{
 		APIID:        apiID,
 		Name:         body.Name,
@@ -408,7 +408,7 @@ func (p *APIGatewayV2Plugin) createRoute(ctx *RequestContext, req *AWSRequest, a
 	}
 
 	route := V2RouteState{
-		RouteID:           generateAPIGatewayID(),
+		RouteID:           generateAPIGatewayID(ctx.IDs),
 		RouteKey:          body.RouteKey,
 		Target:            body.Target,
 		AuthorizationType: body.AuthorizationType,
@@ -493,7 +493,7 @@ func (p *APIGatewayV2Plugin) createIntegration(ctx *RequestContext, req *AWSRequ
 	}
 
 	integ := V2IntegrationState{
-		IntegrationID:        generateAPIGatewayID(),
+		IntegrationID:        generateAPIGatewayID(ctx.IDs),
 		IntegrationType:      body.IntegrationType,
 		IntegrationURI:       body.IntegrationURI,
 		PayloadFormatVersion: body.PayloadFormatVersion,
@@ -667,7 +667,7 @@ func (p *APIGatewayV2Plugin) createAuthorizerV2(ctx *RequestContext, req *AWSReq
 	}
 
 	auth := V2AuthorizerState{
-		AuthorizerID:     generateAPIGatewayID(),
+		AuthorizerID:     generateAPIGatewayID(ctx.IDs),
 		Name:             body.Name,
 		AuthorizerType:   body.AuthorizerType,
 		IdentitySource:   body.IdentitySource,
@@ -750,7 +750,7 @@ func (p *APIGatewayV2Plugin) createDeploymentV2(ctx *RequestContext, req *AWSReq
 	}
 
 	dep := V2DeploymentState{
-		DeploymentID:     generateAPIGatewayID(),
+		DeploymentID:     generateAPIGatewayID(ctx.IDs),
 		DeploymentStatus: "DEPLOYED",
 		Description:      body.Description,
 		CreatedDate:      p.tc.Now(),
@@ -843,7 +843,7 @@ func (p *APIGatewayV2Plugin) createAPIMapping(ctx *RequestContext, req *AWSReque
 		return nil, &AWSError{Code: "BadRequestException", Message: "invalid request body", HTTPStatus: http.StatusBadRequest}
 	}
 
-	mappingID := generateAPIGatewayID()
+	mappingID := generateAPIGatewayID(ctx.IDs)
 	mapping := v2APIMappingState{
 		APIMappingID:  mappingID,
 		APIID:         body.APIID,

emulator/appsync_plugin.go

@@ -140,7 +140,7 @@ func (p *AppSyncPlugin) createGraphqlAPI(reqCtx *RequestContext, req *AWSRequest
 		input.AuthenticationType = "API_KEY"
 	}
 
-	apiID := generateAppSyncAPIID()
+	apiID := generateAppSyncAPIID(reqCtx.IDs)
 	acct := reqCtx.AccountID
 	region := reqCtx.Region
 	arn := fmt.Sprintf("arn:aws:appsync:%s:%s:apis/%s", region, acct, apiID)
@@ -458,7 +458,7 @@ func (p *AppSyncPlugin) createFunction(reqCtx *RequestContext, req *AWSRequest,
 		return nil, &AWSError{Code: "BadRequestException", Message: "name is required", HTTPStatus: http.StatusBadRequest}
 	}
 	acct, region := reqCtx.AccountID, reqCtx.Region
-	funcID := generateAppSyncFunctionID()
+	funcID := generateAppSyncFunctionID(reqCtx.IDs)
 	fn := AppSyncFunction{
 		APIID:          apiID,
 		FunctionID:     funcID,
@@ -540,7 +540,7 @@ func (p *AppSyncPlugin) createAPIKey(reqCtx *RequestContext, req *AWSRequest, ap
 		return nil, expiresErr
 	}
 
-	keyID := generateAppSyncAPIKeyID()
+	keyID := generateAppSyncAPIKeyID(reqCtx.IDs)
 	key := AppSyncAPIKey{
 		ID:          keyID,
 		Description: input.Description,

emulator/appsync_types.go

@@ -109,19 +109,20 @@ func appSyncAPIKeyIDsKey(acct, region, apiID string) string {
 	return fmt.Sprintf("apikey_ids:%s/%s/%s", acct, region, apiID)
 }
 
-// generateAppSyncAPIID returns a new unique AppSync API ID (13 lowercase hex chars).
-func generateAppSyncAPIID() string {
-	return randomHex(13)
+// generateAppSyncAPIID mints a new unique AppSync API ID from m (26 lowercase hex chars,
+// which is the width AppSync publishes an API ID in).
+func generateAppSyncAPIID(m *IDMint) string {
+	return m.Hex(13)
 }
 
-// generateAppSyncFunctionID returns a new unique AppSync function ID.
-func generateAppSyncFunctionID() string {
-	return randomHex(26)
+// generateAppSyncFunctionID mints a new unique AppSync function ID from m.
+func generateAppSyncFunctionID(m *IDMint) string {
+	return m.Hex(26)
 }
 
-// generateAppSyncAPIKeyID returns a new unique AppSync API key ID.
-func generateAppSyncAPIKeyID() string {
-	return randomHex(26)
+// generateAppSyncAPIKeyID mints a new unique AppSync API key ID from m.
+func generateAppSyncAPIKeyID(m *IDMint) string {
+	return m.Hex(26)
 }
 
 // parseAppSyncOperation derives the AppSync operation name from the HTTP method

emulator/batch_plugin.go

@@ -2,7 +2,6 @@ package emulator
 
 import (
 	"context"
-	"crypto/rand"
 	"crypto/sha256"
 	"encoding/json"
 	"fmt"
@@ -578,7 +577,7 @@ func (p *BatchPlugin) submitJob(ctx *RequestContext, req *AWSRequest) (*AWSRespo
 		return nil, &AWSError{Code: "MissingParameter", Message: "jobName is required", HTTPStatus: http.StatusBadRequest}
 	}
 
-	jobID := generateBatchJobID()
+	jobID := generateBatchJobID(ctx.IDs)
 	job := BatchJob{
 		JobID:         jobID,
 		JobName:       body.JobName,
@@ -878,11 +877,9 @@ func (p *BatchPlugin) nextJobDefinitionRevision(goCtx context.Context, ctx *Requ
 	return highest, nil
 }
 
-// generateBatchJobID generates a random UUID-formatted job ID.
-func generateBatchJobID() string {
-	b := make([]byte, 16)
-	_, _ = rand.Read(b)
-	return fmt.Sprintf("%x-%x-%x-%x-%x", b[0:4], b[4:6], b[6:8], b[8:10], b[10:16])
+// generateBatchJobID mints a UUID-shaped job ID from m.
+func generateBatchJobID(m *IDMint) string {
+	return m.HexUUID()
 }
 
 // batchJSONResponse serializes v to JSON and returns an AWSResponse.

emulator/cloudwatchlogs_plugin.go

@@ -419,7 +419,7 @@ func (p *CloudWatchLogsPlugin) createLogStream(ctx *RequestContext, req *AWSRequ
 		LogStreamName:       body.LogStreamName,
 		ARN:                 cwLogStreamARN(ctx.Region, ctx.AccountID, body.LogGroupName, body.LogStreamName),
 		CreationTime:        now,
-		UploadSequenceToken: generateLambdaRevisionID(ctx.IDs),
+		UploadSequenceToken: ctx.IDs.HexUUID(),
 	}
 	data, err := json.Marshal(ls)
 	if err != nil {
@@ -607,7 +607,7 @@ func (p *CloudWatchLogsPlugin) putLogEvents(ctx *RequestContext, req *AWSRequest
 	var ls CWLogStream
 	if json.Unmarshal(streamData, &ls) == nil {
 		ls.LastIngestionTime = now
-		ls.UploadSequenceToken = generateLambdaRevisionID(ctx.IDs)
+		ls.UploadSequenceToken = ctx.IDs.HexUUID()
 		if updated, marshalErr := json.Marshal(ls); marshalErr == nil {
 			_ = p.state.Put(goCtx, cloudwatchLogsNamespace, streamKey, updated)
 		}

emulator/ec2_types.go

@@ -495,7 +495,7 @@ func generateAssociationID(m *IDMint) string {
 // flag day: a caller moves by taking a mint and calling [IDMint.Hex] with the same width.
 // EC2's own ids no longer come through here.
 //
-// TODO(#856): 28 draw sites remain on crypto/rand, tiered by service family on the issue;
+// TODO(#856): 23 draw sites remain on crypto/rand, tiered by service family on the issue;
 // delete this function when the last caller moves.
 func randomHex(n int) string {
 	b := make([]byte, n)

emulator/ecr_plugin.go

@@ -2,7 +2,6 @@ package emulator
 
 import (
 	"context"
-	"crypto/rand"
 	"encoding/base64"
 	"encoding/json"
 	"fmt"
@@ -401,7 +400,7 @@ func (p *ECRPlugin) putImage(ctx *RequestContext, req *AWSRequest) (*AWSResponse
 
 	digest := body.ImageDigest
 	if digest == "" {
-		digest = generateECRDigest()
+		digest = generateECRDigest(ctx.IDs)
 	}
 
 	img := ECRImage{
@@ -1213,11 +1212,16 @@ func (p *ECRPlugin) saveImageTagsMap(goCtx context.Context, tagsKey string, m ma
 	_ = p.state.Put(goCtx, ecrNamespace, tagsKey, b)
 }
 
-// generateECRDigest creates a random sha256 digest string.
-func generateECRDigest() string {
-	b := make([]byte, 32)
-	_, _ = rand.Read(b)
-	return fmt.Sprintf("sha256:%x", b)
+// generateECRDigest mints a sha256-shaped image digest from m, for a PutImage that supplied no
+// `imageDigest` of its own.
+//
+// A real digest is the SHA-256 of the image manifest, so ECR computes the same one for two pushes
+// of identical manifest bytes and treats the second as the same image. Substrate mints an
+// unrelated value instead, which is why every image it stores is distinct even when the manifests
+// are byte-identical; #1283 tracks deriving it from the manifest, which is a change to what the
+// digest *means* rather than to where its bytes come from and so is not #856's to make.
+func generateECRDigest(m *IDMint) string {
+	return "sha256:" + m.Hex(32)
 }
 
 // ecrJSONResponse marshals v as JSON and returns an AWSResponse with