Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -294,6 +294,37 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
route, integration or mapping ID — now looks like one AWS would issue. This is the one identifier
whose *alphabet* changed when it was derived, and it widened rather than narrowed. 23 draw sites
remain on `crypto/rand`.
- **The identity, key and certificate family of draw sites is derived** (#856). Eight generators
across seven services move onto `IDMint`: a Cognito user-pool ID, user-pool client ID and client
secret, a Cognito identity-pool ID and identity ID, an IAM Identity Center permission-set ID, a KMS
key ID, an ACM certificate ID, a Secrets Manager version ID, a WAFv2 web ACL and IP set ID, and an
API Gateway API key's ID and value. Every rendering is byte-for-byte the shape the `crypto/rand`
version produced — the width, the case and the alphabet — so an identifier a previous substrate
recorded is still the shape this one mints; nothing in any of the seven API models distinguishes the
two renderings of a UUID-shaped value, which is why #671's "only what the API model states" leaves
them alone. 15 draw sites remain on `crypto/rand`.
- **A replayed secret version ID answers the read it recorded** (#856). This is the quietest way an
identifier can break a replay, and the reason the family's replay test records the pair: substrate
reports an unknown `VersionId` to `GetSecretValue` by *omitting* `SecretString` rather than by
refusing, so before this a replay answered a recorded read with a **200 that had silently lost a
member**. A version ID is a key a caller hands back, not a bare identifier, which is what puts it
above the rest of the family. Reverting the minter to confirm the assertion is not vacuous produces
21 differences, of which that lost member is one.
- **A replayed WAFv2 `LockToken` still unlocks the update it recorded** (#856). The optimistic-locking
contract requires a caller to hand the token back to `UpdateWebACL`, so a replay that re-minted it
answered the recorded update with `WAFOptimisticLockException` instead of the recorded success. One
minter serves the web ACL ID, the IP set ID and both lock tokens because API_WebACLSummary publishes
the identical 1–36 character `^[0-9a-f]{8}-(?:[0-9a-f]{4}-){3}[0-9a-f]{12}$` constraint on each.
- **An API Gateway API key is no longer minted by ACM's certificate generator** (#856). `CreateApiKey`
reached into `generateACMCertID` for both UUID-shaped strings it returns, so a change to ACM's
rendering would have silently moved API Gateway's. The two now have their own minters and the `id`
and `value` draw in turn, so they differ. `generateACMCertID` no longer returns an error either —
the mint cannot fail where `rand.Read` could, so the caller's dead error branch went with it.
- **A KMS `GenerateDataKey` response is reproducible in both of its members** (#856). The stub data key
is not an identifier, and it is in scope because a caller observes it twice over in one response: the
bytes are the `Plaintext`, and the same bytes are wrapped into the `CiphertextBlob`. A recorded
`Decrypt` was never affected — substrate's ciphertext carries its own plaintext, so a replayed
decrypt answers from the recorded blob regardless — so the break was in the create, not the read.
- **A stream recorded under a seed replays under the same seed** (#1140). Every seedable outcome in
substrate is written through a control-plane endpoint, and only the AWS path recorded anything — so
a seed never entered the event stream. A replay opens by resetting the whole `StateManager`, and a
Expand Down
47 changes: 41 additions & 6 deletions docs/services.md
Original file line number Diff line number Diff line change
Expand Up @@ -2208,11 +2208,12 @@ Three kinds of value stay random, and one more is still migrating:
which predate this rule and are what generalising it was modelled on.
- EC2, IAM, STS, SQS, SNS, Lambda, EFS, FSx, Transfer, ECS, Step Functions, EventBridge,
CloudWatch Logs, CloudFront, Service Quotas, API Gateway (v1 and v2), AppSync, Batch, EMR
Serverless, ECR, ELB and Route 53 identifiers are derived today. A CloudFront distribution,
invalidation and origin access control all draw from one generator, so the three moved together
with the origin access control family (#1277). The remaining services are migrating one family
at a time, tracked on #856; until a service moves, its identifiers are still drawn from
`crypto/rand` and a replay of a stream creating one of its resources still diverges.
Serverless, ECR, ELB, Route 53, Cognito (both the user-pool and the identity-pool API), IAM
Identity Center, KMS, ACM, Secrets Manager and WAFv2 identifiers are derived today. A CloudFront
distribution, invalidation and origin access control all draw from one generator, so the three
moved together with the origin access control family (#1277). The remaining services are
migrating one family at a time, tracked on #856; until a service moves, its identifiers are still
drawn from `crypto/rand` and a replay of a stream creating one of its resources still diverges.

Nine of those services publish an identifier from one shared generator rather than declaring their
own, so they moved together: an ECS task ID, a Step Functions execution name, an SQS message ID,
Expand All @@ -2229,6 +2230,23 @@ digit never did. Drawing a byte per character reaches the whole published set, s
resource ID, deployment ID, authorizer ID, usage-plan ID or API Gateway v2 route, integration and
mapping ID now looks like one AWS would issue.

**Not every derived value is an identifier.** Three kinds of minted string are in scope because a
caller hands them *back*, so a re-minted one turns a recorded success into a refusal or a silently
incomplete response. A WAFv2 `LockToken` is the clearest: the optimistic-locking contract requires a
caller to return the token to `UpdateWebACL`, so a replay that re-minted it would answer the recorded
update with `WAFOptimisticLockException` instead of the recorded success. A Secrets Manager version
ID is the quietest: `GetSecretValue` reports an unknown `VersionId` by *omitting* `SecretString`
rather than refusing, so a re-minted version ID produces a 200 that has lost a member. And a KMS
`GenerateDataKey` stub data key is observed twice in one response — as `Plaintext` and wrapped inside
`CiphertextBlob` — so both members of a recorded response depend on it. (A recorded `Decrypt` does
not: substrate's ciphertext carries its plaintext, so a replayed decrypt answers from the recorded
blob either way.)

**One service used to mint another's identifiers.** An API Gateway API key's `id` and `value` were
drawn from ACM's certificate-ID generator, which meant a change to ACM's rendering would silently
move API Gateway's. #856 split them into separate minters; both still render the UUID shape they
always did, because neither API publishes a pattern that would decide the question (#671).

An ECR image digest is minted rather than computed from the manifest, so it is reproducible across
a replay but is not the SHA-256 of the image it names, and two pushes of identical manifest bytes
store two images where AWS stores one. [#1283](https://github.com/scttfrdmn/substrate/issues/1283)
Expand Down Expand Up @@ -12532,6 +12550,14 @@ SNS publish: $0.0000005 per message.
| UntagResource | Idempotent — an absent key is not an error — but a secret scheduled for deletion is refused even then, see [A deleted secret is scheduled, not removed](#a-deleted-secret-is-scheduled-not-removed) |
| RotateSecret | Records the rotation function and schedule and echoes `ClientRequestToken` as `VersionId`; no rotation function is executed, and a secret scheduled for deletion is refused — see [A rotation is configured, not run](#a-rotation-is-configured-not-run) |

A version ID is minted from the request ID (#856), so a replayed `CreateSecret` or `PutSecretValue`
reports the version the recording reported and the recorded `GetSecretValue` naming it still answers
its value. Two things about the value are known-unfaithful and tracked on
[#1285](https://github.com/scttfrdmn/substrate/issues/1285): it is sixteen characters where
`VersionId` publishes a minimum of 32, and `CreateSecret` and `PutSecretValue` mint their own rather
than using the caller's `ClientRequestToken`, which AWS documents as *becoming* the version ID and
builds an idempotency contract on. `RotateSecret` already echoes the token it was sent.

### A `SecretId` addresses the secret its own ARN names

`SecretId` is documented as "the ARN or name of the secret", and the two halves have
Expand Down Expand Up @@ -16251,7 +16277,7 @@ ECS Fargate vCPU: $0.04048 per vCPU-hour. Memory: $0.004445 per GB-hour.

| Operation | Notes |
|-----------|-------|
| CreateUserPool | Pool ID format: `{region}_{12-char alphanum}` |
| CreateUserPool | Pool ID format: `{region}_{12-char alphanum}`, derived from the request ID (#856) |
| DescribeUserPool | |
| UpdateUserPool | Replaces the published configuration and answers an empty body — see below |
| DeleteUserPool | |
Expand Down Expand Up @@ -16324,6 +16350,15 @@ the tag set outright like every other published member. Note that `DescribeUserP
set as `Tags` rather than the published `UserPoolTags`
([#1136](https://github.com/scttfrdmn/substrate/issues/1136)).

`CreateUserPool`, `DescribeUserPool` and `UpdateUserPool` report the pool's identifier as
`UserPool.UserPoolId`, where `UserPoolType` publishes it as `Id` and carries no `UserPoolId` member at
all ([#1286](https://github.com/scttfrdmn/substrate/issues/1286)). The cause is that the state struct
is also the wire struct, so its storage-side tag reaches the response; `ListUserPools` escapes it only
because it builds a separate summary type, and already renders `Id`. So the two disagree inside one
service, which is what makes it a shape bug rather than a naming preference — an SDK caller reads
`CreateUserPoolOutput.UserPool.Id` as nil and must fall back to `ListUserPools` to learn the ID of the
pool it just created.

### CloudFormation resource types

| Type | Ref | Notes |
Expand Down
3 changes: 2 additions & 1 deletion docs/testing-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -337,7 +337,8 @@ tests above are built on caller-chosen bucket and key names instead.

Two caveats. **Not every service's identifiers are derived yet.** EC2, IAM, STS, SQS, SNS,
Lambda, EFS, FSx, Transfer, ECS, Step Functions, EventBridge, CloudWatch Logs, CloudFront,
Service Quotas, API Gateway (v1 and v2), AppSync, Batch, EMR Serverless, ECR, ELB and Route 53
Service Quotas, API Gateway (v1 and v2), AppSync, Batch, EMR Serverless, ECR, ELB, Route 53,
Cognito (both APIs), IAM Identity Center, KMS, ACM, Secrets Manager and WAFv2
are; the rest are migrating one family at a time, and until a service moves, a
replay of a stream creating one of its resources still diverges. And a recording made against an
**unfrozen** clock can still diverge on a `state_hash_after` even when every identifier
Expand Down
28 changes: 15 additions & 13 deletions emulator/acm_plugin.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@ package emulator

import (
"context"
"crypto/rand"
"encoding/json"
"fmt"
"net/http"
Expand Down Expand Up @@ -91,10 +90,7 @@ func (p *ACMPlugin) requestCertificate(ctx *RequestContext, req *AWSRequest) (*A
keyAlgo = "RSA_2048"
}

certID, err := generateACMCertID()
if err != nil {
return nil, fmt.Errorf("acm requestCertificate generateACMCertID: %w", err)
}
certID := generateACMCertID(ctx.IDs)
certArn := fmt.Sprintf("arn:aws:acm:%s:%s:certificate/%s", ctx.Region, ctx.AccountID, certID)

tags := make(map[string]string, len(body.Tags))
Expand Down Expand Up @@ -405,14 +401,20 @@ func (p *ACMPlugin) listTagsForCertificate(ctx *RequestContext, req *AWSRequest)
return acmJSONResponse(http.StatusOK, response{Tags: tags})
}

// generateACMCertID generates a UUID-like string for an ACM certificate ID
// using cryptographically random bytes.
func generateACMCertID() (string, error) {
b := make([]byte, 16)
if _, err := rand.Read(b); err != nil {
return "", fmt.Errorf("generateACMCertID rand.Read: %w", err)
}
return fmt.Sprintf("%x-%x-%x-%x-%x", b[0:4], b[4:6], b[6:8], b[8:10], b[10:16]), nil
// generateACMCertID mints an ACM certificate ID from m, in the UUID shape API_RequestCertificate
// writes into the ARN form it documents:
// `arn:aws:acm:us-east-1:123456789012:certificate/12345678-1234-1234-1234-123456789012`.
//
// [IDMint.HexUUID] rather than [IDMint.UUID]: the crypto/rand form reshaped sixteen raw bytes
// without setting the RFC 4122 version and variant nibbles, and #856 does not change which bytes a
// caller sees. `CertificateArn`'s published pattern ends `[\w+=,.@-]+(/[\w+=,.@-]+)*`, which admits
// either rendering, so nothing in the API model distinguishes them (#671).
//
// It no longer returns an error. The mint cannot fail where rand.Read could, so the caller's error
// branch went with it — and it no longer serves API Gateway's API keys either, which drew from it
// before [generateAPIGatewayAPIKey] existed.
func generateACMCertID(m *IDMint) string {
return m.HexUUID()
}

// acmJSONResponse marshals v as JSON and returns an AWSResponse with
Expand Down
24 changes: 16 additions & 8 deletions emulator/apigateway_plugin.go
Original file line number Diff line number Diff line change
Expand Up @@ -1162,14 +1162,8 @@ func (p *APIGatewayPlugin) createAPIKey(ctx *RequestContext, req *AWSRequest) (*
}
}

keyID, err := generateACMCertID()
if err != nil {
return nil, fmt.Errorf("apigateway createApiKey generateID: %w", err)
}
keyValue, err := generateACMCertID()
if err != nil {
return nil, fmt.Errorf("apigateway createApiKey generateValue: %w", err)
}
keyID := generateAPIGatewayAPIKey(ctx.IDs)
keyValue := generateAPIGatewayAPIKey(ctx.IDs)

key := APIKeyState{
ID: keyID,
Expand Down Expand Up @@ -1514,6 +1508,20 @@ func generateAPIGatewayID(m *IDMint) string {
return m.Chars(10, apigwIDAlphabet)
}

// generateAPIGatewayAPIKey mints one of the two UUID-shaped strings a `CreateApiKey` response
// carries — the key's `id` and its `value` — each drawn from m in turn, so the two differ.
//
// It replaces a call into ACM's certificate-ID generator, which is where `createAPIKey` used to
// reach for a UUID; one service minting another's identifiers meant a change to ACM's rendering
// silently moved API Gateway's, and #856 splits them.
//
// The rendering is [IDMint.HexUUID]'s, unchanged from the crypto/rand form, and nothing in the API
// model asks otherwise: API_ApiKey publishes **no** length constraint and **no** pattern on either
// `id` or `value`, so the shape is substrate's to keep rather than #856's to revisit (#671).
func generateAPIGatewayAPIKey(m *IDMint) string {
return m.HexUUID()
}

// --- Response helper ---------------------------------------------------------

// apigwJSONResponse marshals v as JSON and returns an AWSResponse with
Expand Down
38 changes: 14 additions & 24 deletions emulator/cognito_identity_plugin.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,6 @@ package emulator

import (
"context"
"crypto/rand"
"encoding/hex"
"encoding/json"
"fmt"
"net/http"
Expand Down Expand Up @@ -84,7 +82,7 @@ func (p *CognitoIdentityPlugin) createIdentityPool(ctx *RequestContext, req *AWS
return nil, &AWSError{Code: "InvalidParameterException", Message: "IdentityPoolName is required", HTTPStatus: http.StatusBadRequest}
}

poolID := generateIdentityPoolID(ctx.Region)
poolID := generateIdentityPoolID(ctx.IDs, ctx.Region)
now := p.tc.Now()
pool := CognitoIdentityPool{
IdentityPoolID: poolID,
Expand Down Expand Up @@ -224,7 +222,7 @@ func (p *CognitoIdentityPlugin) listIdentityPools(ctx *RequestContext, req *AWSR
// --- Identity operations -----------------------------------------------------

func (p *CognitoIdentityPlugin) getID(ctx *RequestContext, _ *AWSRequest) (*AWSResponse, error) {
identityID := generateIdentityPoolID(ctx.Region)
identityID := generateIdentityPoolID(ctx.IDs, ctx.Region)
type response struct {
IdentityID string `json:"IdentityId"`
}
Expand All @@ -242,7 +240,7 @@ func (p *CognitoIdentityPlugin) getCredentialsForIdentity(ctx *RequestContext, r
}
identityID := body.IdentityID
if identityID == "" {
identityID = generateIdentityPoolID(ctx.Region)
identityID = generateIdentityPoolID(ctx.IDs, ctx.Region)
}

expiration := p.tc.Now().Add(time.Hour)
Expand Down Expand Up @@ -345,25 +343,17 @@ func (p *CognitoIdentityPlugin) loadIdentityPool(ctx *RequestContext, poolID str
return &pool, nil
}

// generateIdentityPoolID returns a Cognito Identity Pool ID of the form
// {region}:{xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx} using crypto/rand.
func generateIdentityPoolID(region string) string {
return region + ":" + generateIdentityUUID()
}

// generateIdentityUUID generates a lowercase hex UUID in the standard
// xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx format using crypto/rand.
func generateIdentityUUID() string {
b := make([]byte, 16)
_, _ = rand.Read(b)
// Set version 4 and variant bits.
b[6] = (b[6] & 0x0f) | 0x40
b[8] = (b[8] & 0x3f) | 0x80
return hex.EncodeToString(b[0:4]) + "-" +
hex.EncodeToString(b[4:6]) + "-" +
hex.EncodeToString(b[6:8]) + "-" +
hex.EncodeToString(b[8:10]) + "-" +
hex.EncodeToString(b[10:16])
// generateIdentityPoolID mints a Cognito identity pool ID from m, in the form
// API_CreateIdentityPool publishes for its `IdentityPoolId` response element: "an identity pool ID
// in the format REGION:GUID", 1–55 characters matching `[\w-]+:[0-9a-f-]+`. The pattern's `[0-9a-f]`
// is why the GUID half is lowercase hex and not the uppercase the IDP plugin's IDs use.
//
// [IDMint.UUID] rather than [IDMint.HexUUID], because the generator this replaces set the RFC 4122
// version and variant bits itself, and UUID is the method that sets them; the rendering is
// unchanged. The same minter serves `GetId`'s identity IDs, which the same page publishes in the
// same form.
func generateIdentityPoolID(m *IDMint, region string) string {
return region + ":" + m.UUID()
}

// cognitoIdentityJSONResponse serializes v as JSON and returns an AWSResponse with
Expand Down
Loading
Loading