Skip to content

fix(#856): the internal dispatch paths — every identifier in every CloudFormation stack - #1293

Merged
scttfrdmn merged 1 commit into
mainfrom
fix/856-internal-dispatch-mints
Sep 26, 2026
Merged

scttfrdmn merged 1 commit into
mainfrom
fix/856-internal-dispatch-mints

Conversation

@scttfrdmn

Copy link
Copy Markdown
Owner

Tier 8 of #856, and the one tier the draw-site counter could not have found.

Every earlier tier migrated a generator: a function that called crypto/rand and now
draws from IDMint instead. That work is finished — two non-test rand.Read sites remain,
one of which is IDMint's own fallback. But counting draw sites answers the wrong question.
A plugin whose minters are all migrated still publishes random identifiers if it is
called with nothing to derive from, and substrate has four places that call a plugin
with a RequestContext it built itself rather than parsed off the wire.

The CloudFormation deployer is the expensive one. Deploying a stack turns each resource into
an internal EC2, IAM or S3 request, and those requests were built with a fresh
generateRequestID() and no mint — so every identifier of every resource in every
CloudFormation stack
was undirected randomness, while StackDeployer itself contains no
draw site at all. A replayed DescribeStackResources reported four different
PhysicalResourceId values in a 200, and CreateStack's state_hash_after never
matched, because that hash covers all of them.

What changed

Two changes, fixing different failures:

  • dispatch sets IDs: NewIDMint(requestID) on the context it builds. That makes a replayed
    internal event reproduce, because a replay dispatches with the recorded request id.
  • WithDeployerMint derives the internal request id itself from the mint of the request that
    asked for the deployment ("req-cfn-" + m.Hex(12)). That makes a replayed outer
    CreateStack reproduce — replaying it re-runs the whole deployment, so the internal ids
    have to come out the same way twice. The sequence is reproducible because
    DeployWithOptions already sorts by type priority then logical id, never by map iteration,
    and a deployment is single-goroutine.

Both halves were measured by reverting each alone against the new replay test:
7 differences without the mint (all four PhysicalResourceId values plus CreateStack's
state_hash_after, critical), 6 differences without the derived request id. In both
cases the replay reported every event as a success — a 200 describing four resources that
were not the recorded ones, which is the silent-wrong-answer class rather than a visible
failure.

Three smaller pieces fall out of the same seam:

  • StartStackDriftDetection answered a StackDriftDetectionId that was substrate's internal
    req-… string verbatim — a shape no AWS reference describes, in an element a consumer hands
    straight back to DescribeStackDriftDetectionStatus. It is now IDMint.HexUUID.
  • deploySSMAssociation was the deployer's last randomHex caller. It mints from the
    deployer's mint directly, because there is no request to dispatch: SSM's plugin models Run
    Command and Parameter Store, not State Manager.
  • An API Gateway proxy integration invokes a Lambda through an internal request, and now
    carries a derived mint into it. The two requestId values inside the proxy event payload
    are deliberately untouched: they are handed to a function whose code never runs, so no API
    observation depends on them.

A Lambda event-source-mapping poll is deliberately left on the fallback, with an in-code note
pointing at #1292: its dispatches come from a wall-clock ticker and are recorded nowhere, so
there is no recorded id to derive from and a seed there would be no more reproducible than
the fallback it replaced.

Client.Deploy seeds a mint from a fresh request id rather than passing nil, so an in-process
deploy derives its resource ids from one seed. The seed is random because an in-process deploy
is not a recorded AWS request — there is no outer event to re-run — and each dispatched request
records the derived id it used, so a replay of those events still reuses it.

Tests

TestReplay_ACloudFormationStackReplaysWithTheIdentifiersItCreated records a four-resource
stack over three services, replays the stream with ValidateState, and asserts zero
Differences, TotalEvents == SuccessEvents and StateValid — the assertion this issue has
never been able to make about a stack. Plus
TestIDs_ADriftDetectionIDIsDerivedAndUUIDShaped (shape, the published bound, and the status
read that takes the id back) and
TestIDs_AProxyIntegrationInvokeIDDerivesFromTheRequestThatReachedIt, which covers the
seedless branch through a new APIGatewayInternalRequestIDForTest wrapper — that branch is
unreachable through the integration, since a request off the wire always carries a mint.

make lint 0 issues, make test green, the five make checks pass with the wire-bookkeeping
ratchet still at 330, patch coverage 29/29 against git diff -U0 main.

Provenance

API_DetectStackDrift publishes StackDriftDetectionId as a String with a maximum length
of 36
and no pattern. The UUID shape is observed from the page's own sample response,
2f2b2d60-df86-11e7-bea1-500c2example. IDMint.HexUUID renders exactly 36 characters, so the
published bound is met at its limit. Reading AWS's example is the provenance #671 permits;
inventing a bound from a sibling operation is not.

Everything else here is a change of where bytes come from, not of what they look like: no
identifier's rendering changes except the drift-detection id, which was not an AWS shape to
begin with.

Compatibility

  • A StackDriftDetectionId no longer looks like req-…. A test asserting on that literal
    prefix — substrate's internal request-id format, in an AWS response element — will need
    updating; one asserting the published 36-character bound will not.
  • Recorded CloudFormation streams are not portable across this change. A stream recorded
    before it holds physical resource ids drawn from crypto/rand, and replaying it now derives
    different ones. That is the same one-way migration every earlier tier of Identifiers minted from crypto/rand make a replay diverge from its recording, and three replay checks are weakened for it #856 carried, and it
    is the direction that makes replay assertable at all: re-record the stream.
  • No request or response shape changes, and no operation gains or loses a member.

Refs #856.

Every earlier tier of #856 migrated a generator. Counting generators answers the
wrong question: a plugin whose minters are all migrated still publishes random
identifiers if it is called with nothing to derive from. Substrate has four
internal dispatch sites — a RequestContext built in-process rather than parsed
off the wire — and the CloudFormation deployer is one of them, so every
identifier of every resource in every CloudFormation stack was undirected
randomness while the draw-site count read two.

dispatch now seeds a mint from its request id, and WithDeployerMint derives that
request id from the request that asked for the deployment. The two fix different
failures: the first makes a replayed internal event reproduce, the second makes a
replayed CreateStack — which re-runs the whole deployment — reproduce too.
Reverting either alone gives 7 and 6 differences respectively against the new
replay test, each reported as a successful 200.

Falling out of the same seam: a StackDriftDetectionId is no longer substrate's
internal req-… string, deploySSMAssociation is no longer the deployer's last
randomHex caller, and an API Gateway proxy integration carries a derived mint
into the Lambda invocation it dispatches. The Lambda ESM poller's two dispatches
stay mintless, with an in-code note, because a wall-clock ticker records nothing
for a replay to derive from (#1292).

Refs #856.
@scttfrdmn
scttfrdmn merged commit 15b217c into main Sep 26, 2026
17 checks passed
@scttfrdmn
scttfrdmn deleted the fix/856-internal-dispatch-mints branch September 26, 2026 15:43
@codecov

codecov Bot commented Sep 26, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant