Implement planned topic: 0011-external-storage

references/core/error-reference.md

@@ -5,7 +5,7 @@
 | **Non-determinism** | TMPRL1100 | `WorkflowTaskFailed` in history | Replay doesn't match history | Analyze error first. **If accidental**: fix code to match history → restart worker. **If intentional v2 change**: terminate → start fresh workflow. | https://github.com/temporalio/rules/blob/main/rules/TMPRL1100.md |
 | **Deadlock** | TMPRL1101 | `WorkflowTaskFailed` in history, worker logs | Workflow blocked too long (deadlock detected) | Remove blocking operations from workflow code (no I/O, no sleep, no threading locks). Use Temporal primitives instead. | https://github.com/temporalio/rules/blob/main/rules/TMPRL1101.md |
 | **Unfinished handlers** | TMPRL1102 | `WorkflowTaskFailed` in history | Workflow completed while update/signal handlers still running | Ensure all handlers complete before workflow finishes. Use `workflow.wait_condition()` to wait for handler completion. | https://github.com/temporalio/rules/blob/main/rules/TMPRL1102.md |
-| **Payload overflow** | TMPRL1103 | `WorkflowTaskFailed` or `ActivityTaskFailed` in history | Payload size limit exceeded (default 2MB) | Reduce payload size. Use external storage (S3, database) for large data and pass references instead. | https://github.com/temporalio/rules/blob/main/rules/TMPRL1103.md |
+| **Payload overflow** | TMPRL1103 | `WorkflowTaskFailed` or `ActivityTaskFailed` in history | Payload size limit exceeded (default 2MB) | Reduce payload size. Use the SDK's built-in External Storage where available (see `references/{your_language}/external-storage.md`; Go, Python, and TypeScript), or pass references to external storage yourself (see the Large Data Handling pattern in `references/core/patterns.md`). | https://github.com/temporalio/rules/blob/main/rules/TMPRL1103.md |
 | **Workflow code bug** | | `WorkflowTaskFailed` in history | Bug in workflow logic | Fix code → Restart worker → Workflow auto-resumes | |
 | **Missing workflow** | | Worker logs | Workflow not registered | Add to worker.py → Restart worker | |
 | **Missing activity** | | Worker logs | Activity not registered | Add to worker.py → Restart worker | |

references/core/gotchas.md

@@ -212,3 +212,5 @@ See language-specific gotchas for details.
 - Workflow history growing unboundedly
 
 **The Fix**: Store large data externally (S3/GCS) and pass references, use compression codecs, or chunk data across multiple activities. See the Large Data Handling pattern in `references/core/patterns.md`.
+
+Before hand-rolling reference passing, check whether the SDK does it for you: the Go, Python, and TypeScript SDKs have built-in External Storage that applies the claim-check pattern automatically. See `references/{your_language}/external-storage.md`, if available.

references/core/patterns.md

@@ -368,6 +368,8 @@ This ensures that on replay, already-completed steps are skipped.
 - Max 4MB per gRPC message
 - Max 50MB for workflow history (aim for < 10MB)
 
+**Check for SDK support first**: the Go, Python, and TypeScript SDKs have built-in External Storage that applies the claim-check pattern for you — Payloads over a size threshold are offloaded to S3 or GCS and replaced in Event History with a small reference, with no changes to Workflow or Activity code. Prefer it where it exists; see `references/{your_language}/external-storage.md`, if available. The rest of this section applies when you need explicit control over which data is offloaded, or when your SDK has no built-in support.
+
 **Key Principle**: Large data should never flow through workflow history. Activities read and write large data directly, passing only small references through the workflow.
 
 **Wrong Approach**:

references/go/go.md

@@ -250,5 +250,6 @@ See `references/go/testing.md` for info on writing tests.
 - **`references/go/testing.md`** - TestWorkflowEnvironment, time-skipping, activity mocking
 - **`references/go/advanced-features.md`** - Schedules, worker tuning, and more
 - **`references/go/data-handling.md`** - Data converters, payload codecs, encryption
+- **`references/go/external-storage.md`** - Claim-check pattern for large payloads (S3 and GCS drivers, custom drivers, codec-server handling, multi-region durability)
 - **`references/go/versioning.md`** - Patching API (`workflow.GetVersion`), Worker Versioning
 - **`references/go/determinism-protection.md`** - Information on **`workflowcheck`** tool to help statically check for determinism issues.

references/python/python.md

@@ -179,6 +179,7 @@ See `references/python/testing.md` for info on writing tests.
 - **`references/python/sync-vs-async.md`** - Sync vs async activities, event loop blocking, executor configuration
 - **`references/python/advanced-features.md`** - Schedules, worker tuning, and more
 - **`references/python/data-handling.md`** - Data converters, Pydantic, payload encryption
+- **`references/python/external-storage.md`** - Claim-check pattern for large payloads (S3 driver, custom drivers, codec-server handling, multi-region durability)
 - **`references/python/versioning.md`** - Patching API, workflow type versioning, Worker Versioning
 - **`references/python/determinism-protection.md`** - Python sandbox specifics, forbidden operations, pass-through imports
 - **`references/python/ai-patterns.md`** - LLM integration, Pydantic data converter, AI workflow patterns

references/typescript/external-storage.md

@@ -0,0 +1,246 @@
+# TypeScript SDK External Storage
+
+> [!NOTE]
+> This feature is in Public Preview. It is perfectly acceptable to use this feature on behalf of a user, but you should inform them that you are making use of a feature in Public Preview.
+
+## What this is
+
+External Storage uses the **claim check pattern**: it offloads each Payload to an external store (e.g. Amazon S3 or Google Cloud Storage), records a small reference token (the "claim check") in Event History, and uses that token to retrieve the Payload when needed. The SDK handles storage and retrieval transparently.
+
+## When to use it
+
+- A Workflow input, Activity input, Activity result, or Workflow result will exceed the **2 MB** per-payload limit (fixed at 2 MB on Temporal Cloud; configurable on self-hosted only).
+- Long Event Histories degrade Workflow Task latency (e.g. AI agent conversations growing per turn).
+- The user wants payload data to live in storage **they** control. Set `payloadSizeThreshold: 0` to externalize all payloads.
+- The user is migrating from self-hosted (with a larger configured limit) to Temporal Cloud.
+
+## Where it sits in the pipeline
+
+Order: **Payload Converter → Payload Codec → External Storage**. Storage runs last on outbound; it reverses on inbound.
+
+Consequences:
+
+- If a Payload Codec encrypts data, the bytes are already encrypted **before** upload.
+- The Temporal UI displays the reference token, not the data; the SDK retrieves the payload transparently before handing it to your Workflow or Client.
+- Every Client and Worker that might read an offloaded payload needs the same External Storage configuration.
+
+## Setup with a built-in driver
+
+The TypeScript SDK provides first-party drivers for Amazon S3 and Google Cloud Storage. Install one driver, its SDK adapter, and the cloud provider's SDK. Keep all `@temporalio/*` packages on the same version.
+
+Amazon S3:
+
+```bash
+npm install @temporalio/external-storage-s3 \
+  @temporalio/external-storage-s3-aws-sdk \
+  @temporalio/envconfig \
+  @aws-sdk/client-s3
+```
+
+Google Cloud Storage:
+
+```bash
+npm install @temporalio/external-storage-gcs \
+  @temporalio/external-storage-gcs-google-sdk \
+  @temporalio/envconfig \
+  @google-cloud/storage
+```
+
+### Amazon S3 driver
+
+```typescript
+import { S3Client } from '@aws-sdk/client-s3';
+import { S3StorageDriver } from '@temporalio/external-storage-s3';
+import { AwsSdkS3StorageDriverClient } from '@temporalio/external-storage-s3-aws-sdk';
+
+const s3Client = new S3Client({ region: 'us-east-2' });
+
+const driver = new S3StorageDriver({
+  client: new AwsSdkS3StorageDriverClient(s3Client),
+  bucket: 'my-temporal-payloads',
+});
+```
+
+The AWS SDK reads standard credentials from environment variables, an IAM role, or the AWS config file.
+
+### Google Cloud Storage driver
+
+```typescript
+import { Storage } from '@google-cloud/storage';
+import { GcsStorageDriver } from '@temporalio/external-storage-gcs';
+import { GoogleCloudGcsStorageDriverClient } from '@temporalio/external-storage-gcs-google-sdk';
+
+const storage = new Storage();
+
+const driver = new GcsStorageDriver({
+  client: new GoogleCloudGcsStorageDriverClient(storage),
+  bucket: 'my-temporal-payloads',
+});
+```
+
+The Google Cloud SDK reads Application Default Credentials.
+
+For either driver, `bucket` can be a function instead of a string. The function receives the store context and Payload and returns a bucket name, allowing runtime routing.
+
+### Configure the Client and Worker
+
+Create one Data Converter configuration and pass it to both the Client and Worker. Load connection settings with `loadClientConnectConfig()`, and remember that `NativeConnection` carries no namespace, so the Worker needs `namespace` passed explicitly:
+
+```typescript
+import { Client, Connection } from '@temporalio/client';
+import { ExternalStorage } from '@temporalio/common';
+import { loadClientConnectConfig } from '@temporalio/envconfig';
+import { NativeConnection, Worker } from '@temporalio/worker';
+
+const dataConverter = {
+  externalStorage: new ExternalStorage({ drivers: [driver] }),
+};
+
+const config = loadClientConnectConfig();
+
+const connection = await Connection.connect(config.connectionOptions);
+const client = new Client({ connection, namespace: config.namespace, dataConverter });
+
+const workerConnection = await NativeConnection.connect(config.connectionOptions);
+const worker = await Worker.create({
+  connection: workerConnection,
+  namespace: config.namespace,
+  workflowsPath: require.resolve('./workflows'),
+  taskQueue: 'my-task-queue',
+  dataConverter,
+});
+```
+
+External Storage runs outside the Workflow sandbox, so pass the driver object directly. Workflows and Activities use it automatically; business logic does not change.
+
+## Built-in driver behavior
+
+The S3 and GCS drivers:
+
+- Upload and download Payloads concurrently.
+- Address objects by a SHA-256 hash of their contents, deduplicating identical Payloads.
+- Verify the content hash during retrieval.
+- Reject any single Payload larger than `maxPayloadSize`, which defaults to **50 MiB**.
+- Include diagnostic metadata in storage errors.
+
+The External Storage threshold does not override `maxPayloadSize`. Configure the backing store and driver for the largest Payload the application needs to support.
+
+## Payload size threshold
+
+- Default: **256 KiB**.
+- Set `payloadSizeThreshold: 0` to externalize **all** Payloads regardless of size.
+- Payloads whose serialized size is **greater than or equal to** the threshold are eligible for external storage.
+- The measured size includes Payload metadata after Payload Converter and Payload Codec processing, not only the raw application value.
+
+```typescript
+const dataConverter = {
+  externalStorage: new ExternalStorage({
+    drivers: [driver],
+    payloadSizeThreshold: 0,
+  }),
+};
+```
+
+## Multiple drivers and migration
+
+When registering more than one driver, supply a `driverSelector`. The selector chooses which driver stores each Payload. Unselected registered drivers remain available for **retrieval**, which supports migrations without losing access to existing claims.
+
+- Return `null` from the selector to keep a specific Payload inline in Event History.
+- Every registered driver must have a distinct `name`.
+- `S3StorageDriver` defaults its name to `"aws.s3driver"`; when registering two S3 drivers, set `driverName` on at least one.
+
+```typescript
+const preferredDriver = new S3StorageDriver({
+  client: new AwsSdkS3StorageDriverClient(s3Client),
+  bucket: 'my-bucket',
+});
+const legacyDriver = new LegacyStorageDriver();
+
+const externalStorage = new ExternalStorage({
+  drivers: [preferredDriver, legacyDriver],
+  driverSelector: () => preferredDriver,
+});
+```
+
+Useful routing patterns include driver migration, hot/cold storage tiers, per-tenant storage, and selecting S3 or GCS based on the runtime environment.
+
+## Custom storage driver
+
+Implement the `StorageDriver` interface with two readonly properties and two methods:
+
+- `name: string` — unique identifier for **this driver instance**, stored in the reference so the SDK can route retrieval. Changing it after Payloads are stored **breaks retrieval**.
+- `type: string` — stable identifier for the driver implementation, shared by all instances of that implementation and reported in Worker heartbeats (e.g. `"aws.s3driver"`).
+- `store(context, payloads): Promise<StorageDriverClaim[]>` — serialize and upload each Payload, then return one claim per Payload. Each claim contains string key-value data sufficient to find the object later.
+- `retrieve(context, claims): Promise<Payload[]>` — download and reconstruct one Payload per claim, preserving input order.
+
+The `store()` context includes an optional `abortSignal` and `target`. The target is a discriminated union:
+
+- Check `target.kind` for `"workflow"` or `"activity"`.
+- Read `namespace`, `id`, `runId`, and `type` to scope storage keys.
+
+Honor `abortSignal` in storage calls so sibling operations can be cancelled after the first failure. Content-addressable keys can make retries idempotent and deduplicate identical Payloads.
+
+Return exactly one claim for each Payload passed to `store()` and exactly one Payload for each claim passed to `retrieve()`. Store the complete serialized Payload protobuf: application data has already passed through the Payload Converter and Payload Codec before reaching the driver.
+
+## Multi-region durability with Amazon S3
+
+For regional-failure tolerance, configure S3 Cross-Region Replication and an S3 Multi-Region Access Point (MRAP), then use the MRAP ARN as `bucket`.
+
+MRAP requests require a SigV4A signer. The AWS SDK for JavaScript does not bundle one, so install and register it at application startup:
+
+```bash
+npm install @aws-sdk/signature-v4a
+```
+
+```typescript
+import '@aws-sdk/signature-v4a';
+```
+
+Then configure the driver with the MRAP ARN:
+
+```typescript
+const driver = new S3StorageDriver({
+  client: new AwsSdkS3StorageDriverClient(s3Client),
+  bucket: 'arn:aws:s3::123456789012:accesspoint/mfzwi23gnjvgw.mrap',
+});
+```
+
+`@aws-sdk/signature-v4-crt` is an alternative backed by the AWS Common Runtime. The AWS SDK prefers it when both signer implementations are installed.
+
+Cross-region replication is eventually consistent. Activities reading newly written Payloads from another region need an appropriate Retry Policy. Replication, versioning, and Replication Time Control can add significant cost.
+
+## Codec Server with External Storage
+
+When Workers and Clients use External Storage, Event History contains reference tokens — not payload data. A plain codec server that only implements `/encode` and `/decode` leaves the Web UI and CLI showing raw reference tokens.
+
+The TypeScript SDK does not ship a codec-server handler, so implement the routes yourself (e.g. with Express), wiring in your storage drivers, your pre-storage codecs (the Payload Codecs your Workers use), and any post-storage codecs (applied by a proxy after external storage):
+
+- **`/download`** — retrieves payload data from external storage and decodes it through the Payload Codec. The Web UI calls this when a user clicks to view the full payload behind a reference.
+- **`/decode`** — decodes encoded payloads and, by default, retrieves storage references inline. Support `?preserveStorageRefs=true` to return storage references as-is without retrieval; the Web UI uses it to render history without downloading every blob.
+- **`/encode`** — applies the Payload Codec, then uploads payloads exceeding the threshold and replaces them with reference tokens.
+
+**Don't point a Worker's remote codec at the storage-aware handler** — it runs the full encode-store-encode and decode-retrieve-decode pipeline. Serve remote codecs from a separate non-storage endpoint, configured with the same codecs.
+
+## Lifecycle and failure handling
+
+Temporal does **not** automatically delete Payloads from the external store. Configure a bucket lifecycle policy with:
+
+```
+TTL > Maximum Workflow Run Timeout + Namespace Retention Period
+```
+
+Example: Run Timeout 14 days + Namespace retention 30 days → set TTL to at least 44 days.
+
+For Workflows with no finite Run Timeout, there is no safe finite TTL. Use Continue-as-New so the new run uploads fresh Payloads and the old run's Payloads only need to survive its retention period.
+
+The SDK does not retry a failed `store()` or `retrieve()` call within the same Task attempt. The failure fails the current Workflow Task or Activity Task attempt; Temporal then retries the Task as a whole. Storage operations should therefore be idempotent.
+
+## Anti-patterns
+
+- **Don't change a driver's `name` after Payloads have been stored.** The name is embedded in references; changing it breaks retrieval.
+- **Don't register duplicate driver names.** Give each instance a unique `name` or `driverName`.
+- **Don't register multiple drivers without a `driverSelector`.** Construction fails when more than one driver is registered without one.
+- **Don't omit External Storage configuration from a Client or Worker that may retrieve offloaded data.** It cannot resolve the reference without the matching driver.
+- **Don't assume the 2 MB Temporal limit is the built-in driver's maximum.** The S3 and GCS drivers default `maxPayloadSize` to 50 MiB.
+- **Don't point a Worker's remote codec at a storage-aware codec-server handler.** Serve remote codecs from a separate non-storage endpoint.
+- **Don't omit a lifecycle policy.** Payloads are otherwise retained indefinitely, and failed requests can leave orphaned objects.

references/typescript/typescript.md

@@ -180,5 +180,6 @@ See `references/typescript/testing.md` for info on writing tests.
 - **`references/typescript/testing.md`** - TestWorkflowEnvironment, time-skipping, activity mocking
 - **`references/typescript/advanced-features.md`** - Schedules, worker tuning, and more
 - **`references/typescript/data-handling.md`** - Data converters, payload encryption, etc.
+- **`references/typescript/external-storage.md`** - Claim-check pattern for large Payloads (S3 and GCS drivers, custom drivers, codec-server handling, multi-region durability)
 - **`references/typescript/versioning.md`** - Patching API, workflow type versioning, Worker Versioning
 - **`references/typescript/determinism-protection.md`** - V8 sandbox and bundling