diff --git a/.cursor-plugin/plugin.json b/.cursor-plugin/plugin.json index a4a38de..8f12181 100644 --- a/.cursor-plugin/plugin.json +++ b/.cursor-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "temporal", - "version": "0.4.0", + "version": "0.4.1", "description": "Comprehensive skill for the entire Temporal lifecycle — developing applications, using the Temporal CLI, running and managing Temporal Server, and working with Temporal Cloud.", "author": { "name": "Temporal", @@ -22,4 +22,4 @@ ], "logo": "assets/temporal-logo.svg", "skills": "./skills/" -} +} \ No newline at end of file diff --git a/skills/temporal-cloud-setup/SKILL.md b/skills/temporal-cloud-setup/SKILL.md index b6b9ec4..abd587f 100644 --- a/skills/temporal-cloud-setup/SKILL.md +++ b/skills/temporal-cloud-setup/SKILL.md @@ -1,7 +1,13 @@ --- name: temporal-cloud-setup -description: Set up Temporal Cloud and run a sample Workflow on it for the user, doing the work end to end. Use when the user wants to set up Temporal Cloud, get started on Temporal Cloud, install the unified Temporal CLI (prerelease cloud-cli), create a Cloud namespace or API key, clone a money-transfer sample app, write the client config TOML, or connect a local Worker to Temporal Cloud and run a sample Workflow. This is the Cloud setup path, not the local learning path (see temporal-getting-started). Covers Python, TypeScript, Go, Java, .NET, and Ruby SDKs. -version: 0.8.1 +description: Set up Temporal Cloud and run a sample Workflow on it for the user, doing + the work end to end. Use when the user wants to set up Temporal Cloud, get started + on Temporal Cloud, install the unified Temporal CLI (prerelease cloud-cli), create + a Cloud namespace or API key, clone a money-transfer sample app, write the client + config TOML, or connect a local Worker to Temporal Cloud and run a sample Workflow. + This is the Cloud setup path, not the local learning path (see temporal-getting-started). + Covers Python, TypeScript, Go, Java, .NET, and Ruby SDKs. +version: 0.8.2 disable-model-invocation: true --- @@ -796,4 +802,4 @@ On `status=error`, map the `error_code` via **`references/failure-handling.md`** - `scripts/provision.sh` — **the deterministic executor.** Owns preflight / **detect-tools** / **preview** / install / login / regions / namespace-create / **install-deps (manager-parameterized)** / key-mint+config-write / verify / await-auth / **run-workflow (Worker + starter)** / clone / repair-config / cleanup-info. Invoke it and parse its `=== RESULT ===` block (see "Execution model" above); it is the single source of truth for the pinned CLI flags, the per-SDK run commands, **and the per-(SDK,manager) install matrix + minimum-version table**. Pure bash, portable across Claude Code, Codex, and Cursor. **Read-only during a run — invoke it, never edit it (read-only script).** - `references/unified-cli.md` — background on the prerelease CLI and the client-config TOML: `login`/`whoami`, `region list`, `namespace create`, `apikey create-for-me`, file locations, and the auth-override gotcha. The script encodes these; read the reference when a flag drifts and you need to update the script. - `references/sdk-cloud.md` — per-SDK table: repo + cloud branch, task-queue name, how each connects (`cloud-setup` profile), and worker/starter run commands. No connection edits — the branch is pre-wired. -- `references/failure-handling.md` — the `error_code` → remediation map. Read it **only when a subcommand returns `status=error`** (progressive disclosure — the happy path never opens it); the Failure Handling section above is a one-line pointer to it, and the Steps spine's On-error column is the index. +- `references/failure-handling.md` — the `error_code` → remediation map. Read it **only when a subcommand returns `status=error`** (progressive disclosure — the happy path never opens it); the Failure Handling section above is a one-line pointer to it, and the Steps spine's On-error column is the index. \ No newline at end of file diff --git a/skills/temporal-developer/SKILL.md b/skills/temporal-developer/SKILL.md index 50a1603..a770333 100644 --- a/skills/temporal-developer/SKILL.md +++ b/skills/temporal-developer/SKILL.md @@ -1,7 +1,15 @@ --- name: temporal-developer -description: Develop, debug, and manage Temporal applications across Python, TypeScript, Go, Java, .NET, Ruby, and Rust. Use when the user is building workflows, activities, or workers with a Temporal SDK, debugging issues like non-determinism errors, stuck workflows, or activity retries, using Temporal CLI, Temporal Server, or Temporal Cloud, or working with durable execution concepts like signals, queries, heartbeats, versioning, continue-as-new, child workflows, or saga patterns. Also use when the user mentions "run a Temporal workflow from the CLI", "start a dev server", "run temporal server start-dev", "temporal workflow start", "temporal workflow execute", "temporal workflow signal", "temporal workflow query", "temporal workflow update". -version: 0.6.0 +description: Develop, debug, and manage Temporal applications across Python, TypeScript, + Go, Java, .NET, Ruby, and Rust. Use when the user is building workflows, activities, + or workers with a Temporal SDK, debugging issues like non-determinism errors, stuck + workflows, or activity retries, using Temporal CLI, Temporal Server, or Temporal + Cloud, or working with durable execution concepts like signals, queries, heartbeats, + versioning, continue-as-new, child workflows, or saga patterns. Also use when the + user mentions "run a Temporal workflow from the CLI", "start a dev server", "run + temporal server start-dev", "temporal workflow start", "temporal workflow execute", + "temporal workflow signal", "temporal workflow query", "temporal workflow update". +version: 0.6.1 --- # Skill: temporal-developer @@ -103,4 +111,4 @@ For Temporal plugins and integrations with third-party frameworks and SDKs (Spri ### Reporting Issues in This Skill -If you (the AI) find this skill's explanations are unclear, misleading, or missing important information—or if Temporal concepts are proving unexpectedly difficult to work with—draft a GitHub issue body describing the problem encountered and what would have helped, then ask the user to file it at https://github.com/temporalio/skill-temporal-developer/issues/new. Do not file the issue autonomously. +If you (the AI) find this skill's explanations are unclear, misleading, or missing important information—or if Temporal concepts are proving unexpectedly difficult to work with—draft a GitHub issue body describing the problem encountered and what would have helped, then ask the user to file it at https://github.com/temporalio/skill-temporal-developer/issues/new. Do not file the issue autonomously. \ No newline at end of file diff --git a/skills/temporal-developer/references/core/error-reference.md b/skills/temporal-developer/references/core/error-reference.md index 29a40b7..5c108c9 100644 --- a/skills/temporal-developer/references/core/error-reference.md +++ b/skills/temporal-developer/references/core/error-reference.md @@ -5,7 +5,7 @@ | **Non-determinism** | TMPRL1100 | `WorkflowTaskFailed` in history | Replay doesn't match history | Analyze error first. **If accidental**: fix code to match history → restart worker. **If intentional v2 change**: terminate → start fresh workflow. | https://github.com/temporalio/rules/blob/main/rules/TMPRL1100.md | | **Deadlock** | TMPRL1101 | `WorkflowTaskFailed` in history, worker logs | Workflow blocked too long (deadlock detected) | Remove blocking operations from workflow code (no I/O, no sleep, no threading locks). Use Temporal primitives instead. | https://github.com/temporalio/rules/blob/main/rules/TMPRL1101.md | | **Unfinished handlers** | TMPRL1102 | `WorkflowTaskFailed` in history | Workflow completed while update/signal handlers still running | Ensure all handlers complete before workflow finishes. Use `workflow.wait_condition()` to wait for handler completion. | https://github.com/temporalio/rules/blob/main/rules/TMPRL1102.md | -| **Payload overflow** | TMPRL1103 | `WorkflowTaskFailed` or `ActivityTaskFailed` in history | Payload size limit exceeded (default 2MB) | Reduce payload size. Use external storage (S3, database) for large data and pass references instead. | https://github.com/temporalio/rules/blob/main/rules/TMPRL1103.md | +| **Payload overflow** | TMPRL1103 | `WorkflowTaskFailed` or `ActivityTaskFailed` in history | Payload size limit exceeded (default 2MB) | Reduce payload size. Use the SDK's built-in External Storage where available (see `references/{your_language}/external-storage.md`; Go, Python, and TypeScript), or pass references to external storage yourself (see the Large Data Handling pattern in `references/core/patterns.md`). | https://github.com/temporalio/rules/blob/main/rules/TMPRL1103.md | | **Workflow code bug** | | `WorkflowTaskFailed` in history | Bug in workflow logic | Fix code → Restart worker → Workflow auto-resumes | | | **Missing workflow** | | Worker logs | Workflow not registered | Add to worker.py → Restart worker | | | **Missing activity** | | Worker logs | Activity not registered | Add to worker.py → Restart worker | | diff --git a/skills/temporal-developer/references/core/gotchas.md b/skills/temporal-developer/references/core/gotchas.md index 372caed..8b6568f 100644 --- a/skills/temporal-developer/references/core/gotchas.md +++ b/skills/temporal-developer/references/core/gotchas.md @@ -235,3 +235,5 @@ When resetting a workflow with `temporal workflow reset`, `--reapply-type` contr - Workflow history growing unboundedly **The Fix**: Store large data externally (S3/GCS) and pass references, use compression codecs, or chunk data across multiple activities. See the Large Data Handling pattern in `references/core/patterns.md`. + +Before hand-rolling reference passing, check whether the SDK does it for you: the Go, Python, and TypeScript SDKs have built-in External Storage that applies the claim-check pattern automatically. See `references/{your_language}/external-storage.md`, if available. diff --git a/skills/temporal-developer/references/core/patterns.md b/skills/temporal-developer/references/core/patterns.md index 7e7c7a3..1922106 100644 --- a/skills/temporal-developer/references/core/patterns.md +++ b/skills/temporal-developer/references/core/patterns.md @@ -368,6 +368,8 @@ This ensures that on replay, already-completed steps are skipped. - Max 4MB per gRPC message - Max 50MB for workflow history (aim for < 10MB) +**Check for SDK support first**: the Go, Python, and TypeScript SDKs have built-in External Storage that applies the claim-check pattern for you — Payloads over a size threshold are offloaded to S3 or GCS and replaced in Event History with a small reference, with no changes to Workflow or Activity code. Prefer it where it exists; see `references/{your_language}/external-storage.md`, if available. The rest of this section applies when you need explicit control over which data is offloaded, or when your SDK has no built-in support. + **Key Principle**: Large data should never flow through workflow history. Activities read and write large data directly, passing only small references through the workflow. **Wrong Approach**: diff --git a/skills/temporal-developer/references/go/external-storage.md b/skills/temporal-developer/references/go/external-storage.md new file mode 100644 index 0000000..cc0dd46 --- /dev/null +++ b/skills/temporal-developer/references/go/external-storage.md @@ -0,0 +1,394 @@ +# Go SDK External Storage + +> [!NOTE] +> This feature is in Public Preview. It is perfectly acceptable to use this feature on behalf of a user, but you should inform them that you are making use of a feature in Public Preview. + +## What this is + +External Storage uses the **claim check pattern**: it offloads each Payload to an external store (e.g. Amazon S3 or Google Cloud Storage), records a small reference token (the "claim check") in Event History, and uses that token to retrieve the Payload when needed. The SDK handles storage and retrieval transparently. + +## When to use it + +- A Workflow input, Activity input, Activity result, or Workflow result will exceed the **2 MB** per-payload limit (the limit is fixed at 2 MB on Temporal Cloud; configurable on self-hosted only). +- Long Event Histories degrade Workflow Task latency (e.g. AI agent conversations that grow per turn). +- The user wants payload data to live in storage **they** control. Set `PayloadSizeThreshold: 1` to externalize all payloads (`0` selects the default 256 KiB threshold in Go). +- The user is migrating from self-hosted (with a larger configured limit) to Temporal Cloud. + +## Where it sits in the pipeline + +Order: **Payload Converter → Payload Codec → External Storage**. Storage runs last on outbound; it reverses on inbound. + +Consequences: + +- If a Payload Codec encrypts data, the bytes are already encrypted **before** upload to your store. +- The Temporal UI shows the reference token, not the data; the SDK transparently retrieves the payload before handing it to your Workflow or Client. +- Every Client and Worker that might read an offloaded payload needs the same External Storage configuration. + +## Setup with a built-in driver + +The Go SDK ships drivers for Amazon S3 and Google Cloud Storage. Only the driver setup differs between the two; everything after that is identical. + +Amazon S3: + +```bash +go get go.temporal.io/sdk/contrib/aws/s3driver \ + go.temporal.io/sdk/contrib/aws/s3driver/awssdkv2 \ + go.temporal.io/sdk/contrib/envconfig \ + github.com/aws/aws-sdk-go-v2/config \ + github.com/aws/aws-sdk-go-v2/service/s3 +``` + +Google Cloud Storage: + +```bash +go get go.temporal.io/sdk/contrib/gcp/gcsdriver \ + go.temporal.io/sdk/contrib/gcp/gcsdriver/gcssdk \ + go.temporal.io/sdk/contrib/envconfig \ + cloud.google.com/go/storage +``` + +### Amazon S3 driver + +```go +import ( + "context" + "log" + + "github.com/aws/aws-sdk-go-v2/config" + "github.com/aws/aws-sdk-go-v2/service/s3" + "go.temporal.io/sdk/contrib/aws/s3driver" + "go.temporal.io/sdk/contrib/aws/s3driver/awssdkv2" +) + +cfg, err := config.LoadDefaultConfig(context.Background(), + config.WithRegion("us-east-2"), +) +if err != nil { + log.Fatalf("load AWS config: %v", err) +} + +driver, err := s3driver.NewDriver(s3driver.Options{ + Client: awssdkv2.NewClient(s3.NewFromConfig(cfg)), + Bucket: s3driver.StaticBucket("my-temporal-payloads"), +}) +if err != nil { + log.Fatalf("create S3 driver: %v", err) +} +``` + +The AWS SDK reads standard credentials from the environment (env vars, IAM role, or AWS config file). + +### Google Cloud Storage driver + +```go +import ( + "context" + "log" + + "cloud.google.com/go/storage" + "go.temporal.io/sdk/contrib/gcp/gcsdriver" + "go.temporal.io/sdk/contrib/gcp/gcsdriver/gcssdk" +) + +gcsClient, err := storage.NewClient(context.Background()) +if err != nil { + log.Fatalf("create GCS client: %v", err) +} + +driver, err := gcsdriver.NewDriver(gcsdriver.Options{ + Client: gcssdk.NewClient(gcsClient), + Bucket: gcsdriver.StaticBucket("my-temporal-payloads"), +}) +if err != nil { + log.Fatalf("create GCS driver: %v", err) +} +``` + +The Google Cloud SDK reads Application Default Credentials. + +For either driver, pass a `BucketFunc` as `Bucket` instead of `StaticBucket` to route payloads at runtime. The function receives the store context and the payload and returns a bucket name. + +### Configure the Client and Worker + +```go +import ( + "log" + + "go.temporal.io/sdk/client" + "go.temporal.io/sdk/contrib/envconfig" + "go.temporal.io/sdk/converter" + "go.temporal.io/sdk/worker" +) + +opts := envconfig.MustLoadDefaultClientOptions() +opts.ExternalStorage = converter.ExternalStorage{ + Drivers: []converter.StorageDriver{driver}, +} + +c, err := client.Dial(opts) +if err != nil { + log.Fatalf("connect to Temporal: %v", err) +} +defer c.Close() + +w := worker.New(c, "my-task-queue", worker.Options{}) +``` + +A Worker inherits External Storage from the Client it is created with. When your Workers run in their own process, repeat this setup there — a Client or Worker without the matching driver cannot resolve a reference. + +Workflows and Activities running on the Worker use the driver automatically — no changes to business logic. + +## Built-in driver behavior + +Both the S3 and GCS drivers: + +- Upload and download payloads **concurrently**. Multiple offloaded payloads in a single Workflow Task are stored or retrieved in parallel, not sequentially. +- Address objects by a SHA-256 hash of the contents, scoped by Namespace, Workflow ID, and Run ID, and verify that hash on retrieval. One Run passing the same payload to several Activities uploads it once; a different Run, Workflow, or Namespace stores its own copy, so storage scales with the number of Runs rather than with how often a Run passes a payload around. +- Reject any single payload larger than `MaxPayloadSize`, which defaults to **50 MiB**. `PayloadSizeThreshold` does not raise this ceiling — set `MaxPayloadSize` for the largest payload the application must support, and size the backing store to match. +- Include diagnostic metadata, such as the AWS region, in storage errors. + +## Payload size threshold + +- Default: **256 KiB**. +- Set `PayloadSizeThreshold: 1` to externalize **all** payloads regardless of size. +- `PayloadSizeThreshold: 0` is **interpreted as the default (256 KiB)** — it does **not** mean "externalize everything". +- The size compared against the threshold is that of the serialized Payload, including its metadata, not just your data. + +```go +opts := envconfig.MustLoadDefaultClientOptions() +opts.ExternalStorage = converter.ExternalStorage{ + Drivers: []converter.StorageDriver{driver}, + PayloadSizeThreshold: 1, +} + +c, err := client.Dial(opts) +``` + +## Multiple drivers and migration + +When you register more than one driver, you **must** supply a `DriverSelector` implementing `StorageDriverSelector`. The selector chooses which driver stores each payload. Unselected drivers remain available for **retrieval** — this is how you migrate between storage backends without losing access to existing claims. + +- Return `nil` from the selector to keep a specific payload inline in Event History. +- Every registered driver must have a distinct `Name()`; duplicates are rejected when the Client or Worker is constructed. `s3driver` defaults its name to `"aws.s3driver"` and `gcsdriver` to `"gcp.gcsdriver"`, so registering two drivers of the same kind requires setting `DriverName` on at least one. + +```go +import ( + commonpb "go.temporal.io/api/common/v1" + + "go.temporal.io/sdk/converter" +) + +type PreferredSelector struct { + preferred converter.StorageDriver +} + +func (s *PreferredSelector) SelectDriver( + ctx converter.StorageDriverStoreContext, + payload *commonpb.Payload, +) (converter.StorageDriver, error) { + return s.preferred, nil +} + +func MultipleDriversSetup(preferredDriver, legacyDriver converter.StorageDriver) converter.ExternalStorage { + return converter.ExternalStorage{ + Drivers: []converter.StorageDriver{preferredDriver, legacyDriver}, + DriverSelector: &PreferredSelector{preferred: preferredDriver}, + } +} +``` + +Useful routing patterns include driver migration, hot/cold storage tiers, per-tenant storage, and selecting S3 or GCS based on the runtime environment. + +## Custom storage driver + +Implement `converter.StorageDriver` with **four** methods: + +- `Name() string` — unique identifier for **this driver instance**, stored in the claim reference so the SDK can route retrieval. Renaming after payloads are stored **breaks retrieval**. +- `Type() string` — identifier for the driver **implementation**, same across all instances regardless of configuration (e.g. `"aws.s3driver"`, `"local-disk"`). It is reported in Worker heartbeats. +- `Store(ctx, payloads) ([]StorageDriverClaim, error)` — upload each Payload protobuf and return one claim per payload, in the same order. A claim is a `map[string]string` the driver uses to locate the payload later. +- `Retrieve(ctx, claims) ([]*commonpb.Payload, error)` — download bytes using claim data and reconstruct each Payload, one per claim, in the same order. + +Inside `Store()`, marshal each payload with `proto.Marshal(payload)`; in `Retrieve()`, reconstruct with `proto.Unmarshal(data, payload)`. The application data has already been serialized by the Payload Converter and Payload Codec before it reaches the driver. + +`ctx.Context` carries the context of the operation that triggered the driver call — pass it to your storage calls so cancellation and deadlines propagate, and so sibling operations stop after the first failure. + +`ctx.Target` provides identity information. Type-switch over `StorageDriverWorkflowInfo` and `StorageDriverActivityInfo` to access the namespace / Workflow ID / Activity ID, and use it to scope storage keys. Hash or encode identifiers before using them as path segments because identifiers can contain path separators or traversal sequences. `StorageDriverActivityInfo` is only used for standalone (non-workflow-bound) Activities; Activities started by a Workflow get `StorageDriverWorkflowInfo`. + +Validate claim data in `Retrieve()` as untrusted input. A driver that resolves a filesystem path, object key, or URL straight out of the claim will follow whatever a hand-crafted reference payload puts there, so re-check that the resolved location stays inside the store the driver owns. + +Worked example — local-disk driver (development/testing only): + +```go +import ( + "crypto/sha256" + "encoding/hex" + "fmt" + "os" + "path/filepath" + "strings" + + commonpb "go.temporal.io/api/common/v1" + "google.golang.org/protobuf/proto" + + "go.temporal.io/sdk/converter" +) + +type LocalDiskStorageDriver struct { + storeDir string +} + +func safePathSegment(value string) string { + sum := sha256.Sum256([]byte(value)) + return hex.EncodeToString(sum[:]) +} + +func NewLocalDiskStorageDriver(storeDir string) converter.StorageDriver { + return &LocalDiskStorageDriver{storeDir: storeDir} +} + +// resolvePath rejects claim data that points outside the store directory. +func (d *LocalDiskStorageDriver) resolvePath(claimPath string) (string, error) { + root, err := filepath.Abs(d.storeDir) + if err != nil { + return "", fmt.Errorf("resolve store directory: %w", err) + } + resolved, err := filepath.Abs(claimPath) + if err != nil { + return "", fmt.Errorf("resolve claim path: %w", err) + } + if resolved != root && !strings.HasPrefix(resolved, root+string(os.PathSeparator)) { + return "", fmt.Errorf("claim path %q escapes the store directory", claimPath) + } + return resolved, nil +} + +func (d *LocalDiskStorageDriver) Name() string { return "my-local-disk" } +func (d *LocalDiskStorageDriver) Type() string { return "local-disk" } + +func (d *LocalDiskStorageDriver) Store( + ctx converter.StorageDriverStoreContext, + payloads []*commonpb.Payload, +) ([]converter.StorageDriverClaim, error) { + dir := d.storeDir + switch info := ctx.Target.(type) { + case converter.StorageDriverWorkflowInfo: + if info.WorkflowID != "" { + dir = filepath.Join( + d.storeDir, + safePathSegment(info.Namespace), + safePathSegment(info.WorkflowID), + ) + } + case converter.StorageDriverActivityInfo: + if info.ActivityID != "" { + dir = filepath.Join( + d.storeDir, + safePathSegment(info.Namespace), + safePathSegment(info.ActivityID), + ) + } + } + if err := os.MkdirAll(dir, 0o755); err != nil { + return nil, fmt.Errorf("create store directory: %w", err) + } + + claims := make([]converter.StorageDriverClaim, len(payloads)) + for i, payload := range payloads { + data, err := proto.Marshal(payload) + if err != nil { + return nil, fmt.Errorf("marshal payload: %w", err) + } + sum := sha256.Sum256(data) + key := hex.EncodeToString(sum[:]) + ".bin" + filePath := filepath.Join(dir, key) + if err := os.WriteFile(filePath, data, 0o644); err != nil { + return nil, fmt.Errorf("write payload: %w", err) + } + claims[i] = converter.StorageDriverClaim{ + ClaimData: map[string]string{"path": filePath}, + } + } + return claims, nil +} + +func (d *LocalDiskStorageDriver) Retrieve( + ctx converter.StorageDriverRetrieveContext, + claims []converter.StorageDriverClaim, +) ([]*commonpb.Payload, error) { + payloads := make([]*commonpb.Payload, len(claims)) + for i, claim := range claims { + filePath, err := d.resolvePath(claim.ClaimData["path"]) + if err != nil { + return nil, err + } + data, err := os.ReadFile(filePath) + if err != nil { + return nil, fmt.Errorf("read payload: %w", err) + } + payload := &commonpb.Payload{} + if err := proto.Unmarshal(data, payload); err != nil { + return nil, fmt.Errorf("unmarshal payload: %w", err) + } + payloads[i] = payload + } + return payloads, nil +} +``` + +You can package a custom driver as a [plugin](https://docs.temporal.io/develop/plugins-guide) for reuse across services. + +## Multi-region durability with Amazon S3 + +For regional-failure tolerance, configure S3 Cross-Region Replication (CRR) and an S3 Multi-Region Access Point (MRAP), then pass the MRAP ARN as the bucket: + +```go +driver, err := s3driver.NewDriver(s3driver.Options{ + Client: awssdkv2.NewClient(s3.NewFromConfig(cfg)), + Bucket: s3driver.StaticBucket("arn:aws:s3::123456789012:accesspoint/mfzwi23gnjvgw.mrap"), +}) +``` + +The AWS SDK for Go v2 uses SigV4A signing automatically when the bucket value is an MRAP ARN, so no additional client configuration is required. + +Cross-region replication is eventually consistent. Activities reading newly written Payloads from another region need an appropriate Retry Policy. Replication, versioning, and Replication Time Control can add significant cost. + +## Codec Server with External Storage + +When Workers and Clients use External Storage, Event History contains reference tokens — not payload data. For the Web UI and CLI to display decoded payloads, the Codec Server must download from external storage **and** decode through the Payload Codec in the correct order. + +Build the Codec Server with `NewPayloadHTTPHandler` and `PayloadHTTPHandlerOptions`. Pass it your storage drivers, your pre-storage codecs (the Payload Codecs your Workers use), and any post-storage codecs (applied by a proxy after external storage). + +When configured with storage drivers, the handler exposes: + +- **`/download`** — retrieves payload data from external storage and decodes it through the Payload Codec. The Web UI calls this when a user clicks to view the full payload behind a reference. +- **`/decode`** — decodes encoded payloads and, by default, retrieves storage references inline. Pass `?preserveStorageRefs=true` to return storage references as-is without retrieval. +- **`/encode`** — applies the Payload Codec, then uploads payloads exceeding the threshold and replaces them with reference tokens. + +**Don't use `NewPayloadHTTPHandler` as a remote Data Converter or remote codec target for your Workers** — it runs the full encode-store-encode and decode-retrieve-decode pipeline. For remote codecs use `NewPayloadCodecHTTPHandler` separately. If you need both, run both handlers, configured with the same codecs. + +The [Go External Storage sample](https://github.com/temporalio/samples-go/tree/main/external-storage) is a working end-to-end setup to copy from: a Worker with an S3 driver behind a zlib Payload Codec, a Codec Server built on `NewPayloadHTTPHandler` (`codec-server/main.go`), and a mock S3 service so it runs locally without an AWS account. + +## Lifecycle and failure handling + +Temporal does **not** auto-delete payloads from your store. Configure a TTL on your bucket: + +``` +TTL > Maximum Workflow Run Timeout + Namespace Retention Period +``` + +Example: Run Timeout 14 days + Namespace retention 30 days → set TTL to at least 44 days. + +For Workflows with no finite Run Timeout, there is no safe finite TTL. Use Continue-as-New so the new run uploads fresh payloads and the old run's payloads only need to survive its retention period. + +The SDK does not retry a failed `Store` or `Retrieve` call within the same Task attempt. The failure fails the current Workflow Task or Activity Task attempt; Temporal then retries the Task as a whole, and the new attempt retries the storage operation along with it. For Activities, the Retry Policy controls the timing. Storage operations should therefore be idempotent — content-addressable keys are one way to get that. + +## Anti-patterns + +- **Don't change `Name()` after payloads have been stored.** The name is embedded in the claim reference; renaming breaks retrieval of existing claims. +- **Don't use `PayloadSizeThreshold: 0` to mean "externalize all".** `0` is interpreted as the default (256 KiB). Use `PayloadSizeThreshold: 1`. +- **Don't register multiple drivers without a `DriverSelector`.** The selector is required when there are multiple drivers. +- **Don't register duplicate driver names.** Two same-kind drivers share a default name; set `DriverName` on at least one. +- **Don't omit External Storage configuration from a Client or Worker that may retrieve offloaded data.** It cannot resolve the reference without the matching driver. +- **Don't assume the 2 MB Temporal limit is the driver's maximum.** The S3 and GCS drivers reject payloads above `MaxPayloadSize`, which defaults to 50 MiB. +- **Don't point a Worker's remote codec at `NewPayloadHTTPHandler`.** Use `NewPayloadCodecHTTPHandler` for remote codec endpoints. +- **Don't omit a TTL on the bucket.** Payloads are orphaned otherwise; orphaned objects can also remain if a request fails after upload. diff --git a/skills/temporal-developer/references/go/go.md b/skills/temporal-developer/references/go/go.md index 3e259de..4fe4c6b 100644 --- a/skills/temporal-developer/references/go/go.md +++ b/skills/temporal-developer/references/go/go.md @@ -252,6 +252,7 @@ See `references/go/testing.md` for info on writing tests. - **`references/go/testing.md`** - TestWorkflowEnvironment, time-skipping, activity mocking - **`references/go/advanced-features.md`** - Schedules, worker tuning, and more - **`references/go/data-handling.md`** - Data converters, payload codecs, encryption +- **`references/go/external-storage.md`** - Claim-check pattern for large payloads (S3 and GCS drivers, custom drivers, codec-server handling, multi-region durability) - **`references/go/versioning.md`** - Patching API (`workflow.GetVersion`), Worker Versioning - **`references/go/determinism-protection.md`** - Information on **`workflowcheck`** tool to help statically check for determinism issues. - **`references/go/standalone-activities.md`** - Standalone Activities (Public Preview): run an Activity directly from a Client without a Workflow; see also `references/core/standalone-activities.md` for cross-SDK concepts. diff --git a/skills/temporal-developer/references/python/external-storage.md b/skills/temporal-developer/references/python/external-storage.md new file mode 100644 index 0000000..20866fb --- /dev/null +++ b/skills/temporal-developer/references/python/external-storage.md @@ -0,0 +1,299 @@ +# Python SDK External Storage + +> [!NOTE] +> This feature is in Public Preview. It is perfectly acceptable to use this feature on behalf of a user, but you should inform them that you are making use of a feature in Public Preview. + +## What this is + +External Storage uses the **claim check pattern**: it offloads each Payload to an external store (e.g. Amazon S3), records a small reference token (the "claim check") in Event History, and uses that token to retrieve the Payload when needed. The SDK handles storage and retrieval transparently. + +## When to use it + +- A Workflow input, Activity input, Activity result, or Workflow result will exceed the **2 MB** per-payload limit (fixed at 2 MB on Temporal Cloud; configurable on self-hosted only). +- Long Event Histories degrade Workflow Task latency (e.g. AI agent conversations growing per turn). +- The user wants payload data to live in storage **they** control. Set `payload_size_threshold=0` to externalize all payloads. +- The user is migrating from self-hosted (with a larger configured limit) to Temporal Cloud. + +## Where it sits in the pipeline + +Order: **Payload Converter → Payload Codec → External Storage**. Storage runs last on outbound; it reverses on inbound. + +Consequences: + +- If a Payload Codec encrypts data, the bytes are already encrypted **before** upload. +- The Temporal UI displays the reference token, not the data; the SDK retrieves the payload transparently before handing it to your Workflow or Client. +- Every Client and Worker that might read an offloaded payload needs the same External Storage configuration. + +## Setup with the built-in S3 driver + +The Python SDK ships an Amazon S3 driver (there is no built-in GCS driver — use a custom driver for other backends). Install the `aioboto3` extra: + +```bash +python -m pip install "temporalio[aioboto3]" +``` + +Create the driver, attach it to a `DataConverter`, and pass the converter to `Client.connect`. A Worker inherits the Data Converter from the Client it is created with — `Worker` takes no `data_converter` argument of its own: + +```python +import asyncio +import dataclasses + +import aioboto3 +from temporalio.client import Client +from temporalio.contrib.aws.s3driver import S3StorageDriver +from temporalio.contrib.aws.s3driver.aioboto3 import new_aioboto3_client +from temporalio.converter import DataConverter, ExternalStorage +from temporalio.envconfig import ClientConfig +from temporalio.worker import Worker + +from activities.greet import greet +from workflows.greeting import GreetingWorkflow + + +async def main() -> None: + session = aioboto3.Session(region_name="us-east-2") + async with session.client("s3") as s3_client: + driver = S3StorageDriver( + client=new_aioboto3_client(s3_client), + bucket="my-temporal-payloads", + ) + + data_converter = dataclasses.replace( + DataConverter.default, + external_storage=ExternalStorage(drivers=[driver]), + ) + + connect_config = ClientConfig.load_client_connect_config() + connect_config.setdefault("target_host", "localhost:7233") + client = await Client.connect(**connect_config, data_converter=data_converter) + + worker = Worker( + client, + task_queue="my-task-queue", + workflows=[GreetingWorkflow], + activities=[greet], + ) + await worker.run() + + +if __name__ == "__main__": + asyncio.run(main()) +``` + +`ClientConfig` for connection settings comes from `temporalio.envconfig`, not `temporalio.client`. The S3 driver uses standard AWS credentials from the environment (env vars, IAM role, or AWS config file); pass `profile_name=` to `aioboto3.Session` to select a named profile. Keep the `async with session.client("s3")` block open for as long as the Worker runs — the driver uses that client for every upload and download. + +Workflows and Activities on the Worker use the driver automatically — no business-logic changes. + +## Built-in driver behavior + +The S3 driver: + +- Uploads and downloads payloads **concurrently**. Multiple offloaded payloads in a single Workflow Task are stored or retrieved in parallel, not sequentially. +- Addresses objects by a SHA-256 hash of their contents, segmented by Namespace and Workflow/Activity identifiers, and validates payload integrity on retrieval. +- Rejects any single payload larger than `max_payload_size`, which defaults to **50 MiB**. `payload_size_threshold` does not raise this ceiling — set `max_payload_size` for the largest payload the application must support, and size the backing store to match. +- Includes diagnostic metadata, such as the AWS region, in error messages. + +## Payload size threshold + +- Default: **256 KiB**. +- Set `payload_size_threshold=0` to externalize **all** payloads regardless of size. +- Payloads whose serialized size is **greater than or equal to** the threshold are eligible; smaller ones stay inline. The measured size includes Payload metadata, not just your data. + +```python +data_converter = dataclasses.replace( + DataConverter.default, + external_storage=ExternalStorage( + drivers=[driver], + payload_size_threshold=0, + ), +) +``` + +## Multiple drivers and migration + +When you register more than one driver, you **must** supply a `driver_selector` function. The selector chooses which driver stores each payload. Unselected drivers remain available for **retrieval** — this is how you migrate between storage backends without losing access to existing claims. + +- Return `None` from the selector to keep a specific payload inline in Event History. +- Every registered driver must have a distinct name; duplicates raise `ValueError` at construction. `S3StorageDriver` defaults its name to `"aws.s3driver"`, so registering two S3 drivers requires passing `driver_name=` to at least one. + +```python +preferred_driver = S3StorageDriver( + client=new_aioboto3_client(s3_client), + bucket="my-bucket", + driver_name="s3-primary", +) +legacy_driver = LegacyStorageDriver() + +ExternalStorage( + drivers=[preferred_driver, legacy_driver], + driver_selector=lambda context, payload: preferred_driver, +) +``` + +Useful routing patterns include driver migration, hot/cold storage tiers, and per-tenant storage. + +## Custom storage driver + +Extend `StorageDriver` and implement **three** methods: + +- `name() -> str` — unique identifier for the driver, stored in the claim reference so the SDK can route retrieval. Renaming after payloads are stored **breaks retrieval**. +- `async store(context, payloads) -> list[StorageDriverClaim]` — upload each Payload and return one claim per payload, in the same order. A claim is a `dict[str, str]` the driver uses to locate the payload later. +- `async retrieve(context, claims) -> list[Payload]` — download bytes using claim data and reconstruct each Payload, one per claim, in the same order. + +`type() -> str` is optional and defaults to the class name. Override it with a stable identifier shared by every instance of the implementation (e.g. `"aws.s3driver"`) so the driver reports the same type as its equivalents in other languages. + +Inside `store()`, serialize each payload with `payload.SerializeToString()`; in `retrieve()`, reconstruct with `payload.ParseFromString(data)`. The application data has already been serialized by the Payload Converter and Payload Codec before reaching the driver. + +`context.target` provides identity information (namespace, Workflow ID, or Activity ID). Check the target type with `isinstance(target, StorageDriverWorkflowInfo)`; the Workflow info exposes `target.namespace` and `target.id`. Use this to scope storage keys per Workflow, but hash or encode identifiers before using them as path segments because identifiers can contain path separators or traversal sequences. Within that scope, content-addressable keys (such as a SHA-256 hash of the payload bytes) deduplicate identical payloads and make retries idempotent. + +Treat claim data in `retrieve()` as untrusted input. A driver that resolves a filesystem path, object key, or URL straight out of the claim will follow whatever a hand-crafted reference payload puts there, so re-check that the resolved location stays inside the store the driver owns. + +Worked example — local-disk driver (development/testing only): + +```python +import hashlib +import os +from typing import Sequence + +from temporalio.api.common.v1 import Payload +from temporalio.converter import ( + StorageDriver, + StorageDriverClaim, + StorageDriverRetrieveContext, + StorageDriverStoreContext, + StorageDriverWorkflowInfo, +) + + +def safe_path_segment(value: str) -> str: + return hashlib.sha256(value.encode("utf-8")).hexdigest() + + +class LocalDiskStorageDriver(StorageDriver): + def __init__(self, store_dir: str = "/tmp/temporal-payload-store") -> None: + self._store_dir = store_dir + + def _resolve_path(self, claim_path: str) -> str: + """Reject claim data that points outside the store directory.""" + root = os.path.realpath(self._store_dir) + resolved = os.path.realpath(claim_path) + if resolved != root and not resolved.startswith(root + os.sep): + raise ValueError(f"claim path {claim_path!r} escapes the store directory") + return resolved + + def name(self) -> str: + return "local-disk" + + def type(self) -> str: + return "local-disk" + + async def store( + self, + context: StorageDriverStoreContext, + payloads: Sequence[Payload], + ) -> list[StorageDriverClaim]: + os.makedirs(self._store_dir, exist_ok=True) + + prefix = self._store_dir + target = context.target + if isinstance(target, StorageDriverWorkflowInfo) and target.id: + prefix = os.path.join( + self._store_dir, + safe_path_segment(target.namespace), + safe_path_segment(target.id), + ) + os.makedirs(prefix, exist_ok=True) + + claims = [] + for payload in payloads: + data = payload.SerializeToString() + key = f"{hashlib.sha256(data).hexdigest()}.bin" + file_path = os.path.join(prefix, key) + with open(file_path, "wb") as f: + f.write(data) + claims.append(StorageDriverClaim(claim_data={"path": file_path})) + return claims + + async def retrieve( + self, + context: StorageDriverRetrieveContext, + claims: Sequence[StorageDriverClaim], + ) -> list[Payload]: + payloads = [] + for claim in claims: + file_path = self._resolve_path(claim.claim_data["path"]) + with open(file_path, "rb") as f: + raw = f.read() + payload = Payload() + payload.ParseFromString(raw) + payloads.append(payload) + return payloads +``` + +Wire the custom driver into the Data Converter the same way as the S3 driver: + +```python +data_converter = dataclasses.replace( + DataConverter.default, + external_storage=ExternalStorage( + drivers=[LocalDiskStorageDriver()], + ), +) +``` + +You can package a custom driver as a [plugin](https://docs.temporal.io/develop/plugins-guide) for reuse across services. + +## Multi-region durability with Amazon S3 + +For regional-failure tolerance, configure S3 Cross-Region Replication (CRR) and an S3 Multi-Region Access Point (MRAP), then pass the MRAP ARN as `bucket`: + +```python +driver = S3StorageDriver( + client=new_aioboto3_client(s3_client), + bucket="arn:aws:s3::123456789012:accesspoint/mfzwi23gnjvgw.mrap", +) +``` + +`aioboto3` (via `botocore`) uses SigV4A signing automatically when the bucket value is an MRAP ARN. Make sure `botocore` is recent enough to support SigV4A. + +Cross-region replication is eventually consistent. Activities reading newly written payloads from another region need an appropriate Retry Policy. Replication, versioning, and Replication Time Control can add significant cost. + +## Codec Server with External Storage + +When Workers and Clients use External Storage, Event History contains reference tokens — not payload data. For the Web UI and CLI to show decoded payloads, the Codec Server must download from external storage **and** decode through the Payload Codec in the correct order. + +The Python SDK does not ship a storage-aware Codec Server handler — implement the routes yourself (e.g. with `aiohttp`), giving them your storage drivers, your pre-storage codecs (the Payload Codecs your Workers use), and any post-storage codecs (applied by a proxy after external storage). The [Python External Storage sample](https://github.com/temporalio/samples-python/tree/main/external_storage) has a working implementation (`payload_routes` in `handler.py`) to copy from. + +Endpoints to expose when storage drivers are configured: + +- **`/download`** — retrieves payload data from external storage and decodes it through the Payload Codec. The Web UI calls this when a user clicks to view the full payload behind a reference. +- **`/decode`** — decodes encoded payloads and, by default, retrieves storage references inline. Support `?preserveStorageRefs=true` to return storage references as-is without retrieval; the Web UI uses it to render history without downloading every blob. +- **`/encode`** — applies the Payload Codec, then uploads payloads exceeding the threshold and replaces them with reference tokens. + +**Don't point a Worker's remote codec at the storage-aware handler** — it runs the full encode-store-encode and decode-retrieve-decode pipeline. Run a separate non-storage codec HTTP handler for remote codecs, configured with the same codecs. + +## Lifecycle and failure handling + +Temporal does **not** auto-delete payloads from your store. Configure a TTL on your bucket: + +``` +TTL > Maximum Workflow Run Timeout + Namespace Retention Period +``` + +Example: Run Timeout 14 days + Namespace retention 30 days → set TTL to at least 44 days. + +For Workflows with no finite Run Timeout, there is no safe finite TTL. Use Continue-as-New so the new run uploads fresh payloads and the old run's payloads only need to survive its retention period. + +The SDK does not retry a failed `store()` or `retrieve()` call within the same Task attempt. The failure fails the current Workflow Task or Activity Task attempt; Temporal then retries the Task as a whole, and the new attempt retries the storage operation along with it. For Activities, the Retry Policy controls the timing. Storage operations should therefore be idempotent — content-addressable keys are one way to get that. + +## Anti-patterns + +- **Don't change the value returned by `name()` after payloads have been stored.** The name is embedded in the claim reference; renaming breaks retrieval of existing claims. +- **Don't use `payload_size_threshold=1` to mean "externalize all"** — use `payload_size_threshold=0`. (This sentinel differs from Go, where `0` is the default and `1` externalizes all.) +- **Don't register multiple drivers without a `driver_selector`.** The selector is required when there is more than one driver. +- **Don't register duplicate driver names.** Two `S3StorageDriver` instances share a default name; pass `driver_name=` to at least one. +- **Don't omit External Storage configuration from a Client or Worker that may retrieve offloaded data.** It cannot resolve the reference without the matching driver. +- **Don't assume the 2 MB Temporal limit is the driver's maximum.** The S3 driver rejects payloads above `max_payload_size`, which defaults to 50 MiB. +- **Don't import `ClientConfig` from `temporalio.client` for connection settings.** `load_client_connect_config()` lives on `temporalio.envconfig.ClientConfig`. +- **Don't pass the storage-aware payload HTTP handler as a Worker's remote codec target.** Use a separate non-storage codec HTTP handler for that role. +- **Don't omit a TTL on the bucket.** Payloads can be orphaned if a request fails after upload. diff --git a/skills/temporal-developer/references/python/python.md b/skills/temporal-developer/references/python/python.md index c48c6dc..e035da1 100644 --- a/skills/temporal-developer/references/python/python.md +++ b/skills/temporal-developer/references/python/python.md @@ -182,6 +182,7 @@ See `references/python/testing.md` for info on writing tests. - **`references/python/sync-vs-async.md`** - Sync vs async activities, event loop blocking, executor configuration - **`references/python/advanced-features.md`** - Schedules, worker tuning, and more - **`references/python/data-handling.md`** - Data converters, Pydantic, payload encryption +- **`references/python/external-storage.md`** - Claim-check pattern for large payloads (S3 driver, custom drivers, codec-server handling, multi-region durability) - **`references/python/versioning.md`** - Patching API, workflow type versioning, Worker Versioning - **`references/python/standalone-activities.md`** - Standalone Activities: run an Activity directly from a Client without a Workflow (Public Preview). Concept overview at `references/core/standalone-activities.md`. - **`references/python/determinism-protection.md`** - Python sandbox specifics, forbidden operations, pass-through imports diff --git a/skills/temporal-developer/references/typescript/external-storage.md b/skills/temporal-developer/references/typescript/external-storage.md new file mode 100644 index 0000000..52b8fe4 --- /dev/null +++ b/skills/temporal-developer/references/typescript/external-storage.md @@ -0,0 +1,246 @@ +# TypeScript SDK External Storage + +> [!NOTE] +> This feature is in Public Preview. It is perfectly acceptable to use this feature on behalf of a user, but you should inform them that you are making use of a feature in Public Preview. + +## What this is + +External Storage uses the **claim check pattern**: it offloads each Payload to an external store (e.g. Amazon S3 or Google Cloud Storage), records a small reference token (the "claim check") in Event History, and uses that token to retrieve the Payload when needed. The SDK handles storage and retrieval transparently. + +## When to use it + +- A Workflow input, Activity input, Activity result, or Workflow result will exceed the **2 MB** per-payload limit (fixed at 2 MB on Temporal Cloud; configurable on self-hosted only). +- Long Event Histories degrade Workflow Task latency (e.g. AI agent conversations growing per turn). +- The user wants payload data to live in storage **they** control. Set `payloadSizeThreshold: 0` to externalize all payloads. +- The user is migrating from self-hosted (with a larger configured limit) to Temporal Cloud. + +## Where it sits in the pipeline + +Order: **Payload Converter → Payload Codec → External Storage**. Storage runs last on outbound; it reverses on inbound. + +Consequences: + +- If a Payload Codec encrypts data, the bytes are already encrypted **before** upload. +- The Temporal UI displays the reference token, not the data; the SDK retrieves the payload transparently before handing it to your Workflow or Client. +- Every Client and Worker that might read an offloaded payload needs the same External Storage configuration. + +## Setup with a built-in driver + +The TypeScript SDK provides first-party drivers for Amazon S3 and Google Cloud Storage. Install one driver, its SDK adapter, and the cloud provider's SDK. Keep all `@temporalio/*` packages on the same version. + +Amazon S3: + +```bash +npm install @temporalio/external-storage-s3 \ + @temporalio/external-storage-s3-aws-sdk \ + @temporalio/envconfig \ + @aws-sdk/client-s3 +``` + +Google Cloud Storage: + +```bash +npm install @temporalio/external-storage-gcs \ + @temporalio/external-storage-gcs-google-sdk \ + @temporalio/envconfig \ + @google-cloud/storage +``` + +### Amazon S3 driver + +```typescript +import { S3Client } from '@aws-sdk/client-s3'; +import { S3StorageDriver } from '@temporalio/external-storage-s3'; +import { AwsSdkS3StorageDriverClient } from '@temporalio/external-storage-s3-aws-sdk'; + +const s3Client = new S3Client({ region: 'us-east-2' }); + +const driver = new S3StorageDriver({ + client: new AwsSdkS3StorageDriverClient(s3Client), + bucket: 'my-temporal-payloads', +}); +``` + +The AWS SDK reads standard credentials from environment variables, an IAM role, or the AWS config file. + +### Google Cloud Storage driver + +```typescript +import { Storage } from '@google-cloud/storage'; +import { GcsStorageDriver } from '@temporalio/external-storage-gcs'; +import { GoogleCloudGcsStorageDriverClient } from '@temporalio/external-storage-gcs-google-sdk'; + +const storage = new Storage(); + +const driver = new GcsStorageDriver({ + client: new GoogleCloudGcsStorageDriverClient(storage), + bucket: 'my-temporal-payloads', +}); +``` + +The Google Cloud SDK reads Application Default Credentials. + +For either driver, `bucket` can be a function instead of a string. The function receives the store context and Payload and returns a bucket name, allowing runtime routing. + +### Configure the Client and Worker + +Create one Data Converter configuration and pass it to both the Client and Worker. Load connection settings with `loadClientConnectConfig()`, and remember that `NativeConnection` carries no namespace, so the Worker needs `namespace` passed explicitly: + +```typescript +import { Client, Connection } from '@temporalio/client'; +import { ExternalStorage } from '@temporalio/common'; +import { loadClientConnectConfig } from '@temporalio/envconfig'; +import { NativeConnection, Worker } from '@temporalio/worker'; + +const dataConverter = { + externalStorage: new ExternalStorage({ drivers: [driver] }), +}; + +const config = loadClientConnectConfig(); + +const connection = await Connection.connect(config.connectionOptions); +const client = new Client({ connection, namespace: config.namespace, dataConverter }); + +const workerConnection = await NativeConnection.connect(config.connectionOptions); +const worker = await Worker.create({ + connection: workerConnection, + namespace: config.namespace, + workflowsPath: require.resolve('./workflows'), + taskQueue: 'my-task-queue', + dataConverter, +}); +``` + +External Storage runs outside the Workflow sandbox, so pass the driver object directly. Workflows and Activities use it automatically; business logic does not change. + +## Built-in driver behavior + +The S3 and GCS drivers: + +- Upload and download Payloads concurrently. +- Address objects by a SHA-256 hash of their contents, deduplicating identical Payloads. +- Verify the content hash during retrieval. +- Reject any single Payload larger than `maxPayloadSize`, which defaults to **50 MiB**. +- Include diagnostic metadata in storage errors. + +The External Storage threshold does not override `maxPayloadSize`. Configure the backing store and driver for the largest Payload the application needs to support. + +## Payload size threshold + +- Default: **256 KiB**. +- Set `payloadSizeThreshold: 0` to externalize **all** Payloads regardless of size. +- Payloads whose serialized size is **greater than or equal to** the threshold are eligible for external storage. +- The measured size includes Payload metadata after Payload Converter and Payload Codec processing, not only the raw application value. + +```typescript +const dataConverter = { + externalStorage: new ExternalStorage({ + drivers: [driver], + payloadSizeThreshold: 0, + }), +}; +``` + +## Multiple drivers and migration + +When registering more than one driver, supply a `driverSelector`. The selector chooses which driver stores each Payload. Unselected registered drivers remain available for **retrieval**, which supports migrations without losing access to existing claims. + +- Return `null` from the selector to keep a specific Payload inline in Event History. +- Every registered driver must have a distinct `name`. +- `S3StorageDriver` defaults its name to `"aws.s3driver"`; when registering two S3 drivers, set `driverName` on at least one. + +```typescript +const preferredDriver = new S3StorageDriver({ + client: new AwsSdkS3StorageDriverClient(s3Client), + bucket: 'my-bucket', +}); +const legacyDriver = new LegacyStorageDriver(); + +const externalStorage = new ExternalStorage({ + drivers: [preferredDriver, legacyDriver], + driverSelector: () => preferredDriver, +}); +``` + +Useful routing patterns include driver migration, hot/cold storage tiers, per-tenant storage, and selecting S3 or GCS based on the runtime environment. + +## Custom storage driver + +Implement the `StorageDriver` interface with two readonly properties and two methods: + +- `name: string` — unique identifier for **this driver instance**, stored in the reference so the SDK can route retrieval. Changing it after Payloads are stored **breaks retrieval**. +- `type: string` — stable identifier for the driver implementation, shared by all instances of that implementation and reported in Worker heartbeats (e.g. `"aws.s3driver"`). +- `store(context, payloads): Promise` — serialize and upload each Payload, then return one claim per Payload. Each claim contains string key-value data sufficient to find the object later. +- `retrieve(context, claims): Promise` — download and reconstruct one Payload per claim, preserving input order. + +The `store()` context includes an optional `abortSignal` and `target`. The target is a discriminated union: + +- Check `target.kind` for `"workflow"` or `"activity"`. +- Read `namespace`, `id`, `runId`, and `type` to scope storage keys. + +Honor `abortSignal` in storage calls so sibling operations can be cancelled after the first failure. Content-addressable keys can make retries idempotent and deduplicate identical Payloads. + +Return exactly one claim for each Payload passed to `store()` and exactly one Payload for each claim passed to `retrieve()`. Store the complete serialized Payload protobuf: application data has already passed through the Payload Converter and Payload Codec before reaching the driver. + +## Multi-region durability with Amazon S3 + +For regional-failure tolerance, configure S3 Cross-Region Replication and an S3 Multi-Region Access Point (MRAP), then use the MRAP ARN as `bucket`. + +MRAP requests require a SigV4A signer. The AWS SDK for JavaScript does not bundle one, so install and register it at application startup: + +```bash +npm install @aws-sdk/signature-v4a +``` + +```typescript +import '@aws-sdk/signature-v4a'; +``` + +Then configure the driver with the MRAP ARN: + +```typescript +const driver = new S3StorageDriver({ + client: new AwsSdkS3StorageDriverClient(s3Client), + bucket: 'arn:aws:s3::123456789012:accesspoint/mfzwi23gnjvgw.mrap', +}); +``` + +`@aws-sdk/signature-v4-crt` is an alternative backed by the AWS Common Runtime. The AWS SDK prefers it when both signer implementations are installed. + +Cross-region replication is eventually consistent. Activities reading newly written Payloads from another region need an appropriate Retry Policy. Replication, versioning, and Replication Time Control can add significant cost. + +## Codec Server with External Storage + +When Workers and Clients use External Storage, Event History contains reference tokens — not payload data. A plain codec server that only implements `/encode` and `/decode` leaves the Web UI and CLI showing raw reference tokens. + +The TypeScript SDK does not ship a codec-server handler, so implement the routes yourself (e.g. with Express), wiring in your storage drivers, your pre-storage codecs (the Payload Codecs your Workers use), and any post-storage codecs (applied by a proxy after external storage): + +- **`/download`** — retrieves payload data from external storage and decodes it through the Payload Codec. The Web UI calls this when a user clicks to view the full payload behind a reference. +- **`/decode`** — decodes encoded payloads and, by default, retrieves storage references inline. Support `?preserveStorageRefs=true` to return storage references as-is without retrieval; the Web UI uses it to render history without downloading every blob. +- **`/encode`** — applies the Payload Codec, then uploads payloads exceeding the threshold and replaces them with reference tokens. + +**Don't point a Worker's remote codec at the storage-aware handler** — it runs the full encode-store-encode and decode-retrieve-decode pipeline. Serve remote codecs from a separate non-storage endpoint, configured with the same codecs. + +## Lifecycle and failure handling + +Temporal does **not** automatically delete Payloads from the external store. Configure a bucket lifecycle policy with: + +``` +TTL > Maximum Workflow Run Timeout + Namespace Retention Period +``` + +Example: Run Timeout 14 days + Namespace retention 30 days → set TTL to at least 44 days. + +For Workflows with no finite Run Timeout, there is no safe finite TTL. Use Continue-as-New so the new run uploads fresh Payloads and the old run's Payloads only need to survive its retention period. + +The SDK does not retry a failed `store()` or `retrieve()` call within the same Task attempt. The failure fails the current Workflow Task or Activity Task attempt; Temporal then retries the Task as a whole. Storage operations should therefore be idempotent. + +## Anti-patterns + +- **Don't change a driver's `name` after Payloads have been stored.** The name is embedded in references; changing it breaks retrieval. +- **Don't register duplicate driver names.** Give each instance a unique `name` or `driverName`. +- **Don't register multiple drivers without a `driverSelector`.** Construction fails when more than one driver is registered without one. +- **Don't omit External Storage configuration from a Client or Worker that may retrieve offloaded data.** It cannot resolve the reference without the matching driver. +- **Don't assume the 2 MB Temporal limit is the built-in driver's maximum.** The S3 and GCS drivers default `maxPayloadSize` to 50 MiB. +- **Don't point a Worker's remote codec at a storage-aware codec-server handler.** Serve remote codecs from a separate non-storage endpoint. +- **Don't omit a lifecycle policy.** Payloads are otherwise retained indefinitely, and failed requests can leave orphaned objects. diff --git a/skills/temporal-developer/references/typescript/typescript.md b/skills/temporal-developer/references/typescript/typescript.md index 1c4ff4f..fceea30 100644 --- a/skills/temporal-developer/references/typescript/typescript.md +++ b/skills/temporal-developer/references/typescript/typescript.md @@ -190,6 +190,7 @@ See `references/typescript/testing.md` for info on writing tests. - **`references/typescript/testing.md`** - TestWorkflowEnvironment, time-skipping, activity mocking - **`references/typescript/advanced-features.md`** - Schedules, worker tuning, and more - **`references/typescript/data-handling.md`** - Data converters, payload encryption, etc. +- **`references/typescript/external-storage.md`** - Claim-check pattern for large Payloads (S3 and GCS drivers, custom drivers, codec-server handling, multi-region durability) - **`references/typescript/versioning.md`** - Patching API, workflow type versioning, Worker Versioning - **`references/typescript/standalone-activities.md`** - Standalone Activities: run an Activity directly from a Client without a Workflow (Public Preview). Concept overview at `references/core/standalone-activities.md`. - **`references/typescript/determinism-protection.md`** - V8 sandbox and bundling diff --git a/skills/temporal-ops/SKILL.md b/skills/temporal-ops/SKILL.md index 23b24d5..4dcc5b9 100644 --- a/skills/temporal-ops/SKILL.md +++ b/skills/temporal-ops/SKILL.md @@ -1,8 +1,15 @@ --- name: temporal-ops -description: 'Administer and diagnose running Temporal Cloud or self-hosted Temporal Server environments via CLI (temporal, tcld) — not SDK code. Operations: namespace CRUD, Cloud capacity/APS, API-key rotation, mTLS cert rotation, workflow health, batch cancel/terminate/reset, export, search attributes, Ops API, billing, audit logs, Terraform, SAML/SCIM, migration. Diagnosis: bottom-up triage of stuck workflows, non-determinism, worker-health, task-queue problems, HA failover, payload size limits, performance bottlenecks, missed schedules. Do NOT trigger for generic TLS/gRPC errors unrelated to Temporal, writing application code (temporal-developer), or worker tuning/sizing (temporal-workertuning).' -version: 0.2.0 -disable-model-invocation: true +description: 'Administer and diagnose running Temporal Cloud or self-hosted Temporal + Server environments via CLI (temporal, tcld) — not SDK code. Operations: namespace + CRUD, Cloud capacity/APS, API-key rotation, mTLS cert rotation, workflow health, + batch cancel/terminate/reset, export, search attributes, Ops API, billing, audit + logs, Terraform, SAML/SCIM, migration. Diagnosis: bottom-up triage of stuck workflows, + non-determinism, worker-health, task-queue problems, HA failover, payload size limits, + performance bottlenecks, missed schedules. Do NOT trigger for generic TLS/gRPC errors + unrelated to Temporal, writing application code (temporal-developer), or worker + tuning/sizing (temporal-workertuning).' +version: 0.2.1 --- # Skill: temporal-ops @@ -294,16 +301,7 @@ If the layer above the fix is still failing, return to step 4 and continue walki - [schedule-missed.md](references/triage/schedule-missed.md) — Missed Schedule Actions: alerting via `temporal_cloud_v1_schedule_missed_catchup_window_count` / `schedule_missed_catchup_window`, investigation via `temporal schedule list` + `temporal schedule describe`, DescribeSchedule fields (`missedCatchupWindow`, `overlapSkipped`, `bufferDropped`), default catchup window (one year), root causes, overlap policies (6 values), backfill remediation. - [recipes.md](references/triage/recipes.md) — four end-to-end triage walkthroughs: stuck workflow at 3am, cert expired with workers offline, task-queue backlog mystery, non-determinism caught in prod. -## Feedback -### Feedback prompt +## Reporting Issues in This Skill -When you have **first** loaded this skill, output this message to the user: - -"Thank you for trying out the public preview of the Temporal Ops skill! We would love to hear your feedback - positive or negative - over in the [Community Slack](https://t.mp/slack), in the [#topic-ai channel](https://temporalio.slack.com/archives/C0818FQPYKY)." - -Do not output this message multiple times in the same conversation. - -### Reporting Issues in This Skill - -If you (the AI) find this skill's explanations are unclear, misleading, or missing important information, draft a GitHub issue body describing the problem encountered and what would have helped, then ask the user to file it at https://github.com/temporalio/skill-temporal-ops/issues/new. Do not file the issue autonomously. +If you (the AI) find this skill's explanations are unclear, misleading, or missing important information, draft a GitHub issue body describing the problem encountered and what would have helped, then ask the user to file it at https://github.com/temporalio/skill-temporal-ops/issues/new. Do not file the issue autonomously. \ No newline at end of file diff --git a/skills/temporal-serverless/SKILL.md b/skills/temporal-serverless/SKILL.md index 54ece14..3d01090 100644 --- a/skills/temporal-serverless/SKILL.md +++ b/skills/temporal-serverless/SKILL.md @@ -1,8 +1,11 @@ --- name: temporal-serverless -description: 'Deploy and operate Temporal Workers on serverless compute (AWS Lambda) driven by the Worker Controller Instance (WCI). Use when the user mentions: "serverless worker", "Temporal serverless", "Worker Controller Instance", "WCI", "deploy Temporal worker on Lambda", "Lambda packaging", "Lambda timeout", "WCI inspection", "CloudFormation Temporal".' -version: 0.6.0 -disable-model-invocation: true +description: 'Deploy and operate Temporal Workers on serverless compute (AWS Lambda) + driven by the Worker Controller Instance (WCI). Use when the user mentions: "serverless + worker", "Temporal serverless", "Worker Controller Instance", "WCI", "deploy Temporal + worker on Lambda", "Lambda packaging", "Lambda timeout", "WCI inspection", "CloudFormation + Temporal".' +version: 0.6.1 --- # Skill: temporal-serverless @@ -20,7 +23,15 @@ This skill helps users deploy and operate Temporal Workers on serverless compute Only a provider marked Supported is covered. If a request names another, say it is not supported and stop; do not adapt a supported provider's material to it. **Never let the provider be an unstated assumption:** when the request does not name one, it is confirmed in the step 1 questions, not silently defaulted. -Every supported provider's directory carries the same layout — `setup.md`, `iam.md`, `versioning.md`, `diagnostics.md`, `observability.md`, `self-hosted.md`. Paths below are written `references//…`; substitute the directory from the table. Provider-specific commands, templates, permissions, and defaults live there — this file stays at the workflow level. When a step needs concrete commands, go to the reference file named at the end of that step. +Every supported provider's directory carries the same shared layout — `setup.md`, `iam.md`, `versioning.md`, `diagnostics.md`, `observability.md`, `self-hosted.md` — plus one `sdk-.md` file for each supported SDK. Paths below are written `references//…`; substitute the directory from the table. Provider-specific commands, templates, permissions, SDK APIs, and defaults live there — this file stays at the workflow level. When a step needs concrete commands or SDK details, go to the reference file named at the end of that step. + +| SDK language | AWS Lambda reference | +|---|---| +| Go | `references/aws-lambda/sdk-go.md` | +| Python | `references/aws-lambda/sdk-python.md` | +| TypeScript | `references/aws-lambda/sdk-typescript.md` | +| Java | `references/aws-lambda/sdk-java.md` | +| .NET | `references/aws-lambda/sdk-dotnet.md` | **Public Preview is not GA.** The APIs are still evolving and may change: pin SDK and CLI versions for anything long-lived, and read the installed package's actual API surface rather than writing from memory. @@ -84,7 +95,7 @@ Where the harness has a todo list, use it *in addition to* the printed checklist **A step is complete when its verification passed — not when its command exited zero.** Several commands in this workflow exit clean having done nothing: the traffic-shifting and key-revocation commands no-op when their confirmation prompt goes unanswered, and providers return from create and update calls while the resource is still settling. Check an item off against state you read back, not against an exit code. When a step's verification fails, say which step you are on and what it is blocked on rather than moving down the list. -1. **Scope the task.** Identify the SDK language (Go, Python, or TypeScript), the deployment target (Temporal Cloud or self-hosted — self-hosted has its own server prerequisites), the compute provider, and whether this is a new setup, a configuration change, or troubleshooting. Confirm the deployment target is compatible with the chosen provider — see "A Namespace on the target cloud provider is required" under Provider-neutral principles. Ensure a Temporal client/CLI is available and authenticated to the target. Each changes the specifics. → `references/concepts.md` for what the user is building; `references//setup.md` for the compatibility and client-setup details. +1. **Scope the task.** Identify the SDK language (Go, Python, TypeScript, Java, or .NET), the deployment target (Temporal Cloud or self-hosted — self-hosted has its own server prerequisites), the compute provider, and whether this is a new setup, a configuration change, or troubleshooting. Confirm the deployment target is compatible with the chosen provider — see "A Namespace on the target cloud provider is required" under Provider-neutral principles. Ensure a Temporal client/CLI is available and authenticated to the target. Each changes the specifics. → `references/concepts.md` for what the user is building; `references//setup.md` for the compatibility and client-setup details. **Put the compute provider in that batch of questions as a confirmable default, not a free choice.** Pre-select the supported provider from the table above and carry its support status in the option's description. The user confirms rather than chooses, so it costs no extra turn, but the provider is never something they were assumed into. Skip the question only when the request already names a provider. Do not restate any of this in a paragraph before the questions; the option description is where it belongs. @@ -133,9 +144,9 @@ Where the harness has a todo list, use it *in addition to* the printed checklist **Before the first account-mutating command, list what you are about to create — with final names — and get approval.** Name the target account and region, then every resource: compute unit, execution role, infrastructure stack, log group, deployment name, and Task Queue. Say plainly that they are live and billable. This is the mirror of the inventory in step 8, and it is worth more here than there: it makes the naming prefix concrete while changing it is still free, and the deployment name, build ID, and Task Queue become expensive to change once step 3 compiles them into the Worker. Skip it only when nothing will be created — a troubleshooting or inspection task. -3. **Author the Worker.** *Install the SDK's serverless Worker package before writing any code* — it is often shipped separately from the main SDK, with its own version line, so having the base SDK installed does not mean it is importable. Then read the installed package's actual API surface and write against that; these are Public Preview APIs that drift between versions, and generating code from memory costs a build cycle. Every Workflow must declare a versioning behavior (`Pinned` or `AutoUpgrade`), per-Workflow or as a Worker-level default — code without it fails at runtime. → `references/sdk-configuration.md` (package, install, entry point, tuned defaults) and `references//setup.md` (install commands, API-inspection recipes, handler shape). +3. **Author the Worker.** *Install the SDK's serverless Worker package before writing any code* — it is usually shipped separately from the main SDK — sometimes on its own version line, sometimes in lockstep with it, and in one SDK not separately at all — so having the base SDK installed does not mean it is importable. Then read the installed package's actual API surface and write against that; these are Public Preview APIs that drift between versions, and generating code from memory costs a build cycle. Entry-point names are not consistent between SDKs, so inspect first rather than pattern-matching from another language. Every Workflow must declare a versioning behavior (`Pinned` or `AutoUpgrade`), per-Workflow or as a Worker-level default — code without it fails at runtime. → `references//sdk-.md` (package, install, API inspection, entry point, handler shape, versioning behavior, tuned defaults). -4. **Package and deploy the compute unit.** Build and package per SDK, deploy the compute unit, and set the invocation deadline high enough for the Worker to start, connect, register the Task Queue, and shut down gracefully. Match the build's target architecture to the deployed compute unit's — a mismatch fails only at invocation time, not at build time. After a create or update, wait for the compute unit to reach a ready state before the next step; providers return from these calls while the unit is still settling. → `references//setup.md`. +4. **Package and deploy the compute unit.** Build and package per SDK, deploy the compute unit, and set the invocation deadline high enough for the Worker to start, connect, register the Task Queue, and shut down gracefully. Match the build's target architecture to the deployed compute unit's — a mismatch fails only at invocation time, not at build time. After a create or update, wait for the compute unit to reach a ready state before the next step; providers return from these calls while the unit is still settling. → `references//sdk-.md` (build, packaging, runtime, handler, architecture, and SDK-specific deployment values) and `references//setup.md` (shared deployment lifecycle). 5. **Grant Temporal permission to invoke the Worker.** Configure the compute provider's access so Temporal can invoke and inspect the Worker. This access is separate from the compute unit's own execution role — do not confuse the two. Two things to get right before you create anything: (a) this grant is **shared, account-wide infrastructure** that a previous deployment may already have created — look for an existing one and extend it to cover your new Worker rather than creating a parallel copy, and never delete or repurpose one you did not create without asking; (b) scope the grant so that *future* immutable builds are covered, not just today's — a grant pinned to one build breaks the next release in a way that surfaces later as an unrelated-looking invocation failure. → `references//iam.md`. @@ -184,7 +195,7 @@ Surface these early — they apply regardless of compute provider: - **Deployment name and build ID must match exactly** between the Worker code and the Worker Deployment Version. A mismatch causes an invocation loop (Temporal invokes → Worker polls with the wrong version → Task not processed → invoke again). Signature: rapid repeated invocations with no Workflow progress. - **Set the invocation deadline high enough.** Providers often default to a very short timeout. If the first invocation times out before the Worker registers the Task Queue, the binding is never created and the Worker is never invoked again. → `references//setup.md` for the exact default. - **Use an immutable, versioned build per Build ID in production.** Pointing the provider at a mutable "latest" target lets code change under in-flight Workflows and cause non-determinism errors, even for Pinned Workflows. Keep a 1-to-1 mapping between each Build ID and one immutable build. → `references//versioning.md`. -- **Tune the timeout triple together for long-running Activities:** (1) worker stop timeout > longest Activity runtime, (2) shutdown deadline buffer > worker stop timeout + shutdown hook time, (3) invocation deadline > longest Activity runtime + shutdown deadline buffer. Raising one alone does not help. If the longest Activity exceeds half the maximum invocation deadline, recommend Activity Heartbeats. → `references/concepts.md`, `references/sdk-configuration.md`. +- **Tune the timeout triple together for long-running Activities:** (1) worker stop timeout > longest Activity runtime, (2) shutdown deadline buffer > worker stop timeout + shutdown hook time, (3) invocation deadline > longest Activity runtime + shutdown deadline buffer. Raising one alone does not help. If the longest Activity exceeds half the maximum invocation deadline, recommend Activity Heartbeats. → `references/concepts.md`, `references//sdk-.md`. - **Eager Activities are always disabled** — serverless invocations don't maintain persistent connections. Don't suggest them as an optimization. - **Activities are bounded by the invocation limit** (minus the shutdown deadline buffer); Workflow duration is unbounded and can span many invocations. Flag Activities that approach the provider's limit early. → `references/concepts.md`. - **Mixed serverless + long-lived Workers on one Task Queue:** do not enable dynamic scaling on the long-lived Workers — the two groups can't coordinate scaling and will cause unnecessary invocations. @@ -215,18 +226,22 @@ Most questions need 2–3 reference files. | User intent | Reference file(s) | |---|---| | What is a Serverless Worker / the WCI? How do invocation and autoscaling work? What are the constraints? Serverless vs long-lived Workers? | `references/concepts.md` | -| Deploy a Serverless Worker (happy path): write code, package, deploy, register + set-current version, verify, tear down. | `references//setup.md` (+ `references/concepts.md`) | +| Deploy a Serverless Worker (happy path): write code, package, deploy, register + set-current version, verify, tear down. | `references//setup.md` + the selected `references//sdk-.md` (+ `references/concepts.md`) | | Operator permissions and preflight; execution role vs Temporal invocation role; CloudFormation (Cloud + self-hosted). | `references//iam.md` | | Update or redeploy; version the build, use a qualified ARN, roll back. | `references//versioning.md` (+ `references/concepts.md`) | | Self-hosted server enablement (dynamic config, WCI, server AWS credentials). | `references//self-hosted.md` (+ `references//iam.md`) | -| SDK-specific options and tuned defaults, which package to install and how it is distributed, imports, versioning-behavior configuration, connection config (TOML, env vars). Reduce cold start / pre-bundle Workflow code. | `references/sdk-configuration.md` | -| Add OpenTelemetry observability, collector config, tracing. | `references//observability.md` | -| Worker not invoked, Workflows not progressing, inspect the WCI. | `references//diagnostics.md` (+ `references/concepts.md`) | -| Long-running Activities and timeout relationships. Isolate Activities from resource exhaustion. | `references/concepts.md` (+ `references/sdk-configuration.md`) | +| Go SDK-specific options and tuned defaults, package and import, API inspection, handler, build and packaging, runtime and deployment values, versioning-behavior configuration, connection config, OpenTelemetry integration. | `references//sdk-go.md` | +| Python SDK-specific options and tuned defaults, package and import, API inspection, handler, build and packaging, runtime and deployment values, versioning-behavior configuration, connection config, OpenTelemetry integration, diagnostic signatures. | `references//sdk-python.md` | +| TypeScript SDK-specific options and tuned defaults, package and import, API inspection, handler, build and packaging, runtime and deployment values, versioning-behavior configuration, connection config, pre-bundled Workflow code, OpenTelemetry integration. | `references//sdk-typescript.md` | +| Java SDK-specific options and tuned defaults, artifact and imports, API inspection, handler, build and packaging, runtime and deployment values, versioning-behavior configuration, connection config, OpenTelemetry integration, logging and diagnostic signatures. | `references//sdk-java.md` | +| .NET SDK-specific options and tuned defaults, package and imports, API inspection, handler, RID-specific publish and packaging, runtime and deployment values, versioning-behavior configuration, connection config and `SSL_CERT_FILE`, OpenTelemetry integration, logging and diagnostic signatures. | `references//sdk-dotnet.md` | +| Add OpenTelemetry observability, Collector config, X-Ray, and IAM. | `references//observability.md` + the selected `references//sdk-.md` | +| Worker not invoked, Workflows not progressing, inspect the WCI. | `references//diagnostics.md` + the selected `references//sdk-.md` (+ `references/concepts.md`) | +| Long-running Activities and timeout relationships. Isolate Activities from resource exhaustion. | `references/concepts.md` (+ the selected `references//sdk-.md`) | ## Out of Scope - **General SDK development patterns** (Workflows, Activities, signals, queries, Worker Versioning concepts): see `skill-temporal-developer`. - **Traditional Worker tuning** (slot suppliers, tuners, poller autoscaling, resource-based tuning): see `skill-temporal-workertuning`. - **Temporal Cloud administration** (Namespaces, users, certificates, billing): see `skill-temporal-ops`. -- **CLI command reference** (beyond the serverless-specific flags): see `skill-temporal-cli`. +- **CLI command reference** (beyond the serverless-specific flags): see `skill-temporal-cli`. \ No newline at end of file diff --git a/skills/temporal-serverless/references/aws-lambda/diagnostics.md b/skills/temporal-serverless/references/aws-lambda/diagnostics.md index f8f20d9..2d4ac73 100644 --- a/skills/temporal-serverless/references/aws-lambda/diagnostics.md +++ b/skills/temporal-serverless/references/aws-lambda/diagnostics.md @@ -123,6 +123,12 @@ Common errors include: - **TLS errors**: The TLS certificate or key is missing, expired, or does not match the Namespace. - **Authentication errors**: The API key is invalid or does not have access to the Namespace. +### Language-specific signatures + +**No application logs at all, but the Worker clearly ran.** SDK-specific logging configuration can suppress application logs even when the Worker is healthy. Diagnose invocation health from Lambda's runtime markers (`INIT_START`/`START`/`END`/`REPORT`) and CloudWatch metrics. + +For the exact logging cause and fix, see the selected SDK reference. + ### Check for Lambda timeout If the Lambda function reaches its configured timeout before the Worker finishes processing, AWS terminates the invocation. diff --git a/skills/temporal-serverless/references/aws-lambda/observability.md b/skills/temporal-serverless/references/aws-lambda/observability.md index af52f8f..0492608 100644 --- a/skills/temporal-serverless/references/aws-lambda/observability.md +++ b/skills/temporal-serverless/references/aws-lambda/observability.md @@ -1,149 +1,29 @@ -# Observability for Serverless Workers - - +# Observability for AWS Lambda Serverless Workers ## Overview Each SDK provides an OpenTelemetry integration package with defaults configured for the AWS Distro for OpenTelemetry (ADOT) Lambda layer. When enabled, the Worker emits SDK metrics and distributed traces for Workflow and Activity executions. The ADOT Lambda layer collects this telemetry and can forward traces to AWS X-Ray and metrics to Amazon CloudWatch. -## Go SDK - -### OTel package - -Import: `otel "go.temporal.io/sdk/contrib/aws/lambdaworker/otel"` - -### OTel functions - -- `otel.ApplyDefaults` — configures both metrics and tracing. -- `otel.ApplyMetrics` — configures metrics only. -- `otel.ApplyTracing` — configures tracing only. - -Usage in the configure callback: - -```go -if err := otel.ApplyDefaults(opts, &opts.ClientOptions, otel.Options{}); err != nil { - return err -} -``` - -By default, telemetry is sent to `localhost:4317`, which is the ADOT Lambda layer's default collector endpoint. - -### ADOT layer setup (Go) - -Attach the ADOT Collector layer to your Lambda function. -Go does not need a language-specific ADOT layer because the OTel SDK is compiled into the binary. - -### Collector config env var (Go) - -`OPENTELEMETRY_COLLECTOR_CONFIG_URI=/var/task/otel-collector-config.yaml` - ---- - -## Python SDK - -### OTel package - -Import: `from temporalio.contrib.aws.lambda_worker.otel import apply_defaults` - -To install with OTel support: `pip install temporalio[lambda-worker-otel]` - -### OTel functions - -- `apply_defaults` — configures both metrics and tracing. -- `build_metrics_telemetry_config` — configures metrics only. -- `apply_tracing` — configures tracing only. - -Usage in the configure callback: - -```python -def configure(config: LambdaWorkerConfig) -> None: - config.worker_config["task_queue"] = TASK_QUEUE - config.worker_config["workflows"] = [SampleWorkflow] - config.worker_config["activities"] = [hello_activity] - apply_defaults(config) -``` - -By default, telemetry is sent to `localhost:4317`, which is the ADOT Lambda layer's default collector endpoint. - -### ADOT layer setup (Python) - -Attach the ADOT Python Lambda layer to your Lambda function. The layer includes both auto-instrumentation and an OpenTelemetry Collector that receives telemetry on `localhost:4317` and forwards traces to AWS X-Ray and metrics to Amazon CloudWatch. - -### Collector config env var (Python) - -`OPENTELEMETRY_COLLECTOR_CONFIG_FILE=/var/task/otel-collector-config.yaml` - -Note: Python uses `_FILE` while Go and TypeScript use `_URI`. - ---- +Load the selected SDK reference's **Observability** section: -## TypeScript SDK - -### OTel package - -Import: `import { applyDefaults } from '@temporalio/lambda-worker/otel'` - -### OTel functions - -- `applyDefaults` — registers Temporal SDK interceptors for tracing and configures the Core SDK to export metrics via OTLP. -- `makeOtelPlugin` — returns a plugin for pre-bundling Workflow code that includes Workflow interceptor modules. - -Usage in the configure callback: - -```typescript -export const handler = runWorker({ deploymentName: 'sdk-demo', buildId: 'v1' }, (config) => { - config.workerOptions.taskQueue = TASK_QUEUE; - config.workerOptions.workflowBundle = { - codePath: require.resolve('./workflow-bundle.js'), - }; - config.workerOptions.activities = activities; - applyDefaults(config); -}); -``` - -By default, telemetry is sent to `localhost:4317`, which is the ADOT Lambda layer's default collector endpoint. - -### Pre-bundling with OTel - -When pre-bundling Workflow code, pass the plugin from `makeOtelPlugin()` so that Workflow interceptor modules are included in the bundle: - -```typescript -import { bundleWorkflowCode } from '@temporalio/worker'; -import { makeOtelPlugin } from '@temporalio/lambda-worker/otel'; - -const { plugin } = makeOtelPlugin(); -const { code } = await bundleWorkflowCode({ - workflowsPath: require.resolve('./workflows'), - plugins: [plugin], -}); -``` - - -### ADOT layer setup (TypeScript) - -Attach two ADOT Lambda layers: - -1. The ADOT JavaScript layer for Node.js-side auto-instrumentation and trace export. -2. The ADOT Collector layer (`aws-otel-collector-amd64`) to run the OTel Collector as a Lambda extension, receiving telemetry via OTLP on `localhost:4317` and forwarding traces to X-Ray and metrics to CloudWatch. - -### Collector config env var (TypeScript) - -`OPENTELEMETRY_COLLECTOR_CONFIG_URI=/var/task/otel-collector-config.yaml` - ---- +| SDK | Observability reference | +|---|---| +| Go | `sdk-go.md` → Observability | +| Python | `sdk-python.md` → Observability | +| TypeScript | `sdk-typescript.md` → Observability | +| Java | `sdk-java.md` → Observability | +| .NET | `sdk-dotnet.md` → Observability | -## Common across all SDKs +The remaining steps in this file are shared across SDKs. -### Custom Collector configuration required +## Custom Collector configuration required The default ADOT Collector configuration does not route OpenTelemetry Protocol (OTLP) data to the traces pipeline. You must provide a custom Collector configuration that wires the OTLP receiver to both the traces and metrics pipelines. Example `otel-collector-config.yaml` (bundle in your Lambda deployment package): +For the Collector configuration environment variable, see the selected SDK reference. + ```yaml receivers: otlp: @@ -181,7 +61,7 @@ service: ``` -### Enable X-Ray active tracing +## Enable X-Ray active tracing ```bash aws lambda update-function-configuration \ @@ -190,7 +70,7 @@ aws lambda update-function-configuration \ ``` -### Required IAM permissions +## Required IAM permissions The Lambda execution role must have permissions to write to X-Ray and CloudWatch: @@ -200,26 +80,4 @@ The Lambda execution role must have permissions to write to X-Ray and CloudWatch Without these permissions, the Collector fails silently and no telemetry appears. -For Python, the `AWSXRayDaemonWriteAccess` managed policy can be attached instead. - -### Collector config env var summary - - - -| SDK | Environment variable | -|---|---| -| Go | `OPENTELEMETRY_COLLECTOR_CONFIG_URI` | -| Python | `OPENTELEMETRY_COLLECTOR_CONFIG_FILE` | -| TypeScript | `OPENTELEMETRY_COLLECTOR_CONFIG_URI` | - -### ADOT layer summary - -| SDK | Layers needed | -|---|---| -| Go | ADOT Collector layer only (no language-specific layer; OTel SDK is compiled into the binary) | -| Python | ADOT Python Lambda layer (includes collector and auto-instrumentation) | -| TypeScript | ADOT JavaScript layer + ADOT Collector layer (`aws-otel-collector-amd64`) | - - - - +For language-specific IAM notes, see the selected SDK reference. diff --git a/skills/temporal-serverless/references/aws-lambda/sdk-dotnet.md b/skills/temporal-serverless/references/aws-lambda/sdk-dotnet.md new file mode 100644 index 0000000..6cad1af --- /dev/null +++ b/skills/temporal-serverless/references/aws-lambda/sdk-dotnet.md @@ -0,0 +1,268 @@ +# .NET SDK on AWS Lambda + +Sources: [`Temporalio.Extensions.Aws.Lambda` 1.18.0](https://www.nuget.org/packages/Temporalio.Extensions.Aws.Lambda/1.18.0), [Lambda extension source](https://github.com/temporalio/sdk-dotnet/tree/1.18.0/src/Temporalio.Extensions.Aws.Lambda), [OpenTelemetry extension source](https://github.com/temporalio/sdk-dotnet/tree/1.18.0/src/Temporalio.Extensions.Aws.Lambda.OpenTelemetry), and the [maintained Lambda Worker sample](https://github.com/temporalio/samples-dotnet/tree/6aba4fb9ea08177e303352ec9a4c61e303cefb0e/src/LambdaWorker). + +Use this reference for .NET SDK-specific package, entry-point, Worker configuration, tuned defaults, observability, and diagnostic details. For shared AWS Lambda deployment, observability infrastructure, and diagnostic flow, see `setup.md`, `observability.md`, and `diagnostics.md`. + +## Package + +Import: `using Temporalio.Extensions.Aws.Lambda;` plus `Temporalio.Common` (for `WorkerDeploymentVersion`) and `Amazon.Lambda.Core` (for `ILambdaContext`). + +Install: `dotnet add package Temporalio.Extensions.Aws.Lambda` — a **separate NuGet package** from `Temporalio`, published in **lockstep** with it (both 1.18.0). Published versions: 1.17.0 and 1.18.0. The package targets `netstandard2.0` and declares `Temporalio` 1.18.0 and `Amazon.Lambda.Core` 3.1.0. + +OpenTelemetry lives in a **second package**, `Temporalio.Extensions.Aws.Lambda.OpenTelemetry` (also 1.18.0). → Observability below. + +- .NET: [.NET Lambda Worker sample](https://github.com/temporalio/samples-dotnet/tree/6aba4fb9ea08177e303352ec9a4c61e303cefb0e/src/LambdaWorker) — `Worker/`, `Starter/`, and `Deploy/` (deploy, IAM-role, execution-role, and telemetry scripts plus a CloudFormation template), with a test project under `tests/LambdaWorker`. + +List the real public API of the resolved package before generating code — the `.nupkg` is a zip and ships full XML documentation: + +```bash +curl -sO https://api.nuget.org/v3-flatcontainer/temporalio.extensions.aws.lambda//temporalio.extensions.aws.lambda..nupkg +unzip -p temporalio.extensions.aws.lambda..nupkg \ + lib/netstandard2.0/Temporalio.Extensions.Aws.Lambda.xml +# the .nuspec lists the exact dependency versions: +unzip -p ...nupkg Temporalio.Extensions.Aws.Lambda.nuspec | grep dependency +``` + +If sources disagree, use the installed artifact's public API, followed by the maintained sample and the prose documentation. + +## Entry point + +**`TemporalLambdaWorker.CreateHandler(version, configure)`** — returns a `Func` that your handler method delegates to. Overloads take either a synchronous `Action` or an asynchronous `Func` for setup that must await. + +## Configure callback + +Receives a `TemporalLambdaWorkerOptions` with public `ClientOptions`, `WorkerOptions`, `ShutdownDeadlineBuffer`, and `AddShutdownHook(Func)` members. The Task Queue and registrations go through `WorkerOptions` — an ordinary `TemporalWorkerOptions`, so `TaskQueue`, `AddWorkflow()` and `AddActivity(...)` behave exactly as they do for a long-lived Worker. The callback runs **per invocation**. + +## Versioning behavior + +Per-Workflow via the `[Workflow]` attribute: + +```csharp +[Workflow(VersioningBehavior = VersioningBehavior.Pinned)] +public class MyWorkflow +{ + [WorkflowRun] + public async Task RunAsync(string name) => /* ... */; +} +``` + +Or a Worker-level default through `DefaultVersioningBehavior` in `DeploymentOptions`. + +**The .NET Worker-level default is `AutoUpgrade`.** Prefer setting the behavior explicitly per Workflow. + +## Handler example + +A plain class exposes an async method that delegates to the handler returned by `TemporalLambdaWorker.CreateHandler`. + +```csharp +namespace MyCompany.Temporal.Worker; + +using Amazon.Lambda.Core; +using Temporalio.Common; +using Temporalio.Extensions.Aws.Lambda; + +public class LambdaFunction +{ + private static readonly Func WorkerHandler = + TemporalLambdaWorker.CreateHandler( + new WorkerDeploymentVersion("my-app", "build-1"), + config => + { + config.WorkerOptions.TaskQueue = + Environment.GetEnvironmentVariable("TEMPORAL_TASK_QUEUE") ?? "my-task-queue"; + config.WorkerOptions + .AddWorkflow() + .AddActivity(MyActivities.MyActivity); + }); + + public Task HandlerAsync(Stream input, ILambdaContext context) => + WorkerHandler(input, context); +} +``` + +Registrations go through `config.WorkerOptions`, an ordinary `TemporalWorkerOptions` — the same API a long-lived Worker uses. Use the `Func` overload when setup must await. + +## Lambda-tuned defaults + + + +| Setting | Lambda default | +|---|---| +| `MaxConcurrentActivities` | 2 | +| `MaxConcurrentWorkflowTasks` | 10 | +| `MaxConcurrentLocalActivities` | 2 | +| `MaxConcurrentNexusTasks` | 5 | +| `MaxConcurrentWorkflowTaskPolls` | 2 | +| `MaxConcurrentActivityTaskPolls` | 1 | +| `MaxConcurrentNexusTaskPolls` | 1 | +| `MaxCachedWorkflows` | 30 | +| `GracefulShutdownTimeout` | 5 seconds | +| `ShutdownDeadlineBuffer` | 7 seconds | +| `DisableEagerActivityExecution` | Always `true`, cannot be overridden | + +## Logging — set a LoggerFactory or Workflow logs vanish + +`TemporalWorkerOptions.LoggerFactory` is unset by default and "defaults to the client logger factory", which is also unset — so `Workflow.Logger` output is discarded. Activity `Console.WriteLine` still reaches CloudWatch, which makes the gap look selective rather than total. + +Install the console logging provider: + +```bash +dotnet add package Microsoft.Extensions.Logging.Console +``` + +```csharp +using Microsoft.Extensions.Logging; + +config.WorkerOptions.LoggerFactory = + LoggerFactory.Create(b => b.AddSimpleConsole().SetMinimumLevel(LogLevel.Information)); +``` + +## Connection configuration + +Loaded automatically from environment variables and an optional TOML config file, in this resolution order: + +1. `TEMPORAL_CONFIG_FILE` environment variable, if set. +2. `temporal.toml` in the Lambda task root (typically `/var/task`). +3. `temporal.toml` in the current working directory. + +When using `temporal.toml`, copy it into the publish directory before zipping so it lands in the task root. Keep the API key in `TEMPORAL_API_KEY` rather than in the file; supplying an API key enables TLS automatically. + +**TLS caveat specific to .NET — set `SSL_CERT_FILE` or the first invocation fails.** AWS's .NET 8 Lambda images force-override `SSL_CERT_FILE`, which prevents the SDK's Rust core from loading system root CAs. Set it explicitly on the function: + +``` +SSL_CERT_FILE=/etc/pki/tls/certs/ca-bundle.crt # or /etc/ssl/certs/ca-certificates.crt +``` + +**This is server-certificate verification, not client credentials.** The API key is unaffected and is not the problem — an API key auto-enables TLS, and TLS requires verifying Temporal Cloud's certificate chain against root CAs. The connection fails before authentication is ever attempted. See `diagnostics.md` for the corresponding failure signature and recovery steps. + +## Build and package + +### Native dependency — publish must be RID-specific + +The .NET SDK wraps a **native Rust core** (`libtemporalio_sdk_core_c_bridge.so`). For Lambda, publish for an explicit runtime identifier matching the function's architecture and confirm the native library is present before creating the zip: + +| `--runtime` | `--architectures` | +|---|---| +| `linux-x64` | `x86_64` | +| `linux-arm64` | `arm64` | + +Publish for an explicit Linux runtime identifier, then zip the publish output. + +```bash +dotnet publish path/to/Worker.csproj \ + --configuration Release \ + --runtime linux-x64 \ + --self-contained false \ + --output ./publish + +# Guard: the SDK's native Rust bridge must be in the output, or the function +# fails at FIRST INVOCATION, not at build time. +[[ -f ./publish/libtemporalio_sdk_core_c_bridge.so ]] || { + echo "Publish output is missing the linux-x64 Temporal native bridge." >&2; exit 1; } + +# Copy each optional configuration file that this deployment uses so it lands +# in the Lambda task root: +if [[ -f temporal.toml ]]; then + cp temporal.toml ./publish/ +fi +if [[ -f otel-collector-config.yaml ]]; then + cp otel-collector-config.yaml ./publish/ +fi + +cd ./publish && zip -r ../function.zip . && cd .. +``` + +Keep the native-library check before zipping. `--self-contained false` is correct because the `dotnet8` managed runtime supplies the framework. + +## Deploy the Lambda function + +```bash +aws lambda create-function \ + --function-name my-temporal-worker \ + --runtime dotnet8 \ + --architectures x86_64 \ + --handler 'MyAssembly::MyCompany.Temporal.Worker.LambdaFunction::HandlerAsync' \ + --role \ + --zip-file fileb://function.zip \ + --timeout 600 \ + --memory-size 256 \ + --environment file:///tmp/lambda-env.json +``` + +**The environment block for .NET must include `SSL_CERT_FILE`**, in addition to the usual `TEMPORAL_*` variables: + +```json +{"Variables":{ + "TEMPORAL_ADDRESS":"...", "TEMPORAL_NAMESPACE":"...", "TEMPORAL_API_KEY":"...", + "SSL_CERT_FILE":"/etc/pki/tls/certs/ca-bundle.crt"}} +``` + +Without it the **first invocation fails**, the Task Queue is never bound, and the Worker is never invoked again. AWS's .NET 8 Lambda images force-override `SSL_CERT_FILE`, which stops the SDK's Rust core from loading system root CAs. `/etc/ssl/certs/ca-certificates.crt` also works; try the other if one fails. This is server-certificate verification, unrelated to your API key. → `diagnostics.md`. + +- `--runtime`: `dotnet8` for a `net8.0` build. +- `--handler`: **`ASSEMBLY::NAMESPACE.TYPE::METHOD` — three colon-separated parts.** Getting this wrong presents as a handler-not-found error at first invocation. +- `--timeout 600` / `--memory-size 256` are example values. The timeout must accommodate Worker startup and registration, Task and Activity processing, and graceful shutdown. Memory contributes directly to Lambda cost. → `setup.md` for how to choose both values. +- `--architectures` must match the publish RID (`linux-x64` → `x86_64`, `linux-arm64` → `arm64`). + +## Observability + +### OTel package + +A **second NuGet package**, separate from the Lambda extension itself: + +```bash +dotnet add package Temporalio.Extensions.Aws.Lambda.OpenTelemetry +``` + +Published in lockstep with `Temporalio` and `Temporalio.Extensions.Aws.Lambda` (all 1.18.0). + +### OTel functions + +The package contributes an extension method on the options object, applied inside the configure callback: + +```csharp +using Temporalio.Extensions.Aws.Lambda.OpenTelemetry; + +TemporalLambdaWorker.CreateHandler( + new WorkerDeploymentVersion("my-app", "build-1"), + config => + { + config.ApplyOpenTelemetryDefaults(); + config.WorkerOptions.TaskQueue = "my-task-queue"; + config.WorkerOptions.AddWorkflow().AddActivity(MyActivities.MyActivity); + }); +``` + +`ApplyOpenTelemetryDefaults()` configures metrics and tracing against the ADOT layer's collector. Telemetry must be exported before the invocation ends — keep any metrics export interval shorter than the Lambda timeout. + +### ADOT layer setup + +Attach an **ADOT Collector layer** for the target region and architecture. No language-specific auto-instrumentation layer is needed because the OpenTelemetry SDK arrives as an ordinary package dependency. Supply the collector layer ARN for the target region. + +Set `OPENTELEMETRY_COLLECTOR_CONFIG_URI=/var/task/otel-collector-config.yaml` and copy `otel-collector-config.yaml` into the publish directory before zipping so it lands in the task root. + +For the shared Collector configuration, X-Ray enablement, and execution-role permissions, see `observability.md`. + +## Diagnostic signatures + +| SDK | Cause | Fix | +|---|---|---| +| .NET | `TemporalWorkerOptions.LoggerFactory` is unset and "defaults to the client logger factory", which is also unset — so `Workflow.Logger` output is discarded. Activity `Console.WriteLine` still reaches CloudWatch, which makes the gap look selective rather than total | set `config.WorkerOptions.LoggerFactory` (e.g. `LoggerFactory.Create(b => b.AddSimpleConsole().SetMinimumLevel(LogLevel.Information))`) | + +**.NET — `DllNotFoundException` / missing `libtemporalio_sdk_core_c_bridge.so` at first invocation.** Republish with an explicit runtime identifier matching the function's architecture (`--runtime linux-x64` for `x86_64`, `linux-arm64` for `arm64`) and check the native library is in the publish output before zipping. → Build and package above. + +**.NET — `NativeCertsNotFound` at first invocation, despite a correct address, Namespace, and API key.** + +``` +System.InvalidOperationException: Connection failed: Server connection error: + tonic::transport::Error(Transport, NativeCertsNotFound) + at Temporalio.Bridge.Client.ConnectAsync(...) + at Temporalio.Client.TemporalConnection.ConnectAsync(...) +``` + +*Cause:* AWS's .NET 8 Lambda images force-override `SSL_CERT_FILE`, so the SDK's Rust core cannot load system root CAs. *Fix:* set `SSL_CERT_FILE=/etc/pki/tls/certs/ca-bundle.crt` (or `/etc/ssl/certs/ca-certificates.crt`) on the function, then recover the binding as described under "Failed first invocation" in `diagnostics.md` — the failed validation invocation means no Task Queue was bound and Temporal will not retry on its own. + +"Certs" here means the operating system's root CA store, not client credentials. An API key auto-enables TLS, and TLS requires verifying the server's certificate chain. The connection fails before authentication is attempted, so changing the API key, Namespace, invocation role, or External ID will not fix this error. + +**.NET — handler not found at first invocation.** The handler string has **three** colon-separated parts, `ASSEMBLY::NAMESPACE.TYPE::METHOD`. Compare against the assembly name (not the project name, if they differ) and the fully-qualified type. diff --git a/skills/temporal-serverless/references/aws-lambda/sdk-go.md b/skills/temporal-serverless/references/aws-lambda/sdk-go.md new file mode 100644 index 0000000..0b57603 --- /dev/null +++ b/skills/temporal-serverless/references/aws-lambda/sdk-go.md @@ -0,0 +1,192 @@ +# Go SDK on AWS Lambda + + + +Use this reference for Go SDK-specific package, entry-point, Worker configuration, tuned defaults, and observability details. For shared AWS Lambda deployment and observability infrastructure, see `setup.md` and `observability.md`. + +## Package + +Import: `lambdaworker "go.temporal.io/sdk/contrib/aws/lambdaworker"` + +Install: `go get go.temporal.io/sdk/contrib/aws/lambdaworker` — **this is a separate Go module** from `go.temporal.io/sdk`, versioned independently (`v0.1.1` at the time of writing). Having the main SDK in `go.mod` does not make it importable; add it explicitly, then `go mod tidy`. Verify the installed surface with `go doc go.temporal.io/sdk/contrib/aws/lambdaworker` before generating code — the API is Public Preview and drifts. + +- Go: [Go Lambda Worker sample](https://github.com/temporalio/samples-go/tree/main/lambda-worker) + +List the exported API of the installed module version before generating code: + +```bash +go doc go.temporal.io/sdk/contrib/aws/lambdaworker +go doc go.temporal.io/sdk/contrib/aws/lambdaworker.Options +``` + +**Ordering when sources disagree:** the installed artifact first, the SDK's maintained samples second (they are built in CI, so they cannot reference a method that does not exist), the prose docs last. Entry-point names are not consistent across SDKs, so check rather than pattern-match from another language. + +## Entry point + +`lambdaworker.RunWorker` — starts a Lambda-based Worker. Pass a `WorkerDeploymentVersion` and a callback that registers Workflows and Activities. + +## Configure callback + +The `Options` callback gives access to the same registration methods as a traditional Worker: `RegisterWorkflow`, `RegisterWorkflowWithOptions`, `RegisterActivity`, `RegisterActivityWithOptions`, and `RegisterNexusService`. + +Go, Python and TypeScript invoke it per invocation. + +In Go it is a direct field on the options object (`opts.TaskQueue`). + +## Versioning behavior + +Set per-Workflow at registration time with `workflow.VersioningBehaviorPinned` or `workflow.VersioningBehaviorAutoUpgrade`. +Or set a Worker-level default with `DefaultVersioningBehavior` in `DeploymentOptions`. + +**Worker Versioning is always on.** The run-worker entry point enables it, so the only remaining decision is `Pinned` vs `AutoUpgrade` per Workflow (or a Worker-level default). + +## Handler example + +Use the Go SDK's `lambdaworker` package. + +```go +package main + +import ( + lambdaworker "go.temporal.io/sdk/contrib/aws/lambdaworker" + "go.temporal.io/sdk/worker" + "go.temporal.io/sdk/workflow" +) + +func main() { + lambdaworker.RunWorker(worker.WorkerDeploymentVersion{ + DeploymentName: "my-app", + BuildID: "build-1", + }, func(opts *lambdaworker.Options) error { + opts.TaskQueue = "my-task-queue" + + opts.RegisterWorkflowWithOptions(MyWorkflow, workflow.RegisterOptions{ + VersioningBehavior: workflow.VersioningBehaviorPinned, + }) + opts.RegisterActivity(MyActivity) + + return nil + }) +} +``` + + +## Lambda-tuned defaults + + + +| Setting | Lambda default | +|---|---| +| `MaxConcurrentActivityExecutionSize` | 2 | +| `MaxConcurrentWorkflowTaskExecutionSize` | 10 | +| `MaxConcurrentLocalActivityExecutionSize` | 2 | +| `MaxConcurrentNexusTaskExecutionSize` | 5 | +| `MaxConcurrentActivityTaskPollers` | 1 | +| `MaxConcurrentWorkflowTaskPollers` | 2 | +| `MaxConcurrentNexusTaskPollers` | 1 | +| `WorkerStopTimeout` | 5 seconds | +| `DisableEagerActivities` | Always true | +| Sticky cache size | 100 | +| `ShutdownDeadlineBuffer` | 7 seconds | + +Note: Go sticky cache size is 100, while Python and TypeScript are 30. These values come from each SDK's own docs and are not interchangeable. + +These are the same `worker.Options` available to any Temporal Worker, just with lower values for Lambda's constrained environment. Except for `ShutdownDeadlineBuffer`, which is specific to the `lambdaworker` package. + +`DisableEagerActivities` is always true and cannot be overridden. Eager Activities require a persistent connection, which Lambda invocations don't maintain. + +`ShutdownDeadlineBuffer` controls how much time before the Lambda deadline the Worker begins its graceful shutdown. The default is `WorkerStopTimeout` + 2 seconds. + +If your Worker handles long-running Activities, increase `WorkerStopTimeout`, `ShutdownDeadlineBuffer`, and the Lambda invocation deadline (`--timeout`) together. + +## Connection configuration + +The `lambdaworker` package automatically loads Temporal client configuration from a TOML config file and environment variables (see the Environment Configuration docs, `/develop/environment-configuration`). + +TOML config file resolution order: + +1. `TEMPORAL_CONFIG_FILE` environment variable, if set. +2. `temporal.toml` in `$LAMBDA_TASK_ROOT` (typically `/var/task`). +3. `temporal.toml` in the current working directory. + +The file is optional. If absent, only environment variables are used. + +## Build and package + +Cross-compile for Lambda's Linux runtime: + +```bash +GOOS=linux GOARCH=amd64 go build -tags lambda.norpc -o bootstrap ./worker +``` + + +Package the binary into a zip file: + +```bash +zip function.zip bootstrap +``` + + +**Add `CGO_ENABLED=0`, and match the architecture you deploy.** The `provided.al2023` runtime expects a self-contained binary; building with cgo enabled links against host libraries that may not resolve inside the runtime. Set `CGO_ENABLED=0` for a statically linked binary, and keep `GOARCH` consistent with the function's `--architectures` (`amd64` ↔ `x86_64`, `arm64` ↔ `arm64`). Also adjust the trailing package path to your layout — `.` when `main` is in the repo root, `./worker` when it is in a `worker/` subdirectory. A reusable script: + +```bash +#!/usr/bin/env bash +set -euo pipefail +go vet ./... # catches a missing import before the cross-compile +CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -tags lambda.norpc -o bootstrap . +zip -q function.zip bootstrap +file bootstrap # expect: ELF 64-bit ... statically linked +``` + +Run `go vet` (or a plain `go build ./...`) before the packaging build. The three-package import block above — `lambdaworker`, `worker` for `WorkerDeploymentVersion`, and `workflow` for the versioning-behavior constants — is easy to write short by one entry, and catching that locally is faster than discovering it in the cross-compile step. + +An architecture mismatch surfaces only at invocation time as an `Runtime.InvalidEntrypoint`/exec-format error, not at build or package time — the same failure class as the Python wheel mismatch described in `sdk-python.md`. + +A typical Go Worker zip lands around 10–15 MB, well under the 50 MB direct-upload limit. + +## Deploy the Lambda function + +```bash +aws lambda create-function \ + --function-name my-temporal-worker \ + --runtime provided.al2023 \ + --handler bootstrap \ + --role \ + --zip-file fileb://function.zip \ + --timeout 600 \ + --memory-size 256 \ + --environment '{"Variables":{"HOME":"/tmp","TEMPORAL_ADDRESS":":7233","TEMPORAL_NAMESPACE":"","TEMPORAL_API_KEY":""}}' +``` + + +- `--runtime`: `provided.al2023` for custom Go binaries. +- `--handler`: `bootstrap` when using the `provided.al2023` custom runtime. + +| Variable | Description | +|---|---| +| `HOME` | Set to `/tmp` in the Go and TypeScript examples above. Lambda's filesystem is read-only outside `/tmp`, so anything the runtime or config loader resolves relative to the home directory needs a writable target. The docs omit it from the Python example; including it there is harmless. | + +## Observability + +Import: `otel "go.temporal.io/sdk/contrib/aws/lambdaworker/otel"` + +- `otel.ApplyDefaults` — configures both metrics and tracing. +- `otel.ApplyMetrics` — configures metrics only. +- `otel.ApplyTracing` — configures tracing only. + +Usage in the configure callback: + +```go +if err := otel.ApplyDefaults(opts, &opts.ClientOptions, otel.Options{}); err != nil { + return err +} +``` + +By default, telemetry is sent to `localhost:4317`, which is the ADOT Lambda layer's default collector endpoint. + +Attach the ADOT Collector layer to your Lambda function. +Go does not need a language-specific ADOT layer because the OTel SDK is compiled into the binary. + +`OPENTELEMETRY_COLLECTOR_CONFIG_URI=/var/task/otel-collector-config.yaml` + +For the shared Collector configuration, X-Ray enablement, and execution-role permissions, see `observability.md`. diff --git a/skills/temporal-serverless/references/aws-lambda/sdk-java.md b/skills/temporal-serverless/references/aws-lambda/sdk-java.md new file mode 100644 index 0000000..a1359c2 --- /dev/null +++ b/skills/temporal-serverless/references/aws-lambda/sdk-java.md @@ -0,0 +1,318 @@ +# Java SDK on AWS Lambda + + + +Use this reference for Java SDK-specific package, entry-point, Worker configuration, tuned defaults, observability, and diagnostic details. For shared AWS Lambda deployment, observability infrastructure, and diagnostic flow, see `setup.md`, `observability.md`, and `diagnostics.md`. + +## Package + +Import: `io.temporal.aws.lambda.LambdaWorker`, `io.temporal.aws.lambda.LambdaWorkerOptions`, `io.temporal.common.WorkerDeploymentVersion` + +Install: `io.temporal:temporal-aws-lambda` — a **separate Maven artifact** from `io.temporal:temporal-sdk`, but published on the **same version line** (both 1.38.0). This is a third packaging pattern: unlike Go and TypeScript it is not independently versioned, and unlike Python it does not ship inside the main SDK. Use `io.temporal:temporal-bom` in `dependencyManagement` to keep them aligned. + +```xml + + + + io.temporaltemporal-bom + 1.38.0pomimport + + + +``` + +`aws-lambda-java-core` (1.4.0) arrives transitively from `temporal-aws-lambda`; declare it explicitly if you compile against `RequestHandler`/`Context`. + +- Java: [Java Lambda Worker sample](https://github.com/temporalio/samples-java/tree/main/lambda-worker) — three Gradle subprojects (`worker/` handler + greeting Workflow/Activity, `starter/` local client, `deploy/` IAM and deploy scripts plus a CloudFormation template) + +List the real public API of the resolved artifact before generating code: + +```bash +javap -cp ~/.m2/repository/io/temporal/temporal-aws-lambda//temporal-aws-lambda-.jar \ + io.temporal.aws.lambda.LambdaWorker +javap -cp 'io.temporal.aws.lambda.LambdaWorkerOptions$Builder' +# or read the source directly — Maven Central publishes a sources jar: +# curl -O https://repo1.maven.org/maven2/io/temporal/temporal-aws-lambda//temporal-aws-lambda--sources.jar +``` + +**A useful ordering when sources disagree:** the installed artifact first, the SDK's maintained samples second (they are built in CI, so they cannot reference a method that does not exist), the prose docs last. Entry-point names are not consistent across SDKs — Java's is `define`, not "run"-shaped like the others — so check rather than pattern-match from another language. + +## Entry point + +**`LambdaWorker.define(version, configure)`** — returns a `RequestHandler` that your handler class delegates to. There are four public overloads: `define` (2- and 3-arg) and `newHandler` (2- and 3-arg, taking a pre-built `LambdaWorkerOptions`). + +Note that Java's entry point is not "run"-shaped like the other SDKs' (`RunWorker`, `run_worker`, `runWorker`) — confirm the method name against the version you install. + +## Configure callback — two phases, unlike the other SDKs + +Java splits configuration in a way no other SDK does, and the distinction matters: + +- The `Consumer` passed to `define` is *"invoked once while the Lambda handler is constructed"* — i.e. at **cold start**, once per container, **not** per invocation. +- Per-invocation configuration is a **separate** callback, `LambdaWorker.InvocationConfigurator`, taking `(LambdaWorkerOptions.Builder, com.amazonaws.services.lambda.runtime.Context)`, *"invoked for each Lambda invocation before Temporal service stubs, client, and worker are created."* Use the 3-arg `define` overload for it. + +Contrast Python, whose `configure` runs **once per invocation**. Do not describe them as equivalent, and do not carry per-invocation logic into Java's cold-start callback. + +Registration methods on `LambdaWorkerOptions.Builder`: `setTaskQueue`, `registerWorkflowImplementationTypes`, `registerDynamicWorkflowImplementationType`, `registerWorkflowImplementationFactory` (3 overloads), `registerActivitiesImplementations`, `registerDynamicActivityImplementation`, `registerNexusServiceImplementation`, `addShutdownHook`, plus `getWorkerOptionsBuilder()` / `getWorkflowClientOptionsBuilder()` / `getWorkflowServiceStubsOptionsBuilder()` for lower-level tuning. + +## Versioning behavior + +Per-Workflow via the **annotation** `io.temporal.workflow.WorkflowVersioningBehavior` on the workflow method, taking `io.temporal.common.VersioningBehavior.PINNED` or `.AUTO_UPGRADE`: + +```java +public class GreetingWorkflowImpl implements GreetingWorkflow { + @Override + @WorkflowVersioningBehavior(VersioningBehavior.PINNED) + public String getGreeting(String name) { ... } +} +``` + +Or a Worker-level default with `DefaultVersioningBehavior` in `DeploymentOptions`. + +**Worker Versioning is always on.** The run-worker entry point enables it, so the only remaining decision is `Pinned` vs `AutoUpgrade` per Workflow (or a Worker-level default). + +Versioning behavior: annotate the Workflow **method** in the implementation class with `io.temporal.workflow.WorkflowVersioningBehavior`, or set a Worker-level default with `DefaultVersioningBehavior` in `DeploymentOptions`. + +```java +public class MyWorkflowImpl implements MyWorkflow { + @Override + @WorkflowVersioningBehavior(VersioningBehavior.PINNED) + public String getGreeting(String name) { ... } +} +``` + +## Handler example + +Use the `temporal-aws-lambda` module. The handler class implements `RequestHandler` and delegates to the handler returned by `LambdaWorker.define`. + +```java +package com.example.temporal; + +import com.amazonaws.services.lambda.runtime.Context; +import com.amazonaws.services.lambda.runtime.RequestHandler; +import io.temporal.aws.lambda.LambdaWorker; +import io.temporal.common.WorkerDeploymentVersion; + +public final class LambdaFunction implements RequestHandler { + + // The callback below runs ONCE, at cold start, when the handler is constructed -- + // not per invocation. Use the 3-arg define(...) overload with an + // InvocationConfigurator for anything that must run per invocation. + private static final RequestHandler WORKER = + LambdaWorker.define( + new WorkerDeploymentVersion("my-app", "build-1"), + builder -> { + builder.setTaskQueue("my-task-queue"); + builder.registerWorkflowImplementationTypes(MyWorkflowImpl.class); + builder.registerActivitiesImplementations(new MyActivitiesImpl()); + }); + + @Override + public Void handleRequest(Object input, Context context) { + return WORKER.handleRequest(input, context); + } +} +``` + +The entry point is `define` (or `newHandler` for pre-built options) — not a "run"-shaped name like the other SDKs use. Temporal's [sample handler](https://github.com/temporalio/samples-java/blob/main/lambda-worker/worker/src/main/java/io/temporal/samples/lambdaworker/LambdaFunction.java) is the reference implementation. + +## Lambda-tuned defaults + + + +| Setting | Lambda default | +|---|---| +| `MaxConcurrentActivityExecutionSize` | 2 | +| `MaxConcurrentWorkflowTaskExecutionSize` | 10 | +| `MaxConcurrentLocalActivityExecutionSize` | 2 | +| `MaxConcurrentNexusExecutionSize` | 5 | +| `MaxConcurrentWorkflowTaskPollers` | 2 | +| `MaxConcurrentActivityTaskPollers` | 1 | +| `MaxConcurrentNexusTaskPollers` | 1 | +| `WorkflowCacheSize` | 30 | +| `MaxWorkflowThreadCount` | 30 | +| `GracefulShutdownTimeout` | 5 seconds | +| `ShutdownDeadlineBuffer` | 7 seconds | + +`MaxWorkflowThreadCount` has no counterpart in the other SDKs — Java runs Workflow code on real threads. + +Eager Activities are disabled: `builder.setDisableEagerExecution(true)` (`LambdaWorkerOptions.java:258`). `ShutdownDeadlineBuffer` defaults to `GracefulShutdownTimeout` + 2s, the same relationship as the other SDKs. + +## Logging — the binding must be SLF4J 1.7.x + +The Java SDK compiles against `org.slf4j:slf4j-api:1.7.36`. A 2.x provider (`slf4j-simple:2.x`, Logback 1.3+) **will not bind to a 1.7 API**, and the Worker runs with no logs at all — the same silent outcome as Python's `logging.basicConfig()` no-op, by a different mechanism. Use a 1.7.x provider: + +```xml + + org.slf4jslf4j-simple1.7.36 + +``` + +With a correct binding the module logs its own lifecycle unprompted, which is more than the other SDKs give you by default: + +``` +[main] INFO io.temporal.aws.lambda.LambdaWorker - Temporal Lambda worker started + awsRequestId= invokedFunctionArn= taskQueue= identity=@ +``` + +## Connection configuration + +Loaded automatically from environment variables and an optional TOML config file. Public constants on `LambdaWorkerOptions`: `TEMPORAL_TASK_QUEUE`, `TEMPORAL_CONFIG_FILE`, `LAMBDA_TASK_ROOT`. + +Resolution order (`LambdaWorkerOptions.resolveConfigFilePath`): + +1. `TEMPORAL_CONFIG_FILE` environment variable, if set. +2. `temporal.toml` in `$LAMBDA_TASK_ROOT` (typically `/var/task`). +3. `temporal.toml` in the current working directory. + +**`HOME=/tmp` is not required for Java** — unlike the Go and TypeScript examples. The module never reads `HOME`; when no file is found it passes a null path to `ClientConfig.load`, which falls back to `/.config/temporalio/temporal.toml` and treats both a missing home directory and a `FileNotFoundException` as "empty config, no error". It is a read, not a write, so Lambda's read-only filesystem is not involved. Note also that Java reads the `user.home` **system property**, not the `HOME` environment variable, so setting `HOME` is not even the right lever — use `-Duser.home` via `JAVA_TOOL_OPTIONS` if you ever need to steer it. + +## Build and package + +Build an uber-jar with all dependencies bundled. A JAR is a valid zip, so it uploads directly with no extra packaging step. + +**Gradle** (what the official sample uses): `./gradlew shadowJar` → `build/libs/-all.jar`. + +**Maven**: `maven-shade-plugin`, bound to `package` → `target/.jar`. + +```xml + + org.apache.maven.plugins + maven-shade-plugin + 3.6.0 + + + package + shade + + false + + + + + + *:* + + META-INF/*.SF + META-INF/*.DSA + META-INF/*.RSA + module-info.class + + + + + + + +``` + +**`ServicesResourceTransformer` is mandatory, not hygiene.** The Temporal client is gRPC-based, and gRPC discovers channel providers, name resolvers, and load balancers through `META-INF/services` files that several jars each contribute to. Without merging, later copies overwrite earlier ones and the client fails at the **first invocation** with a "no functional channel service provider found"-class error — never at build time. Verify the merge before uploading: + +```bash +unzip -p target/.jar META-INF/services/io.grpc.ManagedChannelProvider +# expect MORE THAN ONE provider line, e.g.: +# io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider +# io.grpc.netty.shaded.io.grpc.netty.UdsNettyChannelProvider +``` + +Excluding the signature files matters too: signed-jar signatures are invalid inside an uber-jar and produce a `SecurityException` at class load. + +**Match the bytecode target to the runtime.** Compiling on a newer JDK than the function's runtime needs an explicit target — `17` for `--runtime java17`. This is the Java form of the architecture/wheel mismatch: it fails at invocation, not at build. + +**Watch the artifact size — Java hits the 50 MB direct-upload ceiling early.** A hello-world Worker (one Workflow, one Activity, `slf4j-simple`) measured **41 MB**, versus ~14 MB for the equivalent Python package and 10–15 MB for Go. Anything with real dependencies will exceed 50 MB and must be uploaded via S3 (`--code S3Bucket=…,S3Key=…`) rather than `--zip-file fileb://`. Check before deploying: + +```bash +ls -lh target/.jar +``` + +## Deploy the Lambda function + +```bash +aws lambda create-function \ + --function-name my-temporal-worker \ + --runtime java17 \ + --architectures x86_64 \ + --handler com.example.temporal.LambdaFunction::handleRequest \ + --role \ + --zip-file fileb://target/my-worker.jar \ + --timeout 90 \ + --memory-size 1024 \ + --environment file:///tmp/lambda-env.json +``` + +- `--runtime`: `java17` (or another supported Java version). +- `--handler`: `fully.qualified.Class::method` — **a different format from every other SDK**, which use `module.function` / `module.export`. Point it at the method that delegates to the `LambdaWorker.define` handler. +- `--zip-file`: the shaded jar directly; no separate zip step. Switch to `--code S3Bucket=…,S3Key=…` once the jar exceeds 50 MB, which happens early in Java (see packaging above). +- **`HOME=/tmp` is not needed** — unlike the Go and TypeScript examples. Verified: the Java module never reads `HOME`, and a missing config file is non-fatal. → Connection configuration above. +- `--memory-size`: the docs recommend starting at `1024` because "Java Workers typically need more memory than other runtimes," then adjusting from CloudWatch. A measured hello-world used **240 MB of 1024** (`Max Memory Used` in the invocation's REPORT line), so `512` is usually ample for small Workers — and since Lambda bills GB-seconds, halving memory halves the bill. Start at 1024, read the metric, then cut. + + + +**`--timeout` is a cost setting once it clears startup.** The per-SDK examples differ deliberately — 600 for Go, Python and TypeScript; 90 for Java — and both clear startup easily: a measured Java Worker bound its Task Queue **~10s** after `create-version`, JVM cold start included, against the 83s of polling a 90s deadline allows. Lambda's 3-second default is what fails this; 90 does not. + +Measured cold starts are ~1s for Python and Java alike (`Init Duration` in the REPORT line). + +| SDK example | Memory | Full invocation | GB-seconds | +|---|---|---|---| +| Java 90s / 1024 MB | 1 GB | ~84 s billed | ~84 | + +**Memory is the multiplier, not the deadline.** 1024 MB costs 4× per second at *any* deadline; the deadline only sets how many idle seconds you buy. That is the likeliest reason Java's example caps the tail at 90s, though it is inference rather than a documented rationale. Right-sizing beats it either way — the measured Worker used **240 MB of 1024**, so read `Max Memory Used` and cut. And `Init Duration` is billed, so a shorter deadline buys proportionally more billed inits. + +## Observability + +No extra dependency is required: `temporal-aws-lambda` already depends on `io.temporal:temporal-opentelemetry`, `io.opentelemetry:opentelemetry-api` (BOM 1.25.0), and `io.opentelemetry.contrib:opentelemetry-aws-xray`. The helper class ships inside the module. + +Import: `io.temporal.aws.lambda.OtelLambdaWorkerConfigurationHelper` + + + +- `configure(LambdaWorkerOptions.Builder)` — configures metrics and tracing with defaults. +- `configure(LambdaWorkerOptions.Builder, Consumer)` — same, with customization. +- `configureMetrics(LambdaWorkerOptions.Builder, OpenTelemetry)` — metrics only (an overload also takes a service name and a `Duration` report interval). +- `configureTracing(LambdaWorkerOptions.Builder, OpenTelemetry)` — tracing only. +- `configureFlushHook(LambdaWorkerOptions.Builder, OpenTelemetry, Duration)` — registers a flush before the invocation ends. + +Its own `newBuilder()` exposes `setOpenTelemetry`, `setEndpoint`, `setServiceName`, `setMetricsReportInterval`, `setFlushTimeout`, and `setFlushHook`. + +```java +LambdaWorker.define( + new WorkerDeploymentVersion("my-app", "build-1"), + builder -> { + builder.setTaskQueue("my-task-queue"); + builder.registerWorkflowImplementationTypes(MyWorkflowImpl.class); + builder.registerActivitiesImplementations(new MyActivitiesImpl()); + OtelLambdaWorkerConfigurationHelper.configure(builder); + }); +``` + +Defaults come from the constants `DEFAULT_OTLP_ENDPOINT` and `DEFAULT_SERVICE_NAME`, and the helper reads `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_SERVICE_NAME`, and `AWS_LAMBDA_FUNCTION_NAME` from the environment. As with the other SDKs, the default endpoint is the ADOT layer's collector on `localhost:4317`. + +**Flush before the deadline.** A Serverless Worker's invocation ends on a deadline rather than after a request, so telemetry buffered past that point is lost. Use `configureFlushHook` (or a report interval shorter than the invocation deadline) so metrics and spans are exported before shutdown. This matters more on Java's shorter recommended deadline (90s) than on the 600s used elsewhere. + +Attach the ADOT Collector layer. Because the OpenTelemetry SDK arrives as an ordinary Maven dependency of `temporal-aws-lambda`, no language-specific auto-instrumentation layer is required for the Worker's own telemetry — the same situation as Go. + +`OPENTELEMETRY_COLLECTOR_CONFIG_URI=/var/task/otel-collector-config.yaml` — the `_URI` form, as with Go and TypeScript. The official Java sample packages `otel-collector-config.template.yaml` into the artifact root as `otel-collector-config.yaml` during `shadowJar`; with Maven, add it under `src/main/resources`. + +For the shared Collector configuration, X-Ray enablement, and execution-role permissions, see `observability.md`. + +## Diagnostic signatures + +| SDK | Cause | Fix | +|---|---|---| +| Java | The SDK compiles against `slf4j-api` **1.7.36**; a 2.x provider (`slf4j-simple:2.x`, Logback 1.3+) does not bind to a 1.7 API and nothing is emitted | use a 1.7.x provider, e.g. `org.slf4j:slf4j-simple:1.7.36` | + +**Java — `NullPointerException` in `ShutdownManager` on every invocation (benign).** As of `temporal-aws-lambda` 1.38.0, a normal graceful shutdown logs a `WARN` with a full stack trace: + +``` +[main] WARN io.temporal.internal.worker.ShutdownManager - Exception during waiting for termination +java.lang.NullPointerException: Cannot invoke "SuspendableWorker.awaitTermination(long, TimeUnit)" + because "this.workerCommandWorker" is null + at io.temporal.worker.WorkerFactory.lambda$awaitTermination$12(WorkerFactory.java:519) + at io.temporal.aws.lambda.DefaultLambdaWorkerRuntime$DefaultInvocation.awaitTermination(...:82) + at io.temporal.aws.lambda.LambdaWorker$Handler.shutdownInvocation(LambdaWorker.java:323) +``` + +This is **not** a failure. It appears *after* Tasks have completed, is followed by `Temporal Lambda worker stopped`, a clean `END`/`REPORT`, and no timeout; Workflows complete correctly. Do not change configuration, IAM, or timeouts in response to it. Confirm it is benign by checking that the Workflow completed and that `REPORT` shows a duration below the deadline, then ignore it. + +**Java — `ClassNotFoundException` / `NoClassDefFoundError` at first invocation.** The uber-jar was built without merging `META-INF/services`, or the handler string is wrong. Check the handler format first: Java uses `fully.qualified.Class::method`, not the `module.function` form every other SDK uses. Then verify the services merge — `unzip -p META-INF/services/io.grpc.ManagedChannelProvider` should list more than one provider. → Build and package above. + +**Java — exec-format or `UnsupportedClassVersionError` at first invocation.** Bytecode targets a newer JDK than the runtime. Set `` (or the Gradle toolchain) to match `--runtime`. diff --git a/skills/temporal-serverless/references/aws-lambda/sdk-python.md b/skills/temporal-serverless/references/aws-lambda/sdk-python.md new file mode 100644 index 0000000..5d46172 --- /dev/null +++ b/skills/temporal-serverless/references/aws-lambda/sdk-python.md @@ -0,0 +1,206 @@ +# Python SDK on AWS Lambda + + + +Use this reference for Python SDK-specific package, entry-point, Worker configuration, tuned defaults, observability, and diagnostic details. For shared AWS Lambda deployment, observability infrastructure, and diagnostic flow, see `setup.md`, `observability.md`, and `diagnostics.md`. + +## Package + +Import: `from temporalio.contrib.aws.lambda_worker import LambdaWorkerConfig, run_worker` + +Install: `pip install temporalio` — the contrib module ships inside the main package here (unlike Go and TypeScript, which need a separate dependency). Use `temporalio[lambda-worker-otel]` for OpenTelemetry support. + +- Python: [Python Lambda Worker sample](https://github.com/temporalio/samples-python/tree/main/lambda_worker) + +Read the installed API before generating code: + +```bash +python -c "import temporalio.contrib.aws.lambda_worker as m; help(m.LambdaWorkerConfig)" +``` + +**Ordering when sources disagree:** the installed artifact first, the SDK's maintained samples second (they are built in CI, so they cannot reference a method that does not exist), the prose docs last. Entry-point names are not consistent across SDKs, so check rather than pattern-match from another language. + +**Fastest path:** start from the language sample linked above — it has a working Worker, Workflow, and Activity already wired together. The handler example below imports the Workflow and Activity from separate modules (`my_workflows`, `my_activities`). When writing from scratch, create those modules with at least one registered Workflow (declaring a versioning behavior) and one Activity, and name the entry-point file to match the `--handler` you deploy (for example, `lambda_function.py` → `--handler lambda_function.lambda_handler`). + +## Entry point + +`run_worker` — takes a `WorkerDeploymentVersion` and a configure callback, returns a Lambda handler. + +## Configure callback + +The `configure` callback receives a `LambdaWorkerConfig` dataclass with fields pre-populated with Lambda-appropriate defaults. Set the Task Queue, Workflows, and Activities through `worker_config`, which accepts the same keyword arguments as the `Worker` constructor. + +Go, Python and TypeScript invoke it per invocation. + +Python and TypeScript pass Workflow and Activity collections into the worker config. + +## Versioning behavior + +Set per-Workflow in the `@workflow.defn` decorator: `VersioningBehavior.PINNED` or `VersioningBehavior.AUTO_UPGRADE`. +Or set a Worker-level default with `default_versioning_behavior` in the worker config. + +**Worker Versioning is always on.** The run-worker entry point enables it, so the only remaining decision is `Pinned` vs `AutoUpgrade` per Workflow (or a Worker-level default). + +## Handler example + +Use the Python SDK's `lambda_worker` contrib package. + +```python +from temporalio.common import WorkerDeploymentVersion +from temporalio.contrib.aws.lambda_worker import LambdaWorkerConfig, run_worker + +from my_workflows import MyWorkflow +from my_activities import my_activity + + +def configure(config: LambdaWorkerConfig) -> None: + config.worker_config["task_queue"] = "my-task-queue" + config.worker_config["workflows"] = [MyWorkflow] + config.worker_config["activities"] = [my_activity] + + +lambda_handler = run_worker( + WorkerDeploymentVersion( + deployment_name="my-app", + build_id="build-1", + ), + configure, +) +``` + + +Versioning behavior: set per-Workflow in the `@workflow.defn` decorator with `VersioningBehavior.PINNED` or `VersioningBehavior.AUTO_UPGRADE`, or set a Worker-level default with `default_versioning_behavior` in the worker config. + +```python +from temporalio import workflow +from temporalio.common import VersioningBehavior + + +@workflow.defn(versioning_behavior=VersioningBehavior.PINNED) +class MyWorkflow: + @workflow.run + async def run(self, input: str) -> str: + ... +``` + + +## Lambda-tuned defaults + + + +| Setting | Lambda default | +|---|---| +| `max_concurrent_activities` | 2 | +| `max_concurrent_workflow_tasks` | 10 | +| `max_concurrent_local_activities` | 2 | +| `max_concurrent_nexus_tasks` | 5 | +| `workflow_task_poller_behavior` | `SimpleMaximum(2)` | +| `activity_task_poller_behavior` | `SimpleMaximum(1)` | +| `nexus_task_poller_behavior` | `SimpleMaximum(1)` | +| `graceful_shutdown_timeout` | 5 seconds | +| `max_cached_workflows` | 30 | +| `disable_eager_activity_execution` | Always `True` | +| `shutdown_deadline_buffer` | 7 seconds | + +`disable_eager_activity_execution` is always `True` and cannot be overridden. Eager Activities require a persistent connection, which Lambda invocations don't maintain. + +`shutdown_deadline_buffer` is specific to the `lambda_worker` package. It controls how much time before the Lambda deadline the Worker begins its graceful shutdown. The default is `graceful_shutdown_timeout` + 2 seconds. + +If your Worker handles long-running Activities, increase `graceful_shutdown_timeout`, `shutdown_deadline_buffer`, and the Lambda invocation deadline (`--timeout`) together. + +## Connection configuration + +The `lambda_worker` package automatically loads Temporal client configuration from a TOML config file and environment variables (see the Environment Configuration docs, `/develop/environment-configuration`). + +TOML config file resolution order: + +1. `TEMPORAL_CONFIG_FILE` environment variable, if set. +2. `temporal.toml` in `$LAMBDA_TASK_ROOT` (typically `/var/task`). +3. `temporal.toml` in the current working directory. + +The file is optional. If absent, only environment variables are used. + +## Build and package + +Install dependencies into a local directory for packaging, using `--platform` for Linux-compatible binaries: + +```bash +pip install --target ./package --platform manylinux2014_x86_64 --only-binary=:all: temporalio +``` + + +**Pin the download to the Lambda runtime's Python version and architecture, not your local interpreter's.** If they differ (e.g. local `3.14` vs the function's `python3.13`), add `--python-version 3.13` alongside `--only-binary=:all:` so pip fetches runtime-matching wheels, and keep `--platform` (`manylinux2014_x86_64` for `x86_64`, `manylinux2014_aarch64` for `arm64`) consistent with the function's `--architectures`. Mismatches surface as import errors only at invocation time, not at package time. + +To include OpenTelemetry support, install `temporalio[lambda-worker-otel]` instead. + +Package dependencies and application code: + +```bash +cd package && zip -r ../function.zip . && cd .. +zip function.zip lambda_function.py my_workflows.py my_activities.py +``` + + +## Deploy the Lambda function + +```bash +aws lambda create-function \ + --function-name my-temporal-worker \ + --runtime python3.13 \ + --handler lambda_function.lambda_handler \ + --role \ + --zip-file fileb://function.zip \ + --timeout 600 \ + --memory-size 256 \ + --environment '{"Variables":{"TEMPORAL_ADDRESS":":7233","TEMPORAL_NAMESPACE":"","TEMPORAL_API_KEY":""}}' +``` + + +- `--runtime`: `python3.13` (or another supported Python version). +- `--handler`: `lambda_function.lambda_handler` (entry point in `module.function` format, must point to the handler returned by `run_worker`). + +| SDK example | Memory | Full invocation | GB-seconds | +|---|---|---|---| +| Python 600s / 256 MB | 0.25 GB | ~594 s billed | ~149 | + +Measured cold starts are ~1s for Python and Java alike (`Init Duration` in the REPORT line). + +The `--environment` examples above pass `TEMPORAL_API_KEY` inline for brevity — **that is acceptable for development only.** For production, store the API key (or TLS private key) in AWS Secrets Manager or SSM Parameter Store, grant the *execution* role `secretsmanager:GetSecretValue` (or `ssm:GetParameter`), and load it at cold start before the Worker initializes — for example, at module scope in the handler file, fetch the secret and set `os.environ["TEMPORAL_API_KEY"]` so the serverless Worker package reads it at startup. Do not commit key values into the `--environment` block for production functions. + +## Observability + +Import: `from temporalio.contrib.aws.lambda_worker.otel import apply_defaults` + +To install with OTel support: `pip install temporalio[lambda-worker-otel]` + +- `apply_defaults` — configures both metrics and tracing. +- `build_metrics_telemetry_config` — configures metrics only. +- `apply_tracing` — configures tracing only. + +Usage in the configure callback: + +```python +def configure(config: LambdaWorkerConfig) -> None: + config.worker_config["task_queue"] = TASK_QUEUE + config.worker_config["workflows"] = [SampleWorkflow] + config.worker_config["activities"] = [hello_activity] + apply_defaults(config) +``` + +By default, telemetry is sent to `localhost:4317`, which is the ADOT Lambda layer's default collector endpoint. + +Attach the ADOT Python Lambda layer to your Lambda function. The layer includes both auto-instrumentation and an OpenTelemetry Collector that receives telemetry on `localhost:4317` and forwards traces to AWS X-Ray and metrics to Amazon CloudWatch. + +`OPENTELEMETRY_COLLECTOR_CONFIG_FILE=/var/task/otel-collector-config.yaml` + +Note: Python uses `_FILE` while Go and TypeScript use `_URI`. + +For Python, the `AWSXRayDaemonWriteAccess` managed policy can be attached instead. + +For the shared Collector configuration, X-Ray enablement, and execution-role permissions, see `observability.md`. + +## Diagnostic signatures + +| SDK | Cause | Fix | +|---|---|---| +| Python | `logging.basicConfig()` is a no-op when a root handler already exists, and the Lambda runtime installs one before your module is imported — so the level never changes and `INFO` records are filtered out | `logging.getLogger().setLevel(logging.INFO)` | diff --git a/skills/temporal-serverless/references/aws-lambda/sdk-typescript.md b/skills/temporal-serverless/references/aws-lambda/sdk-typescript.md new file mode 100644 index 0000000..a89a51e --- /dev/null +++ b/skills/temporal-serverless/references/aws-lambda/sdk-typescript.md @@ -0,0 +1,201 @@ +# TypeScript SDK on AWS Lambda + + + +Use this reference for TypeScript SDK-specific package, entry-point, Worker configuration, tuned defaults, and observability details. For shared AWS Lambda deployment and observability infrastructure, see `setup.md` and `observability.md`. + +## Package + +Import: `import { runWorker } from '@temporalio/lambda-worker'` + +Install: `npm install @temporalio/lambda-worker` — a separate npm package from `@temporalio/worker`, versioned independently. + +- TypeScript: [TypeScript Lambda Worker sample](https://github.com/temporalio/samples-typescript/tree/main/lambda-worker) + +Check the installed version, then read its type declarations: + +```bash +npm ls @temporalio/lambda-worker +``` + +**Ordering when sources disagree:** the installed artifact first, the SDK's maintained samples second (they are built in CI, so they cannot reference a method that does not exist), the prose docs last. Entry-point names are not consistent across SDKs, so check rather than pattern-match from another language. + +## Entry point + +`runWorker` — creates a Lambda handler that runs a Temporal Worker. Pass a deployment version and a configure callback. + +## Configure callback + +Set Worker options via `config.workerOptions`. For Workflow code, use `workflowBundle` with pre-bundled code instead of `workflowsPath` to avoid webpack bundling overhead on Lambda cold starts. + +Go, Python and TypeScript invoke it per invocation. + +Python and TypeScript pass Workflow and Activity collections into the worker config. + +## Pre-bundling Workflow code + +Build the bundle as a separate build step: + +```typescript +import { bundleWorkflowCode } from '@temporalio/worker'; +import { writeFile } from 'fs/promises'; + +const { code } = await bundleWorkflowCode({ + workflowsPath: require.resolve('./workflows'), +}); +await writeFile('./workflow-bundle.js', code); +``` + + +Then reference the bundle in your handler with `workflowBundle: { codePath: require.resolve('./workflow-bundle.js') }`. + +## Versioning behavior + +Set per-Workflow with `setWorkflowOptions` in the Workflow file, or set a default for all Workflows with `defaultVersioningBehavior` in the configure callback. +Values are `'PINNED'` or `'AUTO_UPGRADE'`. The default versioning behavior is `PINNED`. + +Access via: `config.workerOptions.workerDeploymentOptions!.defaultVersioningBehavior = 'PINNED'` + +**Worker Versioning is always on.** The run-worker entry point enables it, so the only remaining decision is `Pinned` vs `AutoUpgrade` per Workflow (or a Worker-level default). + +## Handler example + +Use the `@temporalio/lambda-worker` package. + +```typescript +import { runWorker } from '@temporalio/lambda-worker'; +import * as activities from './activities'; + +export const handler = runWorker({ deploymentName: 'my-app', buildId: 'build-1' }, (config) => { + config.workerOptions.taskQueue = 'my-task-queue'; + config.workerOptions.workflowBundle = { + codePath: require.resolve('./workflow-bundle.js'), + }; + config.workerOptions.activities = activities; + config.workerOptions.workerDeploymentOptions!.defaultVersioningBehavior = 'PINNED'; +}); +``` + + +Use `workflowBundle` with pre-bundled code instead of `workflowsPath` to avoid webpack bundling overhead on Lambda cold starts. + +Versioning behavior: set per-Workflow with `setWorkflowOptions` in the Workflow file, or set a default for all Workflows with `defaultVersioningBehavior` in the configure callback. Values are `'AUTO_UPGRADE'` or `'PINNED'`. + +## Lambda-tuned defaults + + + +| Setting | Lambda default | +|---|---| +| `maxConcurrentActivityTaskExecutions` | 2 | +| `maxConcurrentWorkflowTaskExecutions` | 10 | +| `maxConcurrentLocalActivityExecutions` | 2 | +| `maxConcurrentNexusTaskExecutions` | 5 | +| `workflowTaskPollerBehavior` | `SimpleMaximum(2)` | +| `activityTaskPollerBehavior` | `SimpleMaximum(1)` | +| `nexusTaskPollerBehavior` | `SimpleMaximum(1)` | +| `shutdownGraceTime` | 5 seconds | +| `maxCachedWorkflows` | 30 | +| `shutdownDeadlineBufferMs` | 7000 | + +Eager Activities are not supported. Lambda invocations don't maintain persistent connections. + +`shutdownDeadlineBufferMs` is specific to the `@temporalio/lambda-worker` package. It controls how much time before the Lambda deadline the Worker begins its graceful shutdown. The default is `shutdownGraceTime` (5s) + 2s. + +If your Worker handles long-running Activities, increase `shutdownGraceTime`, `shutdownDeadlineBufferMs`, and the Lambda invocation deadline (`--timeout`) together. + +## Connection configuration + +The `@temporalio/lambda-worker` package automatically loads Temporal client configuration from a TOML config file and environment variables (see the Environment Configuration docs, `/develop/environment-configuration`). + +TOML config file resolution order: + +1. `TEMPORAL_CONFIG_FILE` environment variable, if set. +2. `temporal.toml` in `$LAMBDA_TASK_ROOT` (typically `/var/task`). +3. `temporal.toml` in the current working directory. + +The file is optional. If absent, only environment variables are used. + +## Build and package + +Build the Workflow bundle and compile the project: + +```bash +npx ts-node src/scripts/build-workflow-bundle.ts +npx tsc +``` + + +Install production dependencies and package everything: + +```bash +npm install --omit=dev +zip -r function.zip lib/ node_modules/ workflow-bundle.js +``` + + +## Deploy the Lambda function + +```bash +aws lambda create-function \ + --function-name my-temporal-worker \ + --runtime nodejs22.x \ + --handler lib/index.handler \ + --role \ + --zip-file fileb://function.zip \ + --timeout 600 \ + --memory-size 256 \ + --environment '{"Variables":{"HOME":"/tmp","TEMPORAL_ADDRESS":":7233","TEMPORAL_NAMESPACE":"","TEMPORAL_API_KEY":""}}' +``` + + +- `--runtime`: `nodejs22.x` (or another supported Node.js version, 20+). +- `--handler`: `lib/index.handler` (entry point in `module.export` format, must point to the handler exported by `runWorker`). + +| Variable | Description | +|---|---| +| `HOME` | Set to `/tmp` in the Go and TypeScript examples above. Lambda's filesystem is read-only outside `/tmp`, so anything the runtime or config loader resolves relative to the home directory needs a writable target. The docs omit it from the Python example; including it there is harmless. | + +## Observability + +Import: `import { applyDefaults } from '@temporalio/lambda-worker/otel'` + +- `applyDefaults` — registers Temporal SDK interceptors for tracing and configures the Core SDK to export metrics via OTLP. +- `makeOtelPlugin` — returns a plugin for pre-bundling Workflow code that includes Workflow interceptor modules. + +Usage in the configure callback: + +```typescript +export const handler = runWorker({ deploymentName: 'sdk-demo', buildId: 'v1' }, (config) => { + config.workerOptions.taskQueue = TASK_QUEUE; + config.workerOptions.workflowBundle = { + codePath: require.resolve('./workflow-bundle.js'), + }; + config.workerOptions.activities = activities; + applyDefaults(config); +}); +``` + +When pre-bundling Workflow code, pass the plugin from `makeOtelPlugin()` so that Workflow interceptor modules are included in the bundle: + +```typescript +import { bundleWorkflowCode } from '@temporalio/worker'; +import { makeOtelPlugin } from '@temporalio/lambda-worker/otel'; + +const { plugin } = makeOtelPlugin(); +const { code } = await bundleWorkflowCode({ + workflowsPath: require.resolve('./workflows'), + plugins: [plugin], +}); +``` + +By default, telemetry is sent to `localhost:4317`, which is the ADOT Lambda layer's default collector endpoint. + +Attach two ADOT Lambda layers: + +1. The ADOT JavaScript layer for Node.js-side auto-instrumentation and trace export. +2. The ADOT Collector layer (`aws-otel-collector-amd64`) to run the OTel Collector as a Lambda extension, receiving telemetry via OTLP on `localhost:4317` and forwarding traces to X-Ray and metrics to CloudWatch. + +`OPENTELEMETRY_COLLECTOR_CONFIG_URI=/var/task/otel-collector-config.yaml` + +For the shared Collector configuration, X-Ray enablement, and execution-role permissions, see `observability.md`. diff --git a/skills/temporal-serverless/references/aws-lambda/setup.md b/skills/temporal-serverless/references/aws-lambda/setup.md index a026a84..30083bd 100644 --- a/skills/temporal-serverless/references/aws-lambda/setup.md +++ b/skills/temporal-serverless/references/aws-lambda/setup.md @@ -16,13 +16,10 @@ This is the end-to-end golden path: connect, write the Worker, package and deplo - Every Workflow must declare a versioning behavior, or the Worker must set a default versioning behavior. - An AWS account with permissions to create and invoke Lambda functions and create IAM roles. For the exact operator actions and a preflight check, see `iam.md`. - The AWS-specific steps require the `aws` CLI installed and configured with your AWS credentials. You may also use the AWS Console or the AWS SDKs. -- The Go SDK, Python SDK, or TypeScript SDK, depending on your language. +- The Go SDK, Python SDK, TypeScript SDK, Java SDK, or .NET SDK, depending on your language. - The `temporal` CLI, authenticated to the target Temporal Service — Steps 4–6 and the CLI troubleshooting paths use it. See "Temporal CLI and Cloud connection" below. -Sample projects: -- Go: [Go Lambda Worker sample](https://github.com/temporalio/samples-go/tree/main/lambda-worker) -- Python: [Python Lambda Worker sample](https://github.com/temporalio/samples-python/tree/main/lambda_worker) -- TypeScript: [TypeScript Lambda Worker sample](https://github.com/temporalio/samples-typescript/tree/main/lambda-worker) +The selected SDK reference links its maintained sample project. ## Temporal CLI and Cloud connection @@ -118,215 +115,25 @@ Do not proceed to Steps 4–6 on the assumption auth will work — re-run this c The Worker handles the per-invocation lifecycle: connecting to Temporal, polling for tasks, and gracefully shutting down before the invocation deadline. -### Install the serverless Worker package first +Load the reference for the selected SDK alongside this shared deployment guide: -**The serverless Worker package is not always part of the main SDK.** Install it explicitly before writing code — do not assume an `import` resolves just because the base SDK is present. Scaffolding a project and discovering only at build time that the package lives in its own module means backing out and redoing the module setup. - -| SDK | Install | Packaging | -|---|---|---| -| Go | `go get go.temporal.io/sdk/contrib/aws/lambdaworker` | **Separate Go module** from `go.temporal.io/sdk`, with its own version line (`v0.1.1` at the time of writing). It is *not* pulled in by the main SDK — `go get` it directly, then `go mod tidy`. | -| Python | `pip install temporalio` | `temporalio.contrib.aws.lambda_worker` ships inside the main `temporalio` package. Use `temporalio[lambda-worker-otel]` to add OpenTelemetry. | -| TypeScript | `npm install @temporalio/lambda-worker` | Separate npm package from `@temporalio/worker`, versioned independently. | - -### Verify the installed API before generating code - -These are Public Preview APIs and signatures drift between versions. Read the real surface of the version you just installed rather than writing from memory — a wrong field name costs a build cycle: - -```bash -# Go — list the exported API of the installed module version -go doc go.temporal.io/sdk/contrib/aws/lambdaworker -go doc go.temporal.io/sdk/contrib/aws/lambdaworker.Options - -# Python -python -c "import temporalio.contrib.aws.lambda_worker as m; help(m.LambdaWorkerConfig)" - -# TypeScript — check the installed version, then read its type declarations -npm ls @temporalio/lambda-worker -``` +| SDK | Reference | +|---|---| +| Go | `sdk-go.md` | +| Python | `sdk-python.md` | +| TypeScript | `sdk-typescript.md` | +| Java | `sdk-java.md` | +| .NET | `sdk-dotnet.md` | -Specifics worth confirming this way, because they differ by SDK and are easy to get wrong from memory: +Complete the selected SDK reference before continuing. -- **Where the Task Queue lives.** In Go it is a direct field on the options object (`opts.TaskQueue`). In Python it goes through the worker-config mapping (`config.worker_config["task_queue"]`), and in TypeScript through worker options (`config.workerOptions.taskQueue`). Do not carry one shape over to another language. -- **Where registration happens.** In Go the `Register*` methods hang off the same options object; Python and TypeScript pass Workflow and Activity collections into the worker config. - **You do not construct a client.** Connection details (address, namespace, API key) load automatically from the process environment, so `TEMPORAL_*` variables set on the function flow straight through with no client code. In a Lambda that means the `--environment` block at deploy time: no config file is bundled unless you put one there, and the operator's own CLI configuration never reaches the function (see "Operator CLI config does not reach the function" below). -- **Worker Versioning is always on.** The run-worker entry point enables it, so the only remaining decision is `Pinned` vs `AutoUpgrade` per Workflow (or a Worker-level default). - -**Fastest path:** start from the language sample linked in Prerequisites — it has a working Worker, Workflow, and Activity already wired together. The handler examples below import the Workflow and Activity from separate modules (`my_workflows`, `my_activities`). When writing from scratch, create those modules with at least one registered Workflow (declaring a versioning behavior) and one Activity, and name the entry-point file to match the `--handler` you deploy (for example, `lambda_function.py` → `--handler lambda_function.lambda_handler`). - -### Go - -Use the Go SDK's `lambdaworker` package. - -```go -package main - -import ( - lambdaworker "go.temporal.io/sdk/contrib/aws/lambdaworker" - "go.temporal.io/sdk/worker" - "go.temporal.io/sdk/workflow" -) - -func main() { - lambdaworker.RunWorker(worker.WorkerDeploymentVersion{ - DeploymentName: "my-app", - BuildID: "build-1", - }, func(opts *lambdaworker.Options) error { - opts.TaskQueue = "my-task-queue" - - opts.RegisterWorkflowWithOptions(MyWorkflow, workflow.RegisterOptions{ - VersioningBehavior: workflow.VersioningBehaviorPinned, - }) - opts.RegisterActivity(MyActivity) - - return nil - }) -} -``` - - -Versioning behavior: set per-Workflow at registration time with `workflow.VersioningBehaviorPinned` or `workflow.VersioningBehaviorAutoUpgrade`, or set a Worker-level default with `DefaultVersioningBehavior` in `DeploymentOptions`. - -### Python - -Use the Python SDK's `lambda_worker` contrib package. - -```python -from temporalio.common import WorkerDeploymentVersion -from temporalio.contrib.aws.lambda_worker import LambdaWorkerConfig, run_worker - -from my_workflows import MyWorkflow -from my_activities import my_activity - - -def configure(config: LambdaWorkerConfig) -> None: - config.worker_config["task_queue"] = "my-task-queue" - config.worker_config["workflows"] = [MyWorkflow] - config.worker_config["activities"] = [my_activity] - - -lambda_handler = run_worker( - WorkerDeploymentVersion( - deployment_name="my-app", - build_id="build-1", - ), - configure, -) -``` - - -Versioning behavior: set per-Workflow in the `@workflow.defn` decorator with `VersioningBehavior.PINNED` or `VersioningBehavior.AUTO_UPGRADE`, or set a Worker-level default with `default_versioning_behavior` in the worker config. - -```python -from temporalio import workflow -from temporalio.common import VersioningBehavior - - -@workflow.defn(versioning_behavior=VersioningBehavior.PINNED) -class MyWorkflow: - @workflow.run - async def run(self, input: str) -> str: - ... -``` - - -### TypeScript - -Use the `@temporalio/lambda-worker` package. - -```typescript -import { runWorker } from '@temporalio/lambda-worker'; -import * as activities from './activities'; - -export const handler = runWorker({ deploymentName: 'my-app', buildId: 'build-1' }, (config) => { - config.workerOptions.taskQueue = 'my-task-queue'; - config.workerOptions.workflowBundle = { - codePath: require.resolve('./workflow-bundle.js'), - }; - config.workerOptions.activities = activities; - config.workerOptions.workerDeploymentOptions!.defaultVersioningBehavior = 'PINNED'; -}); -``` - - -Use `workflowBundle` with pre-bundled code instead of `workflowsPath` to avoid webpack bundling overhead on Lambda cold starts. - -Versioning behavior: set per-Workflow with `setWorkflowOptions` in the Workflow file, or set a default for all Workflows with `defaultVersioningBehavior` in the configure callback. Values are `'AUTO_UPGRADE'` or `'PINNED'`. ## Step 2: Deploy Lambda function ### Build and package -#### Go - -Cross-compile for Lambda's Linux runtime: - -```bash -GOOS=linux GOARCH=amd64 go build -tags lambda.norpc -o bootstrap ./worker -``` - - -Package the binary into a zip file: - -```bash -zip function.zip bootstrap -``` - - -**Add `CGO_ENABLED=0`, and match the architecture you deploy.** The `provided.al2023` runtime expects a self-contained binary; building with cgo enabled links against host libraries that may not resolve inside the runtime. Set `CGO_ENABLED=0` for a statically linked binary, and keep `GOARCH` consistent with the function's `--architectures` (`amd64` ↔ `x86_64`, `arm64` ↔ `arm64`). Also adjust the trailing package path to your layout — `.` when `main` is in the repo root, `./worker` when it is in a `worker/` subdirectory. A reusable script: - -```bash -#!/usr/bin/env bash -set -euo pipefail -go vet ./... # catches a missing import before the cross-compile -CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -tags lambda.norpc -o bootstrap . -zip -q function.zip bootstrap -file bootstrap # expect: ELF 64-bit ... statically linked -``` - -Run `go vet` (or a plain `go build ./...`) before the packaging build. The three-package import block above — `lambdaworker`, `worker` for `WorkerDeploymentVersion`, and `workflow` for the versioning-behavior constants — is easy to write short by one entry, and catching that locally is faster than discovering it in the cross-compile step. - -An architecture mismatch surfaces only at invocation time as an `Runtime.InvalidEntrypoint`/exec-format error, not at build or package time — the same failure class as the Python wheel mismatch below. - -A typical Go Worker zip lands around 10–15 MB, well under the 50 MB direct-upload limit. - -#### Python - -Install dependencies into a local directory for packaging, using `--platform` for Linux-compatible binaries: - -```bash -pip install --target ./package --platform manylinux2014_x86_64 --only-binary=:all: temporalio -``` - - -**Pin the download to the Lambda runtime's Python version and architecture, not your local interpreter's.** If they differ (e.g. local `3.14` vs the function's `python3.13`), add `--python-version 3.13` alongside `--only-binary=:all:` so pip fetches runtime-matching wheels, and keep `--platform` (`manylinux2014_x86_64` for `x86_64`, `manylinux2014_aarch64` for `arm64`) consistent with the function's `--architectures`. Mismatches surface as import errors only at invocation time, not at package time. - -To include OpenTelemetry support, install `temporalio[lambda-worker-otel]` instead. - -Package dependencies and application code: - -```bash -cd package && zip -r ../function.zip . && cd .. -zip function.zip lambda_function.py my_workflows.py my_activities.py -``` - - -#### TypeScript - -Build the Workflow bundle and compile the project: - -```bash -npx ts-node src/scripts/build-workflow-bundle.ts -npx tsc -``` - - -Install production dependencies and package everything: - -```bash -npm install --omit=dev -zip -r function.zip lib/ node_modules/ workflow-bundle.js -``` - +See the selected SDK reference's **Build and package** section. ### Deploy the Lambda function @@ -357,59 +164,7 @@ rm /tmp/lambda-env.json This is still a plaintext env var on the function — acceptable for a development walkthrough only, and only if you say so explicitly. See "Environment variables" below for the production pattern. -#### Go - -```bash -aws lambda create-function \ - --function-name my-temporal-worker \ - --runtime provided.al2023 \ - --handler bootstrap \ - --role \ - --zip-file fileb://function.zip \ - --timeout 600 \ - --memory-size 256 \ - --environment '{"Variables":{"HOME":"/tmp","TEMPORAL_ADDRESS":":7233","TEMPORAL_NAMESPACE":"","TEMPORAL_API_KEY":""}}' -``` - - -- `--runtime`: `provided.al2023` for custom Go binaries. -- `--handler`: `bootstrap` when using the `provided.al2023` custom runtime. - -#### Python - -```bash -aws lambda create-function \ - --function-name my-temporal-worker \ - --runtime python3.13 \ - --handler lambda_function.lambda_handler \ - --role \ - --zip-file fileb://function.zip \ - --timeout 600 \ - --memory-size 256 \ - --environment '{"Variables":{"TEMPORAL_ADDRESS":":7233","TEMPORAL_NAMESPACE":"","TEMPORAL_API_KEY":""}}' -``` - - -- `--runtime`: `python3.13` (or another supported Python version). -- `--handler`: `lambda_function.lambda_handler` (entry point in `module.function` format, must point to the handler returned by `run_worker`). - -#### TypeScript - -```bash -aws lambda create-function \ - --function-name my-temporal-worker \ - --runtime nodejs22.x \ - --handler lib/index.handler \ - --role \ - --zip-file fileb://function.zip \ - --timeout 600 \ - --memory-size 256 \ - --environment '{"Variables":{"HOME":"/tmp","TEMPORAL_ADDRESS":":7233","TEMPORAL_NAMESPACE":"","TEMPORAL_API_KEY":""}}' -``` - - -- `--runtime`: `nodejs22.x` (or another supported Node.js version, 20+). -- `--handler`: `lib/index.handler` (entry point in `module.export` format, must point to the handler exported by `runWorker`). +Run the selected SDK reference's deployment command before continuing. ### Wait for the function to become Active @@ -439,13 +194,20 @@ aws lambda get-function --function-name my-temporal-worker \ **Caution:** AWS Lambda functions default to a 3-second timeout, which is too short for the Worker to start, connect to Temporal, and register the Task Queue. If the first invocation times out before the Worker polls, the Task Queue binding is never created and the Lambda is never invoked again. Always set `--timeout` high enough for the Worker to start, process Tasks, and shut down gracefully. +Set it from three constraints: + +1. **> cold start + connect + Task Queue registration.** The binding requirement, and what rules out the 3s default. +2. **> longest Activity + shutdown deadline buffer.** An Activity still running when the Worker drains is abandoned and retried. +3. **Beyond those, pure cost.** Longer: fewer invocations and cold starts, warmer sticky cache, room for longer Activities. Shorter: a smaller idle tail — when work stops the Worker polls on until its deadline, so the waste is one deadline's worth. + +Lambda bills **GB-seconds** — allocated memory × billed duration, however idle the Worker was. + ### Environment variables | Variable | Description | |---|---| -| `HOME` | Set to `/tmp` in the Go and TypeScript examples above. Lambda's filesystem is read-only outside `/tmp`, so anything the runtime or config loader resolves relative to the home directory needs a writable target. The docs omit it from the Python example; including it there is harmless. | | `TEMPORAL_ADDRESS` | Temporal frontend address (e.g., `..tmprl.cloud:7233`). | | `TEMPORAL_NAMESPACE` | Temporal Namespace. For Temporal Cloud, the fully-qualified `.`, not the bare name. | | `TEMPORAL_TASK_QUEUE` | Task Queue name. Overrides the value set in code. | @@ -457,8 +219,6 @@ The serverless Worker packages read environment variables and configuration file Sensitive values like TLS keys and API keys should be encrypted at rest. -The `--environment` examples above pass `TEMPORAL_API_KEY` inline for brevity — **that is acceptable for development only.** For production, store the API key (or TLS private key) in AWS Secrets Manager or SSM Parameter Store, grant the *execution* role `secretsmanager:GetSecretValue` (or `ssm:GetParameter`), and load it at cold start before the Worker initializes — for example, at module scope in the handler file, fetch the secret and set `os.environ["TEMPORAL_API_KEY"]` so the serverless Worker package reads it at startup. Do not commit key values into the `--environment` block for production functions. - For updating the function code and publishing immutable versions, see `versioning.md`. ## Step 3: Configure IAM for Temporal invocation diff --git a/skills/temporal-serverless/references/sdk-configuration.md b/skills/temporal-serverless/references/sdk-configuration.md deleted file mode 100644 index c0ccca4..0000000 --- a/skills/temporal-serverless/references/sdk-configuration.md +++ /dev/null @@ -1,221 +0,0 @@ -# SDK Configuration for Serverless Workers - - - -## Go SDK - -### Package - -Import: `lambdaworker "go.temporal.io/sdk/contrib/aws/lambdaworker"` - -Install: `go get go.temporal.io/sdk/contrib/aws/lambdaworker` — **this is a separate Go module** from `go.temporal.io/sdk`, versioned independently (`v0.1.1` at the time of writing). Having the main SDK in `go.mod` does not make it importable; add it explicitly, then `go mod tidy`. Verify the installed surface with `go doc go.temporal.io/sdk/contrib/aws/lambdaworker` before generating code — the API is Public Preview and drifts. - -### Entry point - -`lambdaworker.RunWorker` — starts a Lambda-based Worker. Pass a `WorkerDeploymentVersion` and a callback that registers Workflows and Activities. - -### Configure callback - -The `Options` callback gives access to the same registration methods as a traditional Worker: `RegisterWorkflow`, `RegisterWorkflowWithOptions`, `RegisterActivity`, `RegisterActivityWithOptions`, and `RegisterNexusService`. - -### Versioning behavior - -Set per-Workflow at registration time with `workflow.VersioningBehaviorPinned` or `workflow.VersioningBehaviorAutoUpgrade`. -Or set a Worker-level default with `DefaultVersioningBehavior` in `DeploymentOptions`. - -### Lambda-tuned defaults - - - -| Setting | Lambda default | -|---|---| -| `MaxConcurrentActivityExecutionSize` | 2 | -| `MaxConcurrentWorkflowTaskExecutionSize` | 10 | -| `MaxConcurrentLocalActivityExecutionSize` | 2 | -| `MaxConcurrentNexusTaskExecutionSize` | 5 | -| `MaxConcurrentActivityTaskPollers` | 1 | -| `MaxConcurrentWorkflowTaskPollers` | 2 | -| `MaxConcurrentNexusTaskPollers` | 1 | -| `WorkerStopTimeout` | 5 seconds | -| `DisableEagerActivities` | Always true | -| Sticky cache size | 100 | -| `ShutdownDeadlineBuffer` | 7 seconds | - -These are the same `worker.Options` available to any Temporal Worker, just with lower values for Lambda's constrained environment. Except for `ShutdownDeadlineBuffer`, which is specific to the `lambdaworker` package. - -`DisableEagerActivities` is always true and cannot be overridden. Eager Activities require a persistent connection, which Lambda invocations don't maintain. - -`ShutdownDeadlineBuffer` controls how much time before the Lambda deadline the Worker begins its graceful shutdown. The default is `WorkerStopTimeout` + 2 seconds. - -If your Worker handles long-running Activities, increase `WorkerStopTimeout`, `ShutdownDeadlineBuffer`, and the Lambda invocation deadline (`--timeout`) together. - -### Connection configuration - -The `lambdaworker` package automatically loads Temporal client configuration from a TOML config file and environment variables (see the Environment Configuration docs, `/develop/environment-configuration`). - -TOML config file resolution order: - -1. `TEMPORAL_CONFIG_FILE` environment variable, if set. -2. `temporal.toml` in `$LAMBDA_TASK_ROOT` (typically `/var/task`). -3. `temporal.toml` in the current working directory. - -The file is optional. If absent, only environment variables are used. - ---- - -## Python SDK - -### Package - -Import: `from temporalio.contrib.aws.lambda_worker import LambdaWorkerConfig, run_worker` - -Install: `pip install temporalio` — the contrib module ships inside the main package here (unlike Go and TypeScript, which need a separate dependency). Use `temporalio[lambda-worker-otel]` for OpenTelemetry support. - -### Entry point - -`run_worker` — takes a `WorkerDeploymentVersion` and a configure callback, returns a Lambda handler. - -### Configure callback - -The `configure` callback receives a `LambdaWorkerConfig` dataclass with fields pre-populated with Lambda-appropriate defaults. Set the Task Queue, Workflows, and Activities through `worker_config`, which accepts the same keyword arguments as the `Worker` constructor. - -### Versioning behavior - -Set per-Workflow in the `@workflow.defn` decorator: `VersioningBehavior.PINNED` or `VersioningBehavior.AUTO_UPGRADE`. -Or set a Worker-level default with `default_versioning_behavior` in the worker config. - -### Lambda-tuned defaults - - - -| Setting | Lambda default | -|---|---| -| `max_concurrent_activities` | 2 | -| `max_concurrent_workflow_tasks` | 10 | -| `max_concurrent_local_activities` | 2 | -| `max_concurrent_nexus_tasks` | 5 | -| `workflow_task_poller_behavior` | `SimpleMaximum(2)` | -| `activity_task_poller_behavior` | `SimpleMaximum(1)` | -| `nexus_task_poller_behavior` | `SimpleMaximum(1)` | -| `graceful_shutdown_timeout` | 5 seconds | -| `max_cached_workflows` | 30 | -| `disable_eager_activity_execution` | Always `True` | -| `shutdown_deadline_buffer` | 7 seconds | - -`disable_eager_activity_execution` is always `True` and cannot be overridden. Eager Activities require a persistent connection, which Lambda invocations don't maintain. - -`shutdown_deadline_buffer` is specific to the `lambda_worker` package. It controls how much time before the Lambda deadline the Worker begins its graceful shutdown. The default is `graceful_shutdown_timeout` + 2 seconds. - -If your Worker handles long-running Activities, increase `graceful_shutdown_timeout`, `shutdown_deadline_buffer`, and the Lambda invocation deadline (`--timeout`) together. - -### Connection configuration - -The `lambda_worker` package automatically loads Temporal client configuration from a TOML config file and environment variables (see the Environment Configuration docs, `/develop/environment-configuration`). - -TOML config file resolution order: - -1. `TEMPORAL_CONFIG_FILE` environment variable, if set. -2. `temporal.toml` in `$LAMBDA_TASK_ROOT` (typically `/var/task`). -3. `temporal.toml` in the current working directory. - -The file is optional. If absent, only environment variables are used. - ---- - -## TypeScript SDK - -### Package - -Import: `import { runWorker } from '@temporalio/lambda-worker'` - -Install: `npm install @temporalio/lambda-worker` — a separate npm package from `@temporalio/worker`, versioned independently. - -### Entry point - -`runWorker` — creates a Lambda handler that runs a Temporal Worker. Pass a deployment version and a configure callback. - -### Configure callback - -Set Worker options via `config.workerOptions`. For Workflow code, use `workflowBundle` with pre-bundled code instead of `workflowsPath` to avoid webpack bundling overhead on Lambda cold starts. - -### Pre-bundling Workflow code - -Build the bundle as a separate build step: - -```typescript -import { bundleWorkflowCode } from '@temporalio/worker'; -import { writeFile } from 'fs/promises'; - -const { code } = await bundleWorkflowCode({ - workflowsPath: require.resolve('./workflows'), -}); -await writeFile('./workflow-bundle.js', code); -``` - - -Then reference the bundle in your handler with `workflowBundle: { codePath: require.resolve('./workflow-bundle.js') }`. - -### Versioning behavior - -Set per-Workflow with `setWorkflowOptions` in the Workflow file, or set a default for all Workflows with `defaultVersioningBehavior` in the configure callback. -Values are `'PINNED'` or `'AUTO_UPGRADE'`. The default versioning behavior is `PINNED`. - -Access via: `config.workerOptions.workerDeploymentOptions!.defaultVersioningBehavior = 'PINNED'` - -### Lambda-tuned defaults - - - -| Setting | Lambda default | -|---|---| -| `maxConcurrentActivityTaskExecutions` | 2 | -| `maxConcurrentWorkflowTaskExecutions` | 10 | -| `maxConcurrentLocalActivityExecutions` | 2 | -| `maxConcurrentNexusTaskExecutions` | 5 | -| `workflowTaskPollerBehavior` | `SimpleMaximum(2)` | -| `activityTaskPollerBehavior` | `SimpleMaximum(1)` | -| `nexusTaskPollerBehavior` | `SimpleMaximum(1)` | -| `shutdownGraceTime` | 5 seconds | -| `maxCachedWorkflows` | 30 | -| `shutdownDeadlineBufferMs` | 7000 | - -Eager Activities are not supported. Lambda invocations don't maintain persistent connections. - -`shutdownDeadlineBufferMs` is specific to the `@temporalio/lambda-worker` package. It controls how much time before the Lambda deadline the Worker begins its graceful shutdown. The default is `shutdownGraceTime` (5s) + 2s. - -If your Worker handles long-running Activities, increase `shutdownGraceTime`, `shutdownDeadlineBufferMs`, and the Lambda invocation deadline (`--timeout`) together. - -### Connection configuration - -The `@temporalio/lambda-worker` package automatically loads Temporal client configuration from a TOML config file and environment variables (see the Environment Configuration docs, `/develop/environment-configuration`). - -TOML config file resolution order: - -1. `TEMPORAL_CONFIG_FILE` environment variable, if set. -2. `temporal.toml` in `$LAMBDA_TASK_ROOT` (typically `/var/task`). -3. `temporal.toml` in the current working directory. - -The file is optional. If absent, only environment variables are used. - ---- - -## Cross-SDK comparison: Lambda-tuned defaults - -| Concept | Go | Python | TypeScript | -|---|---|---|---| -| Max concurrent activities | `MaxConcurrentActivityExecutionSize` = 2 | `max_concurrent_activities` = 2 | `maxConcurrentActivityTaskExecutions` = 2 | -| Max concurrent workflow tasks | `MaxConcurrentWorkflowTaskExecutionSize` = 10 | `max_concurrent_workflow_tasks` = 10 | `maxConcurrentWorkflowTaskExecutions` = 10 | -| Sticky cache size | 100 | `max_cached_workflows` = 30 | `maxCachedWorkflows` = 30 | -| Worker stop timeout | `WorkerStopTimeout` = 5s | `graceful_shutdown_timeout` = 5s | `shutdownGraceTime` = 5s | -| Shutdown deadline buffer | `ShutdownDeadlineBuffer` = 7s | `shutdown_deadline_buffer` = 7s | `shutdownDeadlineBufferMs` = 7000 | -| Eager activities | `DisableEagerActivities` always true | `disable_eager_activity_execution` always `True` | Not supported | - - - - - -Note: Go sticky cache size is 100, while Python and TypeScript are 30. These values come from each SDK's own docs and are not interchangeable.