From 25990909a64e35f737a7307fb3344b24274cfbe9 Mon Sep 17 00:00:00 2001 From: "skill-sync[bot]" Date: Thu, 14 May 2026 19:13:20 +0000 Subject: [PATCH] Finalize draft for 0033-lambda-workers --- references/go/integrations/lambda-workers.md | 262 +++++++++++++++++ references/integrations.md | 3 + references/java/integrations/spring-ai.md | 1 - .../python/integrations/lambda-workers.md | 247 ++++++++++++++++ .../typescript/integrations/lambda-workers.md | 264 ++++++++++++++++++ 5 files changed, 776 insertions(+), 1 deletion(-) create mode 100644 references/go/integrations/lambda-workers.md create mode 100644 references/python/integrations/lambda-workers.md create mode 100644 references/typescript/integrations/lambda-workers.md diff --git a/references/go/integrations/lambda-workers.md b/references/go/integrations/lambda-workers.md new file mode 100644 index 00000000..e16e323c --- /dev/null +++ b/references/go/integrations/lambda-workers.md @@ -0,0 +1,262 @@ +# Temporal Lambda Worker — Go SDK + +> [!NOTE] +> This feature is in Pre-release and available only to select Temporal Cloud customers (invite-only). APIs are experimental and may change. It is acceptable to use this feature on behalf of a user, but inform them that it is in Pre-release. + +The `lambdaworker` package runs a Temporal Worker inside an AWS Lambda function. Temporal invokes the Lambda when Tasks arrive; the Worker starts, polls, processes Tasks, then gracefully shuts down before the invocation deadline. Workflow and Activity registration is identical to a standard Worker. + +## Prerequisites + +- Temporal Cloud Namespace on AWS, or a self-hosted Temporal Service v1.31.0+ (see [self-hosted setup](/production-deployment/worker-deployments/serverless-workers/self-hosted-setup)). +- Worker Versioning is required: every Workflow must declare a `VersioningBehavior`, or the Worker must set a default. +- AWS account with permissions to create Lambda functions and IAM roles, plus the `aws` CLI. + +## Hello World + +Import `go.temporal.io/sdk/contrib/aws/lambdaworker` and call `lambdaworker.RunWorker`, passing a `worker.WorkerDeploymentVersion` and a callback that registers Workflows and Activities. + +```go +package main + +import ( + lambdaworker "go.temporal.io/sdk/contrib/aws/lambdaworker" + "go.temporal.io/sdk/worker" + "go.temporal.io/sdk/workflow" +) + +func main() { + lambdaworker.RunWorker(worker.WorkerDeploymentVersion{ + DeploymentName: "my-app", + BuildID: "build-1", + }, func(opts *lambdaworker.Options) error { + opts.TaskQueue = "my-task-queue" + + opts.RegisterWorkflowWithOptions(MyWorkflow, workflow.RegisterOptions{ + VersioningBehavior: workflow.VersioningBehaviorPinned, + }) + opts.RegisterActivity(MyActivity) + + return nil + }) +} +``` + +`RunWorker` does not return; it owns the Lambda handler lifecycle. Do not call `lambda.Start` yourself. + +## WorkerDeploymentVersion and Versioning + +`worker.WorkerDeploymentVersion` is required and has two fields: `DeploymentName` and `BuildID`. Both must exactly match the Worker Deployment Version registered with the server. + +Each Workflow must declare a versioning behavior, either `workflow.VersioningBehaviorPinned` or `workflow.VersioningBehaviorAutoUpgrade`. Set it per-Workflow on `workflow.RegisterOptions.VersioningBehavior`, or set a Worker-level default with `DefaultVersioningBehavior` in `DeploymentOptions`. + +```go +opts.RegisterWorkflowWithOptions(MyWorkflow, workflow.RegisterOptions{ + VersioningBehavior: workflow.VersioningBehaviorPinned, +}) +``` + +## Options callback + +The `*lambdaworker.Options` passed to the callback exposes the standard registration methods: `RegisterWorkflow`, `RegisterWorkflowWithOptions`, `RegisterActivity`, `RegisterActivityWithOptions`, `RegisterNexusService`. Set the Task Queue on `opts.TaskQueue`. Configure the client through `opts.ClientOptions`. + +## Connection configuration + +The package loads Temporal client config from a TOML file and environment variables automatically. Resolution order: + +1. `TEMPORAL_CONFIG_FILE` environment variable, if set. +2. `temporal.toml` in `$LAMBDA_TASK_ROOT` (typically `/var/task`). +3. `temporal.toml` in the current working directory. + +The file is optional; environment variables alone are sufficient. See the Environment Configuration doc for the full variable list. + +## Lambda-tuned defaults + +These are standard `worker.Options` fields with lower values plus the `lambdaworker`-specific `ShutdownDeadlineBuffer`. + +| Setting | Lambda default | +|---|---| +| `MaxConcurrentActivityExecutionSize` | 2 | +| `MaxConcurrentWorkflowTaskExecutionSize` | 10 | +| `MaxConcurrentLocalActivityExecutionSize` | 2 | +| `MaxConcurrentNexusTaskExecutionSize` | 5 | +| `MaxConcurrentActivityTaskPollers` | 1 | +| `MaxConcurrentWorkflowTaskPollers` | 2 | +| `MaxConcurrentNexusTaskPollers` | 1 | +| `WorkerStopTimeout` | 5 seconds | +| `DisableEagerActivities` | Always true | +| Sticky cache size | 100 | +| `ShutdownDeadlineBuffer` | 7 seconds | + +`DisableEagerActivities` is always `true` and cannot be overridden; Eager Activities require a persistent connection that Lambda does not maintain. + +`ShutdownDeadlineBuffer` defaults to `WorkerStopTimeout + 2 seconds` and controls how much time before the Lambda deadline the Worker begins graceful shutdown. + +## Worker lifecycle and tuning + +Each invocation has three phases: + +- **Init** — establishes the client connection to Temporal. +- **Work** — polls the Task Queue and processes Tasks. +- **Shutdown** — stops polling, waits for in-flight Tasks, runs shutdown hooks (such as OTel flushes). + +For long-running Activities, three distinct values must be tuned together — do not conflate them: + +- `WorkerStopTimeout` > longest Activity runtime. Controls how long the Worker waits for in-flight Tasks after polling stops. +- `ShutdownDeadlineBuffer` > `WorkerStopTimeout` + shutdown hook time. Controls when polling stops before the invocation deadline. +- Lambda `--timeout` (invocation deadline) > longest Activity runtime + `ShutdownDeadlineBuffer`. + +If the longest Activity runs longer than half the maximum invocation deadline, use Activity Heartbeats so retries can resume. + +## OpenTelemetry + +The `go.temporal.io/sdk/contrib/aws/lambdaworker/otel` sub-package configures OTel metrics and tracing with defaults that target the AWS Distro for OpenTelemetry (ADOT) collector layer at `localhost:4317`. + +Apply both metrics and tracing: + +```go +import otel "go.temporal.io/sdk/contrib/aws/lambdaworker/otel" + +if err := otel.ApplyDefaults(opts, &opts.ClientOptions, otel.Options{}); err != nil { + return err +} +``` + +For metrics-only or tracing-only, use `otel.ApplyMetrics` or `otel.ApplyTracing`. + +Go does not need a language-specific ADOT layer because the OTel SDK is compiled into the binary; attach only the ADOT Collector layer. + +The default Collector config does not route OTLP to the traces pipeline. Bundle a custom `otel-collector-config.yaml` in the deployment package and set: + +- `OPENTELEMETRY_COLLECTOR_CONFIG_URI=/var/task/otel-collector-config.yaml` + +Enable X-Ray active tracing on the function: + +```bash +aws lambda update-function-configuration \ + --function-name \ + --tracing-config Mode=Active +``` + +The Lambda execution role must include `xray:PutTraceSegments`, `xray:PutTelemetryRecords`, and `cloudwatch:PutMetricData`; without them the Collector fails silently. + +## Deploy + +### Build and package + +Cross-compile for the Lambda Linux runtime; the output binary must be named `bootstrap`: + +```bash +GOOS=linux GOARCH=amd64 go build -tags lambda.norpc -o bootstrap ./worker +zip function.zip bootstrap +``` + +### Create the Lambda function + +```bash +aws lambda create-function \ + --function-name my-temporal-worker \ + --runtime provided.al2023 \ + --handler bootstrap \ + --role \ + --zip-file fileb://function.zip \ + --timeout 600 \ + --memory-size 256 \ + --environment '{"Variables":{"HOME":"/tmp","TEMPORAL_ADDRESS":":7233","TEMPORAL_NAMESPACE":"","TEMPORAL_API_KEY":""}}' +``` + +For Go binaries, `--runtime` must be `provided.al2023` and `--handler` must be `bootstrap`. Set `HOME=/tmp` so the Go runtime has a writable home directory. + +Supported environment variables: + +| Variable | Description | +|---|---| +| `TEMPORAL_ADDRESS` | Temporal frontend address (e.g. `..tmprl.cloud:7233`). | +| `TEMPORAL_NAMESPACE` | Temporal Namespace. | +| `TEMPORAL_TASK_QUEUE` | Task Queue name. Overrides the value set in code. | +| `TEMPORAL_API_KEY` | API key authentication. | +| `TEMPORAL_TLS_CLIENT_CERT_PATH` | mTLS client certificate path. | +| `TEMPORAL_TLS_CLIENT_KEY_PATH` | mTLS client key path. | + +The `--timeout` flag is the Lambda invocation deadline, not `WorkerStopTimeout` or `ShutdownDeadlineBuffer`. + +### Redeploy + +```bash +aws lambda update-function-code \ + --function-name my-temporal-worker \ + --zip-file fileb://function.zip +``` + +Create a 1-to-1 mapping between each `BuildID` and a Lambda function version; if you use an unversioned Lambda, do not change `BuildID` without also creating a new Worker Deployment Version. + +### Configure IAM for Temporal to invoke the Lambda + +Deploy the CloudFormation template from the [deployment guide](/production-deployment/worker-deployments/serverless-workers/aws-lambda#configure-iam) to create the invocation role for Temporal Cloud. For self-hosted, follow the [self-hosted setup](/production-deployment/worker-deployments/serverless-workers/self-hosted-setup#create-invocation-role). The trust policy uses an External ID condition; pass the same External ID when creating the Worker Deployment Version. + +### Create the Worker Deployment Version + +```bash +temporal worker deployment create \ + --namespace \ + --name my-app + +temporal worker deployment create-version \ + --namespace \ + --deployment-name my-app \ + --build-id build-1 \ + --aws-lambda-function-arn \ + --aws-lambda-assume-role-arn \ + --aws-lambda-assume-role-external-id +``` + +`--deployment-name` and `--build-id` must match `DeploymentName` and `BuildID` in the Worker code. `--aws-lambda-assume-role-arn` is the invocation role from the CloudFormation stack, not the Lambda execution role. + +### Set version as current + +```bash +temporal worker deployment set-current-version \ + --deployment-name my-app \ + --build-id build-1 +``` + +CLI-created versions are not current until this command runs; without it, Tasks do not route to the version. + +## Constraints + +- Activity duration must complete within the Lambda invocation limit minus `ShutdownDeadlineBuffer`; Lambda's maximum is 15 minutes. +- Workflow duration is unconstrained; a Workflow spans as many invocations as needed. +- Features requiring persistent connections are unavailable. +- `DisableEagerActivities` is always `true`. +- Worker Versioning is required; every Workflow needs `AutoUpgrade` or `Pinned`. + +## Troubleshooting + +Use the **Validate Connection** action on the deployment version in the Temporal UI to confirm Temporal can assume the IAM role and invoke the Lambda. + +Common failure modes: + +- **Invocation loop with no Workflow progress.** `DeploymentName` / `BuildID` in code do not match the Worker Deployment Version; the WCI keeps re-invoking. Update the code to match and redeploy. +- **Lambda never invoked, no Task Queue binding.** A failed first invocation prevents the Task Queue from binding to the version. Invoke the Lambda manually from the AWS Console to surface configuration errors directly. +- **No WCI Workflow exists.** The Worker Deployment Version has no compute provider; recreate it with the `--aws-lambda-*` flags. +- **Version not set as current (CLI flow).** Run `temporal worker deployment set-current-version` or verify with `temporal worker deployment describe`. +- **Lambda timeout terminates Activities.** Increase Lambda `--timeout`, `WorkerStopTimeout`, and `ShutdownDeadlineBuffer` together per the tuning rules above. + +Inspect WCI state by Workflow ID pattern `temporal-sys-worker-controller-instance::`: + +```bash +temporal workflow show \ + --namespace \ + --workflow-id 'temporal-sys-worker-controller-instance::' +``` + +Check which Task Queues are bound and whether a backlog exists: + +```bash +temporal worker deployment describe-version \ + --namespace \ + --deployment-name \ + --build-id \ + --report-task-queue-stats +``` + +Worker logs are in CloudWatch under `/aws/lambda/`. diff --git a/references/integrations.md b/references/integrations.md index 53cf1dfb..1da2bafe 100644 --- a/references/integrations.md +++ b/references/integrations.md @@ -15,3 +15,6 @@ Temporal ships and supports a growing set of integrations with third-party frame |---|---|---|---|---| | Spring Boot (`temporal-spring-boot-starter`) | Java | Auto-configuration of `WorkflowClient`, worker factories, workflow/activity bean registration, lifecycle, testing | `references/java/integrations/spring-boot.md` | `references/java/java.md` | | Spring AI (`temporal-spring-ai`) | Java | Durable Spring AI agents: chat-model calls run as Activities; tools dispatched per type (Activity stub, Nexus stub, `@SideEffectTool`, plain); vector stores, embeddings, and MCP clients auto-registered | `references/java/integrations/spring-ai.md` | `references/java/integrations/spring-boot.md`, `references/core/ai-patterns.md` | +| Lambda Worker (`lambdaworker`) | Go | Pre-release contrib package to run a Temporal Worker as an AWS Lambda function; Temporal invokes the Worker on demand with Lambda-tuned defaults, Worker Versioning required | `references/go/integrations/lambda-workers.md` | `references/go/go.md` | +| Lambda Worker (`lambda_worker`) | Python | Pre-release contrib package to run a Temporal Worker as an AWS Lambda function; configure callback wires `LambdaWorkerConfig.worker_config`, Worker Versioning required | `references/python/integrations/lambda-workers.md` | `references/python/python.md` | +| Lambda Worker (`@temporalio/lambda-worker`) | TypeScript | Pre-release package to run a Temporal Worker as an AWS Lambda function; pre-bundled `workflowBundle` required, Worker Versioning required | `references/typescript/integrations/lambda-workers.md` | `references/typescript/typescript.md` | diff --git a/references/java/integrations/spring-ai.md b/references/java/integrations/spring-ai.md index 5ee0704a..ae5154fd 100644 --- a/references/java/integrations/spring-ai.md +++ b/references/java/integrations/spring-ai.md @@ -217,7 +217,6 @@ Media image = new Media(MimeTypeUtils.IMAGE_PNG, URI.create("https://cdn.example For anything larger than a small thumbnail, route the bytes to a binary store from an Activity and pass only the URL across the conversation. - ## Vector stores, embeddings, and MCP When the corresponding Spring AI modules (`spring-ai-rag`, `spring-ai-mcp`) are on the classpath, the integration registers Activities for vector stores, embeddings, and MCP tool calls automatically. Inject the matching Spring AI types into your Activities or Workflows and use them as you would in any Spring AI application — each operation executes through a Temporal Activity. diff --git a/references/python/integrations/lambda-workers.md b/references/python/integrations/lambda-workers.md new file mode 100644 index 00000000..a6a9bd99 --- /dev/null +++ b/references/python/integrations/lambda-workers.md @@ -0,0 +1,247 @@ +# Temporal Lambda Worker — Python SDK + +> [!NOTE] +> This feature is in Pre-release and available only to select Temporal Cloud customers (invite-only). APIs are experimental and may change. It is acceptable to use this feature on behalf of a user, but inform them that it is in Pre-release. + +The `lambda_worker` contrib package runs a Temporal Serverless Worker inside an AWS Lambda function. Temporal Cloud invokes the Lambda when Tasks arrive on a bound Task Queue; the Worker starts, polls, processes Tasks, then gracefully shuts down before the invocation deadline. Workflows and Activities are registered the same way as a standard Worker. + +## Prerequisites + +- Worker Versioning is required; every Workflow must have a versioning behavior or the Worker must set a default. +- A Temporal Cloud account with an AWS-hosted Namespace, or self-hosted Temporal Service v1.31.0 or later. +- For self-hosted deployments, complete the [self-hosted setup](/production-deployment/worker-deployments/serverless-workers/self-hosted-setup) first. +- AWS account with permissions to create and invoke Lambda functions and create IAM roles. + +## Hello world + +Create a Lambda handler by calling `run_worker` with a `WorkerDeploymentVersion` and a configure callback; assign the returned handler to a module-level `lambda_handler`. + +```python +from activities import hello_activity +from temporalio.common import WorkerDeploymentVersion +from temporalio.contrib.aws.lambda_worker import LambdaWorkerConfig, run_worker +from workflows import TASK_QUEUE, SampleWorkflow + + +def configure(config: LambdaWorkerConfig) -> None: + config.worker_config["task_queue"] = TASK_QUEUE + config.worker_config["workflows"] = [SampleWorkflow] + config.worker_config["activities"] = [hello_activity] + + +lambda_handler = run_worker( + WorkerDeploymentVersion(deployment_name="my-app", build_id="build-1"), + configure, +) +``` + +- Module path is `temporalio.contrib.aws.lambda_worker`. +- `run_worker` returns the Lambda handler; the AWS `--handler` flag must point at the variable bound to it. +- `configure` is called with a pre-populated `LambdaWorkerConfig` dataclass; set Task Queue, Workflows, and Activities through `worker_config`. + +## `WorkerDeploymentVersion` and versioning + +Import `WorkerDeploymentVersion` from `temporalio.common`; construct it with `deployment_name=...` and `build_id=...`. The deployment name groups related Workers across versions; the Build Id identifies a specific release of your Worker code. + +Each Workflow must declare a versioning behavior — `VersioningBehavior.PINNED` or `VersioningBehavior.AUTO_UPGRADE`, imported from `temporalio.common`. Set it per-Workflow in `@workflow.defn(versioning_behavior=...)`, or set `default_versioning_behavior` in the worker config as a Worker-level fallback. + +```python +from temporalio import workflow +from temporalio.common import VersioningBehavior + + +@workflow.defn(versioning_behavior=VersioningBehavior.PINNED) +class MyWorkflow: + @workflow.run + async def run(self, input: str) -> str: + ... +``` + +## `LambdaWorkerConfig` and the configure callback + +`LambdaWorkerConfig.worker_config` is a dict that accepts the same keyword arguments as the `Worker` constructor. Set, at minimum: + +- `config.worker_config["task_queue"]` +- `config.worker_config["workflows"]` +- `config.worker_config["activities"]` + +Don't overwrite the entire dict — mutate keys on the dataclass field so Lambda-tuned defaults are preserved. + +## Configure the Temporal connection + +The `lambda_worker` package automatically loads Temporal client configuration from a TOML config file and environment variables. See [Environment configuration](/develop/environment-configuration) for the full list of variables and profiles. + +Config file resolution order: + +1. `TEMPORAL_CONFIG_FILE` environment variable, if set. +2. `temporal.toml` in `$LAMBDA_TASK_ROOT` (typically `/var/task`). +3. `temporal.toml` in the current working directory. + +The file is optional; if absent, only environment variables are used. Encrypt sensitive values like TLS keys and API keys at rest. + +## Lambda-tuned defaults + +The `lambda_worker` package applies conservative defaults suited to short-lived Lambda invocations. Transcribed verbatim: + +| Setting | Lambda default | +|---|---| +| `max_concurrent_activities` | 2 | +| `max_concurrent_workflow_tasks` | 10 | +| `max_concurrent_local_activities` | 2 | +| `max_concurrent_nexus_tasks` | 5 | +| `workflow_task_poller_behavior` | `SimpleMaximum(2)` | +| `activity_task_poller_behavior` | `SimpleMaximum(1)` | +| `nexus_task_poller_behavior` | `SimpleMaximum(1)` | +| `graceful_shutdown_timeout` | 5 seconds | +| `max_cached_workflows` | 30 | +| `disable_eager_activity_execution` | Always `True` | +| `shutdown_deadline_buffer` | 7 seconds | + +- `disable_eager_activity_execution` is always `True` and cannot be overridden — Eager Activities require a persistent connection that Lambda invocations don't maintain. +- `shutdown_deadline_buffer` is specific to `lambda_worker`; it controls how much time before the Lambda deadline the Worker begins graceful shutdown. Default is `graceful_shutdown_timeout` + 2 seconds. + +## Worker lifecycle and tuning + +Each invocation has three phases: init (initialize and connect to Temporal), work (poll Task Queue and process Tasks), and shutdown (stop polling, drain in-flight Tasks, run shutdown hooks). + +For long-running Activities, raise three values together — raising only one breaks the chain: + +- **Worker stop timeout > longest Activity runtime** — gives in-flight Activities time to finish after polling stops. +- **`shutdown_deadline_buffer` > Worker stop timeout + shutdown hook time** — ensures drain and shutdown hooks complete before the compute provider terminates the environment. +- **Invocation deadline (`--timeout`) > longest Activity runtime + shutdown deadline buffer** — set on the Lambda function. + +If the longest Activity runs longer than half the maximum invocation deadline, use [Activity Heartbeats](/encyclopedia/detecting-activity-failures#activity-heartbeat) instead so the next retry resumes from the recorded state. + +## OpenTelemetry + +The `temporalio.contrib.aws.lambda_worker.otel` module provides OpenTelemetry integration with defaults configured for the [AWS Distro for OpenTelemetry (ADOT)](https://aws-otel.github.io/docs/getting-started/lambda) Lambda layer. Use it to emit SDK metrics and distributed traces for Workflow and Activity executions. + +```python +from temporalio.contrib.aws.lambda_worker.otel import apply_defaults + + +def configure(config: LambdaWorkerConfig) -> None: + config.worker_config["task_queue"] = TASK_QUEUE + config.worker_config["workflows"] = [SampleWorkflow] + config.worker_config["activities"] = [hello_activity] + apply_defaults(config) +``` + +- `apply_defaults(config)` configures both metrics and tracing; default endpoint is `localhost:4317`, the ADOT Lambda layer's default Collector endpoint. +- If you only need metrics, use `build_metrics_telemetry_config`; for tracing only, use `apply_tracing`. +- Optional install extra: `temporalio[lambda-worker-otel]`. +- Attach the [ADOT Python Lambda layer](https://aws-otel.github.io/docs/getting-started/lambda/lambda-python). +- The default Collector configuration does not route OTLP data to the traces pipeline; bundle a custom `otel-collector-config.yaml` in the deployment package that wires the OTLP receiver to both traces and metrics pipelines. +- Set environment variable `OPENTELEMETRY_COLLECTOR_CONFIG_FILE=/var/task/otel-collector-config.yaml` on the Lambda function. +- Enable X-Ray active tracing: `aws lambda update-function-configuration --function-name --tracing-config Mode=Active`. +- The Lambda execution role must have permissions to write to X-Ray and CloudWatch. Attach the `AWSXRayDaemonWriteAccess` managed policy, or add `xray:PutTraceSegments`, `xray:PutTelemetryRecords`, and `cloudwatch:PutMetricData`. Without these, the Collector fails silently and no telemetry appears. + +## Deploy + +### Build and package + +Install dependencies for the Lambda Linux runtime using `--platform` to fetch Linux-compatible binaries: + +```bash +pip install --target ./package --platform manylinux2014_x86_64 --only-binary=:all: temporalio +``` + +For OpenTelemetry support, install `temporalio[lambda-worker-otel]` instead. + +Package dependencies and application code into a zip: + +```bash +cd package && zip -r ../function.zip . && cd .. +zip function.zip lambda_function.py my_workflows.py my_activities.py +``` + +### Create the Lambda function + +```bash +aws lambda create-function \ + --function-name my-temporal-worker \ + --runtime python3.13 \ + --handler lambda_function.lambda_handler \ + --role \ + --zip-file fileb://function.zip \ + --timeout 600 \ + --memory-size 256 \ + --environment '{"Variables":{"TEMPORAL_ADDRESS":":7233","TEMPORAL_NAMESPACE":"","TEMPORAL_API_KEY":""}}' +``` + +- `--runtime python3.13` (or another supported Python version). +- `--handler lambda_function.lambda_handler` — `module.function` format pointing at the handler returned by `run_worker`. +- `--role` is the Lambda execution role (separate from the role Temporal assumes to invoke the function); must have at least `AWSLambdaBasicExecutionRole`. +- `--timeout` is the invocation deadline in seconds; set it high enough for init, work, and graceful shutdown. + +### Environment variables + +| Variable | Description | +|---|---| +| `TEMPORAL_ADDRESS` | Temporal frontend address (e.g., `..tmprl.cloud:7233`). | +| `TEMPORAL_NAMESPACE` | Temporal Namespace. | +| `TEMPORAL_TASK_QUEUE` | Task Queue name. Overrides the value set in code. | +| `TEMPORAL_API_KEY` | API key for API key authentication. | +| `TEMPORAL_TLS_CLIENT_CERT_PATH` | Path to TLS client certificate for mTLS. | +| `TEMPORAL_TLS_CLIENT_KEY_PATH` | Path to TLS client key for mTLS. | + +### Redeploy code + +```bash +aws lambda update-function-code \ + --function-name my-temporal-worker \ + --zip-file fileb://function.zip +``` + +Create a 1-to-1 mapping between each Build Id in your Worker code and a Lambda function version. If you use an unversioned Lambda, don't change the Build Id without also creating a new Worker Deployment Version. + +### Configure IAM for Temporal invocation + +Temporal needs permission to invoke your Lambda. Deploy the CloudFormation template from the [deploy guide](/production-deployment/worker-deployments/serverless-workers/aws-lambda#configure-iam) to create the invocation role with an External ID condition. + +### Create the Worker Deployment Version + +```bash +temporal worker deployment create \ + --namespace \ + --name my-app + +temporal worker deployment create-version \ + --namespace \ + --deployment-name my-app \ + --build-id build-1 \ + --aws-lambda-function-arn \ + --aws-lambda-assume-role-arn \ + --aws-lambda-assume-role-external-id +``` + +- `--deployment-name` and `--build-id` must exactly match the `deployment_name` and `build_id` in your Worker code. +- `--aws-lambda-assume-role-arn` is the role Temporal assumes (from the CloudFormation stack output), not the Lambda execution role. + +### Set the version as current + +CLI-created versions are not current automatically; set the version as current or Tasks will not route to it. + +```bash +temporal worker deployment set-current-version \ + --deployment-name my-app \ + --build-id build-1 +``` + +## Constraints + +- Activity duration must complete within the Lambda invocation limit minus the shutdown deadline buffer; AWS Lambda's maximum is 15 minutes. +- Workflow duration has no limit — Workflows span as many invocations as needed. +- Worker Versioning is required; every Workflow needs `AUTO_UPGRADE` or `PINNED`. +- Eager Activities are always disabled (`disable_eager_activity_execution=True`); features that require a persistent connection are unavailable. +- Each invocation creates a fresh client connection — no connection reuse or shared state. + +## Common mistakes and troubleshooting + +- **Rapid invocation loop with no Workflow progress** — the `deployment_name` or `build_id` in your code doesn't match the Worker Deployment Version. The WCI invokes the Lambda, the Worker polls under a different deployment version, the Task isn't processed, the WCI invokes again. Fix the values in code and redeploy. +- **Lambda is invoked but Task Queue never binds** — the first invocation after creating the Worker Deployment Version failed (missing env vars, wrong TLS, missing dependencies). Without a successful poll, no Task Queue binding is created. Invoke the Lambda manually from the AWS Console to see the error directly. +- **Lambda not being invoked at all** — verify the version has a compute provider configured and use **Workers > Deployments > Actions > Validate Connection** to confirm Temporal can assume the role and invoke the function. +- **Version created via CLI but no invocations** — confirm the version is set as current with `temporal worker deployment describe`. +- **List WCI Workflows** in a Namespace: query `TemporalNamespaceDivision = "TemporalWorkerControllerInstance"`. WCI Workflow IDs follow the pattern `temporal-sys-worker-controller-instance::`. +- **Connection/TLS/auth errors during startup** — verify `TEMPORAL_ADDRESS`, `TEMPORAL_API_KEY` (or `temporal.toml`) are correct on the Lambda function. +- **Activities abandoned mid-execution** — the Lambda timeout fired before the Worker finished. Increase Lambda `--timeout` and `shutdown_deadline_buffer` together per the [tuning rules](#worker-lifecycle-and-tuning). diff --git a/references/typescript/integrations/lambda-workers.md b/references/typescript/integrations/lambda-workers.md new file mode 100644 index 00000000..fba6d536 --- /dev/null +++ b/references/typescript/integrations/lambda-workers.md @@ -0,0 +1,264 @@ +# Temporal Lambda Worker — TypeScript SDK + +> [!NOTE] +> This feature is in Pre-release and available only to select Temporal Cloud customers (invite-only). APIs are experimental and may change. It is acceptable to use this feature on behalf of a user, but inform them that it is in Pre-release. + +The `@temporalio/lambda-worker` package runs a Temporal Worker as an AWS Lambda function. Temporal Cloud invokes the Lambda when Tasks arrive; each invocation starts a Worker, polls for Tasks, and shuts down before the invocation deadline. You register Workflows and Activities the same way as a standard Worker. + +## Prerequisites + +- Worker Versioning is required; every Workflow must have a versioning behavior or the Worker must set a default. +- A Temporal Cloud account with an AWS-hosted Namespace (invite-only), or a self-hosted Temporal Service v1.31.0 or later. +- For self-hosted setups, complete the [self-hosted setup](/production-deployment/worker-deployments/serverless-workers/self-hosted-setup) before deploying. +- AWS account with permissions to create and invoke Lambda functions and create IAM roles. +- Node.js 20+ runtime (the `nodejs22.x` Lambda runtime is recommended). + +## Hello world + +Use `runWorker` to produce the Lambda `handler`. The first argument is the deployment version (`deploymentName`, `buildId`); the second is a configure callback that mutates `config.workerOptions`. + +```typescript +import { runWorker } from '@temporalio/lambda-worker'; +import * as activities from './activities'; +import { TASK_QUEUE } from './workflows'; + +export const handler = runWorker({ deploymentName: 'sdk-demo', buildId: 'v1' }, (config) => { + config.workerOptions.taskQueue = TASK_QUEUE; + config.workerOptions.workflowBundle = { + codePath: require.resolve('./workflow-bundle.js'), + }; + config.workerOptions.activities = activities; +}); +``` + +The package name is `@temporalio/lambda-worker` (scoped, kebab-case). The handler is exported as the value returned by `runWorker(...)`. + +## Pre-bundle Workflow code (required on Lambda) + +Don't use `workflowsPath` on Lambda; use `workflowBundle` with pre-bundled code instead — this avoids webpack bundling overhead on every cold start. + +Build the bundle in a separate build step with `bundleWorkflowCode` from `@temporalio/worker`: + +```typescript +import { bundleWorkflowCode } from '@temporalio/worker'; +import { writeFile } from 'fs/promises'; + +const { code } = await bundleWorkflowCode({ + workflowsPath: require.resolve('./workflows'), +}); +await writeFile('./workflow-bundle.js', code); +``` + +Reference the bundle in the handler with `workflowBundle: { codePath: require.resolve('./workflow-bundle.js') }`. + +## Deployment version and versioning + +- `deploymentName` and `buildId` are passed as the first argument to `runWorker`. +- Worker Deployment Versioning is always enabled for Serverless Workers; the deployment version is required. +- Each Workflow must declare a versioning behavior, either `'AUTO_UPGRADE'` or `'PINNED'`. +- The default versioning behavior is `'PINNED'`. +- To change the Worker-level default, set `config.workerOptions.workerDeploymentOptions.defaultVersioningBehavior` in the configure callback. +- To set per-Workflow behavior, use `setWorkflowOptions` in the Workflow file. + +Example of overriding the default in the configure callback: + +```typescript +config.workerOptions.workerDeploymentOptions!.defaultVersioningBehavior = 'PINNED'; +``` + +## Connection configuration + +The package auto-loads Temporal client configuration from a TOML config file and environment variables. Config file resolution order: + +1. `TEMPORAL_CONFIG_FILE` environment variable, if set. +2. `temporal.toml` in `$LAMBDA_TASK_ROOT` (typically `/var/task`). +3. `temporal.toml` in the current working directory. + +The file is optional; if absent, only environment variables are used. Encrypt sensitive values (TLS keys, API keys) at rest. + +## Lambda-tuned defaults + +The package applies conservative defaults suited to short-lived Lambda invocations. + +| Setting | Lambda default | +|---|---| +| `maxConcurrentActivityTaskExecutions` | 2 | +| `maxConcurrentWorkflowTaskExecutions` | 10 | +| `maxConcurrentLocalActivityExecutions` | 2 | +| `maxConcurrentNexusTaskExecutions` | 5 | +| `workflowTaskPollerBehavior` | `SimpleMaximum(2)` | +| `activityTaskPollerBehavior` | `SimpleMaximum(1)` | +| `nexusTaskPollerBehavior` | `SimpleMaximum(1)` | +| `shutdownGraceTime` | 5 seconds | +| `maxCachedWorkflows` | 30 | +| `shutdownDeadlineBufferMs` | 7000 | + +Eager Activities are not supported. Lambda invocations don't maintain persistent connections. + +`shutdownDeadlineBufferMs` is specific to `@temporalio/lambda-worker`; it controls how much time before the Lambda deadline the Worker begins graceful shutdown. The default is `shutdownGraceTime` (5s) + 2s. + +## Worker lifecycle and tuning + +Each invocation has three phases: init (client connect), work (poll and process Tasks), and shutdown (stop polling, drain in-flight Tasks, run shutdown hooks). + +For long-running Activities, tune these three values together: + +- Worker stop timeout (`shutdownGraceTime`) > longest Activity runtime. +- `shutdownDeadlineBufferMs` > Worker stop timeout + shutdown hook time. +- Lambda `--timeout` > longest Activity runtime + `shutdownDeadlineBufferMs`. + +Raising only the buffer makes the Worker stop polling earlier without giving in-flight Tasks more time; raising only the stop timeout risks Lambda terminating the function before the drain completes. If an Activity may run longer than half the maximum invocation deadline, use [Activity Heartbeats](/encyclopedia/detecting-activity-failures#activity-heartbeat) instead. + +## OpenTelemetry + +Telemetry support lives in the `@temporalio/lambda-worker/otel` subpath. + +Call `applyDefaults(config)` inside the configure callback to register Temporal SDK tracing interceptors and configure OTLP metric export. Telemetry is sent to `localhost:4317` (the ADOT Lambda layer's default collector endpoint). + +```typescript +import { runWorker } from '@temporalio/lambda-worker'; +import { applyDefaults } from '@temporalio/lambda-worker/otel'; +import * as activities from './activities'; +import { TASK_QUEUE } from './workflows'; + +export const handler = runWorker({ deploymentName: 'sdk-demo', buildId: 'v1' }, (config) => { + config.workerOptions.taskQueue = TASK_QUEUE; + config.workerOptions.workflowBundle = { + codePath: require.resolve('./workflow-bundle.js'), + }; + config.workerOptions.activities = activities; + applyDefaults(config); +}); +``` + +When pre-bundling Workflow code with OTel, pass the plugin from `makeOtelPlugin()` so Workflow interceptor modules are included in the bundle: + +```typescript +import { bundleWorkflowCode } from '@temporalio/worker'; +import { makeOtelPlugin } from '@temporalio/lambda-worker/otel'; + +const { plugin } = makeOtelPlugin(); +const { code } = await bundleWorkflowCode({ + workflowsPath: require.resolve('./workflows'), + plugins: [plugin], +}); +``` + +Attach two ADOT Lambda layers: + +1. The ADOT JavaScript layer for Node.js auto-instrumentation and trace export. +2. The ADOT Collector layer (`aws-otel-collector-amd64`) — runs the OTel Collector as a Lambda extension, receiving OTLP on `localhost:4317` and forwarding traces to X-Ray and metrics to CloudWatch. + +Provide a custom Collector configuration that routes OTLP to both traces and metrics pipelines (default config does not). Set this env var on the Lambda: `OPENTELEMETRY_COLLECTOR_CONFIG_URI=/var/task/otel-collector-config.yaml`. + +Enable X-Ray active tracing on the function: + +```bash +aws lambda update-function-configuration \ + --function-name \ + --tracing-config Mode=Active +``` + +The Lambda execution role needs `xray:PutTraceSegments`, `xray:PutTelemetryRecords`, and `cloudwatch:PutMetricData`. Without these, the Collector fails silently. + +## Deploy + +### Build and package + +```bash +npx ts-node src/scripts/build-workflow-bundle.ts +npx tsc +npm install --omit=dev +zip -r function.zip lib/ node_modules/ workflow-bundle.js +``` + +### Create the Lambda function + +```bash +aws lambda create-function \ + --function-name my-temporal-worker \ + --runtime nodejs22.x \ + --handler lib/index.handler \ + --role \ + --zip-file fileb://function.zip \ + --timeout 600 \ + --memory-size 256 \ + --environment '{"Variables":{"HOME":"/tmp","TEMPORAL_ADDRESS":":7233","TEMPORAL_NAMESPACE":"","TEMPORAL_API_KEY":""}}' +``` + +`--runtime` is `nodejs22.x` or another supported Node.js version (20+); `--handler` is in `module.export` format and must point to the handler exported by `runWorker`. + +### Common Lambda environment variables + +| Variable | Purpose | +|---|---| +| `TEMPORAL_ADDRESS` | Temporal frontend address (e.g., `..tmprl.cloud:7233`). | +| `TEMPORAL_NAMESPACE` | Temporal Namespace. | +| `TEMPORAL_TASK_QUEUE` | Task Queue name; overrides the value set in code. | +| `TEMPORAL_TLS_CLIENT_CERT_PATH` | TLS client certificate path for mTLS. | +| `TEMPORAL_TLS_CLIENT_KEY_PATH` | TLS client key path for mTLS. | +| `TEMPORAL_API_KEY` | API key for API key authentication. | + +For the full list of supported environment variables and TOML profile format, see [Environment configuration](/develop/environment-configuration). + +### Redeploy code + +```bash +aws lambda update-function-code \ + --function-name my-temporal-worker \ + --zip-file fileb://function.zip +``` + +Maintain a 1-to-1 mapping between each build ID in Worker code and a Lambda function version. Don't change the build ID in code without also creating a new Worker Deployment Version. + +### IAM for Temporal invocation + +Temporal Cloud assumes an IAM role in your AWS account to call `lambda:InvokeFunction`. Deploy the [CloudFormation template](/production-deployment/worker-deployments/serverless-workers/aws-lambda#configure-iam) with `AssumeRoleExternalId`, `LambdaFunctionARNs`, and `RoleName`. + +### Create the Worker Deployment Version (CLI) + +```bash +temporal worker deployment create \ + --namespace \ + --name my-app + +temporal worker deployment create-version \ + --namespace \ + --deployment-name my-app \ + --build-id build-1 \ + --aws-lambda-function-arn \ + --aws-lambda-assume-role-arn \ + --aws-lambda-assume-role-external-id +``` + +`--deployment-name` and `--build-id` must match the values in the Worker code. `--aws-lambda-assume-role-arn` is the `RoleARN` output from the CloudFormation stack (not the Lambda execution role). + +### Set the version as current + +```bash +temporal worker deployment set-current-version \ + --deployment-name my-app \ + --build-id build-1 +``` + +Without this step, Tasks on the Task Queue won't route to the version. When the version is created via the UI, it is set as current automatically. + +## Constraints + +| Constraint | Detail | +|---|---| +| Activity duration | Must complete within the Lambda invocation limit minus `shutdownDeadlineBufferMs`. Lambda's hard maximum is 15 minutes. | +| Workflow duration | No limit; a Workflow runs across as many invocations as needed. | +| Versioning | Worker Versioning is required. | +| Persistent connections | Not maintained across invocations; features requiring persistent connections are unavailable. | +| Eager Activities | Not supported. Lambda invocations don't maintain persistent connections. | + +## Troubleshooting + +- **Invocation loop with no Workflow progress.** Deployment name and build ID in code must exactly match the Worker Deployment Version. A mismatch causes the WCI to repeatedly invoke without ever processing the Task. Fix by aligning the values and redeploying. +- **Lambda never invoked / no Task Queue binding.** A failed first invocation prevents the Task Queue binding from being created. Invoke the Lambda manually from the AWS Console to surface the error directly. +- **Lambda never invoked / version not current.** If created via CLI, run `temporal worker deployment set-current-version`. +- **Validate Connection action.** In the Temporal UI under **Workers > Deployments**, the version's **Actions > Validate Connection** confirms IAM role assume and Lambda reachability. +- **Listing WCI Workflows.** WCI Workflow IDs follow `temporal-sys-worker-controller-instance::` and can be listed with `--query 'TemporalNamespaceDivision = "TemporalWorkerControllerInstance"'`. +- **Connection / TLS / auth errors.** Check `TEMPORAL_ADDRESS`, `TEMPORAL_API_KEY`, TLS certificate/key validity, and (self-hosted) network reachability. +- **Lambda timeout abandoning Activities.** If Activities run past the Lambda deadline, AWS terminates the invocation. Increase `--timeout`, `shutdownGraceTime`, and `shutdownDeadlineBufferMs` together.