Skip to content

feat: a plan waits 10 minutes for an executor, like an apply (#513) - #515

Open
vishu42 wants to merge 2 commits into
mainfrom
feat/513-plan-waits-for-executor
Open

vishu42 wants to merge 2 commits into
mainfrom
feat/513-plan-waits-for-executor

Conversation

@vishu42

@vishu42 vishu42 commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator

Part of #513.

Behaviour change

A plan now waits up to 10 minutes for a free executor session before it fails, the same as an apply. Before, a plan that found every session busy failed after 1 minute. With the chart's coming default of one session per executor, a busy slot is the normal case, so a teammate's plan would fail instead of waiting its turn.

planSessionCreationTimeout (1m) and applySessionCreationTimeout (10m) become one sessionCreationTimeout = 10 * time.Minute, used by both phases. The run already records "waiting for executor" before the session is created; nothing else changes. docs/architecture.md now says 10-minute.

No configuration or migration impact. Only the api image carries it (the api registers the workflows).

Replay

CreationTimeout sets the SDK session-creation activity's schedule-to-start timeout: an activity option, not part of a command's identity, so in-flight histories should replay. The workflow-replay CI job on this PR is the evidence.

Validation

Test first: TestTemplateWorkflowsWaitTenMinutesForAnExecutor reads the creation timeout off the options the workflow schedules the SDK's internalSessionCreationActivity with (through a workflow interceptor, since activity.Info doesn't carry schedule-to-start). Before the change: plan failed with session creation timeouts = [1m0s], want [10m0s], apply passed. After: both pass.

In a golang:1.26 Linux container:

  • gofmt -l cmd internal: no output
  • go vet ./internal/workflows/, go test ./internal/workflows/ -count=1: ok
  • go test ./...: ok except the four tests in internal/activities and internal/sandbox that fail only as root (Permission denied on the workspace); go test ./internal/activities/ ./internal/sandbox/ as uid 1000: ok

— Rahul (engineering)

🤖 Generated with Claude Code

With one session per executor in the chart's defaults, every session busy
is the normal case, so a plan that found none free in a minute failed a
teammate's run. Plan and apply now share one 10-minute creation timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vishu42 vishu42 added the rahul Filed by Rahul, the engineering persona label Oct 23, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vishu42

vishu42 commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator Author

Verified on Docker Desktop (#513): all three checks pass

Heads: #515 be6ab89f, #516 71b21bcd, #510 be8c4208. Docker Desktop k8s 1.32.2, one node (12 CPU, 7.55 GiB allocatable), Helm v4.1.1.

Setup:

1. Both pods Guaranteed, one 1536 MiB session: pass

  • status.qosClass is Guaranteed for api and for executor, and both are Running.
  • Both containers have requests = limits = cpu: 2, memory: 2Gi. Inside the executor, memory.max is 2147483648 and cpu.max is 200000 100000.
  • The enable-forwarding init container has requests = limits = 50m/32Mi. It ended Completed, exit 0.
  • envFrom has the openplan-executor ConfigMap first, then openplan-executor-env. The pod carries the checksum/config annotation.
  • env in the executor shows EXECUTOR_MAX_SESSIONS=1 and EXECUTOR_SANDBOX_MEMORY=1536.

2. A real 11-resource hashicorp/aws 6.x plan completes: pass

  • Fixture: openplanhq/demo-templates, branch fixture/aws-plan-513, modules/aws-plan-513. The provider has the skip flags, and the stack's Environment holds fake AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY.
  • The run completed with plan summary add 11, change 0, destroy 0, about 30 s from queued to done.
  • Memory, sampled every 3 s:
    • /sys/fs/cgroup/sandboxes/memory.peak = 1,479,774,208 B (1411 MiB) against the 1536 MiB sandbox limit. That figure includes page cache.
    • Executor container memory.peak = 1,544,810,496 B (1473 MiB) against 2048 MiB.
    • memory.events: oom 0, oom_kill 0.
  • Executor restarts: 0. lastState is empty, so no OOMKilled.
  • Headroom in the sandbox is about 125 MiB on this module. That fits the brief's accepted risk: a heavier AWS state needs the README's path, which raises EXECUTOR_SANDBOX_MEMORY together with the pod's resources.

3. A second plan waits for the executor, then completes: pass

  • First, an apply of long-running-apply with duration_seconds = 150 and auto-approve. Once it reached applying (21:08:19 IST), a plan started on a second stack template.
  • From 21:08:22 to 21:10:47 the plan showed running/waiting_for_executor: about 145 s. Before feat: a plan waits 10 minutes for an executor, like an apply (#513) #515 it would have failed at 60 s.
  • The apply completed at 21:10:50 (add 4). The plan then went fetching_source, then initializing, and completed at 21:10:56 (add 4, no error).
  • api and executor restarts: 0.

Everything is torn down: namespace, ingress-nginx, CoreDNS rewrite, the :gh513 images, and the kubelet's exited containers.

— Rahul (engineering)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

rahul Filed by Rahul, the engineering persona

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant