-
Notifications
You must be signed in to change notification settings - Fork 30
Document safe Workflow patch retirement #257
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -41,18 +41,20 @@ else: | |
|
|
||
| **Phase 1: Patch In** | ||
|
|
||
| - Add both old and new code paths | ||
| - New workflows take new path, old workflows take old path | ||
| - Add both old and new code paths. | ||
| - New executions take the new path; histories that predate the patch replay the old path. | ||
|
|
||
| **Phase 2: Deprecate** | ||
|
|
||
| - After all old workflows complete, remove old code | ||
| - Keep deprecation marker for history compatibility | ||
| - Wait until executions that can return the retired version have left retention before removing that branch or raising the minimum supported version. | ||
| - Keep the Workflow's deprecation/version marker at the same deterministic location for history compatibility. | ||
|
|
||
| **Phase 3: Remove** | ||
|
|
||
| - After all deprecated workflows complete | ||
| - Remove patch entirely, only new code remains | ||
| - Removing the final marker is a separate operation with patching-API-specific rules beyond branch retirement. | ||
| - Retire the patch/change identifier permanently when the patching API requires it; do not reuse an identifier whose first marker call was removed. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This seems unhelpful. It's basically saying "you can remove the marker when you should", whereas the old text tried to tell you when "should" is. Some combination of old/new would be appropriate here. |
||
|
|
||
| See the language-specific versioning reference for exact marker, visibility, retention, and replay rules. | ||
|
|
||
| ### When to Use | ||
|
|
||
|
|
@@ -192,31 +194,28 @@ For long-running Workflows that cannot use Continue-as-New (e.g., compliance aud | |
|
|
||
| ## Best Practices | ||
|
|
||
| 1. **Check for open executions** before removing old code | ||
| 1. **Wait for affected executions to leave retention** before removing old code | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same comment about the difference between open & retention |
||
| 2. **Use descriptive patch IDs** (e.g., "add-fraud-check" not "patch-1") | ||
| 3. **Deploy incrementally**: patch → deprecate → remove | ||
| 4. **Test replay compatibility** before deploying changes | ||
| 5. **Monitor old workflow counts** during migration | ||
|
|
||
| ## Finding Workflows by Version | ||
|
|
||
| ```bash | ||
| # Find workflows with specific patch | ||
| temporal workflow list --query \ | ||
| 'WorkflowType = "OrderWorkflow" AND TemporalChangeVersion = "add-fraud-check"' | ||
| Different patching APIs encode `TemporalChangeVersion` values differently. A pre-patch execution has no marker for that patch, but marker absence can also mean an optional patch site has not executed. Follow the language-specific guide to query and classify the old-version population rather than guessing a marker value. | ||
|
|
||
| # Find pre-patch workflows | ||
| temporal workflow list --query \ | ||
| 'WorkflowType = "OrderWorkflow" AND TemporalChangeVersion IS NULL' | ||
| Worker Deployment versions use a separate Search Attribute: | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This seems to have dropped a bunch of context about how to do queries with The previous paragraph refers to it without explaining what it is |
||
|
|
||
| # Find workflows on specific worker version | ||
| ```bash | ||
| temporal workflow list --query \ | ||
| 'TemporalWorkerDeploymentVersion = "my-service:v1.0.0"' | ||
| ``` | ||
|
|
||
| ## Common Mistakes | ||
|
|
||
| 1. **Removing old code too early** - Breaks replaying workflows | ||
| 2. **Not testing with replay** - Catches issues before production | ||
| 3. **Patching non-Command changes** - Unnecessary complexity | ||
| 4. **Forgetting to deprecate** - Accumulates dead code | ||
| 2. **Treating branch removal and final marker removal as identical changes** - Both can have retention gates, while final marker removal can also retire the identifier permanently | ||
| 3. **Treating marker absence as proof of an old branch** - Optional patch sites and other markers make absence ambiguous | ||
| 4. **Not testing with replay** - Replay catches compatibility failures before production | ||
| 5. **Patching non-Command changes** - Unnecessary complexity | ||
| 6. **Forgetting to deprecate** - Accumulates dead code | ||
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -68,7 +68,7 @@ public class OrderWorkflow | |||||
|
|
||||||
| **Step 2: Deprecate the Patch** | ||||||
|
|
||||||
| Once all pre-patch Workflow Executions have completed: | ||||||
| After all pre-patch Workflow Executions have left retention: | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same on open/retention |
||||||
|
|
||||||
| ```csharp | ||||||
| [Workflow] | ||||||
|
|
@@ -92,7 +92,7 @@ public class OrderWorkflow | |||||
|
|
||||||
| **Step 3: Remove the Patch** | ||||||
|
|
||||||
| After all workflows with the deprecated patch marker have completed, remove the `DeprecatePatch()` call entirely: | ||||||
| In a later deployment, after the deprecation rollout is complete and the pre-patch Workflow Executions have left retention, remove the `DeprecatePatch()` call entirely. A deprecated marker is intentionally safe to omit during replay, so marker-bearing executions do not require another retention wait: | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This last sentence is hard to parse. I think it means something like "If a workflow's history contains a deprecated marker, it can safely be replayed by code without any of the old code or the patch call." |
||||||
|
|
||||||
| ```csharp | ||||||
| [Workflow] | ||||||
|
|
@@ -121,11 +121,13 @@ Use List Filters to find workflows with specific patch versions: | |||||
| temporal workflow list --query \ | ||||||
| 'WorkflowType = "OrderWorkflow" AND ExecutionStatus = "Running" AND TemporalChangeVersion = "add-fraud-check"' | ||||||
|
|
||||||
| # Find running workflows without any patch (pre-patch versions) | ||||||
| # Find retained workflows without any patch marker (simple pre-patch case) | ||||||
| temporal workflow list --query \ | ||||||
| 'WorkflowType = "OrderWorkflow" AND ExecutionStatus = "Running" AND TemporalChangeVersion IS NULL' | ||||||
| 'WorkflowType = "OrderWorkflow" AND TemporalChangeVersion IS NULL' | ||||||
| ``` | ||||||
|
|
||||||
| `TemporalChangeVersion IS NULL` is reliable only when no other patch marker could make the attribute non-null. For multiple or optional patch sites, inspect all retained executions of the Workflow Type and classify the exact marker set and Event History. A zero running count identifies no immediate live blocker, but it does not prove that pre-patch histories have left retention. Replay selected histories to test compatibility; sampled replay does not prove the retained population is empty. | ||||||
|
|
||||||
|
Comment on lines
+129
to
+130
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
| ## Workflow Type Versioning | ||||||
|
|
||||||
| For incompatible changes, create a new Workflow Type instead of using patches: | ||||||
|
|
||||||
|
robzienert marked this conversation as resolved.
|
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -4,7 +4,7 @@ For conceptual overview and guidance on choosing an approach, see `references/co | |||||
|
|
||||||
| ## GetVersion API | ||||||
|
|
||||||
| `workflow.GetVersion` safely performs backwards-incompatible changes to Workflow Definitions. It returns the version to branch on, recording the result as a marker in the Event History. | ||||||
| [`workflow.GetVersion`](https://pkg.go.dev/go.temporal.io/sdk/workflow#GetVersion) safely performs backwards-incompatible changes to Workflow Definitions. It returns the version to branch on, recording the result as a marker in the Event History. | ||||||
|
|
||||||
| ```go | ||||||
| v := workflow.GetVersion(ctx, "changeID", workflow.DefaultVersion, maxSupported) | ||||||
|
|
@@ -15,9 +15,11 @@ v := workflow.GetVersion(ctx, "changeID", workflow.DefaultVersion, maxSupported) | |||||
| - `maxSupported`: current/newest version | ||||||
| - Returns `maxSupported` for new executions; returns the recorded version on replay | ||||||
|
|
||||||
| ### Three-Step Lifecycle | ||||||
| ### Retirement Lifecycle | ||||||
|
|
||||||
| **Step 1: Add GetVersion with both code paths** | ||||||
| Treat `GetVersion` cleanup as replay-safety work, not ordinary dead-code removal. Removing an old branch and deleting the first marker call are separate changes with different gates. | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
|
||||||
| #### Step 1: Add `GetVersion` with both code paths | ||||||
|
|
||||||
| Original code calls `ActivityA`. You want to replace it with `ActivityC`: | ||||||
|
|
||||||
|
|
@@ -32,26 +34,82 @@ if v == workflow.DefaultVersion { | |||||
| } | ||||||
| ``` | ||||||
|
|
||||||
| For new executions, `GetVersion` returns `1` and records a marker. For replay of pre-change workflows (no marker), it returns `DefaultVersion` (`-1`). | ||||||
| For new executions, `GetVersion` returns `1` and records a marker. For replay of pre-change workflows with no marker for `Step1`, it returns `workflow.DefaultVersion` (`-1`). **`DefaultVersion` is not version `0`**; do not search for `Step1-0` unless the code actually recorded version `0`. | ||||||
|
|
||||||
| #### Step 2: Inventory and prove the old version has left retention | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This section got really big in Go for some reason, but not the others (the smaller versions seem preferable. There's a lot of redundancy with the non-language-specific file). |
||||||
|
|
||||||
| Before raising `minSupported`, record enough context to review each change independently: | ||||||
|
|
||||||
| | Field | Example | | ||||||
| |---|---| | ||||||
| | Workflow type | `OrderWorkflow` | | ||||||
| | Change ID | `Step1` | | ||||||
| | Old version | `workflow.DefaultVersion`, `0`, or `1` | | ||||||
| | Retained version | `1` | | ||||||
| | Branch effect | Activity, child workflow, timer, signal/update flow | | ||||||
| | Call location | Startup, loop, selector, or handler | | ||||||
|
|
||||||
| For a recorded integer version, query the exact old `changeID-version` value across open and closed executions still in retention. For example, when retiring version `1` after a later `1 → 2` migration: | ||||||
|
|
||||||
| ```bash | ||||||
| temporal workflow list --query \ | ||||||
| 'WorkflowType = "OrderWorkflow" AND TemporalChangeVersion = "Step1-1"' | ||||||
| ``` | ||||||
|
|
||||||
| A nonzero result means version `1` has not left retention. Keep that branch. In the initial `workflow.DefaultVersion → 1` example above, the old branch has no recorded integer value, so use the marker-absence procedure instead. | ||||||
|
|
||||||
| Histories from before the first `GetVersion` call have no `TemporalChangeVersion` entry for that change ID. If this is the workflow's only patch marker, find the simple pre-marker population still in retention with: | ||||||
|
|
||||||
| ```bash | ||||||
| temporal workflow list --query \ | ||||||
| 'WorkflowType = "OrderWorkflow" AND TemporalChangeVersion IS NULL' | ||||||
| ``` | ||||||
|
|
||||||
| For workflows with multiple patch markers, query all running executions of the workflow type and inspect their `TemporalChangeVersion` values for the absence of the retained marker. Do the same when the `GetVersion` call is on an optional path: absence can mean either a pre-marker history or an execution that has not reached that call. | ||||||
|
|
||||||
| Treat marker absence as an unverified candidate set. Before raising `minSupported`, classify every candidate using its history or establish another authoritative population bound that proves every execution which could return the old version has left retention. Representative replay is still required for compatibility evidence, but it cannot classify an unexamined population. | ||||||
|
|
||||||
| A zero count of running old-version executions is necessary operational evidence, but it is not the retirement gate: closed histories can still be replayed or reset while retained. Do not raise `minSupported` until executions on the old version have left retention. A replay test can show whether selected histories are compatible; it does not prove that the old-version population has left retention. | ||||||
|
|
||||||
| **Step 2: Remove old branch (increase minSupported)** | ||||||
| #### Step 3: Remove the old branch but keep the first marker call | ||||||
|
|
||||||
| After all `DefaultVersion` Workflow Executions have completed: | ||||||
| After executions on the old version have left retention, raise `minSupported` and collapse the branch: | ||||||
|
|
||||||
| ```go | ||||||
| v := workflow.GetVersion(ctx, "Step1", 1, 1) | ||||||
| // Only the new code path remains | ||||||
| _ = workflow.GetVersion(ctx, "Step1", 1, 1) | ||||||
| err = workflow.ExecuteActivity(ctx, ActivityC, data).Get(ctx, &result1) | ||||||
| ``` | ||||||
|
|
||||||
| Keep the `GetVersion` call even with a single branch. This ensures: | ||||||
| Keep the first `GetVersion` call for the change ID at the same deterministic point in the Workflow. Moving it across other Commands can itself break replay. The pinned call causes an older unsupported history to fail at the version boundary instead of continuing with the wrong behavior, and it leaves a safe place to add a later version. | ||||||
|
|
||||||
| 1. If an older execution replays on this code, it fails fast instead of proceeding incorrectly | ||||||
| 2. If you need further changes, you just bump `maxSupported` | ||||||
| Only the first call for a change ID needs to be retained. Subsequent calls with that same change ID return the recorded value and can be removed once their surrounding branches are gone. | ||||||
|
|
||||||
| **Step 3: Further changes (bump maxSupported)** | ||||||
| Deploy and replay representative open and closed histories before making another retirement change. | ||||||
|
|
||||||
| Later, replace `ActivityC` with `ActivityD`: | ||||||
| #### Step 4: Optionally remove the first marker call | ||||||
|
|
||||||
| Deleting the first `GetVersion` call is stricter than removing an old branch. Do it only when: | ||||||
|
|
||||||
| 1. All executions with older versions have left retention. | ||||||
| 2. Replay verification covers the retained behavior. | ||||||
| 3. You will permanently retire that change ID. | ||||||
|
|
||||||
| After deleting the call, you **must never reuse** `Step1`. A future change at the same code location needs a new change ID and must start again from `workflow.DefaultVersion`: | ||||||
|
|
||||||
| ```go | ||||||
| v := workflow.GetVersion(ctx, "Step1-activity-d", workflow.DefaultVersion, 1) | ||||||
| if v == workflow.DefaultVersion { | ||||||
| err = workflow.ExecuteActivity(ctx, ActivityC, data).Get(ctx, &result1) | ||||||
| } else { | ||||||
| err = workflow.ExecuteActivity(ctx, ActivityD, data).Get(ctx, &result1) | ||||||
| } | ||||||
| ``` | ||||||
|
|
||||||
| When in doubt, leave the pinned marker call in place. It is small, explicit replay-safety state. | ||||||
|
|
||||||
| #### Further changes before final marker removal | ||||||
|
|
||||||
| To replace `ActivityC` with `ActivityD` while retaining `Step1`, bump `maxSupported`: | ||||||
|
|
||||||
| ```go | ||||||
| v := workflow.GetVersion(ctx, "Step1", 1, 2) | ||||||
|
|
@@ -62,13 +120,25 @@ if v == 1 { | |||||
| } | ||||||
| ``` | ||||||
|
|
||||||
| After all version-1 executions complete, collapse again: | ||||||
| After executions on version `1` have left retention, collapse again while preserving the first call: | ||||||
|
|
||||||
| ```go | ||||||
| _ = workflow.GetVersion(ctx, "Step1", 2, 2) | ||||||
| err = workflow.ExecuteActivity(ctx, ActivityD, data).Get(ctx, &result1) | ||||||
| ``` | ||||||
|
|
||||||
| #### Replay verification checklist | ||||||
|
|
||||||
| Before raising `minSupported` or deleting a marker call: | ||||||
|
|
||||||
| - Replay saved histories for every known version, including `DefaultVersion` when applicable. | ||||||
| - Replay both open and closed histories, especially long-running and error-path executions. | ||||||
| - Disable test caching so the evidence comes from the current code. | ||||||
| - Inspect targeted histories when marker absence is ambiguous. | ||||||
| - Record the visibility query, result count, replay command, and code version in the change review. | ||||||
|
|
||||||
| Replay and visibility answer different questions: replay tests compatibility for the histories selected, while visibility helps estimate whether an old-version population still exists. Use both; neither substitutes for the other. | ||||||
|
|
||||||
| ### Using GetVersion in Loops | ||||||
|
|
||||||
| The return value for a given `changeID` is immutable once recorded. In loops, append the iteration number to the `changeID`: | ||||||
|
|
@@ -269,11 +339,12 @@ if workflow.GetInfo(ctx).GetTargetWorkerDeploymentVersionChanged() { | |||||
|
|
||||||
| ## Best Practices | ||||||
|
|
||||||
| 1. **Keep GetVersion calls** even when only a single branch remains -- it guards against stale replays and simplifies future changes | ||||||
| 2. **Use `TemporalChangeVersion` search attribute** to find Workflows running on old versions: | ||||||
| 1. **Keep the first `GetVersion` call** when collapsing to a single branch unless the final retention and change-ID retirement gates are satisfied | ||||||
| 2. **Query the exact recorded value** using `changeID-version`: | ||||||
| ```bash | ||||||
| temporal workflow list --query \ | ||||||
| 'WorkflowType = "MyWorkflow" AND ExecutionStatus = "Running" AND TemporalChangeVersion = "Step1"' | ||||||
| 'WorkflowType = "MyWorkflow" AND TemporalChangeVersion = "Step1-1"' | ||||||
| ``` | ||||||
| 3. **Test with replay** before removing old branches to verify determinism is preserved | ||||||
| 4. **Prefer Worker Versioning** for large-scale deployments to avoid accumulating patching branches | ||||||
| 3. **Treat `DefaultVersion` as marker absence, not version `0`**, and inspect histories when absence is ambiguous | ||||||
| 4. **Test with replay** before removing old branches or marker calls, while remembering that sampled replay does not prove a version has left retention | ||||||
| 5. **Prefer Worker Versioning** for large-scale deployments to avoid accumulating patching branches | ||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's worth mentioning you only need to wait until they've left retention if you query them after they've completed, which is somewhat rare. If you know you don't do that, then you can remove the code when they've all completed, but before they've been archived/deleted.