Order Lifecycle on Temporal

An Order → Payment → Shipping lifecycle orchestrated with Temporal: a parent OrderWorkflow with signals and a manual-review timer, a ShippingWorkflow child on its own task queue, Postgres for persistence, and a FastAPI layer for triggering, signalling and inspecting runs.

OrderWorkflow                       (task queue: order-tq)
  receive_order
  validate_order
  ── manual review gate ──         timer + ApproveOrder signal
  charge_payment                    idempotent on payment_id
  └─ ShippingWorkflow               (task queue: shipping-tq)
       prepare_package
       dispatch_carrier  ──failure──▶ DispatchFailed signal to parent
                         ◀──retry──── RetryDispatch signal from parent
  mark_shipped

---

Quick start

docker compose up --build

That brings up Postgres (with migrations applied on first boot), the Temporal dev server, the Temporal Web UI, both workers, and the API.

Service	URL
API	http://localhost:8000 (docs at /docs)
Temporal Web UI	http://localhost:8080
Postgres	localhost:5432, db orders, user/pass postgres

Temporal's auto-setup image provisions its own schema on first boot. The API and worker containers gate on its healthcheck rather than starting into a connection error. From a clean slate (docker compose down -v) the whole stack reaches healthy in about 25s.

Then drive one order end to end:

python scripts/demo.py

It starts an order, waits for the review gate, updates the address, approves, then prints the final workflow status, the persisted Postgres rows, and the duration Temporal recorded for the execution (which is what the 15s SLA is about, as distinct from how long the script spent polling). --cancel and --no-approve exercise the other two paths.

Running without Docker

pip install -r requirements.txt
docker run --rm -d --name temporal -p 7233:7233 temporalio/auto-setup
psql -U postgres -f migrations/001_init.sql
python -m app.worker                                  # terminal 1
uvicorn app.api:app --reload --port 8000              # terminal 2

Both processes read TEMPORAL_ADDRESS and DATABASE_URL from the environment (app/config.py); the defaults point at localhost.

---

API

Method & path	Description
POST /orders/{order_id}/start	Start OrderWorkflow. Body: {"payment_id": "...", "address": {...}}
POST /orders/{order_id}/signals/approve	ApproveOrder, opens the review gate
POST /orders/{order_id}/signals/cancel	CancelOrder. Body: {"reason": "..."}
POST /orders/{order_id}/signals/update-address	UpdateAddress. Body: {"address": {...}}
GET /orders/{order_id}/status	Live state: current step, retry counts, recent errors, budget left
GET /orders/{order_id}/events	The Postgres side: order row, payment row, event log

ORDER=order-1

curl -X POST localhost:8000/orders/$ORDER/start \
  -H 'content-type: application/json' \
  -d '{"payment_id":"pay-1","address":{"line1":"1 Main St"}}'

curl -X POST localhost:8000/orders/$ORDER/signals/update-address \
  -H 'content-type: application/json' \
  -d '{"address":{"line1":"99 New Road"}}'

curl -X POST localhost:8000/orders/$ORDER/signals/approve

curl -X POST localhost:8000/orders/$ORDER/signals/cancel \
  -H 'content-type: application/json' -d '{"reason":"customer request"}'

curl localhost:8000/orders/$ORDER/status
curl localhost:8000/orders/$ORDER/events

Inspecting live state

GET /status merges two sources, because neither alone tells the whole story:

• The workflow's own get_status query: current step, per-step retry counts, last five errors, remaining budget. These counts are recorded from activity.info().attempt and so only land once a step succeeds.
• Temporal's describe(): pending_activities with live attempt, maximum_attempts and last_failure. This is what shows you a step that is currently on attempt 4 of 8 and still failing.

---

The 15-second constraint

flaky_call() raises ~⅓ of the time and sleeps 300s another ⅓, so it succeeds only about a third of the time and the median activity needs ~3 attempts. Retries are the normal case here, not the exception, which makes the cost of a single retry the number that decides whether 15s is achievable.

The measurement that drove the design

The obvious approach (tight start_to_close_timeout, let Temporal's RetryPolicy handle the rest) does not work, and it's worth saying why explicitly.

Temporal's timer queue resolves at about one second, so sub-second retry backoffs are rounded up. Configured at 50ms and 100ms, three consecutive attempts that each raised instantly still landed 1.016s and 0.993s apart:

attempt 1 failed  12:58:32.371
attempt 2 failed  12:58:33.387   <- +1.016s
attempt 3 failed  12:58:34.380   <- +0.993s

At ~1s of enforced backoff per retry, six activities needing ~2 retries each spend ~12s just waiting, and the budget is gone before any work happens. An earlier version of this code failed exactly that way.

What actually keeps runs inside 15s

1. Expected flakiness is absorbed in-activity, where a retry is free. app/activities.py wraps each provided.py call in a bounded loop (12 attempts, 150ms each) so the common case never pays Temporal's ~1s timer granularity. 0.67¹² ≈ 0.8% of activities fall through to the Temporal layer.
2. The Temporal RetryPolicy stays in place as the outer net, for the failures it's genuinely good at: worker crashes, poisoned tasks, infrastructure faults. Both layers log their attempts, and retry_counts in the status query sums them.
3. A wall-clock budget, not just an attempt count. The workflow computes deadline = start_time + 14s and passes the entire remaining budget as schedule_to_close_timeout. Capping per-activity was tried and was wrong: it partitions the slack, giving each step its own independent chance of running out, which compounds across six of them. Pooling lets an unlucky step borrow from the lucky ones.
4. A bounded review gate, closing at whichever comes first: 5s of review, or the point where payment and shipping would no longer fit.

The client sets execution_timeout = 15s as a hard backstop; the internal budget is 14s so the workflow records a clean terminal state before Temporal would kill the execution outright.

Measured

12 consecutive orders through the full stack, happy path:

completed: 12/12
min 1.22s  max 3.32s  mean 2.10s
within 15s SLA: 12/12

Cancel-at-the-gate closes in ~2.1s; letting the review timer expire closes in ~5.4s (the 5s window plus teardown). Reproduce with python scripts/demo.py, --cancel, and --no-approve.

tests/test_budget.py asserts these constants stay mutually consistent, including that the inner loop fits inside start_to_close and that fallthrough to Temporal stays under 1%, so a future tuning change that breaks the SLA fails the suite rather than failing in production.

---

Signals, timers, and the parent/child handshake

Manual review is a timer and a signal. The workflow waits on wait_condition(approved or cancelled, timeout=review_deadline). Approve, and it proceeds to payment; cancel, and it unwinds; do neither, and the timer fires and the order is cancelled with reason manual review window expired. The wait also wakes on a pending address change, so an UpdateAddress that arrives mid-review is written to Postgres immediately instead of sitting in workflow memory.

Dispatch failure is the parent/child handshake. ShippingWorkflow gives dispatch_carrier two attempts, then signals DispatchFailed(reason) to the parent via an external workflow handle and blocks on wait_condition. The parent records the failure in its status, and, if there's budget left, signals RetryDispatch back. The loop is bounded at 3 rounds; the child also gives up if the parent doesn't answer within 3s, so a dead parent can't hang it.

Cancellation is checked at each step boundary and always persisted: orders.state becomes cancelled and an order_cancelled event is written, using a fixed short timeout rather than the (possibly exhausted) budget, so a cancellation is recorded even on a run that's out of time.

One gap worth naming: cancelling after payment has been charged is recorded but not compensated. A real system needs a refund activity there. It's flagged in app/order_workflow.py at the check after charge_payment.

---

Persistence

migrations/001_init.sql is mounted into docker-entrypoint-initdb.d, so it runs once, on first volume creation. After changing it: docker compose down -v && docker compose up.

orders(id PK, state, address_json, created_at, updated_at)
payments(payment_id PK, order_id FK, status, amount, created_at)
events(id PK, order_id, type, payload_json, ts)

Why these three. orders is the state machine: received → validated → shipped, or cancelled. payments is keyed on payment_id rather than order_id precisely so the primary key can carry the idempotency guarantee. events is an append-only trace for debugging; nothing reads it for control flow, and log_event swallows its own errors so a failed trace write can never take down an order.

Idempotent payment

payment_charged does two things, and the second is what actually matters:

existing = await db.get_payment(payment_id)
if existing is not None:
    return {"status": existing["status"], "amount": existing["amount"]}   # fast path

inserted = await db.insert_payment_if_new(payment_id, order_id, "charged", amount)
if not inserted:
    existing = await db.get_payment(payment_id)                           # lost the race
    return {"status": existing["status"], "amount": existing["amount"]}

The get_payment pre-check handles the ordinary case, where Temporal retried the activity after a timeout, and this payment_id was already charged. But it's a read followed by a write, so on its own it's racy: two workers can both read "no payment" and both charge.

insert_payment_if_new closes that with INSERT ... ON CONFLICT (payment_id) DO NOTHING, and returns whether this statement was the one that inserted, by parsing asyncpg's command tag (INSERT 0 1 vs INSERT 0 0). The loser of the race re-reads and returns the winner's result. The charge is recorded exactly once regardless of how many retries or concurrent attempts hit the same payment_id.

The row is only written after flaky_call() returns, so a failed attempt leaves nothing behind and is safe to retry.

---

Tests

pytest

14 pass in ~4s with nothing else running (17 with Postgres up, see below). The workflow tests use Temporal's time-skipping WorkflowEnvironment; provided.py is patched in tests/conftest.py so failures are deterministic instead of being at the mercy of flaky_call().

Test	Covers
test_happy_path	Full run through both workflows, DB state ends shipped
test_cancel_before_shipment_is_persisted	Cancel at the gate; payment untouched, cancellation reaches the DB
test_manual_review_timer_expires	The review timer fires and cancels with no approval
test_update_address_is_persisted_before_dispatch	UpdateAddress mid-review reaches Postgres, not just memory
test_dispatch_failure_and_retry	Child fails past both retry layers → signals parent → parent signals retry → succeeds
test_budget.py (7)	The 15s SLA constants stay mutually consistent
test_payment_idempotency	Duplicate payment_id doesn't double-charge

Note on time skipping: the review gate is a real timer, and the time-skipping server will fast-forward straight through it before a test can send ApproveOrder. Tests that need to signal in real time wrap their body in env.auto_time_skipping_disabled(); test_manual_review_timer_expires leaves skipping on, which is exactly what makes the 5s timer fire instantly.

Integration tests

tests/test_db_idempotency.py exercises the real ON CONFLICT path, including five concurrent charges on one payment_id, which the fake DB in the unit test can't reach. It skips itself when Postgres isn't up, so pytest stays green without Docker:

docker compose up -d postgres
pytest -m integration

---

Logging

Structured JSON, one object per line (app/logging_config.py), so retries and state transitions are queryable rather than needing to be parsed out of a message string:

docker compose logs -f worker | jq 'select(.order_id == "order-1")'
docker compose logs worker | jq 'select(.attempt > 1) | {activity, attempt, order_id}'

Workflow-side logs go through workflow.logger, which suppresses duplicate lines during replay.

---

Layout

File
app/provided.py	The assignment's stubs. flaky_call() is verbatim; every function still calls it. DB writes fill the TODO spots.
app/activities.py	Activity wrappers, timeouts, retry policies
app/order_workflow.py	Parent workflow, signals, review gate, budget
app/shipping_workflow.py	Child workflow, dispatch retry handshake
app/db.py	asyncpg pool and queries
app/api.py	FastAPI: start, signals, status
app/worker.py	Both workers, one per task queue