Skip to content

item 8: the OpenTelemetry sink - #15

Merged
arpanghoshal merged 2 commits into
mainfrom
item-8-otel
Sep 3, 2026
Merged

arpanghoshal merged 2 commits into
mainfrom
item-8-otel

Conversation

@arpanghoshal

Copy link
Copy Markdown
Member

The other half of item 8. SPEC-v0.2 §8, acceptance test T29 — and T30 now exists as a named test, so every §10 acceptance test ID is present.

One span per action, opened on ACTION_PROPOSED, one span event per Event. A retry is a new action and a new span, correlated by ctrlrun.effect_key. Status OK for committed, ERROR for failed and ambiguous, unset for denied and blocked — a refusal is the system working as designed, and marking it an error teaches an on-call rotation to ignore the signal that matters.

Two things §8 leaves to the implementation

The span ends on the receipt, one call after the terminal event §8 names. ctrlrun.receipt_id, ctrlrun.result and the final attempt live on the receipt, and a receipt always follows a terminal event in the same Control call. Ending a step earlier would export a span that cannot say how the action came out — the only thing a reader wants from one. An action awaiting a human gets no receipt, so its span stays open; §8 already accepts that, and the open-span map is bounded so a long-running gateway does not leak one per abandoned approval. Evicted spans are dropped, never ended: ending one asserts an outcome that never happened.

Argument values are withheld by default, and ctrlrun.effect_key is exported always — but an effect key is built from arguments. refund:{payment_id} puts a payment id in every backend receiving these spans. That is intended: §8 names the effect key as what correlates a retry, and a correlation key nobody can see correlates nothing. It is now asserted by a test rather than left to be discovered, because it makes the choice of effect template a choice about what leaves the building.

Mutation table — 23 mutations, all red

# Group Result
M1–M5 one span per action, named, ended; every event a span event red
M6–M11 every §8 attribute; argument values withheld by default; the effect key always red
M12–M14 span status, including unset for denied and blocked red
M15–M18 the bounded map, eviction, receipt-to-span matching, attempt red
M19–M23 MissingDependency, lazy import, the extra's contents, non-exportable values red

Five started green, and the last one was worth the whole exercise.

A receipt whose span had been evicted could have ended somebody else's span — closing a live action and stamping it with a different action's outcome. A wrong trace is worse than a missing one, and nothing tested it.

The window is narrow and real: a suspended action is the only way a receipt arrives long after its ACTION_PROPOSED under the same action_id (§6.9), so if its span is evicted meanwhile, that receipt has no span to end. My first attempt at the test reached for a re-presented approval — which is a new action_id with its own span, so the window never opened. Only Suspended/resume reopens it. That is a small lesson in its own right: a test for a rare window has to reproduce the window, and "something similar happened" is not the same thing.

The other four: an event for an action the sink never saw start (a restarted process, a ctrlrun resolve days later) must open no span; an evicted span is dropped rather than ended; a receipt ends only its own action's span; and an argument value OpenTelemetry cannot carry is dropped rather than stringified — an attribute whose type changed on the way out is worse than one that is absent, because a query written against it silently stops matching.

M20 is an equivalent mutant against the import test and was retargeted: a module-scope opentelemetry import is invisible to import ctrlrun, because __init__ does not import ctrlrun.otel. It is caught by T30's MissingDependency test, which is where the guarantee actually lives.

Also

ctrlrun gateway --otel [--otel-arguments], and Control.sinks so one more sink can be composed onto a Control.from_file(). CI installs .[dev,gateway,otel] in both jobs. The otel extra declares the API, the SDK and the OTLP/HTTP exporter — the API alone would do for a library, but an operator installing this wants to export without assembling a stack.

Where v0.2 stands

Every acceptance test ID in §10 now exists: T13–T31, T26b, T30b, and T51–T55. Items 0–8 are done. Item 9 is the release pass — changelog, docs/CLAIMS.md regenerated, README, ROADMAP reconciled.

1269 tests pass (1241 → 1269), mypy --strict src/, ruff check and ruff format --check clean.

SPEC-v0.2 §8, acceptance test T29, and T30 now exists as a named test.

One span per action, opened on ACTION_PROPOSED with one span event per Event.
A retry is a new action and a new span; the two are correlated by
ctrlrun.effect_key, which is the identity that actually spans attempts.

Two things §8 leaves to the implementation, both resolved in the module's own
words rather than silently:

The span ends on the receipt, one call after the terminal event §8 names.
ctrlrun.receipt_id, ctrlrun.result and the final attempt live on the receipt,
and a receipt always follows a terminal event in the same Control call. Ending
a step earlier would export a span that cannot say how the action came out,
which is the only thing a reader wants from one. An action awaiting a human
gets no receipt, so its span stays open — §8 already accepts that, and the
open-span map is bounded so a long-running gateway does not leak one per
abandoned approval. Evicted spans are dropped rather than ended: ending one
would assert an outcome that never happened.

Argument values are withheld by default, and ctrlrun.effect_key is exported
always — but an effect key is built from arguments, so refund:{payment_id}
puts a payment id in every backend receiving these spans. That is intended:
§8 names the effect key as what correlates a retry, and a correlation key
nobody can see correlates nothing. It is now asserted by a test rather than
discovered, because it makes the choice of effect template a choice about what
leaves the building.

Span status is OK for committed, ERROR for failed and ambiguous, and unset for
denied and blocked. A refusal is the system working as designed; marking it an
error teaches an on-call rotation to ignore the signal that matters.

The gateway grows --otel and --otel-arguments, and Control exposes its sinks
so one more can be composed onto it. CI installs the extra.
Mutation-testing §8 found five gaps, and the last one was worth the whole
exercise.

A receipt whose span had been evicted could have ended somebody else's span —
closing a live action and stamping it with a different action's outcome. The
window is narrow and real: a suspended action is the only way a receipt arrives
long after its ACTION_PROPOSED under the same action_id (§6.9), so if its span
is evicted meanwhile, that receipt has no span to end. A wrong trace is worse
than a missing one, and nothing tested it.

My first attempt at that test was wrong in an instructive way: I reached for a
re-presented approval, which is a *new* action_id with its own span, so the
window never opened. Suspended/resume is the only path that reopens it.

The other four: an event for an action the sink never saw start (a restarted
process, a `ctrlrun resolve` days later) must open no span; an evicted span is
dropped rather than ended, because ending one asserts an outcome that never
happened; a receipt ends only its own action's span; and an argument value
OpenTelemetry cannot carry is dropped rather than stringified, since an
attribute whose type changed on the way out is worse than one that is absent.

M20 is an equivalent mutant against the import test and was retargeted: a
module-scope opentelemetry import is invisible to `import ctrlrun`, because
__init__ does not import ctrlrun.otel. It is caught by T30's MissingDependency
test, which is where the guarantee actually lives.
@arpanghoshal
arpanghoshal merged commit 467766e into main Sep 3, 2026
3 checks passed
@arpanghoshal
arpanghoshal deleted the item-8-otel branch September 3, 2026 21:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant