Repository navigation
item 8: the OpenTelemetry sink - #15
Merged
Merged
Conversation
SPEC-v0.2 §8, acceptance test T29, and T30 now exists as a named test.
One span per action, opened on ACTION_PROPOSED with one span event per Event.
A retry is a new action and a new span; the two are correlated by
ctrlrun.effect_key, which is the identity that actually spans attempts.
Two things §8 leaves to the implementation, both resolved in the module's own
words rather than silently:
The span ends on the receipt, one call after the terminal event §8 names.
ctrlrun.receipt_id, ctrlrun.result and the final attempt live on the receipt,
and a receipt always follows a terminal event in the same Control call. Ending
a step earlier would export a span that cannot say how the action came out,
which is the only thing a reader wants from one. An action awaiting a human
gets no receipt, so its span stays open — §8 already accepts that, and the
open-span map is bounded so a long-running gateway does not leak one per
abandoned approval. Evicted spans are dropped rather than ended: ending one
would assert an outcome that never happened.
Argument values are withheld by default, and ctrlrun.effect_key is exported
always — but an effect key is built from arguments, so refund:{payment_id}
puts a payment id in every backend receiving these spans. That is intended:
§8 names the effect key as what correlates a retry, and a correlation key
nobody can see correlates nothing. It is now asserted by a test rather than
discovered, because it makes the choice of effect template a choice about what
leaves the building.
Span status is OK for committed, ERROR for failed and ambiguous, and unset for
denied and blocked. A refusal is the system working as designed; marking it an
error teaches an on-call rotation to ignore the signal that matters.
The gateway grows --otel and --otel-arguments, and Control exposes its sinks
so one more can be composed onto it. CI installs the extra.
Mutation-testing §8 found five gaps, and the last one was worth the whole exercise. A receipt whose span had been evicted could have ended somebody else's span — closing a live action and stamping it with a different action's outcome. The window is narrow and real: a suspended action is the only way a receipt arrives long after its ACTION_PROPOSED under the same action_id (§6.9), so if its span is evicted meanwhile, that receipt has no span to end. A wrong trace is worse than a missing one, and nothing tested it. My first attempt at that test was wrong in an instructive way: I reached for a re-presented approval, which is a *new* action_id with its own span, so the window never opened. Suspended/resume is the only path that reopens it. The other four: an event for an action the sink never saw start (a restarted process, a `ctrlrun resolve` days later) must open no span; an evicted span is dropped rather than ended, because ending one asserts an outcome that never happened; a receipt ends only its own action's span; and an argument value OpenTelemetry cannot carry is dropped rather than stringified, since an attribute whose type changed on the way out is worse than one that is absent. M20 is an equivalent mutant against the import test and was retargeted: a module-scope opentelemetry import is invisible to `import ctrlrun`, because __init__ does not import ctrlrun.otel. It is caught by T30's MissingDependency test, which is where the guarantee actually lives.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The other half of item 8. SPEC-v0.2 §8, acceptance test T29 — and T30 now exists as a named test, so every §10 acceptance test ID is present.
One span per action, opened on
ACTION_PROPOSED, one span event perEvent. A retry is a new action and a new span, correlated byctrlrun.effect_key. StatusOKfor committed,ERRORfor failed and ambiguous, unset for denied and blocked — a refusal is the system working as designed, and marking it an error teaches an on-call rotation to ignore the signal that matters.Two things §8 leaves to the implementation
The span ends on the receipt, one call after the terminal event §8 names.
ctrlrun.receipt_id,ctrlrun.resultand the finalattemptlive on the receipt, and a receipt always follows a terminal event in the sameControlcall. Ending a step earlier would export a span that cannot say how the action came out — the only thing a reader wants from one. An action awaiting a human gets no receipt, so its span stays open; §8 already accepts that, and the open-span map is bounded so a long-running gateway does not leak one per abandoned approval. Evicted spans are dropped, never ended: ending one asserts an outcome that never happened.Argument values are withheld by default, and
ctrlrun.effect_keyis exported always — but an effect key is built from arguments.refund:{payment_id}puts a payment id in every backend receiving these spans. That is intended: §8 names the effect key as what correlates a retry, and a correlation key nobody can see correlates nothing. It is now asserted by a test rather than left to be discovered, because it makes the choice of effect template a choice about what leaves the building.Mutation table — 23 mutations, all red
MissingDependency, lazy import, the extra's contents, non-exportable valuesFive started green, and the last one was worth the whole exercise.
A receipt whose span had been evicted could have ended somebody else's span — closing a live action and stamping it with a different action's outcome. A wrong trace is worse than a missing one, and nothing tested it.
The window is narrow and real: a suspended action is the only way a receipt arrives long after its
ACTION_PROPOSEDunder the sameaction_id(§6.9), so if its span is evicted meanwhile, that receipt has no span to end. My first attempt at the test reached for a re-presented approval — which is a newaction_idwith its own span, so the window never opened. OnlySuspended/resumereopens it. That is a small lesson in its own right: a test for a rare window has to reproduce the window, and "something similar happened" is not the same thing.The other four: an event for an action the sink never saw start (a restarted process, a
ctrlrun resolvedays later) must open no span; an evicted span is dropped rather than ended; a receipt ends only its own action's span; and an argument value OpenTelemetry cannot carry is dropped rather than stringified — an attribute whose type changed on the way out is worse than one that is absent, because a query written against it silently stops matching.M20 is an equivalent mutant against the import test and was retargeted: a module-scope
opentelemetryimport is invisible toimport ctrlrun, because__init__does not importctrlrun.otel. It is caught by T30'sMissingDependencytest, which is where the guarantee actually lives.Also
ctrlrun gateway --otel [--otel-arguments], andControl.sinksso one more sink can be composed onto aControl.from_file(). CI installs.[dev,gateway,otel]in both jobs. Theotelextra declares the API, the SDK and the OTLP/HTTP exporter — the API alone would do for a library, but an operator installing this wants to export without assembling a stack.Where v0.2 stands
Every acceptance test ID in §10 now exists: T13–T31, T26b, T30b, and T51–T55. Items 0–8 are done. Item 9 is the release pass — changelog,
docs/CLAIMS.mdregenerated, README, ROADMAP reconciled.1269 tests pass (1241 → 1269),
mypy --strict src/,ruff checkandruff format --checkclean.