docs(reflex): add calibration passport guidance

patterns/engine-calibration.md

@@ -46,6 +46,25 @@ Pool slices only when held-out evidence supports doing so. A threshold calibrate
 
 Changing the primitive also changes the measurement contract. A Binary formulation and a two-option Choice may express similar prose, but their probability behavior must be evaluated independently rather than assumed interchangeable.
 
+
+## Use a calibration passport
+
+A threshold is meaningful only together with the evidence that produced it. Treat calibration as a versioned artifact, not an unlabeled number in application code.
+
+Record at least:
+
+- Reflex/use-case and question-set version;
+- engine, model/checkpoint and runtime/backend version;
+- primitive type;
+- Choice cardinality bucket or Score rubric size where relevant;
+- workload/domain and language slice;
+- calibration corpus identity and immutable hash;
+- calibration method and artifact/version;
+- target branch/error-cost policy;
+- evaluation date and hardware/runtime notes when backend conversion can affect outputs.
+
+Two thresholds that happen to have the same numeric value are not interchangeable when their passports differ.
+
 ## Keep threshold policy replayable
 
 Store the raw semantic answer and probabilities separately from the host policy that interprets them.
@@ -61,6 +80,27 @@ For noisy state that crosses a threshold repeatedly, host policy may use hystere
 
 Version threshold and hysteresis policy independently from the Reflex definition and engine identity.
 
+
+## Distinguish answer probabilities from probability of correctness
+
+Some engines expose only a distribution over semantic answers. Others may also expose a separate confidence or correctness estimate.
+
+Those signals answer different questions:
+
+- **answer distribution** — which semantic answer the engine currently prefers and how probability mass is distributed among the declared answers;
+- **correctness/selective-automation signal** — how likely the selected answer is to be reliable enough for a policy decision.
+
+Do not assume the largest answer probability, distribution concentration, entropy-derived confidence or a vendor-named `confidence` field is automatically a calibrated probability of correctness.
+
+If an engine exposes multiple uncertainty signals:
+
+1. document exactly which signal the host policy uses;
+2. calibrate that signal on held-out target-workload data;
+3. keep the semantic answer/distribution separately for audit and replay;
+4. do not mix signals from different engines or model versions on one implicit scale.
+
+A Reflex contract should remain provider-neutral even when one engine has a richer uncertainty head than another.
+
 ## Do not tune away real errors
 
 Lowering a confidence threshold can reduce unnecessary review, but it also converts more model judgments into accepted branches.
@@ -73,6 +113,21 @@ Separate:
 
 Threshold tuning addresses only the first class.
 
+
+## Verify runtime and conversion fidelity separately
+
+Running the same checkpoint through a different serving/runtime path can change numerical behavior even when the public Reflex contract is unchanged.
+
+For local, edge or converted runtimes:
+
+1. freeze representative input/question fixtures;
+2. compare semantic answers and probability vectors against the admitted reference runtime;
+3. measure drift before measuring latency/throughput;
+4. record tokenizer/model/conversion/runtime hashes;
+5. reject a runtime port that materially changes decision quality, even if it is faster.
+
+Runtime fidelity is a separate gate from model quality. A faster backend is not a new engine identity, and a byte-compatible API is not proof of semantic equivalence.
+
 ## Benchmark layers
 
 Keep at least two evaluation layers:

skills/brida-reflex/references/verification.md

@@ -18,7 +18,8 @@ Record:
 
 - use-case ID;
 - definition/question/policy version;
-- engine/model when observable;
+- engine/model plus runtime/backend when observable;
+- calibration-profile/passport identity when policy uses calibrated thresholds;
 - corpus/fixture version;
 - semantic answer;
 - operational branch;
@@ -50,14 +51,27 @@ Measure separately:
 - raw semantic-label accuracy;
 - policy/branch accuracy under the chosen thresholds;
 - abstention/review rate;
-- calibration (for example ECE/Brier where appropriate);
+- calibration (for example ECE/Brier where appropriate), sliced by primitive/cardinality/workload when needed;
+- which uncertainty signal is being calibrated (answer distribution vs separate correctness/selective-automation signal);
 - selective accuracy at admitted coverage;
+- runtime/backend semantic fidelity before comparing runtime performance;
 - latency distribution;
 - error/timeout coverage;
 - per-use-case results.
 
 Calibrate on train/dev data and report holdout separately. Never tune thresholds on the same holdout later reported as unseen evidence.
 
+
+## Calibration passport
+
+When a branch threshold depends on engine probabilities, record the evidence tuple that makes that threshold interpretable:
+
+`Reflex/question version + engine/checkpoint + runtime/backend + primitive/cardinality + workload slice + calibration corpus/hash + calibration artifact/version`.
+
+Do not copy a threshold to another engine, model version, primitive, option-count bucket or converted runtime just because the public request/response schema matches.
+
+If the engine exposes both answer probabilities and a separate confidence/correctness signal, preserve both. Document which one host policy consumes and validate it independently.
+
 ## Claims
 
 Third-party benchmark numbers are research inputs, not Brida claims.