From f8a4dad0844cbd67dd0c8cb97fc9577fb8e6e9e3 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Mon, 31 Aug 2026 07:35:42 +0000 Subject: [PATCH 1/5] feat(analysis): bind validation runs to scientific acceptance evidence GAP-003A first slice for issue #166. submit_validation_run binds cutoff-eligible evidence, snapshot, knowledge cutoff, CPU f64 model, seed, backend, and precision to a hash-stable tepp-validation-{32 hex} receipt that carries no scientific metrics. complete_validation_run emits tepp.scientific_acceptance.v1 through validation_core (RMSE, bias, Wilson coverage, temporal-order accuracy, SE-aware gate). LLM-authored recovery, non-finite inputs, empty or duplicate evidence, snapshot mismatch, and cutoff-empty corpora fail closed. Not implemented-main. Postgres persistence remains GAP-003B. ADR 0026. --- ARCHITECTURE.md | 4 +- CHANGELOG.md | 2 + Cargo.lock | 1 + DOCUMENTATION.md | 2 + crates/analysis_engine/Cargo.toml | 1 + crates/analysis_engine/src/lib.rs | 58 +- crates/analysis_engine/src/validation_run.rs | 891 ++++++++++++++++++ .../tests/validation_run_contract.rs | 103 ++ docs/TRACEABILITY.md | 1 + ...26-validation-run-scientific-acceptance.md | 88 ++ docs/adr/README.md | 1 + docs/product-technical-gap-baseline.md | 2 +- .../validation-run-scientific-acceptance.md | 32 + 13 files changed, 1182 insertions(+), 4 deletions(-) create mode 100644 crates/analysis_engine/src/validation_run.rs create mode 100644 crates/analysis_engine/tests/validation_run_contract.rs create mode 100644 docs/adr/0026-validation-run-scientific-acceptance.md create mode 100644 docs/research/validation-run-scientific-acceptance.md diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index e6dafbcc2..479f84d12 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -70,7 +70,7 @@ boundaries above remain the target modular MSA architecture. | `tepp_simulation` | known-truth temporal/event data generation | | `validation_core` | RMSE, bias, coverage, graph, Monte Carlo, and exact-head claim-promotion metrics | | `tepp_api` | versioned DTO, schema, terminal-result, and export contracts | -| `analysis_engine` | bounded cutoff-safe temporal evidence readiness execution and digest-bound terminal artifacts | +| `analysis_engine` | bounded cutoff-safe temporal evidence readiness execution, digest-bound terminal artifacts, and GAP-003A scientific-acceptance validation runs (`tepp.scientific_acceptance.v1`; not implemented-main) | | `episode_membership` | event-time episode membership containment gate | | `prompt_source` | prompt boilerplate is not unique latent content and not stopword deletion | | `corpus_background` | corpus-background wording is not unique latent content and not stopword deletion | @@ -113,7 +113,7 @@ boundaries above remain the target modular MSA architecture. | `episode_membership` | episode membership cannot escape the episode event-time interval | | `membership_target` | language, episode, template, department, and opportunity-pool targets cannot collapse into entity or project | | `topic_measurement` | logistic-normal ALR/ILR coordinates and the CPU `f64` TRSL-TM reference estimator | -| `analysis_engine` | bounded cutoff-safe temporal evidence readiness execution and digest-bound terminal artifacts | +| `analysis_engine` | bounded cutoff-safe temporal evidence readiness execution, digest-bound terminal artifacts, and GAP-003A scientific-acceptance validation runs (`tepp.scientific_acceptance.v1`; not implemented-main) | | `psychometric_core` | posterior-aware structural input gates, CWC within/between OLS plus the contextual effect, event-time log-rate, unequal-interval discrete-lag remapping, constant-predictor discrete effect, time-varying-predictor discrete effect (Eq. 14), exact scalar discrete process noise (Driver et al., 2017, Eq. 3), lagged latent covariance and unconditional latent variance (Driver et al., 2017, Eq. 3–4), stationary within-subject variance (Driver et al., 2017, Eq. 4 as `Δt → ∞`; `asymDIFFUSION`), trait-plus-state variance (Driver et al., 2017, §4.3 `TRAITVAR`; not process noise), observed-indicator variance and lagged observed covariance (Driver et al., 2017, Eq. 5; Table 2 `MANIFESTVAR` is `Θ`, not `Var(y)`; `MANIFESTTRAITVAR` is not `MANIFESTVAR`; `Θ` does not enter lagged observed covariance; observed-indicator mean is `τ + λ μ`; `MANIFESTMEANS` is not `E(y)`; `CINT` is not `MANIFESTMEANS`; discrete latent mean is `exp(a Δt) μ_0 + (exp(a Δt) − 1)/a κ`; `T0MEANS` is not `μ_t`; `CINT` is not the discrete increment; evolved observed mean is `τ + λ μ_t`; `τ + λ μ_0` is not `E(y_t)`; contemporaneous `TDPREDEFFECT` impulse is `m x`, not `CINT`, not `TIPREDEFFECT`, and not Voelkle Eq. 14; Eq. 5 of that contemporaneous impulse is `τ + λ(μ_t + m x)`, and `τ + λ μ_t` is not that observed mean; time-independent `TIPREDEFFECT` increment is `A^{-1}[e^{A Δt} − I] B z`, not `CINT`, not `M x`, not Voelkle Eq. 14, and not the coefficient `B`; Eq. 5 of that increment is `τ + λ(μ_t + A^{-1}[e^{A Δt} − I] B z)`, and `τ + λ μ_t` is not that observed mean; `τ + λ(μ_t + m x)` is not that observed mean; `τ + λ(μ_t + e^{a(t−u)} m x)` is not that observed mean when `u ≠ t`; within-interval `TDPREDEFFECT` carry is `e^{A(t−u)} M x` for `t0 < u < t`, not the contemporaneous Dirac, not `CINT`, not `TIPREDEFFECT`, and not Voelkle Eq. 14; Eq. 5 of that carry is `τ + λ(μ_t + e^{a(t−u)} m x)`, and `τ + λ μ_t` is not that observed mean; `τ + λ(μ_t + m x)` is not that carried observed mean when `u ≠ t`; first-occasion `T0TIPREDEFFECT` shift is `t0_b z` and Eq. 3 first-summand carry is `e^{A Δt} t0_b z` (`T0TIPREDEFFECT` is not `TIPREDEFFECT` `B`; `t0_b z` is not `A^{-1}[e^{A Δt} − I] B z`; `e^{A Δt} t0_b z` is not `t0_b z`; Eq. 5 of that carry is `τ + λ(μ_t + e^{a Δt} t0_b z)`, and `τ + λ μ_t` is not that observed mean; `τ + λ(μ_t + A^{-1}[e^{A Δt} − I] B z)` is not that observed mean), first-occasion `T0TDPREDEFFECT` shift is `t0_m x0` and Eq. 3 first-summand carry is `e^{A Δt} t0_m x0` (`T0TDPREDEFFECT` is not `TDPREDEFFECT` `M`; `t0_m x0` is not `M x`; `e^{A Δt} t0_m x0` is not `t0_m x0`; `e^{A Δt} t0_m x0` is not `e^{A(t−u)} M x` for `t0 < u < t`; `t0_m x0` is not `t0_b z`; an impulse at `u ≤ t0` that used `M` is already in `η(t0)` as `TDPREDEFFECT`, not as `T0TDPREDEFFECT`; Eq. 5 of that carry is `τ + λ(μ_t + e^{a Δt} t0_m x0)`, and `τ + λ μ_t` is not that observed mean; `τ + λ(μ_t + A^{-1}[e^{A Δt} − I] B z)` is not that observed mean; `τ + λ(μ_t + e^{a Δt} t0_b z)` is not that observed mean; §7.2 level-change `CINT` is `κ = −a m x` with `a < 0` so `−κ / a = m x` (`−a m x` is not the dissipating Dirac, not a free `CINT`, not `TIPREDEFFECT`, and not the extra near-zero-drift latent process also named in §7.2; Eq. 3 of that setting is `(1 − e^{a Δt}) m x`, which is not `m x`, not `κ`, and not `TIPREDEFFECT`; §7.2 extra-process contribution is `a_{ηξ} x (e^{ε Δt} − e^{a Δt}) / (ε − a)` (`ε = a` is `a_{ηξ} x Δt e^{a Δt}`; identification `TDPREDEFFECT` on the extra process is 1; printed extra `DRIFT` is `−0.000001`; not `κ = −a m x`, not `(1 − e^{a Δt}) m x`, and not the dissipating Dirac `m x`; `ε ≥ 0` fails closed; Eq. 5 of that contribution is `τ + λ(μ_t + a_{ηξ} x (e^{ε Δt} − e^{a Δt}) / (ε − a)`; the extra process has `LAMBDA` 0 and is not an observed indicator; `τ + λ μ_t` is not that observed mean; `τ + λ(μ_t + m x)` is not that observed mean; the contribution is not `E(y_t)`; the evolved-plus-contribution latent mean is not `E(y_t)`; after-t0 extra-process `TDPREDEFFECT` is `a_{ηξ} x (e^{ε(t−u)} − e^{a(t−u)}) / (ε − a)` for `t0 < u < t` while `μ_t` uses `Δt`; Eq. 5 of that after-t0 contribution is `τ + λ(μ_t + a_{ηξ} x (e^{ε(t−u)} − e^{a(t−u)}) / (ε − a)`; the first-occasion extra-process observed mean is not that observed mean when `u ≠ t0`; `e^{a(t−u)} m x` is a Dirac on the original process, not this `DRIFT` drive; §7.2 `asymTIPREDEFFECT` is `-B z / a` for `a < 0` (`-B z / a` is not the coefficient `B`, not `A^{-1}[e^{A Δt} − I] B z`, not `CINT`, and not `M x`; §7.2 `addedTIPREDVAR` is `(B / a)² v`, not `TRAITVAR`, not `asymDIFFUSION`, and not `-B z / a`; Table 2 `asymCINT` is `-κ / a` for `a < 0` and is not `κ`, not `A^{-1}[e^{A Δt} − I] κ`, not `T0MEANS`, and not `-B z / a`; p. 16 stationary `T0MEANS` is `-κ / a + −B z / a` and is not free `T0MEANS`, not `asymCINT` alone, not `asymTIPREDEFFECT` alone, and not the finite-interval discrete latent mean; Eq. 5 of that constrained mean is `τ + λ(−κ / a + −B z / a)`; `τ + λ μ_0` is not that observed mean; `τ + λ(−κ / a)` is not that observed mean when `B z ≠ 0`; `τ + λ μ_t` is not that observed mean; `MANIFESTMEANS` is not `E(y_0)`; the constrained latent mean is not `E(y_0)`; stationary `T0VAR` is `trait + −q / (2 a) + (B / a)² v` (not free `T0VAR`, not `asymDIFFUSION` alone, not `TRAITVAR` alone, not `addedTIPREDVAR` alone, and not the finite-interval discrete latent variance. Eq. 5 of that constrained variance is `λ²(trait + −q / (2 a) + (B / a)² v) + θ + ψ` (JSS PDF re-opened 2026-08-22T03:20Z; form the stationary latent variance first, then `λ² p + θ + ψ`; `λ² p_0` is not that observed variance; `λ²(−q / (2 a)) + θ` is not that observed variance when `TRAITVAR` or `addedTIPREDVAR` is nonzero; `MANIFESTVAR` is not `Var(y_0)`; the constrained latent variance is not `Var(y_0)`); lagged stationary `T0VAR` is `trait + e^{a Δt}(−q / (2 a)) + (B / a)² v` (trait and `addedTIPREDVAR` do not decay; contemporaneous `T0VAR` is not that lagged map; decaying the constrained total as if it were all state is not that lagged map; Eq. 5 of that lagged covariance is `λ²(trait + e^{a Δt}(−q / (2 a)) + (B / a)² v) + ψ`; `Θ` does not enter; contemporaneous `Var(y_0)` is not that lagged observed covariance; the lagged latent covariance is not that observed covariance); later-occasion stationary `T0VAR` is `trait + e^{2 a Δt}(−q / (2 a)) + Q_Δt + (B / a)² v` (trait and `addedTIPREDVAR` do not enter `Q_Δt`; under stationarity that composition equals contemporaneous `T0VAR`; evolving the constrained total as if it were all state is not that later map; the lagged covariance omits `Q_Δt`; `Q_Δt` is not that later map; Eq. 5 of that later-occasion variance is `λ²(trait + e^{2 a Δt}(−q / (2 a)) + Q_Δt + (B / a)² v) + θ + ψ`; lagged observed covariance omits `Q_Δt` and `θ`; `MANIFESTVAR` is not `Var(y_t)`; the later-occasion latent variance is not `Var(y_t)`))), irregular already-centered residual lag, Rubin `T` on OLS loadings, and strong-gated latent means (two-observation residual variance is identically `0` and caps at strong/scalar; Putnick & Bornstein, 2016) | | `validation_core` | RMSE, bias, coverage, graph, and Monte Carlo metrics | | `tepp_api` | versioned DTO, schema, and export contracts | diff --git a/CHANGELOG.md b/CHANGELOG.md index 062a69412..31b32a03f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -38,6 +38,8 @@ All notable changes to TEPP are documented here. The format follows Keep a Chang ## [Unreleased] +- `analysis_engine` GAP-003A first slice (ADR 0026, active-PR, not implemented-main): `submit_validation_run` binds cutoff-eligible evidence identities, snapshot, knowledge cutoff, CPU `f64` model, seed, backend, and precision to a hash-stable `tepp-validation-{32 hex}` receipt that carries no scientific metrics. `complete_validation_run` emits `tepp.scientific_acceptance.v1` with RMSE, bias, Wilson coverage, temporal-order accuracy, and an SE-aware gate through `validation_core`. LLM-authored recovery, non-finite inputs, empty or duplicate evidence, snapshot mismatch, and cutoff-empty corpora fail closed. Postgres persistence remains GAP-003B. + - `event_core` adds bounded Allen interval-consistency classification, atomic path-consistency closure, contradiction/resource refusals, and an explicit dependency-error fallback without claiming unrestricted global satisfiability. - `psychometric_core` recovers the Driver, Oud, and Voelkle (2017, Table 2, p. 12 `MANIFESTTRAITVAR`; §7.1, p. 19; p. 16 `MANIFESTTRAITVARstd`; footnote 4; 2017-era ctsem `summary.ctsemFit.R`; JSS PDF re-opened 2026-08-27T14:20Z from https://www.jstatsoft.org/index.php/jss/article/download/v077i05/1104) scalar standardised manifest-trait variance on current main after `0ce16e8` dropped the pre-consolidation code while research notes already named the map (register items 83–84). Table 2 names `MANIFESTTRAITVAR` `Ψ_τ` the additional time-invariant variance-covariance on the measurement level and sets it `NULL` when there is no manifest trait. Equation 5 writes `Γ ~ N(τ, Ψ)` and names that covariance the manifest traits. Section 7.1 names manifest traits stable individual differences in indicator levels, distinct from process-level `TRAITVAR` `φ_ξ`. Page 16 prints standardised matrices with the suffix `std` when appropriate. The printed example on p. 16 is `discreteDRIFTstd`, not `MANIFESTTRAITVARstd`. Footnote 4 standardises using only the relevant variance, not the total. The relevant variance for that named indicator-level correlation is `MANIFESTTRAITVAR`, not process-level `TRAITVAR` and not residual `MANIFESTVAR` `θ`. The 2017-era source forms `MANIFESTTRAITVARstd` only when `MANIFESTTRAITVAR != 0`, as `solve(sqrt(diag(MANIFESTTRAITVAR) + ridging)) %&% MANIFESTTRAITVAR` when `verbose = TRUE`. OpenMx `%&%` is `t(A) %*% B %*% A`. Unlike `TRAITVARstd`, that formation adds `diag(c(ridging), n.manifest)`. The default `ridging = FALSE` adds 0, not `0.0001`; that ridge is a numerical hack and is not this exact map. The scalar correlation is `ψ / ψ = 1` after strictly positive `MANIFESTTRAITVAR`. Form strictly positive `ψ` first, then `1 / √ψ`, then `(1 / √ψ) ψ (1 / √ψ)`. Unstandardised `MANIFESTTRAITVAR` is defined for a zero trait; standardised `MANIFESTTRAITVAR` is not. Zero `MANIFESTTRAITVAR` skips forming `MANIFESTTRAITVARstd` in the 2017-era source and fails closed here. Indicator-level trait variance is an event-time structural quantity, so a non-event clock fails closed. `MANIFESTTRAITVAR` does not require stable `a < 0`. Distinct positive `ψ` recover the same 1. `trait / trait = 1` is `TRAITVARstd` and recovers the same number and remains a distinct named quantity. `θ` is `MANIFESTVAR` and is measurement error, not this correlation. Meredith (1993) remains unread (web search 2026-08-27T14:20Z: Springer/Cambridge Core paywalled; Unpaywall historically `is_oa: false`; Springer `content/pdf` is an HTML stub). Mislevy (1991, *Psychometrika, 56*, 177–196) remains unread on the same terms (DOI `10.1007/bf02294457`). Still not a Kalman filter, not a matrix `expm`, not ESEM estimation, not DSEM, and not ctsem estimation. diff --git a/Cargo.lock b/Cargo.lock index 454a7d612..feb37e96e 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -82,6 +82,7 @@ dependencies = [ "tepp_api", "topic_measurement", "uuid", + "validation_core", ] [[package]] diff --git a/DOCUMENTATION.md b/DOCUMENTATION.md index 6fa4b9683..0d18e0e6f 100644 --- a/DOCUMENTATION.md +++ b/DOCUMENTATION.md @@ -58,6 +58,7 @@ TEPP's approved PRD v0.4 and implementation plan are the primary product baselin | Episode-membership identity doctoring | [`docs/research/episode-membership-identity.md`](docs/research/episode-membership-identity.md) | | Entity/project target SQL doctoring | [`docs/research/entity-project-sql.md`](docs/research/entity-project-sql.md) | | Scientific claim-promotion gate doctoring | [`docs/research/scientific-claim-promotion-gates.md`](docs/research/scientific-claim-promotion-gates.md) | +| Validation-run scientific-acceptance doctoring | [`docs/research/validation-run-scientific-acceptance.md`](docs/research/validation-run-scientific-acceptance.md) | | Retention/deletion/legal-hold doctoring | [`docs/research/retention-deletion-legal-hold.md`](docs/research/retention-deletion-legal-hold.md) | | Provider-payload minimization doctoring | [`docs/research/provider-payload-minimization.md`](docs/research/provider-payload-minimization.md) | | Relation absence is not negative evidence | [`docs/research/relation-absence-not-negative.md`](docs/research/relation-absence-not-negative.md) | @@ -147,6 +148,7 @@ TEPP's approved PRD v0.4 and implementation plan are the primary product baselin | Episode-membership identity doctoring | [`docs/research/episode-membership-identity.md`](docs/research/episode-membership-identity.md) | | Entity/project target SQL doctoring | [`docs/research/entity-project-sql.md`](docs/research/entity-project-sql.md) | | Scientific claim-promotion gate doctoring | [`docs/research/scientific-claim-promotion-gates.md`](docs/research/scientific-claim-promotion-gates.md) | +| Validation-run scientific-acceptance doctoring | [`docs/research/validation-run-scientific-acceptance.md`](docs/research/validation-run-scientific-acceptance.md) | | Retention/deletion/legal-hold doctoring | [`docs/research/retention-deletion-legal-hold.md`](docs/research/retention-deletion-legal-hold.md) | | Stopword-deletion doctoring | [`docs/research/stopword-deletion.md`](docs/research/stopword-deletion.md) | | Provider-payload minimization doctoring | [`docs/research/provider-payload-minimization.md`](docs/research/provider-payload-minimization.md) | diff --git a/crates/analysis_engine/Cargo.toml b/crates/analysis_engine/Cargo.toml index 7322212b2..0d384448b 100644 --- a/crates/analysis_engine/Cargo.toml +++ b/crates/analysis_engine/Cargo.toml @@ -22,6 +22,7 @@ tepp_api = { path = "../tepp_api", version = "0.2.0" } temporal_core = { path = "../temporal_core", version = "0.2.0" } topic_measurement = { path = "../topic_measurement", version = "0.2.0" } uuid.workspace = true +validation_core = { path = "../validation_core", version = "0.2.0" } [dev-dependencies] corpus_split = { path = "../corpus_split", version = "0.2.0" } diff --git a/crates/analysis_engine/src/lib.rs b/crates/analysis_engine/src/lib.rs index 72bd5854c..4c5e4536f 100644 --- a/crates/analysis_engine/src/lib.rs +++ b/crates/analysis_engine/src/lib.rs @@ -14,6 +14,7 @@ mod case_deletion_refit; mod lineage_criterion; mod topic_context_posterior; mod topic_lineage_artifact; +mod validation_run; use serde::Serialize; use sha2::{Digest, Sha256}; @@ -26,6 +27,7 @@ use tepp_api::{ ApiError, }; use topic_measurement::TopicMeasurementError; +use validation_core::ValidationError; /// One document admitted to exhaustive case-deletion fitting. pub use case_deletion_refit::CaseDeletionDocument; @@ -58,6 +60,14 @@ pub use topic_lineage_artifact::{ TOPIC_LINEAGE_MODEL_CONTRACT_VERSION, TOPIC_LINEAGE_OUTPUT_PROFILE, TopicLineageArtifact, TopicLineageArtifactEdge, TopicLineageExecution, execute_topic_lineage_run, }; +/// Durable scientific-acceptance validation-run contracts. +pub use validation_run::{ + RecoveryObservation, SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE, + SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION, ScientificAcceptanceEvidence, VALIDATION_BACKEND, + VALIDATION_CPU_F64_MODEL, VALIDATION_PRECISION, VALIDATION_RUN_ID_HEX_LEN, + VALIDATION_RUN_ID_PREFIX, ValidationRunReceipt, WILSON_Z, complete_validation_run, + submit_validation_run, +}; /// Versioned artifact schema emitted by this engine. pub const ANALYSIS_ARTIFACT_SCHEMA_VERSION: &str = "tepp.temporal_evidence_readiness.v1"; @@ -248,6 +258,16 @@ pub enum AnalysisEngineError { TopicMeasurement(TopicMeasurementError), /// A topic-lineage artifact violated its bounded schema or count invariants. InvalidTopicLineageArtifact, + /// A `validation_core` metric or report rejected the recovery vectors. + Validation(ValidationError), + /// Recovery was authored by an LLM and cannot become scientific evidence. + LlmAuthoredRecovery, + /// The corpus contained evidence, but none was available at the cutoff. + NoEligibleEvidence, + /// A completion receipt did not match the rebound scientific binding. + BindingMismatch, + /// The request did not select the scientific-acceptance profile and model. + InvalidValidationProfile, } impl fmt::Display for AnalysisEngineError { @@ -262,6 +282,13 @@ impl fmt::Display for AnalysisEngineError { Self::LimitExceeded => "analysis corpus exceeded its execution bound", Self::TopicMeasurement(error) => return error.fmt(formatter), Self::InvalidTopicLineageArtifact => "invalid topic lineage artifact", + Self::Validation(error) => return error.fmt(formatter), + Self::LlmAuthoredRecovery => { + "llm-authored recovery is not scientific acceptance evidence" + } + Self::NoEligibleEvidence => "no eligible evidence for validation run", + Self::BindingMismatch => "validation run binding mismatch", + Self::InvalidValidationProfile => "invalid scientific acceptance profile", }; formatter.write_str(message) } @@ -281,6 +308,12 @@ impl From for AnalysisEngineError { } } +impl From for AnalysisEngineError { + fn from(error: ValidationError) -> Self { + Self::Validation(error) + } +} + /// Execute the cutoff-safe temporal evidence readiness analysis. /// /// Evidence whose `available_time` is later than the request cutoff is excluded @@ -402,7 +435,7 @@ fn format_digest(digest: impl AsRef<[u8]>) -> String { output } -fn valid_identifier(value: &str) -> bool { +pub(crate) fn valid_identifier(value: &str) -> bool { !value.trim().is_empty() && value.len() <= MAX_ANALYSIS_IDENTIFIER_BYTES && !value.chars().any(char::is_control) @@ -681,6 +714,26 @@ mod tests { AnalysisEngineError::InvalidTopicLineageArtifact, "invalid topic lineage artifact", ), + ( + AnalysisEngineError::Validation(validation_core::ValidationError::InvalidInput), + "invalid validation input", + ), + ( + AnalysisEngineError::LlmAuthoredRecovery, + "llm-authored recovery is not scientific acceptance evidence", + ), + ( + AnalysisEngineError::NoEligibleEvidence, + "no eligible evidence for validation run", + ), + ( + AnalysisEngineError::BindingMismatch, + "validation run binding mismatch", + ), + ( + AnalysisEngineError::InvalidValidationProfile, + "invalid scientific acceptance profile", + ), ]; for (error, message) in messages { assert_eq!(error.to_string(), message); @@ -689,6 +742,9 @@ mod tests { assert_eq!(converted.to_string(), "invalid API wire payload"); let from_topic: AnalysisEngineError = TopicMeasurementError::DidNotConverge.into(); assert_eq!(from_topic.to_string(), "topic estimator did not converge"); + let from_validation: AnalysisEngineError = + validation_core::ValidationError::InvalidInput.into(); + assert_eq!(from_validation.to_string(), "invalid validation input"); assert_eq!( add_membership_count(u64::MAX, 1), Err(AnalysisEngineError::ArithmeticOverflow) diff --git a/crates/analysis_engine/src/validation_run.rs b/crates/analysis_engine/src/validation_run.rs new file mode 100644 index 000000000..1af31b6ea --- /dev/null +++ b/crates/analysis_engine/src/validation_run.rs @@ -0,0 +1,891 @@ +//! Bind immutable evidence to a durable scientific-acceptance validation run. +//! +//! This is the first GAP-003A slice: cutoff-safe evidence, model, seed, backend, +//! and precision hash to one durable run identity. The accepted receipt carries +//! no scientific metrics. Completion asks `validation_core` for RMSE, bias, +//! coverage, temporal-order accuracy, and an SE-aware gate, then emits +//! `tepp.scientific_acceptance.v1`. LLM-authored recovery, non-finite inputs, +//! empty or duplicate evidence, snapshot mismatch, and cutoff-empty corpora +//! fail closed. Postgres persistence remains GAP-003B. + +use crate::{AnalysisCorpus, AnalysisEngineError, valid_identifier}; +use serde::Serialize; +use sha2::{Digest, Sha256}; +use std::collections::BTreeSet; +use std::fmt::Write as _; +use temporal_core::KnowledgeCutoff; +use tepp_api::{AnalysisRunRequest, ApiError}; +use validation_core::{ + ValidationError, ValidationReport, accept_within_standard_errors, bias_standard_error, + interval_coverage, mean_bias, rmse_standard_error, root_mean_square_error, + temporal_order_accuracy, wilson_coverage_interval, +}; + +/// Versioned scientific-acceptance artifact schema. +pub const SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION: &str = "tepp.scientific_acceptance.v1"; +/// Output profile that selects this validation-run executor. +pub const SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE: &str = "scientific_acceptance_v1"; +/// CPU `f64` reference model identity for scientific acceptance. +pub const VALIDATION_CPU_F64_MODEL: &str = "validation_cpu_f64_v1"; +/// Backend identity bound into the durable run. +pub const VALIDATION_BACKEND: &str = "cpu"; +/// Numeric precision bound into the durable run. +pub const VALIDATION_PRECISION: &str = "f64"; +/// Prefix of the hash-stable durable run identity. +pub const VALIDATION_RUN_ID_PREFIX: &str = "tepp-validation-"; +/// Hex characters taken from the binding digest for `run_id`. +pub const VALIDATION_RUN_ID_HEX_LEN: usize = 32; +/// Wilson critical value for nominal 95% coverage bounds. +pub const WILSON_Z: f64 = 1.96; + +/// Durable identity of one submitted validation run. Receipts never carry +/// scientific metrics. +#[derive(Clone, Debug, Eq, PartialEq, Serialize)] +pub struct ValidationRunReceipt { + /// Hash-stable run identity `tepp-validation-{32 hex}`. + pub run_id: String, + /// Lowercase SHA-256 of the canonical binding. + pub binding_sha256: String, + /// Immutable source snapshot identity. + pub snapshot_id: String, + /// Historical cutoff applied to availability. + pub knowledge_cutoff: String, + /// Bound model identity. + pub model: String, + /// Bound numeric seed. + pub seed: u64, + /// Bound compute backend. + pub backend: String, + /// Bound numeric precision. + pub precision: String, + /// Requested output profile. + pub output_profile: String, + /// Number of cutoff-eligible evidence identities. + pub eligible_evidence_count: u64, +} + +impl ValidationRunReceipt { + /// Serialize the receipt after confirming it carries no metrics. + /// + /// # Errors + /// + /// Returns [`AnalysisEngineError::SerializationFailure`] when JSON encoding + /// fails. + pub fn to_json(&self) -> Result { + serde_json::to_string(self).map_err(|_| AnalysisEngineError::SerializationFailure) + } + + /// Return the durable run identity. + #[must_use] + pub fn run_id(&self) -> &str { + &self.run_id + } + + /// Return the canonical binding digest. + #[must_use] + pub fn binding_sha256(&self) -> &str { + &self.binding_sha256 + } + + /// Return the bound snapshot identity. + #[must_use] + pub fn snapshot_id(&self) -> &str { + &self.snapshot_id + } + + /// Return the bound knowledge cutoff. + #[must_use] + pub fn knowledge_cutoff(&self) -> &str { + &self.knowledge_cutoff + } + + /// Return the bound model identity. + #[must_use] + pub fn model(&self) -> &str { + &self.model + } + + /// Return the bound seed. + #[must_use] + pub const fn seed(&self) -> u64 { + self.seed + } + + /// Return the bound backend. + #[must_use] + pub fn backend(&self) -> &str { + &self.backend + } + + /// Return the bound precision. + #[must_use] + pub fn precision(&self) -> &str { + &self.precision + } + + /// Return the bound output profile. + #[must_use] + pub fn output_profile(&self) -> &str { + &self.output_profile + } + + /// Return the eligible evidence count. + #[must_use] + pub const fn eligible_evidence_count(&self) -> u64 { + self.eligible_evidence_count + } +} + +/// Known-truth recovery vectors offered to complete a validation run. +#[derive(Clone, Debug, PartialEq)] +pub struct RecoveryObservation { + study_label: String, + truth: Vec, + recovered: Vec, + interval_lower: Vec, + interval_upper: Vec, + truth_times: Vec, + recovered_times: Vec, + se_gate_k: f64, + authored_by_llm: bool, +} + +impl RecoveryObservation { + /// Construct a recovery observation. + /// + /// LLM authorship is recorded here and refused at completion. Non-finite + /// `se_gate_k` or a negative multiplier fail closed immediately. + /// + /// # Errors + /// + /// Returns [`AnalysisEngineError::InvalidEvidence`] for an empty, oversized, + /// or control-bearing study label, and + /// [`AnalysisEngineError::Validation`] for a non-finite or negative SE + /// multiplier. + #[allow(clippy::too_many_arguments)] + pub fn new( + study_label: impl Into, + truth: Vec, + recovered: Vec, + interval_lower: Vec, + interval_upper: Vec, + truth_times: Vec, + recovered_times: Vec, + se_gate_k: f64, + authored_by_llm: bool, + ) -> Result { + let study_label = study_label.into(); + if !valid_identifier(&study_label) { + return Err(AnalysisEngineError::InvalidEvidence); + } + if !se_gate_k.is_finite() { + return Err(AnalysisEngineError::Validation( + ValidationError::InvalidInput, + )); + } + if se_gate_k < 0.0 { + return Err(AnalysisEngineError::Validation( + ValidationError::InvalidConfiguration, + )); + } + Ok(Self { + study_label, + truth, + recovered, + interval_lower, + interval_upper, + truth_times, + recovered_times, + se_gate_k, + authored_by_llm, + }) + } + + /// Return the study label. + #[must_use] + pub fn study_label(&self) -> &str { + &self.study_label + } + + /// Return the truth vector. + #[must_use] + pub fn truth(&self) -> &[f64] { + &self.truth + } + + /// Return the recovered vector. + #[must_use] + pub fn recovered(&self) -> &[f64] { + &self.recovered + } + + /// Return interval lower bounds. + #[must_use] + pub fn interval_lower(&self) -> &[f64] { + &self.interval_lower + } + + /// Return interval upper bounds. + #[must_use] + pub fn interval_upper(&self) -> &[f64] { + &self.interval_upper + } + + /// Return truth event times. + #[must_use] + pub fn truth_times(&self) -> &[f64] { + &self.truth_times + } + + /// Return recovered event times. + #[must_use] + pub fn recovered_times(&self) -> &[f64] { + &self.recovered_times + } + + /// Return the SE-gate multiplier. + #[must_use] + pub const fn se_gate_k(&self) -> f64 { + self.se_gate_k + } + + /// Return whether an LLM authored the recovery. + #[must_use] + pub const fn authored_by_llm(&self) -> bool { + self.authored_by_llm + } +} + +/// Operator-usable scientific acceptance evidence for one completed run. +#[derive(Clone, Debug, PartialEq, Serialize)] +pub struct ScientificAcceptanceEvidence { + /// Versioned artifact schema. + pub schema_version: String, + /// Durable run identity. + pub run_id: String, + /// Canonical binding digest. + pub binding_sha256: String, + /// Immutable source snapshot identity. + pub snapshot_id: String, + /// Historical cutoff applied to availability. + pub knowledge_cutoff: String, + /// Bound model identity. + pub model: String, + /// Bound numeric seed. + pub seed: u64, + /// Bound compute backend. + pub backend: String, + /// Bound numeric precision. + pub precision: String, + /// Output profile. + pub output_profile: String, + /// Number of cutoff-eligible evidence identities. + pub eligible_evidence_count: u64, + /// Whether RMSE toward 0 passed the SE-aware gate. + pub se_gate_accepted: bool, + /// SE-gate multiplier used for acceptance. + pub se_gate_k: f64, + /// Machine-readable recovery report. + pub report: ValidationReport, +} + +impl ScientificAcceptanceEvidence { + /// Serialize canonical evidence JSON after report validation. + /// + /// # Errors + /// + /// Returns [`AnalysisEngineError::Validation`] when a report field is + /// non-finite, or [`AnalysisEngineError::SerializationFailure`] when JSON + /// encoding fails. + pub fn to_json(&self) -> Result { + self.report.validate()?; + serde_json::to_string(self).map_err(|_| AnalysisEngineError::SerializationFailure) + } + + /// Return the lowercase SHA-256 digest of the canonical evidence JSON. + /// + /// # Errors + /// + /// Returns the same failures as [`Self::to_json`]. + pub fn sha256(&self) -> Result { + self.to_json() + .map(|json| format_hex(Sha256::digest(json.into_bytes()))) + } + + /// Render the nested recovery summary line. + #[must_use] + pub fn to_human_summary(&self) -> String { + self.report.to_human_summary() + } + + /// Return the artifact schema version. + #[must_use] + pub fn schema_version(&self) -> &str { + &self.schema_version + } + + /// Return the durable run identity. + #[must_use] + pub fn run_id(&self) -> &str { + &self.run_id + } + + /// Return whether the SE-aware gate accepted RMSE toward 0. + #[must_use] + pub const fn se_gate_accepted(&self) -> bool { + self.se_gate_accepted + } +} + +struct CanonicalBinding { + snapshot_id: String, + knowledge_cutoff: String, + model: String, + seed: u64, + backend: String, + precision: String, + eligible_ids: Vec, +} + +impl CanonicalBinding { + fn digest_hex(&self) -> String { + format_hex(Sha256::digest(self.canonical_bytes())) + } + + fn run_id(digest_hex: &str) -> String { + let mut run_id = + String::with_capacity(VALIDATION_RUN_ID_PREFIX.len() + VALIDATION_RUN_ID_HEX_LEN); + run_id.push_str(VALIDATION_RUN_ID_PREFIX); + run_id.push_str(&digest_hex[..VALIDATION_RUN_ID_HEX_LEN]); + run_id + } + + fn eligible_count(&self) -> u64 { + self.eligible_ids.len() as u64 + } + + fn canonical_bytes(&self) -> String { + let mut canonical = String::from("tepp.validation_binding.v1\n"); + let _ = writeln!(canonical, "snapshot={}", self.snapshot_id); + let _ = writeln!(canonical, "cutoff={}", self.knowledge_cutoff); + let _ = writeln!(canonical, "model={}", self.model); + let _ = writeln!(canonical, "seed={}", self.seed); + let _ = writeln!(canonical, "backend={}", self.backend); + let _ = writeln!(canonical, "precision={}", self.precision); + for identity in &self.eligible_ids { + let _ = writeln!(canonical, "evidence={identity}"); + } + canonical + } + + fn identity_record(&self, digest_hex: &str, run_id: &str) -> String { + format!( + "{run_id}\n{digest_hex}\n{}\n{}\n{}\n{}\n{}\n{}\n{}", + self.snapshot_id, + self.knowledge_cutoff, + self.model, + self.seed, + self.backend, + self.precision, + self.eligible_count() + ) + } +} + +impl ValidationRunReceipt { + fn identity_record(&self) -> String { + format!( + "{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}", + self.run_id, + self.binding_sha256, + self.snapshot_id, + self.knowledge_cutoff, + self.model, + self.seed, + self.backend, + self.precision, + self.eligible_evidence_count + ) + } +} + +/// Submit immutable evidence to a durable validation run. +/// +/// The receipt is hash-stable for one scientific binding and never includes +/// RMSE, bias, coverage, or gate results. +/// +/// # Errors +/// +/// Returns a fail-closed engine error for an invalid request, wrong output +/// profile or model, snapshot mismatch, duplicate or empty evidence, or a +/// cutoff that admits no evidence. +pub fn submit_validation_run( + request: &AnalysisRunRequest, + corpus: &AnalysisCorpus, + seed: u64, +) -> Result { + let binding = bind_validation_run(request, corpus, seed)?; + let binding_sha256 = binding.digest_hex(); + let run_id = CanonicalBinding::run_id(&binding_sha256); + let eligible_evidence_count = binding.eligible_count(); + Ok(ValidationRunReceipt { + run_id, + binding_sha256, + snapshot_id: binding.snapshot_id, + knowledge_cutoff: binding.knowledge_cutoff, + model: binding.model, + seed: binding.seed, + backend: binding.backend, + precision: binding.precision, + output_profile: SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE.to_owned(), + eligible_evidence_count, + }) +} + +/// Complete a previously submitted validation run with known-truth recovery. +/// +/// # Errors +/// +/// Returns a fail-closed engine error when recovery was LLM-authored, the +/// receipt does not match the rebound scientific identity, metric inputs are +/// invalid, or report validation fails. +pub fn complete_validation_run( + receipt: &ValidationRunReceipt, + request: &AnalysisRunRequest, + corpus: &AnalysisCorpus, + observation: &RecoveryObservation, +) -> Result { + if observation.authored_by_llm { + return Err(AnalysisEngineError::LlmAuthoredRecovery); + } + let binding = bind_validation_run(request, corpus, receipt.seed)?; + let binding_sha256 = binding.digest_hex(); + let run_id = CanonicalBinding::run_id(&binding_sha256); + if receipt.identity_record() != binding.identity_record(&binding_sha256, &run_id) { + return Err(AnalysisEngineError::BindingMismatch); + } + let report = compute_report(observation)?; + let se_gate_accepted = accept_within_standard_errors( + report.rmse, + 0.0, + report.rmse_standard_error, + observation.se_gate_k, + )?; + let eligible_evidence_count = binding.eligible_count(); + let evidence = ScientificAcceptanceEvidence { + schema_version: SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION.to_owned(), + run_id, + binding_sha256, + snapshot_id: binding.snapshot_id, + knowledge_cutoff: binding.knowledge_cutoff, + model: binding.model, + seed: binding.seed, + backend: binding.backend, + precision: binding.precision, + output_profile: SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE.to_owned(), + eligible_evidence_count, + se_gate_accepted, + se_gate_k: observation.se_gate_k, + report, + }; + evidence.to_json()?; + Ok(evidence) +} + +fn bind_validation_run( + request: &AnalysisRunRequest, + corpus: &AnalysisCorpus, + seed: u64, +) -> Result { + request.to_json()?; + let requested = format!( + "{}\n{}", + request.output_profile, request.model_contract_version + ); + let expected = format!("{SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE}\n{VALIDATION_CPU_F64_MODEL}"); + if requested != expected { + return Err(AnalysisEngineError::InvalidValidationProfile); + } + if request.snapshot_id != corpus.snapshot_id() { + return Err(AnalysisEngineError::SnapshotMismatch); + } + let cutoff = KnowledgeCutoff::parse_rfc3339(&request.knowledge_cutoff) + .map_err(|_| AnalysisEngineError::Api(ApiError::InvalidWirePayload))?; + let mut identities = BTreeSet::new(); + let mut eligible = BTreeSet::new(); + for unit in corpus.evidence_units() { + if !identities.insert(unit.evidence_id()) { + return Err(AnalysisEngineError::DuplicateEvidence); + } + if unit.available_time().instant() <= cutoff.instant() { + eligible.insert(unit.evidence_id().to_owned()); + } + } + if identities.is_empty() { + return Err(AnalysisEngineError::InvalidEvidence); + } + if eligible.is_empty() { + return Err(AnalysisEngineError::NoEligibleEvidence); + } + Ok(CanonicalBinding { + snapshot_id: request.snapshot_id.clone(), + knowledge_cutoff: cutoff.to_rfc3339(), + model: VALIDATION_CPU_F64_MODEL.to_owned(), + seed, + backend: VALIDATION_BACKEND.to_owned(), + precision: VALIDATION_PRECISION.to_owned(), + eligible_ids: eligible.into_iter().collect(), + }) +} + +fn compute_report( + observation: &RecoveryObservation, +) -> Result { + let rmse = root_mean_square_error(&observation.truth, &observation.recovered)?; + let rmse_standard_error = rmse_standard_error(&observation.truth, &observation.recovered)?; + let mean_bias = mean_bias(&observation.truth, &observation.recovered)?; + let bias_standard_error = bias_standard_error(&observation.truth, &observation.recovered)?; + let interval_coverage = interval_coverage( + &observation.truth, + &observation.interval_lower, + &observation.interval_upper, + )?; + let (coverage_wilson_lower, coverage_wilson_upper) = wilson_coverage_interval( + &observation.truth, + &observation.interval_lower, + &observation.interval_upper, + WILSON_Z, + )?; + let temporal_order_accuracy = + temporal_order_accuracy(&observation.truth_times, &observation.recovered_times)?; + let report = ValidationReport { + study_label: observation.study_label.clone(), + rmse, + rmse_standard_error, + mean_bias, + bias_standard_error, + interval_coverage, + coverage_wilson_lower, + coverage_wilson_upper, + temporal_order_accuracy, + monte_carlo_rmse: None, + }; + report.validate()?; + Ok(report) +} + +fn format_hex(digest: impl AsRef<[u8]>) -> String { + let bytes = digest.as_ref(); + let mut output = String::with_capacity(bytes.len() * 2); + for byte in bytes { + let _ = write!(output, "{byte:02x}"); + } + output +} + +#[cfg(test)] +mod tests { + use super::{ + RecoveryObservation, SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE, + SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION, VALIDATION_BACKEND, VALIDATION_CPU_F64_MODEL, + VALIDATION_PRECISION, VALIDATION_RUN_ID_PREFIX, WILSON_Z, complete_validation_run, + submit_validation_run, + }; + use crate::{ + AnalysisCorpus, AnalysisEngineError, AnalysisEvidenceUnit, MAX_ANALYSIS_IDENTIFIER_BYTES, + }; + use temporal_core::{AvailableTime, EventTime}; + use tepp_api::AnalysisRunRequest; + use validation_core::ValidationError; + + fn request() -> AnalysisRunRequest { + AnalysisRunRequest { + contract_version: 1, + idempotency_key: "idem-validation-1".into(), + tenant_workspace_id: "tenant-workspace-1".into(), + snapshot_id: "snapshot-1".into(), + knowledge_cutoff: "2026-08-01T00:00:00Z".into(), + model_contract_version: VALIDATION_CPU_F64_MODEL.into(), + output_profile: SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE.into(), + } + } + + fn unit(id: &str, available: &str) -> AnalysisEvidenceUnit { + AnalysisEvidenceUnit::new( + id, + EventTime::parse_rfc3339("2026-07-01T00:00:00Z").expect("event"), + AvailableTime::parse_rfc3339(available).expect("available"), + 1, + ) + .expect("unit") + } + + fn corpus(units: Vec) -> AnalysisCorpus { + AnalysisCorpus::new("snapshot-1", units).expect("corpus") + } + + fn recovery(truth: Vec, recovered: Vec, k: f64) -> RecoveryObservation { + let times = [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0]; + let times = times[..truth.len()].to_vec(); + let lower: Vec = truth.iter().map(|value| value - 0.5).collect(); + let upper: Vec = truth.iter().map(|value| value + 0.5).collect(); + RecoveryObservation::new( + "foundation-recovery", + truth, + recovered, + lower, + upper, + times.clone(), + times, + k, + false, + ) + .expect("observation") + } + + fn two_point( + label: impl Into, + k: f64, + authored_by_llm: bool, + ) -> Result { + RecoveryObservation::new( + label, + vec![1.0, 2.0], + vec![1.0, 2.0], + vec![0.0, 1.0], + vec![2.0, 3.0], + vec![1.0, 2.0], + vec![1.0, 2.0], + k, + authored_by_llm, + ) + } + + #[test] + fn submit_is_hash_stable_cutoff_safe_and_metric_free() { + let first = corpus(vec![ + unit("evidence-b", "2026-07-15T00:00:00Z"), + unit("evidence-a", "2026-07-10T00:00:00Z"), + unit("late", "2026-08-02T00:00:00Z"), + ]); + let second = corpus(vec![ + unit("evidence-a", "2026-07-10T00:00:00Z"), + unit("evidence-b", "2026-07-15T00:00:00Z"), + ]); + let receipt_a = submit_validation_run(&request(), &first, 7).expect("submit a"); + let receipt_b = submit_validation_run(&request(), &second, 7).expect("submit b"); + assert_eq!(receipt_a.run_id(), receipt_b.run_id()); + assert_eq!(receipt_a.binding_sha256(), receipt_b.binding_sha256()); + assert!(receipt_a.run_id().starts_with(VALIDATION_RUN_ID_PREFIX)); + assert_eq!( + receipt_a.run_id().len(), + VALIDATION_RUN_ID_PREFIX.len() + 32 + ); + assert_eq!(receipt_a.eligible_evidence_count(), 2); + assert_eq!(receipt_a.snapshot_id(), "snapshot-1"); + assert_eq!(receipt_a.knowledge_cutoff(), "2026-08-01T00:00:00Z"); + assert_eq!(receipt_a.model(), VALIDATION_CPU_F64_MODEL); + assert_eq!(receipt_a.seed(), 7); + assert_eq!(receipt_a.backend(), VALIDATION_BACKEND); + assert_eq!(receipt_a.precision(), VALIDATION_PRECISION); + assert_eq!( + receipt_a.output_profile(), + SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE + ); + let json = receipt_a.to_json().expect("json"); + assert!(!json.contains("rmse")); + assert!(!json.contains("bias")); + assert!(!json.contains("coverage")); + let other_seed = submit_validation_run(&request(), &first, 8).expect("seed"); + assert_ne!(receipt_a.run_id(), other_seed.run_id()); + } + + #[test] + fn complete_emits_operator_usable_scientific_acceptance_evidence() { + let offered = corpus(vec![ + unit("evidence-a", "2026-07-10T00:00:00Z"), + unit("evidence-b", "2026-07-15T00:00:00Z"), + ]); + let receipt = submit_validation_run(&request(), &offered, 11).expect("submit"); + let observation = recovery( + vec![0.70, 0.55, 0.40, -0.20, 0.85], + vec![0.70, 0.55, 0.40, -0.20, 0.85], + 3.0, + ); + assert_eq!(observation.study_label(), "foundation-recovery"); + assert_eq!(observation.truth().len(), 5); + assert_eq!(observation.recovered().len(), 5); + assert_eq!(observation.interval_lower().len(), 5); + assert_eq!(observation.interval_upper().len(), 5); + assert_eq!(observation.truth_times().len(), 5); + assert_eq!(observation.recovered_times().len(), 5); + assert!((observation.se_gate_k() - 3.0).abs() < f64::EPSILON); + assert!(!observation.authored_by_llm()); + let evidence = complete_validation_run(&receipt, &request(), &offered, &observation) + .expect("complete"); + assert_eq!( + evidence.schema_version(), + SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION + ); + assert_eq!(evidence.run_id(), receipt.run_id()); + assert!(evidence.se_gate_accepted()); + assert_eq!( + evidence.output_profile, + SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE + ); + assert_eq!(evidence.eligible_evidence_count, 2); + assert!((evidence.se_gate_k - 3.0).abs() < f64::EPSILON); + assert!((WILSON_Z - 1.96).abs() < f64::EPSILON); + let json = evidence.to_json().expect("json"); + assert!(json.contains(SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION)); + assert!(json.contains("rmse")); + assert_eq!(evidence.sha256().expect("digest").len(), 64); + assert!(evidence.to_human_summary().contains("foundation-recovery")); + let rejected = recovery( + vec![0.0, 1.0, 2.0, 3.0, 4.0], + vec![10.0, 11.0, 12.0, 13.0, 14.0], + 1.0, + ); + let refused = + complete_validation_run(&receipt, &request(), &offered, &rejected).expect("refused"); + assert!(!refused.se_gate_accepted()); + } + + #[test] + fn submit_and_complete_fail_closed_on_trust_and_shape_errors() { + let offered = corpus(vec![unit("evidence-a", "2026-07-10T00:00:00Z")]); + let mut wrong_profile = request(); + wrong_profile.output_profile = "validation-report".into(); + assert_eq!( + submit_validation_run(&wrong_profile, &offered, 1), + Err(AnalysisEngineError::InvalidValidationProfile) + ); + let mut wrong_model = request(); + wrong_model.model_contract_version = "temporal-evidence-v1".into(); + assert_eq!( + submit_validation_run(&wrong_model, &offered, 1), + Err(AnalysisEngineError::InvalidValidationProfile) + ); + let mismatched = AnalysisCorpus::new( + "snapshot-other", + vec![unit("evidence-a", "2026-07-10T00:00:00Z")], + ) + .expect("other"); + assert_eq!( + submit_validation_run(&request(), &mismatched, 1), + Err(AnalysisEngineError::SnapshotMismatch) + ); + let duplicate = corpus(vec![ + unit("same", "2026-07-10T00:00:00Z"), + unit("same", "2026-07-11T00:00:00Z"), + ]); + assert_eq!( + submit_validation_run(&request(), &duplicate, 1), + Err(AnalysisEngineError::DuplicateEvidence) + ); + let empty = AnalysisCorpus::new("snapshot-1", Vec::new()).expect("empty"); + assert_eq!( + submit_validation_run(&request(), &empty, 1), + Err(AnalysisEngineError::InvalidEvidence) + ); + let late = corpus(vec![unit("late", "2026-08-02T00:00:00Z")]); + assert_eq!( + submit_validation_run(&request(), &late, 1), + Err(AnalysisEngineError::NoEligibleEvidence) + ); + let mut invalid_request = request(); + invalid_request.idempotency_key.clear(); + assert!(matches!( + submit_validation_run(&invalid_request, &offered, 1), + Err(AnalysisEngineError::Api(_)) + )); + let mut invalid_cutoff = request(); + invalid_cutoff.knowledge_cutoff = "not-a-time".into(); + assert!(matches!( + submit_validation_run(&invalid_cutoff, &offered, 1), + Err(AnalysisEngineError::Api(_)) + )); + } + + #[test] + fn complete_refuses_llm_authored_recovery_and_tampered_receipt() { + let offered = corpus(vec![ + unit("evidence-a", "2026-07-10T00:00:00Z"), + unit("evidence-b", "2026-07-15T00:00:00Z"), + ]); + let receipt = submit_validation_run(&request(), &offered, 3).expect("submit"); + let llm = RecoveryObservation::new( + "llm-study", + vec![1.0, 2.0, 3.0], + vec![1.0, 2.0, 3.0], + vec![0.5, 1.5, 2.5], + vec![1.5, 2.5, 3.5], + vec![1.0, 2.0, 3.0], + vec![1.0, 2.0, 3.0], + 3.0, + true, + ) + .expect("llm"); + assert!(llm.authored_by_llm()); + assert_eq!( + complete_validation_run(&receipt, &request(), &offered, &llm), + Err(AnalysisEngineError::LlmAuthoredRecovery) + ); + let mut tampered = receipt.clone(); + tampered.run_id = "tepp-validation-deadbeefdeadbeefdeadbeefdeadbeef".into(); + let observation = recovery(vec![1.0, 2.0, 3.0], vec![1.0, 2.0, 3.0], 3.0); + assert_eq!( + complete_validation_run(&tampered, &request(), &offered, &observation), + Err(AnalysisEngineError::BindingMismatch) + ); + } + + #[test] + fn complete_refuses_nonfinite_recovery_and_invalid_observation() { + let offered = corpus(vec![ + unit("evidence-a", "2026-07-10T00:00:00Z"), + unit("evidence-b", "2026-07-15T00:00:00Z"), + ]); + let receipt = submit_validation_run(&request(), &offered, 3).expect("submit"); + let nan = RecoveryObservation::new( + "nan-study", + vec![1.0, 2.0, 3.0], + vec![f64::NAN, 2.0, 3.0], + vec![0.5, 1.5, 2.5], + vec![1.5, 2.5, 3.5], + vec![1.0, 2.0, 3.0], + vec![1.0, 2.0, 3.0], + 3.0, + false, + ) + .expect("nan observation"); + assert_eq!( + complete_validation_run(&receipt, &request(), &offered, &nan), + Err(AnalysisEngineError::Validation( + ValidationError::InvalidInput + )) + ); + assert_eq!( + two_point("", 3.0, false), + Err(AnalysisEngineError::InvalidEvidence) + ); + assert_eq!( + two_point("x".repeat(MAX_ANALYSIS_IDENTIFIER_BYTES + 1), 3.0, false), + Err(AnalysisEngineError::InvalidEvidence) + ); + assert_eq!( + two_point("nan-k", f64::NAN, false), + Err(AnalysisEngineError::Validation( + ValidationError::InvalidInput + )) + ); + assert_eq!( + two_point("neg-k", -1.0, false), + Err(AnalysisEngineError::Validation( + ValidationError::InvalidConfiguration + )) + ); + let converted: AnalysisEngineError = ValidationError::InvalidInput.into(); + assert_eq!(converted.to_string(), "invalid validation input"); + } +} diff --git a/crates/analysis_engine/tests/validation_run_contract.rs b/crates/analysis_engine/tests/validation_run_contract.rs new file mode 100644 index 000000000..1ebb164d5 --- /dev/null +++ b/crates/analysis_engine/tests/validation_run_contract.rs @@ -0,0 +1,103 @@ +//! GAP-003A scientific-acceptance validation-run contract. + +use analysis_engine::{ + AnalysisCorpus, AnalysisEngineError, AnalysisEvidenceUnit, RecoveryObservation, + SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE, SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION, + VALIDATION_CPU_F64_MODEL, complete_validation_run, submit_validation_run, +}; +use temporal_core::{AvailableTime, EventTime}; +use tepp_api::AnalysisRunRequest; + +fn request() -> AnalysisRunRequest { + AnalysisRunRequest { + contract_version: 1, + idempotency_key: "idem-validation-contract".into(), + tenant_workspace_id: "tenant-workspace-contract".into(), + snapshot_id: "snapshot-contract".into(), + knowledge_cutoff: "2026-08-01T00:00:00Z".into(), + model_contract_version: VALIDATION_CPU_F64_MODEL.into(), + output_profile: SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE.into(), + } +} + +fn unit(id: &str, available: &str) -> AnalysisEvidenceUnit { + AnalysisEvidenceUnit::new( + id, + EventTime::parse_rfc3339("2026-07-01T00:00:00Z").expect("event"), + AvailableTime::parse_rfc3339(available).expect("available"), + 1, + ) + .expect("unit") +} + +#[test] +fn durable_run_binds_cutoff_eligible_evidence_and_emits_acceptance_evidence() { + let corpus = AnalysisCorpus::new( + "snapshot-contract", + vec![ + unit("evidence-2", "2026-07-20T00:00:00Z"), + unit("evidence-1", "2026-07-10T00:00:00Z"), + unit("future", "2026-08-02T00:00:00Z"), + ], + ) + .expect("corpus"); + let receipt = submit_validation_run(&request(), &corpus, 42).expect("submit"); + assert_eq!(receipt.eligible_evidence_count(), 2); + assert!(!receipt.to_json().expect("json").contains("rmse")); + let observation = RecoveryObservation::new( + "contract-recovery", + vec![0.70, 0.55, 0.40, -0.20, 0.85], + vec![0.70, 0.55, 0.40, -0.20, 0.85], + vec![0.50, 0.35, 0.20, -0.40, 0.65], + vec![0.90, 0.75, 0.60, 0.00, 1.00], + vec![1.0, 2.0, 3.0, 4.0, 5.0], + vec![1.1, 1.9, 3.2, 3.8, 5.1], + 3.0, + false, + ) + .expect("observation"); + let evidence = + complete_validation_run(&receipt, &request(), &corpus, &observation).expect("complete"); + assert_eq!( + evidence.schema_version(), + SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION + ); + assert_eq!(evidence.run_id(), receipt.run_id()); + assert!(evidence.se_gate_accepted()); + assert!(evidence.to_json().expect("json").contains("rmse")); +} + +#[test] +fn llm_authored_recovery_and_cutoff_empty_corpora_fail_closed() { + let corpus = AnalysisCorpus::new( + "snapshot-contract", + vec![unit("only-future", "2026-08-02T00:00:00Z")], + ) + .expect("corpus"); + assert_eq!( + submit_validation_run(&request(), &corpus, 1), + Err(AnalysisEngineError::NoEligibleEvidence) + ); + let eligible = AnalysisCorpus::new( + "snapshot-contract", + vec![unit("evidence-1", "2026-07-10T00:00:00Z")], + ) + .expect("eligible"); + let receipt = submit_validation_run(&request(), &eligible, 1).expect("submit"); + let llm = RecoveryObservation::new( + "llm-contract", + vec![1.0, 2.0, 3.0], + vec![1.0, 2.0, 3.0], + vec![0.5, 1.5, 2.5], + vec![1.5, 2.5, 3.5], + vec![1.0, 2.0, 3.0], + vec![1.0, 2.0, 3.0], + 3.0, + true, + ) + .expect("llm"); + assert_eq!( + complete_validation_run(&receipt, &request(), &eligible, &llm), + Err(AnalysisEngineError::LlmAuthoredRecovery) + ); +} diff --git a/docs/TRACEABILITY.md b/docs/TRACEABILITY.md index 2b783c2ab..c2ab5d13f 100644 --- a/docs/TRACEABILITY.md +++ b/docs/TRACEABILITY.md @@ -50,6 +50,7 @@ The full APA 7th standards/literature register remains `docs/research/standards- | leakage-safe availability/cutoff snapshots | PRD; ADR 0002/0013 | `corpus_split` on protected main | implemented-main | | Unicode NFC/NFD canonical identity for split leakage | ADR 0004/0008/0013; UAX #15 | `corpus_split` canonical-equivalence links on active PR #59 | active-PR | | recovery metrics (RMSE, bias, coverage, graph, temporal order, Monte Carlo SE gates) | PRD; Test Strategy; ADR 0007/0014 | `validation_core` on protected main (PR #19); SE-aware Monte Carlo gates included | implemented-main | +| durable validation-run scientific acceptance evidence | ADR 0026; ADR 0002/0008/0014; GAP-003A; National Academies (2019); Wasserstein & Lazar (2016); Wilson (1927) | `analysis_engine` `submit_validation_run` / `complete_validation_run` on this PR: hash-stable cutoff-safe binding, metric-free receipt, `tepp.scientific_acceptance.v1` RMSE/bias/coverage/temporal-order/SE-aware evidence; LLM/NaN/empty/duplicate/snapshot/cutoff-empty fail closed; not implemented-main; Postgres persistence remains GAP-003B | active-PR | | PostgreSQL bitemporal/lineage persistence | ADR 0013; Architecture/ERD | `persistence_postgres` migration contracts, in-memory adapters, live SQL session/document SQL port, tenant RLS (`0002` + session GUC/role helpers), `DATABASE_URL` SQLx gate, optional session-affine `live-sqlx` `PgPool` driver, exact-head live PostgreSQL CI with isolation proof, append-only immutability triggers (`0004`), temporal interval ordering CHECKs (`0005`), typed membership assignment (`0006` implemented-main), event-relation/mention/instance SQL (#37–#39 implemented-main), source-artifact SQL (#40 implemented-main), audit-event SQL (#41 implemented-main), concurrent document-write stress (#43 implemented-main), backup/restore integrity revalidation (#44 implemented-main), `revision_order` later-revision system-time ordering implemented-main, entity/project target SQL on PR #131; remaining physical ERD constraints | partial | | known-truth temporal/event simulation manifests | PRD; TRD; Test Strategy | `tepp_simulation` on protected main; recovery metrics in `validation_core` | implemented-main | | versioned service/API contracts and exports | PRD; API contract; ADR 0011/0013 | `tepp_api` analysis-run/export/JSON-LD/GraphML contracts on protected main (PR #21); request-bound terminal result active in PR #157; HTTP service remains accepted-target; the `orchestrator_live` loopback interpretation listener is on this PR | partial | diff --git a/docs/adr/0026-validation-run-scientific-acceptance.md b/docs/adr/0026-validation-run-scientific-acceptance.md new file mode 100644 index 000000000..e769539f2 --- /dev/null +++ b/docs/adr/0026-validation-run-scientific-acceptance.md @@ -0,0 +1,88 @@ +# ADR 0026 — Durable validation-run scientific acceptance evidence + +**Decision status:** Accepted +**Implementation maturity:** active-PR — library-level `analysis_engine` binding on this PR; not implemented-main +**Date:** 2026-08-31 +**Supersedes:** None; complements ADR 0002, ADR 0008, ADR 0013, ADR 0014, and ADR 0022. +**Figma File ID:** N/A — this increment changes a Rust service crate and has no user-interface surface. +**Storybook inventory:** N/A — no reusable web object or interaction changed. + +## Context + +GAP-003A records that immutable evidence cannot yet be submitted to a durable +validation run that produces operator-usable scientific acceptance evidence. +`validation_core` already computes RMSE, bias, interval coverage, temporal-order +accuracy, and SE-aware gates, but those metrics remain library-level. An +accepted analysis run is a transport receipt; it must never carry scientific +results. Operators therefore have no hash-stable identity that binds cutoff-safe +evidence, model, seed, backend, and precision to one completion that emits +scientific acceptance evidence. + +Postgres persistence, restart/recovery, and Compose E2E remain GAP-003B. + +## Decision + +Add a bounded `analysis_engine` validation-run executor: + +- `submit_validation_run` binds sorted cutoff-eligible evidence identities, + snapshot, knowledge cutoff, model `validation_cpu_f64_v1`, seed, backend + `cpu`, and precision `f64` into a canonical SHA-256 digest. The durable + `run_id` is `tepp-validation-{32 hex}`. The receipt carries no RMSE, bias, + coverage, or gate fields. +- `complete_validation_run` rebinds the same scientific identity, refuses + LLM-authored recovery, computes `validation_core` recovery metrics, applies + the SE-aware gate `|RMSE − 0| ≤ k · SE(RMSE)`, and emits + `tepp.scientific_acceptance.v1` under output profile + `scientific_acceptance_v1`. +- Empty corpora, duplicate evidence identities, snapshot mismatch, invalid + profiles, non-finite inputs, and cutoff-empty eligibility fail closed. +- A computed recovery that fails the SE-aware gate still emits evidence with + `se_gate_accepted = false` so operators can read the metrics. Invalid or + LLM-authored recovery never emits evidence. +- The engine does not persist rows, claim implemented-main, or replace ADR 0014 + exact-head claim promotion. + +## Alternatives considered + +1. Keep metrics library-only — rejected because GAP-003A is the operator-visible + product-completion gap and has no live implementation PR. +2. Return scientific metrics on `AnalysisRunAccepted` — rejected because + accepted/running receipts must never carry scientific results. +3. Persist the run in PostgreSQL in this slice — rejected because durable + storage, restart, and Compose recovery belong to GAP-003B / issue #287. +4. Bind evidence, cutoff, model, seed, backend, and precision in + `analysis_engine` and emit scientific acceptance evidence through + `validation_core` — accepted because it is independently testable and does + not weaken fail-closed gates. + +## Consequences + +Operators can submit one immutable evidence snapshot and later complete it with +known-truth recovery to obtain a digest-bound scientific acceptance artifact. +The run identity is hash-stable for the scientific binding, independent of +evidence input order and of ineligible post-cutoff units. Persistence, HTTP +ingress, and release promotion remain later slices. LLM output still cannot +become scientific authority. + +## Verification + +The stacked PR includes unit and integration tests for hash-stable identity, +cutoff exclusion, metric-free receipts, SE-aware accept and refuse, and +fail-closed LLM, NaN, empty, duplicate, snapshot, profile, and cutoff-empty +paths. Run: + +```text +cargo fmt --all -- --check +cargo test -p analysis_engine +cargo clippy -p analysis_engine --all-targets -- -D warnings +``` + +Doctoring and APA 7th citations are in +`docs/research/validation-run-scientific-acceptance.md`. + +## Rollback and supersession + +Rollback removes `validation_run` from `analysis_engine` and stops exporting +the scientific-acceptance artifact. No persisted schema migration is +introduced. Supersession requires a new ADR if execution changes cutoff +semantics, binding identity, LLM-refusal, or scientific estimands. diff --git a/docs/adr/README.md b/docs/adr/README.md index 1254c8079..b068bad74 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -30,6 +30,7 @@ Read [`ADR_POLICY.md`](ADR_POLICY.md) first. **Decision status and implementatio | [0022](0022-deterministic-analysis-run-execution.md) | Deterministic cutoff-safe analysis-run execution | Accepted | active-PR | Closes the first executable product path from accepted run to digest-bound terminal result without claiming estimator authority. | | [0024](0024-lineage-pair-criterion-and-project-journey-posterior.md) | Independent Event Lineage pair criterion and posterior Project Journey | Proposed | active-PR | Strict artifacts preserve criterion/event-time draws, branches, ties, and CPU/GPU receipts without claiming the scientific estimator is complete. | | [0025](0025-macos-native-rust-mlx-metal-boundary.md) | macOS-native Rust-owned MLX Metal execution | Accepted | accepted-target | Compose authenticates to a native host service; Linux never claims Metal, and actual backend/parity receipts fail closed. | +| [0026](0026-validation-run-scientific-acceptance.md) | Durable validation-run scientific acceptance evidence | Accepted | active-PR | GAP-003A first slice binds cutoff-safe evidence to a hash-stable run that emits `tepp.scientific_acceptance.v1`; Postgres persistence remains GAP-003B. | | [0023](0023-lineage-criterion-anchor-contract.md) | TEPP-owned Event Lineage criterion anchor | Accepted | active-PR | PR #237 publishes the strict accepted/rejected artifact and identities; estimator execution remains fail-closed future work. | | [0024](0024-independent-topic-importance-anchor.md) | Posterior topic-context producer contract | Accepted | contract-only active-PR | Strict DTO/schema only; the current estimator does not emit it. fast-mlsirm owns case-deletion influence. | | [0001](0001-rust-first-modular-msa.md) | Rust-first numerical core and CPU `f64` reference | Accepted | partial | ADR 0011 owns cross-service/MSA authority; 0001 retains numerical/backend authority. | diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index 5ea0c95bf..d86947e63 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -248,7 +248,7 @@ the coordinated visual workspace, or a supported multi-tenant release. |---|---|---|---|---|---|---|---| | GAP-001 | Submission produces a durable accepted receipt, and the deterministic terminal-result lifecycle is now implemented-main. | `implemented-main` | closed on protected main | `340087494b0a` lineage merged through [PR #157](https://github.com/ContextualWisdomLab/TEPP/pull/157) (merged 2026-08-25T02:53Z); [#156](https://github.com/ContextualWisdomLab/TEPP/issues/156) CLOSED | — | — | Exact request/result/snapshot/cutoff/model/profile binding, typed terminal failures, deterministic retrieval, and cutoff-safe execution are protected-main behavior as of the #157 merge. | | GAP-002 | LineageWeave and other modular consumers can rely on the complete protected-main HTTP evidence/result boundary. | `partial` | consumer hardening remains | Terminal-result lifecycle implemented-main via #157; versioned API contract intact | [#156](https://github.com/ContextualWisdomLab/TEPP/issues/156) (closed) / [PR #155](https://github.com/ContextualWisdomLab/TEPP/pull/155) (merged) | — | Remaining work is consumer-side adoption evidence and any versioned-contract drift discovered during integration; core boundary is no longer the gap it was. | -| GAP-003A | Immutable evidence cannot yet be submitted to a durable validation run that produces operator-usable scientific acceptance evidence. | `accepted-target` | product-completion | `e65cd66` (validation metrics are library-level only) | [#166](https://github.com/ContextualWisdomLab/TEPP/issues/166) | `—` (issue program; no current implementation PR) | Compose/CLI/API execution must bind immutable evidence, cutoffs, model configuration, validation metrics, and reproducibility manifests to one idempotent run. | +| GAP-003A | Immutable evidence cannot yet be submitted to a durable validation run that produces operator-usable scientific acceptance evidence. | `accepted-target` | product-completion | `e65cd66` (validation metrics are library-level only) | [#166](https://github.com/ContextualWisdomLab/TEPP/issues/166) | this PR (`feat/validation-run-scientific-acceptance`; library-level binding; not implemented-main; not Compose persistence) | Compose/CLI/API execution must bind immutable evidence, cutoffs, model configuration, validation metrics, and reproducibility manifests to one idempotent run. | | GAP-003B | Scientific result artifacts cannot yet be persisted, restarted, and recovered as one supported operator workflow. | `accepted-target` | product-completion | `e65cd66` (persistence contracts lack E2E recovery) | [#166](https://github.com/ContextualWisdomLab/TEPP/issues/166) | `—` (issue program; no current implementation PR) | Durable storage, migration/rollback, restart/recovery, artifact digest verification, and terminal retrieval must pass against a real Compose deployment. | | GAP-003C | The persistence slice classifies concurrent-write SQLSTATEs, but has no measured hot-partition detection, routing, or mitigation for tenant/result workloads. | `accepted-target` | product-completion | `e65cd66` (conflict classification only; no measured partition control) | [#166](https://github.com/ContextualWisdomLab/TEPP/issues/166) | `—` (issue program; no current implementation PR) | A real Compose/PostgreSQL workload identifies hot keys and partition skew, applies bounded tenant/time or result routing without weakening 3NF or temporal authority, and proves conflict rate, latency, recovery, and migration/rollback behavior under load. | | GAP-004 | The central shared-latent temporal/relational topic estimator is absent. | `partial` | product vertical | CPU `f64` TRSL-TM reference estimator with ALR/ILR coordinates and refusal gates is implemented-main (v0.2.0 `topic_measurement`); fitted candidate-`K` scoring present | [#167](https://github.com/ContextualWisdomLab/TEPP/issues/167) | — | GPU, method effects, full Bayesian sampling, and topic birth/split/merge remain. This is not full #167 closure. | diff --git a/docs/research/validation-run-scientific-acceptance.md b/docs/research/validation-run-scientific-acceptance.md new file mode 100644 index 000000000..4bc06125c --- /dev/null +++ b/docs/research/validation-run-scientific-acceptance.md @@ -0,0 +1,32 @@ +# Validation-run scientific acceptance evidence + +## Scope + +This note doctors the first GAP-003A slice in `analysis_engine`: + +1. immutable evidence, knowledge cutoff, model, seed, backend, and precision bind to one hash-stable validation run; +2. the accepted receipt carries no scientific metrics; +3. completion emits `tepp.scientific_acceptance.v1` with RMSE, bias, interval coverage (Wilson bounds), temporal-order accuracy, and an SE-aware gate; +4. LLM-authored recovery, non-finite inputs, empty or duplicate evidence, snapshot mismatch, and cutoff-empty corpora fail closed. + +Postgres persistence, restart/recovery, and Compose execution remain GAP-003B. This slice is not implemented-main. + +## Authoritative sources + +National Academies of Sciences, Engineering, and Medicine. (2019). *Reproducibility and replicability in science*. The National Academies Press. https://doi.org/10.17226/25303 + +Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on *p*-values: Context, process, and purpose. *The American Statistician, 70*(2), 129–133. https://doi.org/10.1080/00031305.2016.1154108 + +Wilson, E. B. (1927). Probable inference, the law of succession, and statistical inference. *Journal of the American Statistical Association, 22*(158), 209–212. https://doi.org/10.1080/01621459.1927.10502953 + +## Application + +The National Academies (2019) separate computational reproducibility (same binding, same digest) from a scientific claim that recovery is correct. Wasserstein and Lazar (2016) refuse to treat a passing threshold as automatic scientific authority, so a failed SE-aware gate still reports metrics with `se_gate_accepted = false` rather than inventing a passing claim. Wilson (1927) supplies the coverage interval bounds already implemented in `validation_core`. TEPP therefore binds cutoff-safe evidence before any metric is computed, refuses LLM-authored recovery, and reports RMSE, bias, coverage, temporal order, and the SE-aware gate as operator-usable evidence (National Academies of Sciences, Engineering, and Medicine, 2019; Wasserstein & Lazar, 2016; Wilson, 1927). + +## Verification + +- identical eligible evidence in any order and with extra post-cutoff units yields the same `tepp-validation-{32 hex}` identity; +- receipts serialize without RMSE, bias, or coverage fields; +- known-truth recovery emits `tepp.scientific_acceptance.v1` with a digest-stable JSON body; +- a large residual vector remains operator-readable with `se_gate_accepted = false`; +- LLM authorship, NaN recovery, empty corpora, duplicates, snapshot mismatch, wrong profile/model, and cutoff-empty eligibility return dedicated fail-closed errors. From f51d0424b6f6e66db874304daa60bf62a814768f Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Mon, 31 Aug 2026 07:36:49 +0000 Subject: [PATCH 2/5] docs(gap): record GAP-003A current PR #356 Point the GAP-003A Current head SHA column at exact PR #356 f8a4dad. Library-level binding only; not implemented-main. --- docs/product-technical-gap-baseline.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index d86947e63..ee650024e 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -248,7 +248,7 @@ the coordinated visual workspace, or a supported multi-tenant release. |---|---|---|---|---|---|---|---| | GAP-001 | Submission produces a durable accepted receipt, and the deterministic terminal-result lifecycle is now implemented-main. | `implemented-main` | closed on protected main | `340087494b0a` lineage merged through [PR #157](https://github.com/ContextualWisdomLab/TEPP/pull/157) (merged 2026-08-25T02:53Z); [#156](https://github.com/ContextualWisdomLab/TEPP/issues/156) CLOSED | — | — | Exact request/result/snapshot/cutoff/model/profile binding, typed terminal failures, deterministic retrieval, and cutoff-safe execution are protected-main behavior as of the #157 merge. | | GAP-002 | LineageWeave and other modular consumers can rely on the complete protected-main HTTP evidence/result boundary. | `partial` | consumer hardening remains | Terminal-result lifecycle implemented-main via #157; versioned API contract intact | [#156](https://github.com/ContextualWisdomLab/TEPP/issues/156) (closed) / [PR #155](https://github.com/ContextualWisdomLab/TEPP/pull/155) (merged) | — | Remaining work is consumer-side adoption evidence and any versioned-contract drift discovered during integration; core boundary is no longer the gap it was. | -| GAP-003A | Immutable evidence cannot yet be submitted to a durable validation run that produces operator-usable scientific acceptance evidence. | `accepted-target` | product-completion | `e65cd66` (validation metrics are library-level only) | [#166](https://github.com/ContextualWisdomLab/TEPP/issues/166) | this PR (`feat/validation-run-scientific-acceptance`; library-level binding; not implemented-main; not Compose persistence) | Compose/CLI/API execution must bind immutable evidence, cutoffs, model configuration, validation metrics, and reproducibility manifests to one idempotent run. | +| GAP-003A | Immutable evidence cannot yet be submitted to a durable validation run that produces operator-usable scientific acceptance evidence. | `accepted-target` | product-completion | `e65cd66` (validation metrics are library-level only) | [#166](https://github.com/ContextualWisdomLab/TEPP/issues/166) | [#356](https://github.com/ContextualWisdomLab/TEPP/pull/356) `f8a4dad` (`feat/validation-run-scientific-acceptance`; library-level binding; not implemented-main; not Compose persistence) | Compose/CLI/API execution must bind immutable evidence, cutoffs, model configuration, validation metrics, and reproducibility manifests to one idempotent run. | | GAP-003B | Scientific result artifacts cannot yet be persisted, restarted, and recovered as one supported operator workflow. | `accepted-target` | product-completion | `e65cd66` (persistence contracts lack E2E recovery) | [#166](https://github.com/ContextualWisdomLab/TEPP/issues/166) | `—` (issue program; no current implementation PR) | Durable storage, migration/rollback, restart/recovery, artifact digest verification, and terminal retrieval must pass against a real Compose deployment. | | GAP-003C | The persistence slice classifies concurrent-write SQLSTATEs, but has no measured hot-partition detection, routing, or mitigation for tenant/result workloads. | `accepted-target` | product-completion | `e65cd66` (conflict classification only; no measured partition control) | [#166](https://github.com/ContextualWisdomLab/TEPP/issues/166) | `—` (issue program; no current implementation PR) | A real Compose/PostgreSQL workload identifies hot keys and partition skew, applies bounded tenant/time or result routing without weakening 3NF or temporal authority, and proves conflict rate, latency, recovery, and migration/rollback behavior under load. | | GAP-004 | The central shared-latent temporal/relational topic estimator is absent. | `partial` | product vertical | CPU `f64` TRSL-TM reference estimator with ALR/ILR coordinates and refusal gates is implemented-main (v0.2.0 `topic_measurement`); fitted candidate-`K` scoring present | [#167](https://github.com/ContextualWisdomLab/TEPP/issues/167) | — | GPU, method effects, full Bayesian sampling, and topic birth/split/merge remain. This is not full #167 closure. | From c5315e5b34f55dc87493f38a72c585e41fc00ef9 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Mon, 31 Aug 2026 07:52:35 +0000 Subject: [PATCH 3/5] fix(analysis): stamp recovery to the validation-run binding Refuse completion when recovery vectors belong to a different run, tenant, seed, or eligible evidence set. Include tenant workspace and output profile in the canonical digest so a tampered profile cannot pass. Bound recovery vector length and record a SHA-256 of the stamped vectors on tepp.scientific_acceptance.v1. Evidence fields stay private after completion. This is still the GAP-003A library slice (ADR 0026). It is not implemented-main and does not persist runs (GAP-003B). --- CHANGELOG.md | 2 +- crates/analysis_engine/src/lib.rs | 2 +- crates/analysis_engine/src/validation_run.rs | 370 ++++++++++++++---- .../tests/validation_run_contract.rs | 47 +++ docs/TRACEABILITY.md | 2 +- ...26-validation-run-scientific-acceptance.md | 22 +- .../validation-run-scientific-acceptance.md | 12 +- 7 files changed, 374 insertions(+), 83 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 31b32a03f..05a984036 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -38,7 +38,7 @@ All notable changes to TEPP are documented here. The format follows Keep a Chang ## [Unreleased] -- `analysis_engine` GAP-003A first slice (ADR 0026, active-PR, not implemented-main): `submit_validation_run` binds cutoff-eligible evidence identities, snapshot, knowledge cutoff, CPU `f64` model, seed, backend, and precision to a hash-stable `tepp-validation-{32 hex}` receipt that carries no scientific metrics. `complete_validation_run` emits `tepp.scientific_acceptance.v1` with RMSE, bias, Wilson coverage, temporal-order accuracy, and an SE-aware gate through `validation_core`. LLM-authored recovery, non-finite inputs, empty or duplicate evidence, snapshot mismatch, and cutoff-empty corpora fail closed. Postgres persistence remains GAP-003B. +- `analysis_engine` GAP-003A first slice (ADR 0026, active-PR, not implemented-main): `submit_validation_run` binds cutoff-eligible evidence identities, tenant workspace, snapshot, knowledge cutoff, CPU `f64` model, seed, backend, precision, and output profile to a hash-stable `tepp-validation-{32 hex}` receipt that carries no scientific metrics. `complete_validation_run` requires recovery vectors stamped to that receipt, records a SHA-256 of those vectors, and emits `tepp.scientific_acceptance.v1` with RMSE, bias, Wilson coverage, temporal-order accuracy, and an SE-aware gate through `validation_core`. LLM-authored recovery, a different run/tenant/seed/evidence set, a tampered output profile, oversized vectors, non-finite inputs, empty or duplicate evidence, snapshot mismatch, and cutoff-empty corpora fail closed. Postgres persistence remains GAP-003B. - `event_core` adds bounded Allen interval-consistency classification, atomic path-consistency closure, contradiction/resource refusals, and an explicit dependency-error fallback without claiming unrestricted global satisfiability. diff --git a/crates/analysis_engine/src/lib.rs b/crates/analysis_engine/src/lib.rs index 4c5e4536f..8fe96cd76 100644 --- a/crates/analysis_engine/src/lib.rs +++ b/crates/analysis_engine/src/lib.rs @@ -62,7 +62,7 @@ pub use topic_lineage_artifact::{ }; /// Durable scientific-acceptance validation-run contracts. pub use validation_run::{ - RecoveryObservation, SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE, + MAX_RECOVERY_VECTOR_LEN, RecoveryObservation, SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE, SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION, ScientificAcceptanceEvidence, VALIDATION_BACKEND, VALIDATION_CPU_F64_MODEL, VALIDATION_PRECISION, VALIDATION_RUN_ID_HEX_LEN, VALIDATION_RUN_ID_PREFIX, ValidationRunReceipt, WILSON_Z, complete_validation_run, diff --git a/crates/analysis_engine/src/validation_run.rs b/crates/analysis_engine/src/validation_run.rs index 1af31b6ea..c91d8f41f 100644 --- a/crates/analysis_engine/src/validation_run.rs +++ b/crates/analysis_engine/src/validation_run.rs @@ -1,11 +1,14 @@ //! Bind immutable evidence to a durable scientific-acceptance validation run. //! -//! This is the first GAP-003A slice: cutoff-safe evidence, model, seed, backend, -//! and precision hash to one durable run identity. The accepted receipt carries -//! no scientific metrics. Completion asks `validation_core` for RMSE, bias, -//! coverage, temporal-order accuracy, and an SE-aware gate, then emits -//! `tepp.scientific_acceptance.v1`. LLM-authored recovery, non-finite inputs, -//! empty or duplicate evidence, snapshot mismatch, and cutoff-empty corpora +//! This is the first GAP-003A slice: cutoff-safe evidence, tenant workspace, +//! output profile, model, seed, backend, and precision hash to one durable run +//! identity. The accepted receipt carries no scientific metrics. Completion +//! asks `validation_core` for RMSE, bias, coverage, temporal-order accuracy, +//! and an SE-aware gate, then emits `tepp.scientific_acceptance.v1`. Recovery +//! vectors must be stamped with that same run identity; a different run, +//! model, snapshot, seed, tenant, profile, or eligible evidence set fails +//! closed. LLM-authored recovery, non-finite inputs, empty or duplicate +//! evidence, snapshot mismatch, oversized recovery, and cutoff-empty corpora //! fail closed. Postgres persistence remains GAP-003B. use crate::{AnalysisCorpus, AnalysisEngineError, valid_identifier}; @@ -37,6 +40,8 @@ pub const VALIDATION_RUN_ID_PREFIX: &str = "tepp-validation-"; pub const VALIDATION_RUN_ID_HEX_LEN: usize = 32; /// Wilson critical value for nominal 95% coverage bounds. pub const WILSON_Z: f64 = 1.96; +/// Maximum length of one recovery, interval, or event-time vector. +pub const MAX_RECOVERY_VECTOR_LEN: usize = 10_000; /// Durable identity of one submitted validation run. Receipts never carry /// scientific metrics. @@ -46,6 +51,8 @@ pub struct ValidationRunReceipt { pub run_id: String, /// Lowercase SHA-256 of the canonical binding. pub binding_sha256: String, + /// Tenant workspace that owns the run. + pub tenant_workspace_id: String, /// Immutable source snapshot identity. pub snapshot_id: String, /// Historical cutoff applied to availability. @@ -87,6 +94,12 @@ impl ValidationRunReceipt { &self.binding_sha256 } + /// Return the bound tenant workspace identity. + #[must_use] + pub fn tenant_workspace_id(&self) -> &str { + &self.tenant_workspace_id + } + /// Return the bound snapshot identity. #[must_use] pub fn snapshot_id(&self) -> &str { @@ -134,11 +147,30 @@ impl ValidationRunReceipt { pub const fn eligible_evidence_count(&self) -> u64 { self.eligible_evidence_count } + + fn identity_record(&self) -> String { + format!( + "{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}", + self.run_id, + self.binding_sha256, + self.tenant_workspace_id, + self.snapshot_id, + self.knowledge_cutoff, + self.model, + self.seed, + self.backend, + self.precision, + self.output_profile, + self.eligible_evidence_count + ) + } } /// Known-truth recovery vectors offered to complete a validation run. #[derive(Clone, Debug, PartialEq)] pub struct RecoveryObservation { + run_id: String, + binding_sha256: String, study_label: String, truth: Vec, recovered: Vec, @@ -151,19 +183,24 @@ pub struct RecoveryObservation { } impl RecoveryObservation { - /// Construct a recovery observation. + /// Construct a recovery observation stamped to one submitted receipt. /// /// LLM authorship is recorded here and refused at completion. Non-finite - /// `se_gate_k` or a negative multiplier fail closed immediately. + /// `se_gate_k`, a negative multiplier, or an oversized vector fail closed + /// immediately. Completing a different run, model, snapshot, seed, tenant, + /// profile, or eligible evidence set fails later as a binding mismatch. /// /// # Errors /// /// Returns [`AnalysisEngineError::InvalidEvidence`] for an empty, oversized, - /// or control-bearing study label, and + /// or control-bearing study label, + /// [`AnalysisEngineError::LimitExceeded`] when any vector exceeds + /// [`MAX_RECOVERY_VECTOR_LEN`], and /// [`AnalysisEngineError::Validation`] for a non-finite or negative SE /// multiplier. #[allow(clippy::too_many_arguments)] pub fn new( + receipt: &ValidationRunReceipt, study_label: impl Into, truth: Vec, recovered: Vec, @@ -178,6 +215,19 @@ impl RecoveryObservation { if !valid_identifier(&study_label) { return Err(AnalysisEngineError::InvalidEvidence); } + if [ + truth.len(), + recovered.len(), + interval_lower.len(), + interval_upper.len(), + truth_times.len(), + recovered_times.len(), + ] + .into_iter() + .any(|len| len > MAX_RECOVERY_VECTOR_LEN) + { + return Err(AnalysisEngineError::LimitExceeded); + } if !se_gate_k.is_finite() { return Err(AnalysisEngineError::Validation( ValidationError::InvalidInput, @@ -189,6 +239,8 @@ impl RecoveryObservation { )); } Ok(Self { + run_id: receipt.run_id.clone(), + binding_sha256: receipt.binding_sha256.clone(), study_label, truth, recovered, @@ -201,6 +253,18 @@ impl RecoveryObservation { }) } + /// Return the stamped run identity. + #[must_use] + pub fn run_id(&self) -> &str { + &self.run_id + } + + /// Return the stamped binding digest. + #[must_use] + pub fn binding_sha256(&self) -> &str { + &self.binding_sha256 + } + /// Return the study label. #[must_use] pub fn study_label(&self) -> &str { @@ -254,39 +318,43 @@ impl RecoveryObservation { pub const fn authored_by_llm(&self) -> bool { self.authored_by_llm } + + fn digest_hex(&self) -> String { + let mut canonical = String::from("tepp.recovery_observation.v1\n"); + let _ = writeln!(canonical, "run_id={}", self.run_id); + let _ = writeln!(canonical, "binding={}", self.binding_sha256); + let _ = writeln!(canonical, "study={}", self.study_label); + let _ = writeln!(canonical, "se_gate_k={:016x}", self.se_gate_k.to_bits()); + let _ = writeln!(canonical, "authored_by_llm={}", self.authored_by_llm); + append_f64_vector(&mut canonical, "truth", &self.truth); + append_f64_vector(&mut canonical, "recovered", &self.recovered); + append_f64_vector(&mut canonical, "interval_lower", &self.interval_lower); + append_f64_vector(&mut canonical, "interval_upper", &self.interval_upper); + append_f64_vector(&mut canonical, "truth_times", &self.truth_times); + append_f64_vector(&mut canonical, "recovered_times", &self.recovered_times); + format_hex(Sha256::digest(canonical.into_bytes())) + } } /// Operator-usable scientific acceptance evidence for one completed run. #[derive(Clone, Debug, PartialEq, Serialize)] pub struct ScientificAcceptanceEvidence { - /// Versioned artifact schema. - pub schema_version: String, - /// Durable run identity. - pub run_id: String, - /// Canonical binding digest. - pub binding_sha256: String, - /// Immutable source snapshot identity. - pub snapshot_id: String, - /// Historical cutoff applied to availability. - pub knowledge_cutoff: String, - /// Bound model identity. - pub model: String, - /// Bound numeric seed. - pub seed: u64, - /// Bound compute backend. - pub backend: String, - /// Bound numeric precision. - pub precision: String, - /// Output profile. - pub output_profile: String, - /// Number of cutoff-eligible evidence identities. - pub eligible_evidence_count: u64, - /// Whether RMSE toward 0 passed the SE-aware gate. - pub se_gate_accepted: bool, - /// SE-gate multiplier used for acceptance. - pub se_gate_k: f64, - /// Machine-readable recovery report. - pub report: ValidationReport, + schema_version: String, + run_id: String, + binding_sha256: String, + recovery_sha256: String, + tenant_workspace_id: String, + snapshot_id: String, + knowledge_cutoff: String, + model: String, + seed: u64, + backend: String, + precision: String, + output_profile: String, + eligible_evidence_count: u64, + se_gate_accepted: bool, + se_gate_k: f64, + report: ValidationReport, } impl ScientificAcceptanceEvidence { @@ -330,20 +398,94 @@ impl ScientificAcceptanceEvidence { &self.run_id } + /// Return the scientific binding digest. + #[must_use] + pub fn binding_sha256(&self) -> &str { + &self.binding_sha256 + } + + /// Return the recovery-vector digest stamped into this artifact. + #[must_use] + pub fn recovery_sha256(&self) -> &str { + &self.recovery_sha256 + } + + /// Return the tenant workspace identity. + #[must_use] + pub fn tenant_workspace_id(&self) -> &str { + &self.tenant_workspace_id + } + + /// Return the snapshot identity. + #[must_use] + pub fn snapshot_id(&self) -> &str { + &self.snapshot_id + } + + /// Return the knowledge cutoff. + #[must_use] + pub fn knowledge_cutoff(&self) -> &str { + &self.knowledge_cutoff + } + + /// Return the bound model identity. + #[must_use] + pub fn model(&self) -> &str { + &self.model + } + + /// Return the bound seed. + #[must_use] + pub const fn seed(&self) -> u64 { + self.seed + } + + /// Return the bound backend. + #[must_use] + pub fn backend(&self) -> &str { + &self.backend + } + + /// Return the bound precision. + #[must_use] + pub fn precision(&self) -> &str { + &self.precision + } + + /// Return the output profile. + #[must_use] + pub fn output_profile(&self) -> &str { + &self.output_profile + } + + /// Return the eligible evidence count. + #[must_use] + pub const fn eligible_evidence_count(&self) -> u64 { + self.eligible_evidence_count + } + /// Return whether the SE-aware gate accepted RMSE toward 0. #[must_use] pub const fn se_gate_accepted(&self) -> bool { self.se_gate_accepted } + + /// Return the SE-gate multiplier used for acceptance. + #[must_use] + pub const fn se_gate_k(&self) -> f64 { + self.se_gate_k + } } struct CanonicalBinding { + tenant_workspace_id: String, snapshot_id: String, knowledge_cutoff: String, model: String, seed: u64, backend: String, precision: String, + output_profile: String, eligible_ids: Vec, } @@ -366,12 +508,14 @@ impl CanonicalBinding { fn canonical_bytes(&self) -> String { let mut canonical = String::from("tepp.validation_binding.v1\n"); + let _ = writeln!(canonical, "tenant={}", self.tenant_workspace_id); let _ = writeln!(canonical, "snapshot={}", self.snapshot_id); let _ = writeln!(canonical, "cutoff={}", self.knowledge_cutoff); let _ = writeln!(canonical, "model={}", self.model); let _ = writeln!(canonical, "seed={}", self.seed); let _ = writeln!(canonical, "backend={}", self.backend); let _ = writeln!(canonical, "precision={}", self.precision); + let _ = writeln!(canonical, "profile={}", self.output_profile); for identity in &self.eligible_ids { let _ = writeln!(canonical, "evidence={identity}"); } @@ -380,35 +524,20 @@ impl CanonicalBinding { fn identity_record(&self, digest_hex: &str, run_id: &str) -> String { format!( - "{run_id}\n{digest_hex}\n{}\n{}\n{}\n{}\n{}\n{}\n{}", + "{run_id}\n{digest_hex}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}", + self.tenant_workspace_id, self.snapshot_id, self.knowledge_cutoff, self.model, self.seed, self.backend, self.precision, + self.output_profile, self.eligible_count() ) } } -impl ValidationRunReceipt { - fn identity_record(&self) -> String { - format!( - "{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}", - self.run_id, - self.binding_sha256, - self.snapshot_id, - self.knowledge_cutoff, - self.model, - self.seed, - self.backend, - self.precision, - self.eligible_evidence_count - ) - } -} - /// Submit immutable evidence to a durable validation run. /// /// The receipt is hash-stable for one scientific binding and never includes @@ -431,13 +560,14 @@ pub fn submit_validation_run( Ok(ValidationRunReceipt { run_id, binding_sha256, + tenant_workspace_id: binding.tenant_workspace_id, snapshot_id: binding.snapshot_id, knowledge_cutoff: binding.knowledge_cutoff, model: binding.model, seed: binding.seed, backend: binding.backend, precision: binding.precision, - output_profile: SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE.to_owned(), + output_profile: binding.output_profile, eligible_evidence_count, }) } @@ -447,8 +577,9 @@ pub fn submit_validation_run( /// # Errors /// /// Returns a fail-closed engine error when recovery was LLM-authored, the -/// receipt does not match the rebound scientific identity, metric inputs are -/// invalid, or report validation fails. +/// observation is stamped to a different run identity, the receipt does not +/// match the rebound scientific identity, metric inputs are invalid, or +/// report validation fails. pub fn complete_validation_run( receipt: &ValidationRunReceipt, request: &AnalysisRunRequest, @@ -458,6 +589,12 @@ pub fn complete_validation_run( if observation.authored_by_llm { return Err(AnalysisEngineError::LlmAuthoredRecovery); } + if observation.run_id != receipt.run_id + || observation.binding_sha256 != receipt.binding_sha256 + || receipt.output_profile != SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE + { + return Err(AnalysisEngineError::BindingMismatch); + } let binding = bind_validation_run(request, corpus, receipt.seed)?; let binding_sha256 = binding.digest_hex(); let run_id = CanonicalBinding::run_id(&binding_sha256); @@ -476,6 +613,8 @@ pub fn complete_validation_run( schema_version: SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION.to_owned(), run_id, binding_sha256, + recovery_sha256: observation.digest_hex(), + tenant_workspace_id: binding.tenant_workspace_id, snapshot_id: binding.snapshot_id, knowledge_cutoff: binding.knowledge_cutoff, model: binding.model, @@ -528,12 +667,14 @@ fn bind_validation_run( return Err(AnalysisEngineError::NoEligibleEvidence); } Ok(CanonicalBinding { + tenant_workspace_id: request.tenant_workspace_id.clone(), snapshot_id: request.snapshot_id.clone(), knowledge_cutoff: cutoff.to_rfc3339(), model: VALIDATION_CPU_F64_MODEL.to_owned(), seed, backend: VALIDATION_BACKEND.to_owned(), precision: VALIDATION_PRECISION.to_owned(), + output_profile: SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE.to_owned(), eligible_ids: eligible.into_iter().collect(), }) } @@ -574,6 +715,13 @@ fn compute_report( Ok(report) } +fn append_f64_vector(canonical: &mut String, label: &str, values: &[f64]) { + let _ = writeln!(canonical, "{label}_len={}", values.len()); + for value in values { + let _ = writeln!(canonical, "{label}={:016x}", value.to_bits()); + } +} + fn format_hex(digest: impl AsRef<[u8]>) -> String { let bytes = digest.as_ref(); let mut output = String::with_capacity(bytes.len() * 2); @@ -586,7 +734,7 @@ fn format_hex(digest: impl AsRef<[u8]>) -> String { #[cfg(test)] mod tests { use super::{ - RecoveryObservation, SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE, + MAX_RECOVERY_VECTOR_LEN, RecoveryObservation, SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE, SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION, VALIDATION_BACKEND, VALIDATION_CPU_F64_MODEL, VALIDATION_PRECISION, VALIDATION_RUN_ID_PREFIX, WILSON_Z, complete_validation_run, submit_validation_run, @@ -624,12 +772,18 @@ mod tests { AnalysisCorpus::new("snapshot-1", units).expect("corpus") } - fn recovery(truth: Vec, recovered: Vec, k: f64) -> RecoveryObservation { + fn recovery( + receipt: &super::ValidationRunReceipt, + truth: Vec, + recovered: Vec, + k: f64, + ) -> RecoveryObservation { let times = [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0]; let times = times[..truth.len()].to_vec(); let lower: Vec = truth.iter().map(|value| value - 0.5).collect(); let upper: Vec = truth.iter().map(|value| value + 0.5).collect(); RecoveryObservation::new( + receipt, "foundation-recovery", truth, recovered, @@ -644,11 +798,13 @@ mod tests { } fn two_point( + receipt: &super::ValidationRunReceipt, label: impl Into, k: f64, authored_by_llm: bool, ) -> Result { RecoveryObservation::new( + receipt, label, vec![1.0, 2.0], vec![1.0, 2.0], @@ -682,6 +838,7 @@ mod tests { VALIDATION_RUN_ID_PREFIX.len() + 32 ); assert_eq!(receipt_a.eligible_evidence_count(), 2); + assert_eq!(receipt_a.tenant_workspace_id(), "tenant-workspace-1"); assert_eq!(receipt_a.snapshot_id(), "snapshot-1"); assert_eq!(receipt_a.knowledge_cutoff(), "2026-08-01T00:00:00Z"); assert_eq!(receipt_a.model(), VALIDATION_CPU_F64_MODEL); @@ -698,6 +855,10 @@ mod tests { assert!(!json.contains("coverage")); let other_seed = submit_validation_run(&request(), &first, 8).expect("seed"); assert_ne!(receipt_a.run_id(), other_seed.run_id()); + let mut other_tenant = request(); + other_tenant.tenant_workspace_id = "tenant-workspace-2".into(); + let other_tenant_receipt = submit_validation_run(&other_tenant, &first, 7).expect("tenant"); + assert_ne!(receipt_a.run_id(), other_tenant_receipt.run_id()); } #[test] @@ -708,10 +869,13 @@ mod tests { ]); let receipt = submit_validation_run(&request(), &offered, 11).expect("submit"); let observation = recovery( + &receipt, vec![0.70, 0.55, 0.40, -0.20, 0.85], vec![0.70, 0.55, 0.40, -0.20, 0.85], 3.0, ); + assert_eq!(observation.run_id(), receipt.run_id()); + assert_eq!(observation.binding_sha256(), receipt.binding_sha256()); assert_eq!(observation.study_label(), "foundation-recovery"); assert_eq!(observation.truth().len(), 5); assert_eq!(observation.recovered().len(), 5); @@ -728,20 +892,31 @@ mod tests { SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION ); assert_eq!(evidence.run_id(), receipt.run_id()); + assert_eq!(evidence.binding_sha256(), receipt.binding_sha256()); + assert_eq!(evidence.recovery_sha256().len(), 64); + assert_eq!(evidence.tenant_workspace_id(), "tenant-workspace-1"); assert!(evidence.se_gate_accepted()); assert_eq!( - evidence.output_profile, + evidence.output_profile(), SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE ); - assert_eq!(evidence.eligible_evidence_count, 2); - assert!((evidence.se_gate_k - 3.0).abs() < f64::EPSILON); + assert_eq!(evidence.eligible_evidence_count(), 2); + assert!((evidence.se_gate_k() - 3.0).abs() < f64::EPSILON); + assert_eq!(evidence.snapshot_id(), "snapshot-1"); + assert_eq!(evidence.knowledge_cutoff(), "2026-08-01T00:00:00Z"); + assert_eq!(evidence.model(), VALIDATION_CPU_F64_MODEL); + assert_eq!(evidence.seed(), 11); + assert_eq!(evidence.backend(), VALIDATION_BACKEND); + assert_eq!(evidence.precision(), VALIDATION_PRECISION); assert!((WILSON_Z - 1.96).abs() < f64::EPSILON); let json = evidence.to_json().expect("json"); assert!(json.contains(SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION)); assert!(json.contains("rmse")); + assert!(json.contains("recovery_sha256")); assert_eq!(evidence.sha256().expect("digest").len(), 64); assert!(evidence.to_human_summary().contains("foundation-recovery")); let rejected = recovery( + &receipt, vec![0.0, 1.0, 2.0, 3.0, 4.0], vec![10.0, 11.0, 12.0, 13.0, 14.0], 1.0, @@ -749,6 +924,7 @@ mod tests { let refused = complete_validation_run(&receipt, &request(), &offered, &rejected).expect("refused"); assert!(!refused.se_gate_accepted()); + assert_ne!(refused.recovery_sha256(), evidence.recovery_sha256()); } #[test] @@ -815,6 +991,7 @@ mod tests { ]); let receipt = submit_validation_run(&request(), &offered, 3).expect("submit"); let llm = RecoveryObservation::new( + &receipt, "llm-study", vec![1.0, 2.0, 3.0], vec![1.0, 2.0, 3.0], @@ -833,11 +1010,46 @@ mod tests { ); let mut tampered = receipt.clone(); tampered.run_id = "tepp-validation-deadbeefdeadbeefdeadbeefdeadbeef".into(); - let observation = recovery(vec![1.0, 2.0, 3.0], vec![1.0, 2.0, 3.0], 3.0); + let observation = recovery(&receipt, vec![1.0, 2.0, 3.0], vec![1.0, 2.0, 3.0], 3.0); assert_eq!( complete_validation_run(&tampered, &request(), &offered, &observation), Err(AnalysisEngineError::BindingMismatch) ); + let mut profile_tampered = receipt.clone(); + profile_tampered.output_profile = "tampered-profile".into(); + assert_eq!( + complete_validation_run(&profile_tampered, &request(), &offered, &observation), + Err(AnalysisEngineError::BindingMismatch) + ); + } + + #[test] + fn complete_refuses_recovery_stamped_to_a_different_run() { + let first = corpus(vec![unit("evidence-a", "2026-07-10T00:00:00Z")]); + let second = corpus(vec![ + unit("evidence-a", "2026-07-10T00:00:00Z"), + unit("evidence-b", "2026-07-15T00:00:00Z"), + ]); + let receipt_a = submit_validation_run(&request(), &first, 3).expect("a"); + let receipt_b = submit_validation_run(&request(), &second, 3).expect("b"); + assert_ne!(receipt_a.run_id(), receipt_b.run_id()); + let foreign = recovery(&receipt_a, vec![1.0, 2.0, 3.0], vec![1.0, 2.0, 3.0], 3.0); + assert_eq!( + complete_validation_run(&receipt_b, &request(), &second, &foreign), + Err(AnalysisEngineError::BindingMismatch) + ); + let other_seed = submit_validation_run(&request(), &first, 9).expect("seed"); + assert_eq!( + complete_validation_run(&other_seed, &request(), &first, &foreign), + Err(AnalysisEngineError::BindingMismatch) + ); + let mut other_tenant = request(); + other_tenant.tenant_workspace_id = "tenant-workspace-other".into(); + let other_tenant_receipt = submit_validation_run(&other_tenant, &first, 3).expect("tenant"); + assert_eq!( + complete_validation_run(&other_tenant_receipt, &other_tenant, &first, &foreign), + Err(AnalysisEngineError::BindingMismatch) + ); } #[test] @@ -848,6 +1060,7 @@ mod tests { ]); let receipt = submit_validation_run(&request(), &offered, 3).expect("submit"); let nan = RecoveryObservation::new( + &receipt, "nan-study", vec![1.0, 2.0, 3.0], vec![f64::NAN, 2.0, 3.0], @@ -866,25 +1079,46 @@ mod tests { )) ); assert_eq!( - two_point("", 3.0, false), + two_point(&receipt, "", 3.0, false), Err(AnalysisEngineError::InvalidEvidence) ); assert_eq!( - two_point("x".repeat(MAX_ANALYSIS_IDENTIFIER_BYTES + 1), 3.0, false), + two_point( + &receipt, + "x".repeat(MAX_ANALYSIS_IDENTIFIER_BYTES + 1), + 3.0, + false + ), Err(AnalysisEngineError::InvalidEvidence) ); assert_eq!( - two_point("nan-k", f64::NAN, false), + two_point(&receipt, "nan-k", f64::NAN, false), Err(AnalysisEngineError::Validation( ValidationError::InvalidInput )) ); assert_eq!( - two_point("neg-k", -1.0, false), + two_point(&receipt, "neg-k", -1.0, false), Err(AnalysisEngineError::Validation( ValidationError::InvalidConfiguration )) ); + let oversized = vec![0.0; MAX_RECOVERY_VECTOR_LEN + 1]; + assert_eq!( + RecoveryObservation::new( + &receipt, + "oversized", + oversized.clone(), + oversized.clone(), + oversized.clone(), + oversized.clone(), + oversized.clone(), + oversized, + 3.0, + false, + ), + Err(AnalysisEngineError::LimitExceeded) + ); let converted: AnalysisEngineError = ValidationError::InvalidInput.into(); assert_eq!(converted.to_string(), "invalid validation input"); } diff --git a/crates/analysis_engine/tests/validation_run_contract.rs b/crates/analysis_engine/tests/validation_run_contract.rs index 1ebb164d5..82d97896d 100644 --- a/crates/analysis_engine/tests/validation_run_contract.rs +++ b/crates/analysis_engine/tests/validation_run_contract.rs @@ -45,6 +45,7 @@ fn durable_run_binds_cutoff_eligible_evidence_and_emits_acceptance_evidence() { assert_eq!(receipt.eligible_evidence_count(), 2); assert!(!receipt.to_json().expect("json").contains("rmse")); let observation = RecoveryObservation::new( + &receipt, "contract-recovery", vec![0.70, 0.55, 0.40, -0.20, 0.85], vec![0.70, 0.55, 0.40, -0.20, 0.85], @@ -63,6 +64,7 @@ fn durable_run_binds_cutoff_eligible_evidence_and_emits_acceptance_evidence() { SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION ); assert_eq!(evidence.run_id(), receipt.run_id()); + assert_eq!(evidence.recovery_sha256().len(), 64); assert!(evidence.se_gate_accepted()); assert!(evidence.to_json().expect("json").contains("rmse")); } @@ -85,6 +87,7 @@ fn llm_authored_recovery_and_cutoff_empty_corpora_fail_closed() { .expect("eligible"); let receipt = submit_validation_run(&request(), &eligible, 1).expect("submit"); let llm = RecoveryObservation::new( + &receipt, "llm-contract", vec![1.0, 2.0, 3.0], vec![1.0, 2.0, 3.0], @@ -101,3 +104,47 @@ fn llm_authored_recovery_and_cutoff_empty_corpora_fail_closed() { Err(AnalysisEngineError::LlmAuthoredRecovery) ); } + +#[test] +fn recovery_from_a_different_run_or_tenant_fails_closed() { + let first = AnalysisCorpus::new( + "snapshot-contract", + vec![unit("evidence-1", "2026-07-10T00:00:00Z")], + ) + .expect("first"); + let second = AnalysisCorpus::new( + "snapshot-contract", + vec![ + unit("evidence-1", "2026-07-10T00:00:00Z"), + unit("evidence-2", "2026-07-20T00:00:00Z"), + ], + ) + .expect("second"); + let receipt_a = submit_validation_run(&request(), &first, 1).expect("a"); + let receipt_b = submit_validation_run(&request(), &second, 1).expect("b"); + let foreign = RecoveryObservation::new( + &receipt_a, + "foreign-recovery", + vec![1.0, 2.0, 3.0], + vec![1.0, 2.0, 3.0], + vec![0.5, 1.5, 2.5], + vec![1.5, 2.5, 3.5], + vec![1.0, 2.0, 3.0], + vec![1.0, 2.0, 3.0], + 3.0, + false, + ) + .expect("foreign"); + assert_eq!( + complete_validation_run(&receipt_b, &request(), &second, &foreign), + Err(AnalysisEngineError::BindingMismatch) + ); + let mut other_tenant = request(); + other_tenant.tenant_workspace_id = "tenant-workspace-other".into(); + let other_receipt = submit_validation_run(&other_tenant, &first, 1).expect("tenant"); + assert_ne!(receipt_a.run_id(), other_receipt.run_id()); + assert_eq!( + complete_validation_run(&other_receipt, &other_tenant, &first, &foreign), + Err(AnalysisEngineError::BindingMismatch) + ); +} diff --git a/docs/TRACEABILITY.md b/docs/TRACEABILITY.md index c2ab5d13f..6768092ec 100644 --- a/docs/TRACEABILITY.md +++ b/docs/TRACEABILITY.md @@ -50,7 +50,7 @@ The full APA 7th standards/literature register remains `docs/research/standards- | leakage-safe availability/cutoff snapshots | PRD; ADR 0002/0013 | `corpus_split` on protected main | implemented-main | | Unicode NFC/NFD canonical identity for split leakage | ADR 0004/0008/0013; UAX #15 | `corpus_split` canonical-equivalence links on active PR #59 | active-PR | | recovery metrics (RMSE, bias, coverage, graph, temporal order, Monte Carlo SE gates) | PRD; Test Strategy; ADR 0007/0014 | `validation_core` on protected main (PR #19); SE-aware Monte Carlo gates included | implemented-main | -| durable validation-run scientific acceptance evidence | ADR 0026; ADR 0002/0008/0014; GAP-003A; National Academies (2019); Wasserstein & Lazar (2016); Wilson (1927) | `analysis_engine` `submit_validation_run` / `complete_validation_run` on this PR: hash-stable cutoff-safe binding, metric-free receipt, `tepp.scientific_acceptance.v1` RMSE/bias/coverage/temporal-order/SE-aware evidence; LLM/NaN/empty/duplicate/snapshot/cutoff-empty fail closed; not implemented-main; Postgres persistence remains GAP-003B | active-PR | +| durable validation-run scientific acceptance evidence | ADR 0026; ADR 0002/0008/0014; GAP-003A; National Academies (2019); Wasserstein & Lazar (2016); Wilson (1927) | `analysis_engine` `submit_validation_run` / `complete_validation_run` on this PR: hash-stable cutoff-safe tenant/profile binding, metric-free receipt, recovery stamped to that `run_id`, `tepp.scientific_acceptance.v1` RMSE/bias/coverage/temporal-order/SE-aware evidence with recovery digest; LLM/NaN/empty/duplicate/snapshot/foreign-run/tampered-profile/oversized/cutoff-empty fail closed; not implemented-main; Postgres persistence remains GAP-003B | active-PR | | PostgreSQL bitemporal/lineage persistence | ADR 0013; Architecture/ERD | `persistence_postgres` migration contracts, in-memory adapters, live SQL session/document SQL port, tenant RLS (`0002` + session GUC/role helpers), `DATABASE_URL` SQLx gate, optional session-affine `live-sqlx` `PgPool` driver, exact-head live PostgreSQL CI with isolation proof, append-only immutability triggers (`0004`), temporal interval ordering CHECKs (`0005`), typed membership assignment (`0006` implemented-main), event-relation/mention/instance SQL (#37–#39 implemented-main), source-artifact SQL (#40 implemented-main), audit-event SQL (#41 implemented-main), concurrent document-write stress (#43 implemented-main), backup/restore integrity revalidation (#44 implemented-main), `revision_order` later-revision system-time ordering implemented-main, entity/project target SQL on PR #131; remaining physical ERD constraints | partial | | known-truth temporal/event simulation manifests | PRD; TRD; Test Strategy | `tepp_simulation` on protected main; recovery metrics in `validation_core` | implemented-main | | versioned service/API contracts and exports | PRD; API contract; ADR 0011/0013 | `tepp_api` analysis-run/export/JSON-LD/GraphML contracts on protected main (PR #21); request-bound terminal result active in PR #157; HTTP service remains accepted-target; the `orchestrator_live` loopback interpretation listener is on this PR | partial | diff --git a/docs/adr/0026-validation-run-scientific-acceptance.md b/docs/adr/0026-validation-run-scientific-acceptance.md index e769539f2..c692eb29a 100644 --- a/docs/adr/0026-validation-run-scientific-acceptance.md +++ b/docs/adr/0026-validation-run-scientific-acceptance.md @@ -25,17 +25,22 @@ Postgres persistence, restart/recovery, and Compose E2E remain GAP-003B. Add a bounded `analysis_engine` validation-run executor: - `submit_validation_run` binds sorted cutoff-eligible evidence identities, - snapshot, knowledge cutoff, model `validation_cpu_f64_v1`, seed, backend - `cpu`, and precision `f64` into a canonical SHA-256 digest. The durable + tenant workspace, snapshot, knowledge cutoff, model `validation_cpu_f64_v1`, + seed, backend `cpu`, precision `f64`, and output profile + `scientific_acceptance_v1` into a canonical SHA-256 digest. The durable `run_id` is `tepp-validation-{32 hex}`. The receipt carries no RMSE, bias, coverage, or gate fields. -- `complete_validation_run` rebinds the same scientific identity, refuses +- `complete_validation_run` rebinds the same scientific identity, requires + recovery vectors stamped to that `run_id` and binding digest, refuses LLM-authored recovery, computes `validation_core` recovery metrics, applies the SE-aware gate `|RMSE − 0| ≤ k · SE(RMSE)`, and emits `tepp.scientific_acceptance.v1` under output profile - `scientific_acceptance_v1`. + `scientific_acceptance_v1`. The artifact records a SHA-256 of the stamped + recovery vectors. Evidence fields are private after completion. - Empty corpora, duplicate evidence identities, snapshot mismatch, invalid - profiles, non-finite inputs, and cutoff-empty eligibility fail closed. + profiles, non-finite inputs, oversized recovery vectors, a different run / + tenant / seed / eligible evidence set, a tampered output profile, and + cutoff-empty eligibility fail closed. - A computed recovery that fails the SE-aware gate still emits evidence with `se_gate_accepted = false` so operators can read the metrics. Invalid or LLM-authored recovery never emits evidence. @@ -67,9 +72,10 @@ become scientific authority. ## Verification The stacked PR includes unit and integration tests for hash-stable identity, -cutoff exclusion, metric-free receipts, SE-aware accept and refuse, and -fail-closed LLM, NaN, empty, duplicate, snapshot, profile, and cutoff-empty -paths. Run: +cutoff exclusion, metric-free receipts, SE-aware accept and refuse, recovery +stamped to a foreign run or tenant, a tampered output profile, oversized +vectors, and fail-closed LLM, NaN, empty, duplicate, snapshot, profile, and +cutoff-empty paths. Run: ```text cargo fmt --all -- --check diff --git a/docs/research/validation-run-scientific-acceptance.md b/docs/research/validation-run-scientific-acceptance.md index 4bc06125c..287c0bfdc 100644 --- a/docs/research/validation-run-scientific-acceptance.md +++ b/docs/research/validation-run-scientific-acceptance.md @@ -4,10 +4,11 @@ This note doctors the first GAP-003A slice in `analysis_engine`: -1. immutable evidence, knowledge cutoff, model, seed, backend, and precision bind to one hash-stable validation run; +1. immutable evidence, tenant workspace, knowledge cutoff, model, seed, backend, precision, and output profile bind to one hash-stable validation run; 2. the accepted receipt carries no scientific metrics; -3. completion emits `tepp.scientific_acceptance.v1` with RMSE, bias, interval coverage (Wilson bounds), temporal-order accuracy, and an SE-aware gate; -4. LLM-authored recovery, non-finite inputs, empty or duplicate evidence, snapshot mismatch, and cutoff-empty corpora fail closed. +3. completion emits `tepp.scientific_acceptance.v1` with RMSE, bias, interval coverage (Wilson bounds), temporal-order accuracy, an SE-aware gate, and a SHA-256 of the stamped recovery vectors; +4. recovery vectors must be constructed against that receipt; a different run, tenant, seed, snapshot, profile, or eligible evidence set fails closed; +5. LLM-authored recovery, non-finite inputs, empty or duplicate evidence, snapshot mismatch, oversized recovery, and cutoff-empty corpora fail closed. Postgres persistence, restart/recovery, and Compose execution remain GAP-003B. This slice is not implemented-main. @@ -21,7 +22,7 @@ Wilson, E. B. (1927). Probable inference, the law of succession, and statistical ## Application -The National Academies (2019) separate computational reproducibility (same binding, same digest) from a scientific claim that recovery is correct. Wasserstein and Lazar (2016) refuse to treat a passing threshold as automatic scientific authority, so a failed SE-aware gate still reports metrics with `se_gate_accepted = false` rather than inventing a passing claim. Wilson (1927) supplies the coverage interval bounds already implemented in `validation_core`. TEPP therefore binds cutoff-safe evidence before any metric is computed, refuses LLM-authored recovery, and reports RMSE, bias, coverage, temporal order, and the SE-aware gate as operator-usable evidence (National Academies of Sciences, Engineering, and Medicine, 2019; Wasserstein & Lazar, 2016; Wilson, 1927). +The National Academies (2019) separate computational reproducibility (same binding, same digest) from a scientific claim that recovery is correct. Wasserstein and Lazar (2016) refuse to treat a passing threshold as automatic scientific authority, so a failed SE-aware gate still reports metrics with `se_gate_accepted = false` rather than inventing a passing claim. Wilson (1927) supplies the coverage interval bounds already implemented in `validation_core`. TEPP therefore binds cutoff-safe evidence and tenant/profile identity before any metric is computed, stamps recovery vectors to that binding, refuses LLM-authored recovery, and reports RMSE, bias, coverage, temporal order, and the SE-aware gate as operator-usable evidence (National Academies of Sciences, Engineering, and Medicine, 2019; Wasserstein & Lazar, 2016; Wilson, 1927). Meredith (1993) remains unread (Unpaywall/OpenAlex 2026-08-31T07:45Z: `is_oa: false`, 0 locations). Mislevy (1991, *Psychometrika, 56*, 177–196) remains unread on the same terms (DOI `10.1007/bf02294457`). ## Verification @@ -29,4 +30,7 @@ The National Academies (2019) separate computational reproducibility (same bindi - receipts serialize without RMSE, bias, or coverage fields; - known-truth recovery emits `tepp.scientific_acceptance.v1` with a digest-stable JSON body; - a large residual vector remains operator-readable with `se_gate_accepted = false`; +- recovery stamped to a different run, tenant, or seed returns `BindingMismatch`; +- a tampered receipt `output_profile` returns `BindingMismatch`; +- an oversized recovery vector returns `LimitExceeded`; - LLM authorship, NaN recovery, empty corpora, duplicates, snapshot mismatch, wrong profile/model, and cutoff-empty eligibility return dedicated fail-closed errors. From 791660336458cf99176e6bd3019d5fcb03dabf7c Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Mon, 31 Aug 2026 08:06:26 +0000 Subject: [PATCH 4/5] fix(analysis): pre-register the validation SE-gate multiplier GAP-003A Devin repair on #356. se_gate_k is part of the submitted scientific binding, not a completion-time choice. Changing k changes run identity. k must be finite, non-negative, and at most MAX_SE_GATE_K = 8. RecoveryObservation must match the receipt k; post-hoc or oversized multipliers fail closed. Receipt fields are private. Empty and length-mismatched recovery vectors fail at construction. Wasserstein and Lazar (2016) refuse post-hoc threshold shopping. Not implemented-main. Postgres persistence remains GAP-003B. ADR 0026. --- CHANGELOG.md | 2 +- crates/analysis_engine/src/lib.rs | 10 +- crates/analysis_engine/src/validation_run.rs | 272 ++++++++++++------ .../tests/validation_run_contract.rs | 30 +- docs/TRACEABILITY.md | 2 +- ...26-validation-run-scientific-acceptance.md | 27 +- .../validation-run-scientific-acceptance.md | 16 +- 7 files changed, 241 insertions(+), 118 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 05a984036..ccf1bacdb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -38,7 +38,7 @@ All notable changes to TEPP are documented here. The format follows Keep a Chang ## [Unreleased] -- `analysis_engine` GAP-003A first slice (ADR 0026, active-PR, not implemented-main): `submit_validation_run` binds cutoff-eligible evidence identities, tenant workspace, snapshot, knowledge cutoff, CPU `f64` model, seed, backend, precision, and output profile to a hash-stable `tepp-validation-{32 hex}` receipt that carries no scientific metrics. `complete_validation_run` requires recovery vectors stamped to that receipt, records a SHA-256 of those vectors, and emits `tepp.scientific_acceptance.v1` with RMSE, bias, Wilson coverage, temporal-order accuracy, and an SE-aware gate through `validation_core`. LLM-authored recovery, a different run/tenant/seed/evidence set, a tampered output profile, oversized vectors, non-finite inputs, empty or duplicate evidence, snapshot mismatch, and cutoff-empty corpora fail closed. Postgres persistence remains GAP-003B. +- `analysis_engine` GAP-003A first slice (ADR 0026, active-PR, not implemented-main): `submit_validation_run` binds cutoff-eligible evidence identities, tenant workspace, snapshot, knowledge cutoff, CPU `f64` model, seed, backend, precision, output profile, and a pre-registered SE-gate multiplier `k` (`0 ≤ k ≤ MAX_SE_GATE_K` = 8) to a hash-stable `tepp-validation-{32 hex}` receipt that carries no scientific metrics. Receipt fields are private. `complete_validation_run` requires recovery vectors stamped to that receipt with the same `k`, records a SHA-256 of those vectors, and emits `tepp.scientific_acceptance.v1` with RMSE, bias, Wilson coverage, temporal-order accuracy, and an SE-aware gate through `validation_core` using the submitted `k`. LLM-authored recovery, a different run/tenant/seed/evidence set, a post-hoc or oversized `k`, a tampered output profile, oversized/empty/mismatched vectors, non-finite inputs, empty or duplicate evidence, snapshot mismatch, and cutoff-empty corpora fail closed. Postgres persistence remains GAP-003B. - `event_core` adds bounded Allen interval-consistency classification, atomic path-consistency closure, contradiction/resource refusals, and an explicit dependency-error fallback without claiming unrestricted global satisfiability. diff --git a/crates/analysis_engine/src/lib.rs b/crates/analysis_engine/src/lib.rs index 8fe96cd76..d0d544ba4 100644 --- a/crates/analysis_engine/src/lib.rs +++ b/crates/analysis_engine/src/lib.rs @@ -62,11 +62,11 @@ pub use topic_lineage_artifact::{ }; /// Durable scientific-acceptance validation-run contracts. pub use validation_run::{ - MAX_RECOVERY_VECTOR_LEN, RecoveryObservation, SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE, - SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION, ScientificAcceptanceEvidence, VALIDATION_BACKEND, - VALIDATION_CPU_F64_MODEL, VALIDATION_PRECISION, VALIDATION_RUN_ID_HEX_LEN, - VALIDATION_RUN_ID_PREFIX, ValidationRunReceipt, WILSON_Z, complete_validation_run, - submit_validation_run, + MAX_RECOVERY_VECTOR_LEN, MAX_SE_GATE_K, RecoveryObservation, + SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE, SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION, + ScientificAcceptanceEvidence, VALIDATION_BACKEND, VALIDATION_CPU_F64_MODEL, + VALIDATION_PRECISION, VALIDATION_RUN_ID_HEX_LEN, VALIDATION_RUN_ID_PREFIX, + ValidationRunReceipt, WILSON_Z, complete_validation_run, submit_validation_run, }; /// Versioned artifact schema emitted by this engine. diff --git a/crates/analysis_engine/src/validation_run.rs b/crates/analysis_engine/src/validation_run.rs index c91d8f41f..2a31ae445 100644 --- a/crates/analysis_engine/src/validation_run.rs +++ b/crates/analysis_engine/src/validation_run.rs @@ -1,15 +1,16 @@ //! Bind immutable evidence to a durable scientific-acceptance validation run. //! //! This is the first GAP-003A slice: cutoff-safe evidence, tenant workspace, -//! output profile, model, seed, backend, and precision hash to one durable run -//! identity. The accepted receipt carries no scientific metrics. Completion -//! asks `validation_core` for RMSE, bias, coverage, temporal-order accuracy, -//! and an SE-aware gate, then emits `tepp.scientific_acceptance.v1`. Recovery -//! vectors must be stamped with that same run identity; a different run, -//! model, snapshot, seed, tenant, profile, or eligible evidence set fails -//! closed. LLM-authored recovery, non-finite inputs, empty or duplicate -//! evidence, snapshot mismatch, oversized recovery, and cutoff-empty corpora -//! fail closed. Postgres persistence remains GAP-003B. +//! output profile, model, seed, backend, precision, and the SE-gate multiplier +//! hash to one durable run identity. The accepted receipt carries no scientific +//! metrics. Completion asks `validation_core` for RMSE, bias, coverage, +//! temporal-order accuracy, and an SE-aware gate, then emits +//! `tepp.scientific_acceptance.v1`. Recovery vectors must be stamped with that +//! same run identity and the pre-registered multiplier; a different run, model, +//! snapshot, seed, tenant, profile, eligible evidence set, or post-hoc `k` +//! fails closed. LLM-authored recovery, non-finite inputs, empty or duplicate +//! evidence, snapshot mismatch, oversized recovery, oversized `k`, and +//! cutoff-empty corpora fail closed. Postgres persistence remains GAP-003B. use crate::{AnalysisCorpus, AnalysisEngineError, valid_identifier}; use serde::Serialize; @@ -42,33 +43,29 @@ pub const VALIDATION_RUN_ID_HEX_LEN: usize = 32; pub const WILSON_Z: f64 = 1.96; /// Maximum length of one recovery, interval, or event-time vector. pub const MAX_RECOVERY_VECTOR_LEN: usize = 10_000; +/// Largest finite SE-gate multiplier that may be pre-registered on a run. +/// +/// Conventional three-SE gates sit inside this bound. A larger `k` would make +/// `|RMSE| ≤ k · SE(RMSE)` an effectively unlimited post-hoc acceptance rule. +pub const MAX_SE_GATE_K: f64 = 8.0; /// Durable identity of one submitted validation run. Receipts never carry -/// scientific metrics. -#[derive(Clone, Debug, Eq, PartialEq, Serialize)] +/// scientific metrics. Fields are private so callers cannot rewrite the +/// binding or the pre-registered SE-gate multiplier after submit. +#[derive(Clone, Debug, PartialEq, Serialize)] pub struct ValidationRunReceipt { - /// Hash-stable run identity `tepp-validation-{32 hex}`. - pub run_id: String, - /// Lowercase SHA-256 of the canonical binding. - pub binding_sha256: String, - /// Tenant workspace that owns the run. - pub tenant_workspace_id: String, - /// Immutable source snapshot identity. - pub snapshot_id: String, - /// Historical cutoff applied to availability. - pub knowledge_cutoff: String, - /// Bound model identity. - pub model: String, - /// Bound numeric seed. - pub seed: u64, - /// Bound compute backend. - pub backend: String, - /// Bound numeric precision. - pub precision: String, - /// Requested output profile. - pub output_profile: String, - /// Number of cutoff-eligible evidence identities. - pub eligible_evidence_count: u64, + run_id: String, + binding_sha256: String, + tenant_workspace_id: String, + snapshot_id: String, + knowledge_cutoff: String, + model: String, + seed: u64, + backend: String, + precision: String, + output_profile: String, + eligible_evidence_count: u64, + se_gate_k: f64, } impl ValidationRunReceipt { @@ -148,9 +145,15 @@ impl ValidationRunReceipt { self.eligible_evidence_count } + /// Return the pre-registered SE-gate multiplier. + #[must_use] + pub const fn se_gate_k(&self) -> f64 { + self.se_gate_k + } + fn identity_record(&self) -> String { format!( - "{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}", + "{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{:016x}", self.run_id, self.binding_sha256, self.tenant_workspace_id, @@ -161,7 +164,8 @@ impl ValidationRunReceipt { self.backend, self.precision, self.output_profile, - self.eligible_evidence_count + self.eligible_evidence_count, + self.se_gate_k.to_bits() ) } } @@ -185,18 +189,19 @@ pub struct RecoveryObservation { impl RecoveryObservation { /// Construct a recovery observation stamped to one submitted receipt. /// - /// LLM authorship is recorded here and refused at completion. Non-finite - /// `se_gate_k`, a negative multiplier, or an oversized vector fail closed - /// immediately. Completing a different run, model, snapshot, seed, tenant, - /// profile, or eligible evidence set fails later as a binding mismatch. + /// LLM authorship is recorded here and refused at completion. The SE-gate + /// multiplier must equal the pre-registered receipt value; a post-hoc `k` + /// is a binding mismatch. Empty, length-mismatched, or oversized vectors + /// fail closed immediately. /// /// # Errors /// /// Returns [`AnalysisEngineError::InvalidEvidence`] for an empty, oversized, - /// or control-bearing study label, + /// or control-bearing study label or for empty/mismatched vectors, /// [`AnalysisEngineError::LimitExceeded`] when any vector exceeds - /// [`MAX_RECOVERY_VECTOR_LEN`], and - /// [`AnalysisEngineError::Validation`] for a non-finite or negative SE + /// [`MAX_RECOVERY_VECTOR_LEN`], [`AnalysisEngineError::BindingMismatch`] + /// when `se_gate_k` differs from the receipt, and + /// [`AnalysisEngineError::Validation`] for a non-finite or out-of-policy SE /// multiplier. #[allow(clippy::too_many_arguments)] pub fn new( @@ -215,28 +220,24 @@ impl RecoveryObservation { if !valid_identifier(&study_label) { return Err(AnalysisEngineError::InvalidEvidence); } - if [ + let lengths = [ truth.len(), recovered.len(), interval_lower.len(), interval_upper.len(), truth_times.len(), recovered_times.len(), - ] - .into_iter() - .any(|len| len > MAX_RECOVERY_VECTOR_LEN) - { + ]; + if lengths.iter().any(|len| *len > MAX_RECOVERY_VECTOR_LEN) { return Err(AnalysisEngineError::LimitExceeded); } - if !se_gate_k.is_finite() { - return Err(AnalysisEngineError::Validation( - ValidationError::InvalidInput, - )); + let n = truth.len(); + if n == 0 || lengths.iter().any(|len| *len != n) { + return Err(AnalysisEngineError::InvalidEvidence); } - if se_gate_k < 0.0 { - return Err(AnalysisEngineError::Validation( - ValidationError::InvalidConfiguration, - )); + let se_gate_k = require_se_gate_k(se_gate_k)?; + if se_gate_k.to_bits() != receipt.se_gate_k.to_bits() { + return Err(AnalysisEngineError::BindingMismatch); } Ok(Self { run_id: receipt.run_id.clone(), @@ -486,6 +487,7 @@ struct CanonicalBinding { backend: String, precision: String, output_profile: String, + se_gate_k: f64, eligible_ids: Vec, } @@ -516,6 +518,7 @@ impl CanonicalBinding { let _ = writeln!(canonical, "backend={}", self.backend); let _ = writeln!(canonical, "precision={}", self.precision); let _ = writeln!(canonical, "profile={}", self.output_profile); + let _ = writeln!(canonical, "se_gate_k={:016x}", self.se_gate_k.to_bits()); for identity in &self.eligible_ids { let _ = writeln!(canonical, "evidence={identity}"); } @@ -524,7 +527,7 @@ impl CanonicalBinding { fn identity_record(&self, digest_hex: &str, run_id: &str) -> String { format!( - "{run_id}\n{digest_hex}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}", + "{run_id}\n{digest_hex}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{:016x}", self.tenant_workspace_id, self.snapshot_id, self.knowledge_cutoff, @@ -533,7 +536,8 @@ impl CanonicalBinding { self.backend, self.precision, self.output_profile, - self.eligible_count() + self.eligible_count(), + self.se_gate_k.to_bits() ) } } @@ -546,14 +550,15 @@ impl CanonicalBinding { /// # Errors /// /// Returns a fail-closed engine error for an invalid request, wrong output -/// profile or model, snapshot mismatch, duplicate or empty evidence, or a -/// cutoff that admits no evidence. +/// profile or model, an out-of-policy SE-gate multiplier, snapshot mismatch, +/// duplicate or empty evidence, or a cutoff that admits no evidence. pub fn submit_validation_run( request: &AnalysisRunRequest, corpus: &AnalysisCorpus, seed: u64, + se_gate_k: f64, ) -> Result { - let binding = bind_validation_run(request, corpus, seed)?; + let binding = bind_validation_run(request, corpus, seed, se_gate_k)?; let binding_sha256 = binding.digest_hex(); let run_id = CanonicalBinding::run_id(&binding_sha256); let eligible_evidence_count = binding.eligible_count(); @@ -569,6 +574,7 @@ pub fn submit_validation_run( precision: binding.precision, output_profile: binding.output_profile, eligible_evidence_count, + se_gate_k: binding.se_gate_k, }) } @@ -591,11 +597,12 @@ pub fn complete_validation_run( } if observation.run_id != receipt.run_id || observation.binding_sha256 != receipt.binding_sha256 + || observation.se_gate_k.to_bits() != receipt.se_gate_k.to_bits() || receipt.output_profile != SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE { return Err(AnalysisEngineError::BindingMismatch); } - let binding = bind_validation_run(request, corpus, receipt.seed)?; + let binding = bind_validation_run(request, corpus, receipt.seed, receipt.se_gate_k)?; let binding_sha256 = binding.digest_hex(); let run_id = CanonicalBinding::run_id(&binding_sha256); if receipt.identity_record() != binding.identity_record(&binding_sha256, &run_id) { @@ -606,7 +613,7 @@ pub fn complete_validation_run( report.rmse, 0.0, report.rmse_standard_error, - observation.se_gate_k, + receipt.se_gate_k, )?; let eligible_evidence_count = binding.eligible_count(); let evidence = ScientificAcceptanceEvidence { @@ -624,19 +631,35 @@ pub fn complete_validation_run( output_profile: SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE.to_owned(), eligible_evidence_count, se_gate_accepted, - se_gate_k: observation.se_gate_k, + se_gate_k: receipt.se_gate_k, report, }; evidence.to_json()?; Ok(evidence) } +fn require_se_gate_k(se_gate_k: f64) -> Result { + if !se_gate_k.is_finite() { + return Err(AnalysisEngineError::Validation( + ValidationError::InvalidInput, + )); + } + if se_gate_k < 0.0 || se_gate_k > MAX_SE_GATE_K { + return Err(AnalysisEngineError::Validation( + ValidationError::InvalidConfiguration, + )); + } + Ok(if se_gate_k == 0.0 { 0.0 } else { se_gate_k }) +} + fn bind_validation_run( request: &AnalysisRunRequest, corpus: &AnalysisCorpus, seed: u64, + se_gate_k: f64, ) -> Result { request.to_json()?; + let se_gate_k = require_se_gate_k(se_gate_k)?; let requested = format!( "{}\n{}", request.output_profile, request.model_contract_version @@ -675,6 +698,7 @@ fn bind_validation_run( backend: VALIDATION_BACKEND.to_owned(), precision: VALIDATION_PRECISION.to_owned(), output_profile: SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE.to_owned(), + se_gate_k, eligible_ids: eligible.into_iter().collect(), }) } @@ -734,10 +758,10 @@ fn format_hex(digest: impl AsRef<[u8]>) -> String { #[cfg(test)] mod tests { use super::{ - MAX_RECOVERY_VECTOR_LEN, RecoveryObservation, SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE, - SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION, VALIDATION_BACKEND, VALIDATION_CPU_F64_MODEL, - VALIDATION_PRECISION, VALIDATION_RUN_ID_PREFIX, WILSON_Z, complete_validation_run, - submit_validation_run, + MAX_RECOVERY_VECTOR_LEN, MAX_SE_GATE_K, RecoveryObservation, + SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE, SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION, + VALIDATION_BACKEND, VALIDATION_CPU_F64_MODEL, VALIDATION_PRECISION, + VALIDATION_RUN_ID_PREFIX, WILSON_Z, complete_validation_run, submit_validation_run, }; use crate::{ AnalysisCorpus, AnalysisEngineError, AnalysisEvidenceUnit, MAX_ANALYSIS_IDENTIFIER_BYTES, @@ -828,8 +852,8 @@ mod tests { unit("evidence-a", "2026-07-10T00:00:00Z"), unit("evidence-b", "2026-07-15T00:00:00Z"), ]); - let receipt_a = submit_validation_run(&request(), &first, 7).expect("submit a"); - let receipt_b = submit_validation_run(&request(), &second, 7).expect("submit b"); + let receipt_a = submit_validation_run(&request(), &first, 7, 3.0).expect("submit a"); + let receipt_b = submit_validation_run(&request(), &second, 7, 3.0).expect("submit b"); assert_eq!(receipt_a.run_id(), receipt_b.run_id()); assert_eq!(receipt_a.binding_sha256(), receipt_b.binding_sha256()); assert!(receipt_a.run_id().starts_with(VALIDATION_RUN_ID_PREFIX)); @@ -849,16 +873,21 @@ mod tests { receipt_a.output_profile(), SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE ); + assert!((receipt_a.se_gate_k() - 3.0).abs() < f64::EPSILON); let json = receipt_a.to_json().expect("json"); assert!(!json.contains("rmse")); assert!(!json.contains("bias")); assert!(!json.contains("coverage")); - let other_seed = submit_validation_run(&request(), &first, 8).expect("seed"); + let other_seed = submit_validation_run(&request(), &first, 8, 3.0).expect("seed"); assert_ne!(receipt_a.run_id(), other_seed.run_id()); let mut other_tenant = request(); other_tenant.tenant_workspace_id = "tenant-workspace-2".into(); - let other_tenant_receipt = submit_validation_run(&other_tenant, &first, 7).expect("tenant"); + let other_tenant_receipt = + submit_validation_run(&other_tenant, &first, 7, 3.0).expect("tenant"); assert_ne!(receipt_a.run_id(), other_tenant_receipt.run_id()); + let other_k = submit_validation_run(&request(), &first, 7, 4.0).expect("other k"); + assert_ne!(receipt_a.run_id(), other_k.run_id()); + assert!((other_k.se_gate_k() - 4.0).abs() < f64::EPSILON); } #[test] @@ -867,7 +896,7 @@ mod tests { unit("evidence-a", "2026-07-10T00:00:00Z"), unit("evidence-b", "2026-07-15T00:00:00Z"), ]); - let receipt = submit_validation_run(&request(), &offered, 11).expect("submit"); + let receipt = submit_validation_run(&request(), &offered, 11, 3.0).expect("submit"); let observation = recovery( &receipt, vec![0.70, 0.55, 0.40, -0.20, 0.85], @@ -919,7 +948,7 @@ mod tests { &receipt, vec![0.0, 1.0, 2.0, 3.0, 4.0], vec![10.0, 11.0, 12.0, 13.0, 14.0], - 1.0, + 3.0, ); let refused = complete_validation_run(&receipt, &request(), &offered, &rejected).expect("refused"); @@ -933,13 +962,13 @@ mod tests { let mut wrong_profile = request(); wrong_profile.output_profile = "validation-report".into(); assert_eq!( - submit_validation_run(&wrong_profile, &offered, 1), + submit_validation_run(&wrong_profile, &offered, 1, 3.0), Err(AnalysisEngineError::InvalidValidationProfile) ); let mut wrong_model = request(); wrong_model.model_contract_version = "temporal-evidence-v1".into(); assert_eq!( - submit_validation_run(&wrong_model, &offered, 1), + submit_validation_run(&wrong_model, &offered, 1, 3.0), Err(AnalysisEngineError::InvalidValidationProfile) ); let mismatched = AnalysisCorpus::new( @@ -948,7 +977,7 @@ mod tests { ) .expect("other"); assert_eq!( - submit_validation_run(&request(), &mismatched, 1), + submit_validation_run(&request(), &mismatched, 1, 3.0), Err(AnalysisEngineError::SnapshotMismatch) ); let duplicate = corpus(vec![ @@ -956,29 +985,29 @@ mod tests { unit("same", "2026-07-11T00:00:00Z"), ]); assert_eq!( - submit_validation_run(&request(), &duplicate, 1), + submit_validation_run(&request(), &duplicate, 1, 3.0), Err(AnalysisEngineError::DuplicateEvidence) ); let empty = AnalysisCorpus::new("snapshot-1", Vec::new()).expect("empty"); assert_eq!( - submit_validation_run(&request(), &empty, 1), + submit_validation_run(&request(), &empty, 1, 3.0), Err(AnalysisEngineError::InvalidEvidence) ); let late = corpus(vec![unit("late", "2026-08-02T00:00:00Z")]); assert_eq!( - submit_validation_run(&request(), &late, 1), + submit_validation_run(&request(), &late, 1, 3.0), Err(AnalysisEngineError::NoEligibleEvidence) ); let mut invalid_request = request(); invalid_request.idempotency_key.clear(); assert!(matches!( - submit_validation_run(&invalid_request, &offered, 1), + submit_validation_run(&invalid_request, &offered, 1, 3.0), Err(AnalysisEngineError::Api(_)) )); let mut invalid_cutoff = request(); invalid_cutoff.knowledge_cutoff = "not-a-time".into(); assert!(matches!( - submit_validation_run(&invalid_cutoff, &offered, 1), + submit_validation_run(&invalid_cutoff, &offered, 1, 3.0), Err(AnalysisEngineError::Api(_)) )); } @@ -989,7 +1018,7 @@ mod tests { unit("evidence-a", "2026-07-10T00:00:00Z"), unit("evidence-b", "2026-07-15T00:00:00Z"), ]); - let receipt = submit_validation_run(&request(), &offered, 3).expect("submit"); + let receipt = submit_validation_run(&request(), &offered, 3, 3.0).expect("submit"); let llm = RecoveryObservation::new( &receipt, "llm-study", @@ -1021,6 +1050,12 @@ mod tests { complete_validation_run(&profile_tampered, &request(), &offered, &observation), Err(AnalysisEngineError::BindingMismatch) ); + let mut k_tampered = observation.clone(); + k_tampered.se_gate_k = 8.0; + assert_eq!( + complete_validation_run(&receipt, &request(), &offered, &k_tampered), + Err(AnalysisEngineError::BindingMismatch) + ); } #[test] @@ -1030,22 +1065,23 @@ mod tests { unit("evidence-a", "2026-07-10T00:00:00Z"), unit("evidence-b", "2026-07-15T00:00:00Z"), ]); - let receipt_a = submit_validation_run(&request(), &first, 3).expect("a"); - let receipt_b = submit_validation_run(&request(), &second, 3).expect("b"); + let receipt_a = submit_validation_run(&request(), &first, 3, 3.0).expect("a"); + let receipt_b = submit_validation_run(&request(), &second, 3, 3.0).expect("b"); assert_ne!(receipt_a.run_id(), receipt_b.run_id()); let foreign = recovery(&receipt_a, vec![1.0, 2.0, 3.0], vec![1.0, 2.0, 3.0], 3.0); assert_eq!( complete_validation_run(&receipt_b, &request(), &second, &foreign), Err(AnalysisEngineError::BindingMismatch) ); - let other_seed = submit_validation_run(&request(), &first, 9).expect("seed"); + let other_seed = submit_validation_run(&request(), &first, 9, 3.0).expect("seed"); assert_eq!( complete_validation_run(&other_seed, &request(), &first, &foreign), Err(AnalysisEngineError::BindingMismatch) ); let mut other_tenant = request(); other_tenant.tenant_workspace_id = "tenant-workspace-other".into(); - let other_tenant_receipt = submit_validation_run(&other_tenant, &first, 3).expect("tenant"); + let other_tenant_receipt = + submit_validation_run(&other_tenant, &first, 3, 3.0).expect("tenant"); assert_eq!( complete_validation_run(&other_tenant_receipt, &other_tenant, &first, &foreign), Err(AnalysisEngineError::BindingMismatch) @@ -1058,7 +1094,7 @@ mod tests { unit("evidence-a", "2026-07-10T00:00:00Z"), unit("evidence-b", "2026-07-15T00:00:00Z"), ]); - let receipt = submit_validation_run(&request(), &offered, 3).expect("submit"); + let receipt = submit_validation_run(&request(), &offered, 3, 3.0).expect("submit"); let nan = RecoveryObservation::new( &receipt, "nan-study", @@ -1122,4 +1158,66 @@ mod tests { let converted: AnalysisEngineError = ValidationError::InvalidInput.into(); assert_eq!(converted.to_string(), "invalid validation input"); } + + #[test] + fn se_gate_k_is_pre_registered_and_empty_vectors_fail_closed() { + let offered = corpus(vec![ + unit("evidence-a", "2026-07-10T00:00:00Z"), + unit("evidence-b", "2026-07-15T00:00:00Z"), + ]); + let receipt = submit_validation_run(&request(), &offered, 3, 3.0).expect("submit"); + assert_eq!( + two_point(&receipt, "huge-k", MAX_SE_GATE_K + 0.01, false), + Err(AnalysisEngineError::Validation( + ValidationError::InvalidConfiguration + )) + ); + assert_eq!( + two_point(&receipt, "post-hoc-k", 4.0, false), + Err(AnalysisEngineError::BindingMismatch) + ); + assert_eq!( + submit_validation_run(&request(), &offered, 3, MAX_SE_GATE_K + 0.01), + Err(AnalysisEngineError::Validation( + ValidationError::InvalidConfiguration + )) + ); + assert_eq!( + submit_validation_run(&request(), &offered, 3, f64::NAN), + Err(AnalysisEngineError::Validation( + ValidationError::InvalidInput + )) + ); + assert!(submit_validation_run(&request(), &offered, 3, MAX_SE_GATE_K).is_ok()); + assert_eq!( + RecoveryObservation::new( + &receipt, + "empty", + Vec::new(), + Vec::new(), + Vec::new(), + Vec::new(), + Vec::new(), + Vec::new(), + 3.0, + false, + ), + Err(AnalysisEngineError::InvalidEvidence) + ); + assert_eq!( + RecoveryObservation::new( + &receipt, + "mismatch", + vec![1.0, 2.0], + vec![1.0], + vec![0.0, 1.0], + vec![2.0, 3.0], + vec![1.0, 2.0], + vec![1.0, 2.0], + 3.0, + false, + ), + Err(AnalysisEngineError::InvalidEvidence) + ); + } } diff --git a/crates/analysis_engine/tests/validation_run_contract.rs b/crates/analysis_engine/tests/validation_run_contract.rs index 82d97896d..f7a7bc7be 100644 --- a/crates/analysis_engine/tests/validation_run_contract.rs +++ b/crates/analysis_engine/tests/validation_run_contract.rs @@ -41,8 +41,9 @@ fn durable_run_binds_cutoff_eligible_evidence_and_emits_acceptance_evidence() { ], ) .expect("corpus"); - let receipt = submit_validation_run(&request(), &corpus, 42).expect("submit"); + let receipt = submit_validation_run(&request(), &corpus, 42, 3.0).expect("submit"); assert_eq!(receipt.eligible_evidence_count(), 2); + assert!((receipt.se_gate_k() - 3.0).abs() < f64::EPSILON); assert!(!receipt.to_json().expect("json").contains("rmse")); let observation = RecoveryObservation::new( &receipt, @@ -77,7 +78,7 @@ fn llm_authored_recovery_and_cutoff_empty_corpora_fail_closed() { ) .expect("corpus"); assert_eq!( - submit_validation_run(&request(), &corpus, 1), + submit_validation_run(&request(), &corpus, 1, 3.0), Err(AnalysisEngineError::NoEligibleEvidence) ); let eligible = AnalysisCorpus::new( @@ -85,7 +86,7 @@ fn llm_authored_recovery_and_cutoff_empty_corpora_fail_closed() { vec![unit("evidence-1", "2026-07-10T00:00:00Z")], ) .expect("eligible"); - let receipt = submit_validation_run(&request(), &eligible, 1).expect("submit"); + let receipt = submit_validation_run(&request(), &eligible, 1, 3.0).expect("submit"); let llm = RecoveryObservation::new( &receipt, "llm-contract", @@ -120,8 +121,8 @@ fn recovery_from_a_different_run_or_tenant_fails_closed() { ], ) .expect("second"); - let receipt_a = submit_validation_run(&request(), &first, 1).expect("a"); - let receipt_b = submit_validation_run(&request(), &second, 1).expect("b"); + let receipt_a = submit_validation_run(&request(), &first, 1, 3.0).expect("a"); + let receipt_b = submit_validation_run(&request(), &second, 1, 3.0).expect("b"); let foreign = RecoveryObservation::new( &receipt_a, "foreign-recovery", @@ -141,10 +142,27 @@ fn recovery_from_a_different_run_or_tenant_fails_closed() { ); let mut other_tenant = request(); other_tenant.tenant_workspace_id = "tenant-workspace-other".into(); - let other_receipt = submit_validation_run(&other_tenant, &first, 1).expect("tenant"); + let other_receipt = submit_validation_run(&other_tenant, &first, 1, 3.0).expect("tenant"); assert_ne!(receipt_a.run_id(), other_receipt.run_id()); assert_eq!( complete_validation_run(&other_receipt, &other_tenant, &first, &foreign), Err(AnalysisEngineError::BindingMismatch) ); + let other_k = submit_validation_run(&request(), &first, 1, 4.0).expect("other k"); + assert_ne!(receipt_a.run_id(), other_k.run_id()); + assert_eq!( + RecoveryObservation::new( + &receipt_a, + "post-hoc-k", + vec![1.0, 2.0, 3.0], + vec![1.0, 2.0, 3.0], + vec![0.5, 1.5, 2.5], + vec![1.5, 2.5, 3.5], + vec![1.0, 2.0, 3.0], + vec![1.0, 2.0, 3.0], + 4.0, + false, + ), + Err(AnalysisEngineError::BindingMismatch) + ); } diff --git a/docs/TRACEABILITY.md b/docs/TRACEABILITY.md index 6768092ec..2bd5f446b 100644 --- a/docs/TRACEABILITY.md +++ b/docs/TRACEABILITY.md @@ -50,7 +50,7 @@ The full APA 7th standards/literature register remains `docs/research/standards- | leakage-safe availability/cutoff snapshots | PRD; ADR 0002/0013 | `corpus_split` on protected main | implemented-main | | Unicode NFC/NFD canonical identity for split leakage | ADR 0004/0008/0013; UAX #15 | `corpus_split` canonical-equivalence links on active PR #59 | active-PR | | recovery metrics (RMSE, bias, coverage, graph, temporal order, Monte Carlo SE gates) | PRD; Test Strategy; ADR 0007/0014 | `validation_core` on protected main (PR #19); SE-aware Monte Carlo gates included | implemented-main | -| durable validation-run scientific acceptance evidence | ADR 0026; ADR 0002/0008/0014; GAP-003A; National Academies (2019); Wasserstein & Lazar (2016); Wilson (1927) | `analysis_engine` `submit_validation_run` / `complete_validation_run` on this PR: hash-stable cutoff-safe tenant/profile binding, metric-free receipt, recovery stamped to that `run_id`, `tepp.scientific_acceptance.v1` RMSE/bias/coverage/temporal-order/SE-aware evidence with recovery digest; LLM/NaN/empty/duplicate/snapshot/foreign-run/tampered-profile/oversized/cutoff-empty fail closed; not implemented-main; Postgres persistence remains GAP-003B | active-PR | +| durable validation-run scientific acceptance evidence | ADR 0026; ADR 0002/0008/0014; GAP-003A; National Academies (2019); Wasserstein & Lazar (2016); Wilson (1927) | `analysis_engine` `submit_validation_run` / `complete_validation_run` on this PR: hash-stable cutoff-safe tenant/profile/`k` binding, metric-free private-field receipt, recovery stamped to that `run_id` with pre-registered `k` (`MAX_SE_GATE_K` = 8), `tepp.scientific_acceptance.v1` RMSE/bias/coverage/temporal-order/SE-aware evidence with recovery digest; LLM/NaN/empty/duplicate/snapshot/foreign-run/post-hoc-k/tampered-profile/oversized/cutoff-empty fail closed; not implemented-main; Postgres persistence remains GAP-003B | active-PR | | PostgreSQL bitemporal/lineage persistence | ADR 0013; Architecture/ERD | `persistence_postgres` migration contracts, in-memory adapters, live SQL session/document SQL port, tenant RLS (`0002` + session GUC/role helpers), `DATABASE_URL` SQLx gate, optional session-affine `live-sqlx` `PgPool` driver, exact-head live PostgreSQL CI with isolation proof, append-only immutability triggers (`0004`), temporal interval ordering CHECKs (`0005`), typed membership assignment (`0006` implemented-main), event-relation/mention/instance SQL (#37–#39 implemented-main), source-artifact SQL (#40 implemented-main), audit-event SQL (#41 implemented-main), concurrent document-write stress (#43 implemented-main), backup/restore integrity revalidation (#44 implemented-main), `revision_order` later-revision system-time ordering implemented-main, entity/project target SQL on PR #131; remaining physical ERD constraints | partial | | known-truth temporal/event simulation manifests | PRD; TRD; Test Strategy | `tepp_simulation` on protected main; recovery metrics in `validation_core` | implemented-main | | versioned service/API contracts and exports | PRD; API contract; ADR 0011/0013 | `tepp_api` analysis-run/export/JSON-LD/GraphML contracts on protected main (PR #21); request-bound terminal result active in PR #157; HTTP service remains accepted-target; the `orchestrator_live` loopback interpretation listener is on this PR | partial | diff --git a/docs/adr/0026-validation-run-scientific-acceptance.md b/docs/adr/0026-validation-run-scientific-acceptance.md index c692eb29a..8e9290a67 100644 --- a/docs/adr/0026-validation-run-scientific-acceptance.md +++ b/docs/adr/0026-validation-run-scientific-acceptance.md @@ -26,19 +26,23 @@ Add a bounded `analysis_engine` validation-run executor: - `submit_validation_run` binds sorted cutoff-eligible evidence identities, tenant workspace, snapshot, knowledge cutoff, model `validation_cpu_f64_v1`, - seed, backend `cpu`, precision `f64`, and output profile - `scientific_acceptance_v1` into a canonical SHA-256 digest. The durable + seed, backend `cpu`, precision `f64`, output profile + `scientific_acceptance_v1`, and the SE-gate multiplier `k` into a canonical + SHA-256 digest. `k` must be finite, non-negative, and at most + `MAX_SE_GATE_K` (8). Changing `k` changes the run identity. The durable `run_id` is `tepp-validation-{32 hex}`. The receipt carries no RMSE, bias, - coverage, or gate fields. + coverage, or gate-result fields. Receipt fields are private. - `complete_validation_run` rebinds the same scientific identity, requires - recovery vectors stamped to that `run_id` and binding digest, refuses - LLM-authored recovery, computes `validation_core` recovery metrics, applies - the SE-aware gate `|RMSE − 0| ≤ k · SE(RMSE)`, and emits + recovery vectors stamped to that `run_id` and binding digest with the same + pre-registered `k`, refuses LLM-authored recovery, computes + `validation_core` recovery metrics, applies the SE-aware gate + `|RMSE − 0| ≤ k · SE(RMSE)` using the submitted `k`, and emits `tepp.scientific_acceptance.v1` under output profile `scientific_acceptance_v1`. The artifact records a SHA-256 of the stamped recovery vectors. Evidence fields are private after completion. - Empty corpora, duplicate evidence identities, snapshot mismatch, invalid - profiles, non-finite inputs, oversized recovery vectors, a different run / + profiles, non-finite inputs, oversized recovery vectors, empty or + length-mismatched recovery, a post-hoc or oversized `k`, a different run / tenant / seed / eligible evidence set, a tampered output profile, and cutoff-empty eligibility fail closed. - A computed recovery that fails the SE-aware gate still emits evidence with @@ -72,10 +76,11 @@ become scientific authority. ## Verification The stacked PR includes unit and integration tests for hash-stable identity, -cutoff exclusion, metric-free receipts, SE-aware accept and refuse, recovery -stamped to a foreign run or tenant, a tampered output profile, oversized -vectors, and fail-closed LLM, NaN, empty, duplicate, snapshot, profile, and -cutoff-empty paths. Run: +pre-registered SE-gate `k`, cutoff exclusion, metric-free private receipts, +SE-aware accept and refuse, recovery stamped to a foreign run or tenant, a +post-hoc `k`, a tampered output profile, oversized or empty vectors, and +fail-closed LLM, NaN, empty, duplicate, snapshot, profile, and cutoff-empty +paths. Run: ```text cargo fmt --all -- --check diff --git a/docs/research/validation-run-scientific-acceptance.md b/docs/research/validation-run-scientific-acceptance.md index 287c0bfdc..2ce820517 100644 --- a/docs/research/validation-run-scientific-acceptance.md +++ b/docs/research/validation-run-scientific-acceptance.md @@ -4,11 +4,11 @@ This note doctors the first GAP-003A slice in `analysis_engine`: -1. immutable evidence, tenant workspace, knowledge cutoff, model, seed, backend, precision, and output profile bind to one hash-stable validation run; -2. the accepted receipt carries no scientific metrics; -3. completion emits `tepp.scientific_acceptance.v1` with RMSE, bias, interval coverage (Wilson bounds), temporal-order accuracy, an SE-aware gate, and a SHA-256 of the stamped recovery vectors; -4. recovery vectors must be constructed against that receipt; a different run, tenant, seed, snapshot, profile, or eligible evidence set fails closed; -5. LLM-authored recovery, non-finite inputs, empty or duplicate evidence, snapshot mismatch, oversized recovery, and cutoff-empty corpora fail closed. +1. immutable evidence, tenant workspace, knowledge cutoff, model, seed, backend, precision, output profile, and the SE-gate multiplier `k` bind to one hash-stable validation run; +2. the accepted receipt carries no scientific metrics and cannot be rewritten after submit; +3. completion emits `tepp.scientific_acceptance.v1` with RMSE, bias, interval coverage (Wilson bounds), temporal-order accuracy, an SE-aware gate using that pre-registered `k`, and a SHA-256 of the stamped recovery vectors; +4. recovery vectors must be constructed against that receipt with the same `k`; a different run, tenant, seed, snapshot, profile, eligible evidence set, or post-hoc `k` fails closed; +5. LLM-authored recovery, non-finite inputs, empty or duplicate evidence, snapshot mismatch, oversized recovery, oversized `k` (`MAX_SE_GATE_K` = 8), and cutoff-empty corpora fail closed. Postgres persistence, restart/recovery, and Compose execution remain GAP-003B. This slice is not implemented-main. @@ -22,15 +22,17 @@ Wilson, E. B. (1927). Probable inference, the law of succession, and statistical ## Application -The National Academies (2019) separate computational reproducibility (same binding, same digest) from a scientific claim that recovery is correct. Wasserstein and Lazar (2016) refuse to treat a passing threshold as automatic scientific authority, so a failed SE-aware gate still reports metrics with `se_gate_accepted = false` rather than inventing a passing claim. Wilson (1927) supplies the coverage interval bounds already implemented in `validation_core`. TEPP therefore binds cutoff-safe evidence and tenant/profile identity before any metric is computed, stamps recovery vectors to that binding, refuses LLM-authored recovery, and reports RMSE, bias, coverage, temporal order, and the SE-aware gate as operator-usable evidence (National Academies of Sciences, Engineering, and Medicine, 2019; Wasserstein & Lazar, 2016; Wilson, 1927). Meredith (1993) remains unread (Unpaywall/OpenAlex 2026-08-31T07:45Z: `is_oa: false`, 0 locations). Mislevy (1991, *Psychometrika, 56*, 177–196) remains unread on the same terms (DOI `10.1007/bf02294457`). +The National Academies (2019) separate computational reproducibility (same binding, same digest) from a scientific claim that recovery is correct. Wasserstein and Lazar (2016) refuse to treat a passing threshold as automatic scientific authority and refuse post-hoc threshold shopping, so `k` is pre-registered in the run binding and a failed SE-aware gate still reports metrics with `se_gate_accepted = false` rather than inventing a passing claim. Wilson (1927) supplies the coverage interval bounds already implemented in `validation_core`. TEPP therefore binds cutoff-safe evidence, tenant/profile identity, and `k` before any metric is computed, stamps recovery vectors to that binding, refuses LLM-authored or post-hoc-`k` recovery, and reports RMSE, bias, coverage, temporal order, and the SE-aware gate as operator-usable evidence (National Academies of Sciences, Engineering, and Medicine, 2019; Wasserstein & Lazar, 2016; Wilson, 1927). Meredith (1993) remains unread (Unpaywall/OpenAlex 2026-08-31T07:45Z: `is_oa: false`, 0 locations). Mislevy (1991, *Psychometrika, 56*, 177–196) remains unread on the same terms (DOI `10.1007/bf02294457`). ## Verification - identical eligible evidence in any order and with extra post-cutoff units yields the same `tepp-validation-{32 hex}` identity; +- distinct SE-gate multipliers yield distinct run identities; `k > 8` fails closed; - receipts serialize without RMSE, bias, or coverage fields; - known-truth recovery emits `tepp.scientific_acceptance.v1` with a digest-stable JSON body; - a large residual vector remains operator-readable with `se_gate_accepted = false`; -- recovery stamped to a different run, tenant, or seed returns `BindingMismatch`; +- recovery stamped to a different run, tenant, seed, or post-hoc `k` returns `BindingMismatch`; - a tampered receipt `output_profile` returns `BindingMismatch`; - an oversized recovery vector returns `LimitExceeded`; +- empty or length-mismatched recovery vectors return `InvalidEvidence`; - LLM authorship, NaN recovery, empty corpora, duplicates, snapshot mismatch, wrong profile/model, and cutoff-empty eligibility return dedicated fail-closed errors. From df33bfa3e61ae4de3dbfae16df0deac12d2f4003 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Mon, 31 Aug 2026 08:06:54 +0000 Subject: [PATCH 5/5] docs(gap): record GAP-003A pre-registered k head on #356 Point the GAP-003A Current head SHA column at exact PR #356 7916603. Pre-registered SE-gate k; not implemented-main. --- docs/product-technical-gap-baseline.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index ee650024e..c3e5a0780 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -248,7 +248,7 @@ the coordinated visual workspace, or a supported multi-tenant release. |---|---|---|---|---|---|---|---| | GAP-001 | Submission produces a durable accepted receipt, and the deterministic terminal-result lifecycle is now implemented-main. | `implemented-main` | closed on protected main | `340087494b0a` lineage merged through [PR #157](https://github.com/ContextualWisdomLab/TEPP/pull/157) (merged 2026-08-25T02:53Z); [#156](https://github.com/ContextualWisdomLab/TEPP/issues/156) CLOSED | — | — | Exact request/result/snapshot/cutoff/model/profile binding, typed terminal failures, deterministic retrieval, and cutoff-safe execution are protected-main behavior as of the #157 merge. | | GAP-002 | LineageWeave and other modular consumers can rely on the complete protected-main HTTP evidence/result boundary. | `partial` | consumer hardening remains | Terminal-result lifecycle implemented-main via #157; versioned API contract intact | [#156](https://github.com/ContextualWisdomLab/TEPP/issues/156) (closed) / [PR #155](https://github.com/ContextualWisdomLab/TEPP/pull/155) (merged) | — | Remaining work is consumer-side adoption evidence and any versioned-contract drift discovered during integration; core boundary is no longer the gap it was. | -| GAP-003A | Immutable evidence cannot yet be submitted to a durable validation run that produces operator-usable scientific acceptance evidence. | `accepted-target` | product-completion | `e65cd66` (validation metrics are library-level only) | [#166](https://github.com/ContextualWisdomLab/TEPP/issues/166) | [#356](https://github.com/ContextualWisdomLab/TEPP/pull/356) `f8a4dad` (`feat/validation-run-scientific-acceptance`; library-level binding; not implemented-main; not Compose persistence) | Compose/CLI/API execution must bind immutable evidence, cutoffs, model configuration, validation metrics, and reproducibility manifests to one idempotent run. | +| GAP-003A | Immutable evidence cannot yet be submitted to a durable validation run that produces operator-usable scientific acceptance evidence. | `accepted-target` | product-completion | `e65cd66` (validation metrics are library-level only) | [#166](https://github.com/ContextualWisdomLab/TEPP/issues/166) | [#356](https://github.com/ContextualWisdomLab/TEPP/pull/356) `7916603` (`feat/validation-run-scientific-acceptance`; pre-registered SE-gate `k`; not implemented-main; not Compose persistence) | Compose/CLI/API execution must bind immutable evidence, cutoffs, model configuration, validation metrics, and reproducibility manifests to one idempotent run. | | GAP-003B | Scientific result artifacts cannot yet be persisted, restarted, and recovered as one supported operator workflow. | `accepted-target` | product-completion | `e65cd66` (persistence contracts lack E2E recovery) | [#166](https://github.com/ContextualWisdomLab/TEPP/issues/166) | `—` (issue program; no current implementation PR) | Durable storage, migration/rollback, restart/recovery, artifact digest verification, and terminal retrieval must pass against a real Compose deployment. | | GAP-003C | The persistence slice classifies concurrent-write SQLSTATEs, but has no measured hot-partition detection, routing, or mitigation for tenant/result workloads. | `accepted-target` | product-completion | `e65cd66` (conflict classification only; no measured partition control) | [#166](https://github.com/ContextualWisdomLab/TEPP/issues/166) | `—` (issue program; no current implementation PR) | A real Compose/PostgreSQL workload identifies hot keys and partition skew, applies bounded tenant/time or result routing without weakening 3NF or temporal authority, and proves conflict rate, latency, recovery, and migration/rollback behavior under load. | | GAP-004 | The central shared-latent temporal/relational topic estimator is absent. | `partial` | product vertical | CPU `f64` TRSL-TM reference estimator with ALR/ILR coordinates and refusal gates is implemented-main (v0.2.0 `topic_measurement`); fitted candidate-`K` scoring present | [#167](https://github.com/ContextualWisdomLab/TEPP/issues/167) | — | GPU, method effects, full Bayesian sampling, and topic birth/split/merge remain. This is not full #167 closure. |