From f8a4dad0844cbd67dd0c8cb97fc9577fb8e6e9e3 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Mon, 31 Aug 2026 07:35:42 +0000 Subject: [PATCH 1/6] feat(analysis): bind validation runs to scientific acceptance evidence GAP-003A first slice for issue #166. submit_validation_run binds cutoff-eligible evidence, snapshot, knowledge cutoff, CPU f64 model, seed, backend, and precision to a hash-stable tepp-validation-{32 hex} receipt that carries no scientific metrics. complete_validation_run emits tepp.scientific_acceptance.v1 through validation_core (RMSE, bias, Wilson coverage, temporal-order accuracy, SE-aware gate). LLM-authored recovery, non-finite inputs, empty or duplicate evidence, snapshot mismatch, and cutoff-empty corpora fail closed. Not implemented-main. Postgres persistence remains GAP-003B. ADR 0026. --- ARCHITECTURE.md | 4 +- CHANGELOG.md | 2 + Cargo.lock | 1 + DOCUMENTATION.md | 2 + crates/analysis_engine/Cargo.toml | 1 + crates/analysis_engine/src/lib.rs | 58 +- crates/analysis_engine/src/validation_run.rs | 891 ++++++++++++++++++ .../tests/validation_run_contract.rs | 103 ++ docs/TRACEABILITY.md | 1 + ...26-validation-run-scientific-acceptance.md | 88 ++ docs/adr/README.md | 1 + docs/product-technical-gap-baseline.md | 2 +- .../validation-run-scientific-acceptance.md | 32 + 13 files changed, 1182 insertions(+), 4 deletions(-) create mode 100644 crates/analysis_engine/src/validation_run.rs create mode 100644 crates/analysis_engine/tests/validation_run_contract.rs create mode 100644 docs/adr/0026-validation-run-scientific-acceptance.md create mode 100644 docs/research/validation-run-scientific-acceptance.md diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index e6dafbcc2..479f84d12 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -70,7 +70,7 @@ boundaries above remain the target modular MSA architecture. | `tepp_simulation` | known-truth temporal/event data generation | | `validation_core` | RMSE, bias, coverage, graph, Monte Carlo, and exact-head claim-promotion metrics | | `tepp_api` | versioned DTO, schema, terminal-result, and export contracts | -| `analysis_engine` | bounded cutoff-safe temporal evidence readiness execution and digest-bound terminal artifacts | +| `analysis_engine` | bounded cutoff-safe temporal evidence readiness execution, digest-bound terminal artifacts, and GAP-003A scientific-acceptance validation runs (`tepp.scientific_acceptance.v1`; not implemented-main) | | `episode_membership` | event-time episode membership containment gate | | `prompt_source` | prompt boilerplate is not unique latent content and not stopword deletion | | `corpus_background` | corpus-background wording is not unique latent content and not stopword deletion | @@ -113,7 +113,7 @@ boundaries above remain the target modular MSA architecture. | `episode_membership` | episode membership cannot escape the episode event-time interval | | `membership_target` | language, episode, template, department, and opportunity-pool targets cannot collapse into entity or project | | `topic_measurement` | logistic-normal ALR/ILR coordinates and the CPU `f64` TRSL-TM reference estimator | -| `analysis_engine` | bounded cutoff-safe temporal evidence readiness execution and digest-bound terminal artifacts | +| `analysis_engine` | bounded cutoff-safe temporal evidence readiness execution, digest-bound terminal artifacts, and GAP-003A scientific-acceptance validation runs (`tepp.scientific_acceptance.v1`; not implemented-main) | | `psychometric_core` | posterior-aware structural input gates, CWC within/between OLS plus the contextual effect, event-time log-rate, unequal-interval discrete-lag remapping, constant-predictor discrete effect, time-varying-predictor discrete effect (Eq. 14), exact scalar discrete process noise (Driver et al., 2017, Eq. 3), lagged latent covariance and unconditional latent variance (Driver et al., 2017, Eq. 3–4), stationary within-subject variance (Driver et al., 2017, Eq. 4 as `Δt → ∞`; `asymDIFFUSION`), trait-plus-state variance (Driver et al., 2017, §4.3 `TRAITVAR`; not process noise), observed-indicator variance and lagged observed covariance (Driver et al., 2017, Eq. 5; Table 2 `MANIFESTVAR` is `Θ`, not `Var(y)`; `MANIFESTTRAITVAR` is not `MANIFESTVAR`; `Θ` does not enter lagged observed covariance; observed-indicator mean is `τ + λ μ`; `MANIFESTMEANS` is not `E(y)`; `CINT` is not `MANIFESTMEANS`; discrete latent mean is `exp(a Δt) μ_0 + (exp(a Δt) − 1)/a κ`; `T0MEANS` is not `μ_t`; `CINT` is not the discrete increment; evolved observed mean is `τ + λ μ_t`; `τ + λ μ_0` is not `E(y_t)`; contemporaneous `TDPREDEFFECT` impulse is `m x`, not `CINT`, not `TIPREDEFFECT`, and not Voelkle Eq. 14; Eq. 5 of that contemporaneous impulse is `τ + λ(μ_t + m x)`, and `τ + λ μ_t` is not that observed mean; time-independent `TIPREDEFFECT` increment is `A^{-1}[e^{A Δt} − I] B z`, not `CINT`, not `M x`, not Voelkle Eq. 14, and not the coefficient `B`; Eq. 5 of that increment is `τ + λ(μ_t + A^{-1}[e^{A Δt} − I] B z)`, and `τ + λ μ_t` is not that observed mean; `τ + λ(μ_t + m x)` is not that observed mean; `τ + λ(μ_t + e^{a(t−u)} m x)` is not that observed mean when `u ≠ t`; within-interval `TDPREDEFFECT` carry is `e^{A(t−u)} M x` for `t0 < u < t`, not the contemporaneous Dirac, not `CINT`, not `TIPREDEFFECT`, and not Voelkle Eq. 14; Eq. 5 of that carry is `τ + λ(μ_t + e^{a(t−u)} m x)`, and `τ + λ μ_t` is not that observed mean; `τ + λ(μ_t + m x)` is not that carried observed mean when `u ≠ t`; first-occasion `T0TIPREDEFFECT` shift is `t0_b z` and Eq. 3 first-summand carry is `e^{A Δt} t0_b z` (`T0TIPREDEFFECT` is not `TIPREDEFFECT` `B`; `t0_b z` is not `A^{-1}[e^{A Δt} − I] B z`; `e^{A Δt} t0_b z` is not `t0_b z`; Eq. 5 of that carry is `τ + λ(μ_t + e^{a Δt} t0_b z)`, and `τ + λ μ_t` is not that observed mean; `τ + λ(μ_t + A^{-1}[e^{A Δt} − I] B z)` is not that observed mean), first-occasion `T0TDPREDEFFECT` shift is `t0_m x0` and Eq. 3 first-summand carry is `e^{A Δt} t0_m x0` (`T0TDPREDEFFECT` is not `TDPREDEFFECT` `M`; `t0_m x0` is not `M x`; `e^{A Δt} t0_m x0` is not `t0_m x0`; `e^{A Δt} t0_m x0` is not `e^{A(t−u)} M x` for `t0 < u < t`; `t0_m x0` is not `t0_b z`; an impulse at `u ≤ t0` that used `M` is already in `η(t0)` as `TDPREDEFFECT`, not as `T0TDPREDEFFECT`; Eq. 5 of that carry is `τ + λ(μ_t + e^{a Δt} t0_m x0)`, and `τ + λ μ_t` is not that observed mean; `τ + λ(μ_t + A^{-1}[e^{A Δt} − I] B z)` is not that observed mean; `τ + λ(μ_t + e^{a Δt} t0_b z)` is not that observed mean; §7.2 level-change `CINT` is `κ = −a m x` with `a < 0` so `−κ / a = m x` (`−a m x` is not the dissipating Dirac, not a free `CINT`, not `TIPREDEFFECT`, and not the extra near-zero-drift latent process also named in §7.2; Eq. 3 of that setting is `(1 − e^{a Δt}) m x`, which is not `m x`, not `κ`, and not `TIPREDEFFECT`; §7.2 extra-process contribution is `a_{ηξ} x (e^{ε Δt} − e^{a Δt}) / (ε − a)` (`ε = a` is `a_{ηξ} x Δt e^{a Δt}`; identification `TDPREDEFFECT` on the extra process is 1; printed extra `DRIFT` is `−0.000001`; not `κ = −a m x`, not `(1 − e^{a Δt}) m x`, and not the dissipating Dirac `m x`; `ε ≥ 0` fails closed; Eq. 5 of that contribution is `τ + λ(μ_t + a_{ηξ} x (e^{ε Δt} − e^{a Δt}) / (ε − a)`; the extra process has `LAMBDA` 0 and is not an observed indicator; `τ + λ μ_t` is not that observed mean; `τ + λ(μ_t + m x)` is not that observed mean; the contribution is not `E(y_t)`; the evolved-plus-contribution latent mean is not `E(y_t)`; after-t0 extra-process `TDPREDEFFECT` is `a_{ηξ} x (e^{ε(t−u)} − e^{a(t−u)}) / (ε − a)` for `t0 < u < t` while `μ_t` uses `Δt`; Eq. 5 of that after-t0 contribution is `τ + λ(μ_t + a_{ηξ} x (e^{ε(t−u)} − e^{a(t−u)}) / (ε − a)`; the first-occasion extra-process observed mean is not that observed mean when `u ≠ t0`; `e^{a(t−u)} m x` is a Dirac on the original process, not this `DRIFT` drive; §7.2 `asymTIPREDEFFECT` is `-B z / a` for `a < 0` (`-B z / a` is not the coefficient `B`, not `A^{-1}[e^{A Δt} − I] B z`, not `CINT`, and not `M x`; §7.2 `addedTIPREDVAR` is `(B / a)² v`, not `TRAITVAR`, not `asymDIFFUSION`, and not `-B z / a`; Table 2 `asymCINT` is `-κ / a` for `a < 0` and is not `κ`, not `A^{-1}[e^{A Δt} − I] κ`, not `T0MEANS`, and not `-B z / a`; p. 16 stationary `T0MEANS` is `-κ / a + −B z / a` and is not free `T0MEANS`, not `asymCINT` alone, not `asymTIPREDEFFECT` alone, and not the finite-interval discrete latent mean; Eq. 5 of that constrained mean is `τ + λ(−κ / a + −B z / a)`; `τ + λ μ_0` is not that observed mean; `τ + λ(−κ / a)` is not that observed mean when `B z ≠ 0`; `τ + λ μ_t` is not that observed mean; `MANIFESTMEANS` is not `E(y_0)`; the constrained latent mean is not `E(y_0)`; stationary `T0VAR` is `trait + −q / (2 a) + (B / a)² v` (not free `T0VAR`, not `asymDIFFUSION` alone, not `TRAITVAR` alone, not `addedTIPREDVAR` alone, and not the finite-interval discrete latent variance. Eq. 5 of that constrained variance is `λ²(trait + −q / (2 a) + (B / a)² v) + θ + ψ` (JSS PDF re-opened 2026-08-22T03:20Z; form the stationary latent variance first, then `λ² p + θ + ψ`; `λ² p_0` is not that observed variance; `λ²(−q / (2 a)) + θ` is not that observed variance when `TRAITVAR` or `addedTIPREDVAR` is nonzero; `MANIFESTVAR` is not `Var(y_0)`; the constrained latent variance is not `Var(y_0)`); lagged stationary `T0VAR` is `trait + e^{a Δt}(−q / (2 a)) + (B / a)² v` (trait and `addedTIPREDVAR` do not decay; contemporaneous `T0VAR` is not that lagged map; decaying the constrained total as if it were all state is not that lagged map; Eq. 5 of that lagged covariance is `λ²(trait + e^{a Δt}(−q / (2 a)) + (B / a)² v) + ψ`; `Θ` does not enter; contemporaneous `Var(y_0)` is not that lagged observed covariance; the lagged latent covariance is not that observed covariance); later-occasion stationary `T0VAR` is `trait + e^{2 a Δt}(−q / (2 a)) + Q_Δt + (B / a)² v` (trait and `addedTIPREDVAR` do not enter `Q_Δt`; under stationarity that composition equals contemporaneous `T0VAR`; evolving the constrained total as if it were all state is not that later map; the lagged covariance omits `Q_Δt`; `Q_Δt` is not that later map; Eq. 5 of that later-occasion variance is `λ²(trait + e^{2 a Δt}(−q / (2 a)) + Q_Δt + (B / a)² v) + θ + ψ`; lagged observed covariance omits `Q_Δt` and `θ`; `MANIFESTVAR` is not `Var(y_t)`; the later-occasion latent variance is not `Var(y_t)`))), irregular already-centered residual lag, Rubin `T` on OLS loadings, and strong-gated latent means (two-observation residual variance is identically `0` and caps at strong/scalar; Putnick & Bornstein, 2016) | | `validation_core` | RMSE, bias, coverage, graph, and Monte Carlo metrics | | `tepp_api` | versioned DTO, schema, and export contracts | diff --git a/CHANGELOG.md b/CHANGELOG.md index 062a69412..31b32a03f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -38,6 +38,8 @@ All notable changes to TEPP are documented here. The format follows Keep a Chang ## [Unreleased] +- `analysis_engine` GAP-003A first slice (ADR 0026, active-PR, not implemented-main): `submit_validation_run` binds cutoff-eligible evidence identities, snapshot, knowledge cutoff, CPU `f64` model, seed, backend, and precision to a hash-stable `tepp-validation-{32 hex}` receipt that carries no scientific metrics. `complete_validation_run` emits `tepp.scientific_acceptance.v1` with RMSE, bias, Wilson coverage, temporal-order accuracy, and an SE-aware gate through `validation_core`. LLM-authored recovery, non-finite inputs, empty or duplicate evidence, snapshot mismatch, and cutoff-empty corpora fail closed. Postgres persistence remains GAP-003B. + - `event_core` adds bounded Allen interval-consistency classification, atomic path-consistency closure, contradiction/resource refusals, and an explicit dependency-error fallback without claiming unrestricted global satisfiability. - `psychometric_core` recovers the Driver, Oud, and Voelkle (2017, Table 2, p. 12 `MANIFESTTRAITVAR`; §7.1, p. 19; p. 16 `MANIFESTTRAITVARstd`; footnote 4; 2017-era ctsem `summary.ctsemFit.R`; JSS PDF re-opened 2026-08-27T14:20Z from https://www.jstatsoft.org/index.php/jss/article/download/v077i05/1104) scalar standardised manifest-trait variance on current main after `0ce16e8` dropped the pre-consolidation code while research notes already named the map (register items 83–84). Table 2 names `MANIFESTTRAITVAR` `Ψ_τ` the additional time-invariant variance-covariance on the measurement level and sets it `NULL` when there is no manifest trait. Equation 5 writes `Γ ~ N(τ, Ψ)` and names that covariance the manifest traits. Section 7.1 names manifest traits stable individual differences in indicator levels, distinct from process-level `TRAITVAR` `φ_ξ`. Page 16 prints standardised matrices with the suffix `std` when appropriate. The printed example on p. 16 is `discreteDRIFTstd`, not `MANIFESTTRAITVARstd`. Footnote 4 standardises using only the relevant variance, not the total. The relevant variance for that named indicator-level correlation is `MANIFESTTRAITVAR`, not process-level `TRAITVAR` and not residual `MANIFESTVAR` `θ`. The 2017-era source forms `MANIFESTTRAITVARstd` only when `MANIFESTTRAITVAR != 0`, as `solve(sqrt(diag(MANIFESTTRAITVAR) + ridging)) %&% MANIFESTTRAITVAR` when `verbose = TRUE`. OpenMx `%&%` is `t(A) %*% B %*% A`. Unlike `TRAITVARstd`, that formation adds `diag(c(ridging), n.manifest)`. The default `ridging = FALSE` adds 0, not `0.0001`; that ridge is a numerical hack and is not this exact map. The scalar correlation is `ψ / ψ = 1` after strictly positive `MANIFESTTRAITVAR`. Form strictly positive `ψ` first, then `1 / √ψ`, then `(1 / √ψ) ψ (1 / √ψ)`. Unstandardised `MANIFESTTRAITVAR` is defined for a zero trait; standardised `MANIFESTTRAITVAR` is not. Zero `MANIFESTTRAITVAR` skips forming `MANIFESTTRAITVARstd` in the 2017-era source and fails closed here. Indicator-level trait variance is an event-time structural quantity, so a non-event clock fails closed. `MANIFESTTRAITVAR` does not require stable `a < 0`. Distinct positive `ψ` recover the same 1. `trait / trait = 1` is `TRAITVARstd` and recovers the same number and remains a distinct named quantity. `θ` is `MANIFESTVAR` and is measurement error, not this correlation. Meredith (1993) remains unread (web search 2026-08-27T14:20Z: Springer/Cambridge Core paywalled; Unpaywall historically `is_oa: false`; Springer `content/pdf` is an HTML stub). Mislevy (1991, *Psychometrika, 56*, 177–196) remains unread on the same terms (DOI `10.1007/bf02294457`). Still not a Kalman filter, not a matrix `expm`, not ESEM estimation, not DSEM, and not ctsem estimation. diff --git a/Cargo.lock b/Cargo.lock index 454a7d612..feb37e96e 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -82,6 +82,7 @@ dependencies = [ "tepp_api", "topic_measurement", "uuid", + "validation_core", ] [[package]] diff --git a/DOCUMENTATION.md b/DOCUMENTATION.md index 6fa4b9683..0d18e0e6f 100644 --- a/DOCUMENTATION.md +++ b/DOCUMENTATION.md @@ -58,6 +58,7 @@ TEPP's approved PRD v0.4 and implementation plan are the primary product baselin | Episode-membership identity doctoring | [`docs/research/episode-membership-identity.md`](docs/research/episode-membership-identity.md) | | Entity/project target SQL doctoring | [`docs/research/entity-project-sql.md`](docs/research/entity-project-sql.md) | | Scientific claim-promotion gate doctoring | [`docs/research/scientific-claim-promotion-gates.md`](docs/research/scientific-claim-promotion-gates.md) | +| Validation-run scientific-acceptance doctoring | [`docs/research/validation-run-scientific-acceptance.md`](docs/research/validation-run-scientific-acceptance.md) | | Retention/deletion/legal-hold doctoring | [`docs/research/retention-deletion-legal-hold.md`](docs/research/retention-deletion-legal-hold.md) | | Provider-payload minimization doctoring | [`docs/research/provider-payload-minimization.md`](docs/research/provider-payload-minimization.md) | | Relation absence is not negative evidence | [`docs/research/relation-absence-not-negative.md`](docs/research/relation-absence-not-negative.md) | @@ -147,6 +148,7 @@ TEPP's approved PRD v0.4 and implementation plan are the primary product baselin | Episode-membership identity doctoring | [`docs/research/episode-membership-identity.md`](docs/research/episode-membership-identity.md) | | Entity/project target SQL doctoring | [`docs/research/entity-project-sql.md`](docs/research/entity-project-sql.md) | | Scientific claim-promotion gate doctoring | [`docs/research/scientific-claim-promotion-gates.md`](docs/research/scientific-claim-promotion-gates.md) | +| Validation-run scientific-acceptance doctoring | [`docs/research/validation-run-scientific-acceptance.md`](docs/research/validation-run-scientific-acceptance.md) | | Retention/deletion/legal-hold doctoring | [`docs/research/retention-deletion-legal-hold.md`](docs/research/retention-deletion-legal-hold.md) | | Stopword-deletion doctoring | [`docs/research/stopword-deletion.md`](docs/research/stopword-deletion.md) | | Provider-payload minimization doctoring | [`docs/research/provider-payload-minimization.md`](docs/research/provider-payload-minimization.md) | diff --git a/crates/analysis_engine/Cargo.toml b/crates/analysis_engine/Cargo.toml index 7322212b2..0d384448b 100644 --- a/crates/analysis_engine/Cargo.toml +++ b/crates/analysis_engine/Cargo.toml @@ -22,6 +22,7 @@ tepp_api = { path = "../tepp_api", version = "0.2.0" } temporal_core = { path = "../temporal_core", version = "0.2.0" } topic_measurement = { path = "../topic_measurement", version = "0.2.0" } uuid.workspace = true +validation_core = { path = "../validation_core", version = "0.2.0" } [dev-dependencies] corpus_split = { path = "../corpus_split", version = "0.2.0" } diff --git a/crates/analysis_engine/src/lib.rs b/crates/analysis_engine/src/lib.rs index 72bd5854c..4c5e4536f 100644 --- a/crates/analysis_engine/src/lib.rs +++ b/crates/analysis_engine/src/lib.rs @@ -14,6 +14,7 @@ mod case_deletion_refit; mod lineage_criterion; mod topic_context_posterior; mod topic_lineage_artifact; +mod validation_run; use serde::Serialize; use sha2::{Digest, Sha256}; @@ -26,6 +27,7 @@ use tepp_api::{ ApiError, }; use topic_measurement::TopicMeasurementError; +use validation_core::ValidationError; /// One document admitted to exhaustive case-deletion fitting. pub use case_deletion_refit::CaseDeletionDocument; @@ -58,6 +60,14 @@ pub use topic_lineage_artifact::{ TOPIC_LINEAGE_MODEL_CONTRACT_VERSION, TOPIC_LINEAGE_OUTPUT_PROFILE, TopicLineageArtifact, TopicLineageArtifactEdge, TopicLineageExecution, execute_topic_lineage_run, }; +/// Durable scientific-acceptance validation-run contracts. +pub use validation_run::{ + RecoveryObservation, SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE, + SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION, ScientificAcceptanceEvidence, VALIDATION_BACKEND, + VALIDATION_CPU_F64_MODEL, VALIDATION_PRECISION, VALIDATION_RUN_ID_HEX_LEN, + VALIDATION_RUN_ID_PREFIX, ValidationRunReceipt, WILSON_Z, complete_validation_run, + submit_validation_run, +}; /// Versioned artifact schema emitted by this engine. pub const ANALYSIS_ARTIFACT_SCHEMA_VERSION: &str = "tepp.temporal_evidence_readiness.v1"; @@ -248,6 +258,16 @@ pub enum AnalysisEngineError { TopicMeasurement(TopicMeasurementError), /// A topic-lineage artifact violated its bounded schema or count invariants. InvalidTopicLineageArtifact, + /// A `validation_core` metric or report rejected the recovery vectors. + Validation(ValidationError), + /// Recovery was authored by an LLM and cannot become scientific evidence. + LlmAuthoredRecovery, + /// The corpus contained evidence, but none was available at the cutoff. + NoEligibleEvidence, + /// A completion receipt did not match the rebound scientific binding. + BindingMismatch, + /// The request did not select the scientific-acceptance profile and model. + InvalidValidationProfile, } impl fmt::Display for AnalysisEngineError { @@ -262,6 +282,13 @@ impl fmt::Display for AnalysisEngineError { Self::LimitExceeded => "analysis corpus exceeded its execution bound", Self::TopicMeasurement(error) => return error.fmt(formatter), Self::InvalidTopicLineageArtifact => "invalid topic lineage artifact", + Self::Validation(error) => return error.fmt(formatter), + Self::LlmAuthoredRecovery => { + "llm-authored recovery is not scientific acceptance evidence" + } + Self::NoEligibleEvidence => "no eligible evidence for validation run", + Self::BindingMismatch => "validation run binding mismatch", + Self::InvalidValidationProfile => "invalid scientific acceptance profile", }; formatter.write_str(message) } @@ -281,6 +308,12 @@ impl From for AnalysisEngineError { } } +impl From for AnalysisEngineError { + fn from(error: ValidationError) -> Self { + Self::Validation(error) + } +} + /// Execute the cutoff-safe temporal evidence readiness analysis. /// /// Evidence whose `available_time` is later than the request cutoff is excluded @@ -402,7 +435,7 @@ fn format_digest(digest: impl AsRef<[u8]>) -> String { output } -fn valid_identifier(value: &str) -> bool { +pub(crate) fn valid_identifier(value: &str) -> bool { !value.trim().is_empty() && value.len() <= MAX_ANALYSIS_IDENTIFIER_BYTES && !value.chars().any(char::is_control) @@ -681,6 +714,26 @@ mod tests { AnalysisEngineError::InvalidTopicLineageArtifact, "invalid topic lineage artifact", ), + ( + AnalysisEngineError::Validation(validation_core::ValidationError::InvalidInput), + "invalid validation input", + ), + ( + AnalysisEngineError::LlmAuthoredRecovery, + "llm-authored recovery is not scientific acceptance evidence", + ), + ( + AnalysisEngineError::NoEligibleEvidence, + "no eligible evidence for validation run", + ), + ( + AnalysisEngineError::BindingMismatch, + "validation run binding mismatch", + ), + ( + AnalysisEngineError::InvalidValidationProfile, + "invalid scientific acceptance profile", + ), ]; for (error, message) in messages { assert_eq!(error.to_string(), message); @@ -689,6 +742,9 @@ mod tests { assert_eq!(converted.to_string(), "invalid API wire payload"); let from_topic: AnalysisEngineError = TopicMeasurementError::DidNotConverge.into(); assert_eq!(from_topic.to_string(), "topic estimator did not converge"); + let from_validation: AnalysisEngineError = + validation_core::ValidationError::InvalidInput.into(); + assert_eq!(from_validation.to_string(), "invalid validation input"); assert_eq!( add_membership_count(u64::MAX, 1), Err(AnalysisEngineError::ArithmeticOverflow) diff --git a/crates/analysis_engine/src/validation_run.rs b/crates/analysis_engine/src/validation_run.rs new file mode 100644 index 000000000..1af31b6ea --- /dev/null +++ b/crates/analysis_engine/src/validation_run.rs @@ -0,0 +1,891 @@ +//! Bind immutable evidence to a durable scientific-acceptance validation run. +//! +//! This is the first GAP-003A slice: cutoff-safe evidence, model, seed, backend, +//! and precision hash to one durable run identity. The accepted receipt carries +//! no scientific metrics. Completion asks `validation_core` for RMSE, bias, +//! coverage, temporal-order accuracy, and an SE-aware gate, then emits +//! `tepp.scientific_acceptance.v1`. LLM-authored recovery, non-finite inputs, +//! empty or duplicate evidence, snapshot mismatch, and cutoff-empty corpora +//! fail closed. Postgres persistence remains GAP-003B. + +use crate::{AnalysisCorpus, AnalysisEngineError, valid_identifier}; +use serde::Serialize; +use sha2::{Digest, Sha256}; +use std::collections::BTreeSet; +use std::fmt::Write as _; +use temporal_core::KnowledgeCutoff; +use tepp_api::{AnalysisRunRequest, ApiError}; +use validation_core::{ + ValidationError, ValidationReport, accept_within_standard_errors, bias_standard_error, + interval_coverage, mean_bias, rmse_standard_error, root_mean_square_error, + temporal_order_accuracy, wilson_coverage_interval, +}; + +/// Versioned scientific-acceptance artifact schema. +pub const SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION: &str = "tepp.scientific_acceptance.v1"; +/// Output profile that selects this validation-run executor. +pub const SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE: &str = "scientific_acceptance_v1"; +/// CPU `f64` reference model identity for scientific acceptance. +pub const VALIDATION_CPU_F64_MODEL: &str = "validation_cpu_f64_v1"; +/// Backend identity bound into the durable run. +pub const VALIDATION_BACKEND: &str = "cpu"; +/// Numeric precision bound into the durable run. +pub const VALIDATION_PRECISION: &str = "f64"; +/// Prefix of the hash-stable durable run identity. +pub const VALIDATION_RUN_ID_PREFIX: &str = "tepp-validation-"; +/// Hex characters taken from the binding digest for `run_id`. +pub const VALIDATION_RUN_ID_HEX_LEN: usize = 32; +/// Wilson critical value for nominal 95% coverage bounds. +pub const WILSON_Z: f64 = 1.96; + +/// Durable identity of one submitted validation run. Receipts never carry +/// scientific metrics. +#[derive(Clone, Debug, Eq, PartialEq, Serialize)] +pub struct ValidationRunReceipt { + /// Hash-stable run identity `tepp-validation-{32 hex}`. + pub run_id: String, + /// Lowercase SHA-256 of the canonical binding. + pub binding_sha256: String, + /// Immutable source snapshot identity. + pub snapshot_id: String, + /// Historical cutoff applied to availability. + pub knowledge_cutoff: String, + /// Bound model identity. + pub model: String, + /// Bound numeric seed. + pub seed: u64, + /// Bound compute backend. + pub backend: String, + /// Bound numeric precision. + pub precision: String, + /// Requested output profile. + pub output_profile: String, + /// Number of cutoff-eligible evidence identities. + pub eligible_evidence_count: u64, +} + +impl ValidationRunReceipt { + /// Serialize the receipt after confirming it carries no metrics. + /// + /// # Errors + /// + /// Returns [`AnalysisEngineError::SerializationFailure`] when JSON encoding + /// fails. + pub fn to_json(&self) -> Result { + serde_json::to_string(self).map_err(|_| AnalysisEngineError::SerializationFailure) + } + + /// Return the durable run identity. + #[must_use] + pub fn run_id(&self) -> &str { + &self.run_id + } + + /// Return the canonical binding digest. + #[must_use] + pub fn binding_sha256(&self) -> &str { + &self.binding_sha256 + } + + /// Return the bound snapshot identity. + #[must_use] + pub fn snapshot_id(&self) -> &str { + &self.snapshot_id + } + + /// Return the bound knowledge cutoff. + #[must_use] + pub fn knowledge_cutoff(&self) -> &str { + &self.knowledge_cutoff + } + + /// Return the bound model identity. + #[must_use] + pub fn model(&self) -> &str { + &self.model + } + + /// Return the bound seed. + #[must_use] + pub const fn seed(&self) -> u64 { + self.seed + } + + /// Return the bound backend. + #[must_use] + pub fn backend(&self) -> &str { + &self.backend + } + + /// Return the bound precision. + #[must_use] + pub fn precision(&self) -> &str { + &self.precision + } + + /// Return the bound output profile. + #[must_use] + pub fn output_profile(&self) -> &str { + &self.output_profile + } + + /// Return the eligible evidence count. + #[must_use] + pub const fn eligible_evidence_count(&self) -> u64 { + self.eligible_evidence_count + } +} + +/// Known-truth recovery vectors offered to complete a validation run. +#[derive(Clone, Debug, PartialEq)] +pub struct RecoveryObservation { + study_label: String, + truth: Vec, + recovered: Vec, + interval_lower: Vec, + interval_upper: Vec, + truth_times: Vec, + recovered_times: Vec, + se_gate_k: f64, + authored_by_llm: bool, +} + +impl RecoveryObservation { + /// Construct a recovery observation. + /// + /// LLM authorship is recorded here and refused at completion. Non-finite + /// `se_gate_k` or a negative multiplier fail closed immediately. + /// + /// # Errors + /// + /// Returns [`AnalysisEngineError::InvalidEvidence`] for an empty, oversized, + /// or control-bearing study label, and + /// [`AnalysisEngineError::Validation`] for a non-finite or negative SE + /// multiplier. + #[allow(clippy::too_many_arguments)] + pub fn new( + study_label: impl Into, + truth: Vec, + recovered: Vec, + interval_lower: Vec, + interval_upper: Vec, + truth_times: Vec, + recovered_times: Vec, + se_gate_k: f64, + authored_by_llm: bool, + ) -> Result { + let study_label = study_label.into(); + if !valid_identifier(&study_label) { + return Err(AnalysisEngineError::InvalidEvidence); + } + if !se_gate_k.is_finite() { + return Err(AnalysisEngineError::Validation( + ValidationError::InvalidInput, + )); + } + if se_gate_k < 0.0 { + return Err(AnalysisEngineError::Validation( + ValidationError::InvalidConfiguration, + )); + } + Ok(Self { + study_label, + truth, + recovered, + interval_lower, + interval_upper, + truth_times, + recovered_times, + se_gate_k, + authored_by_llm, + }) + } + + /// Return the study label. + #[must_use] + pub fn study_label(&self) -> &str { + &self.study_label + } + + /// Return the truth vector. + #[must_use] + pub fn truth(&self) -> &[f64] { + &self.truth + } + + /// Return the recovered vector. + #[must_use] + pub fn recovered(&self) -> &[f64] { + &self.recovered + } + + /// Return interval lower bounds. + #[must_use] + pub fn interval_lower(&self) -> &[f64] { + &self.interval_lower + } + + /// Return interval upper bounds. + #[must_use] + pub fn interval_upper(&self) -> &[f64] { + &self.interval_upper + } + + /// Return truth event times. + #[must_use] + pub fn truth_times(&self) -> &[f64] { + &self.truth_times + } + + /// Return recovered event times. + #[must_use] + pub fn recovered_times(&self) -> &[f64] { + &self.recovered_times + } + + /// Return the SE-gate multiplier. + #[must_use] + pub const fn se_gate_k(&self) -> f64 { + self.se_gate_k + } + + /// Return whether an LLM authored the recovery. + #[must_use] + pub const fn authored_by_llm(&self) -> bool { + self.authored_by_llm + } +} + +/// Operator-usable scientific acceptance evidence for one completed run. +#[derive(Clone, Debug, PartialEq, Serialize)] +pub struct ScientificAcceptanceEvidence { + /// Versioned artifact schema. + pub schema_version: String, + /// Durable run identity. + pub run_id: String, + /// Canonical binding digest. + pub binding_sha256: String, + /// Immutable source snapshot identity. + pub snapshot_id: String, + /// Historical cutoff applied to availability. + pub knowledge_cutoff: String, + /// Bound model identity. + pub model: String, + /// Bound numeric seed. + pub seed: u64, + /// Bound compute backend. + pub backend: String, + /// Bound numeric precision. + pub precision: String, + /// Output profile. + pub output_profile: String, + /// Number of cutoff-eligible evidence identities. + pub eligible_evidence_count: u64, + /// Whether RMSE toward 0 passed the SE-aware gate. + pub se_gate_accepted: bool, + /// SE-gate multiplier used for acceptance. + pub se_gate_k: f64, + /// Machine-readable recovery report. + pub report: ValidationReport, +} + +impl ScientificAcceptanceEvidence { + /// Serialize canonical evidence JSON after report validation. + /// + /// # Errors + /// + /// Returns [`AnalysisEngineError::Validation`] when a report field is + /// non-finite, or [`AnalysisEngineError::SerializationFailure`] when JSON + /// encoding fails. + pub fn to_json(&self) -> Result { + self.report.validate()?; + serde_json::to_string(self).map_err(|_| AnalysisEngineError::SerializationFailure) + } + + /// Return the lowercase SHA-256 digest of the canonical evidence JSON. + /// + /// # Errors + /// + /// Returns the same failures as [`Self::to_json`]. + pub fn sha256(&self) -> Result { + self.to_json() + .map(|json| format_hex(Sha256::digest(json.into_bytes()))) + } + + /// Render the nested recovery summary line. + #[must_use] + pub fn to_human_summary(&self) -> String { + self.report.to_human_summary() + } + + /// Return the artifact schema version. + #[must_use] + pub fn schema_version(&self) -> &str { + &self.schema_version + } + + /// Return the durable run identity. + #[must_use] + pub fn run_id(&self) -> &str { + &self.run_id + } + + /// Return whether the SE-aware gate accepted RMSE toward 0. + #[must_use] + pub const fn se_gate_accepted(&self) -> bool { + self.se_gate_accepted + } +} + +struct CanonicalBinding { + snapshot_id: String, + knowledge_cutoff: String, + model: String, + seed: u64, + backend: String, + precision: String, + eligible_ids: Vec, +} + +impl CanonicalBinding { + fn digest_hex(&self) -> String { + format_hex(Sha256::digest(self.canonical_bytes())) + } + + fn run_id(digest_hex: &str) -> String { + let mut run_id = + String::with_capacity(VALIDATION_RUN_ID_PREFIX.len() + VALIDATION_RUN_ID_HEX_LEN); + run_id.push_str(VALIDATION_RUN_ID_PREFIX); + run_id.push_str(&digest_hex[..VALIDATION_RUN_ID_HEX_LEN]); + run_id + } + + fn eligible_count(&self) -> u64 { + self.eligible_ids.len() as u64 + } + + fn canonical_bytes(&self) -> String { + let mut canonical = String::from("tepp.validation_binding.v1\n"); + let _ = writeln!(canonical, "snapshot={}", self.snapshot_id); + let _ = writeln!(canonical, "cutoff={}", self.knowledge_cutoff); + let _ = writeln!(canonical, "model={}", self.model); + let _ = writeln!(canonical, "seed={}", self.seed); + let _ = writeln!(canonical, "backend={}", self.backend); + let _ = writeln!(canonical, "precision={}", self.precision); + for identity in &self.eligible_ids { + let _ = writeln!(canonical, "evidence={identity}"); + } + canonical + } + + fn identity_record(&self, digest_hex: &str, run_id: &str) -> String { + format!( + "{run_id}\n{digest_hex}\n{}\n{}\n{}\n{}\n{}\n{}\n{}", + self.snapshot_id, + self.knowledge_cutoff, + self.model, + self.seed, + self.backend, + self.precision, + self.eligible_count() + ) + } +} + +impl ValidationRunReceipt { + fn identity_record(&self) -> String { + format!( + "{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}", + self.run_id, + self.binding_sha256, + self.snapshot_id, + self.knowledge_cutoff, + self.model, + self.seed, + self.backend, + self.precision, + self.eligible_evidence_count + ) + } +} + +/// Submit immutable evidence to a durable validation run. +/// +/// The receipt is hash-stable for one scientific binding and never includes +/// RMSE, bias, coverage, or gate results. +/// +/// # Errors +/// +/// Returns a fail-closed engine error for an invalid request, wrong output +/// profile or model, snapshot mismatch, duplicate or empty evidence, or a +/// cutoff that admits no evidence. +pub fn submit_validation_run( + request: &AnalysisRunRequest, + corpus: &AnalysisCorpus, + seed: u64, +) -> Result { + let binding = bind_validation_run(request, corpus, seed)?; + let binding_sha256 = binding.digest_hex(); + let run_id = CanonicalBinding::run_id(&binding_sha256); + let eligible_evidence_count = binding.eligible_count(); + Ok(ValidationRunReceipt { + run_id, + binding_sha256, + snapshot_id: binding.snapshot_id, + knowledge_cutoff: binding.knowledge_cutoff, + model: binding.model, + seed: binding.seed, + backend: binding.backend, + precision: binding.precision, + output_profile: SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE.to_owned(), + eligible_evidence_count, + }) +} + +/// Complete a previously submitted validation run with known-truth recovery. +/// +/// # Errors +/// +/// Returns a fail-closed engine error when recovery was LLM-authored, the +/// receipt does not match the rebound scientific identity, metric inputs are +/// invalid, or report validation fails. +pub fn complete_validation_run( + receipt: &ValidationRunReceipt, + request: &AnalysisRunRequest, + corpus: &AnalysisCorpus, + observation: &RecoveryObservation, +) -> Result { + if observation.authored_by_llm { + return Err(AnalysisEngineError::LlmAuthoredRecovery); + } + let binding = bind_validation_run(request, corpus, receipt.seed)?; + let binding_sha256 = binding.digest_hex(); + let run_id = CanonicalBinding::run_id(&binding_sha256); + if receipt.identity_record() != binding.identity_record(&binding_sha256, &run_id) { + return Err(AnalysisEngineError::BindingMismatch); + } + let report = compute_report(observation)?; + let se_gate_accepted = accept_within_standard_errors( + report.rmse, + 0.0, + report.rmse_standard_error, + observation.se_gate_k, + )?; + let eligible_evidence_count = binding.eligible_count(); + let evidence = ScientificAcceptanceEvidence { + schema_version: SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION.to_owned(), + run_id, + binding_sha256, + snapshot_id: binding.snapshot_id, + knowledge_cutoff: binding.knowledge_cutoff, + model: binding.model, + seed: binding.seed, + backend: binding.backend, + precision: binding.precision, + output_profile: SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE.to_owned(), + eligible_evidence_count, + se_gate_accepted, + se_gate_k: observation.se_gate_k, + report, + }; + evidence.to_json()?; + Ok(evidence) +} + +fn bind_validation_run( + request: &AnalysisRunRequest, + corpus: &AnalysisCorpus, + seed: u64, +) -> Result { + request.to_json()?; + let requested = format!( + "{}\n{}", + request.output_profile, request.model_contract_version + ); + let expected = format!("{SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE}\n{VALIDATION_CPU_F64_MODEL}"); + if requested != expected { + return Err(AnalysisEngineError::InvalidValidationProfile); + } + if request.snapshot_id != corpus.snapshot_id() { + return Err(AnalysisEngineError::SnapshotMismatch); + } + let cutoff = KnowledgeCutoff::parse_rfc3339(&request.knowledge_cutoff) + .map_err(|_| AnalysisEngineError::Api(ApiError::InvalidWirePayload))?; + let mut identities = BTreeSet::new(); + let mut eligible = BTreeSet::new(); + for unit in corpus.evidence_units() { + if !identities.insert(unit.evidence_id()) { + return Err(AnalysisEngineError::DuplicateEvidence); + } + if unit.available_time().instant() <= cutoff.instant() { + eligible.insert(unit.evidence_id().to_owned()); + } + } + if identities.is_empty() { + return Err(AnalysisEngineError::InvalidEvidence); + } + if eligible.is_empty() { + return Err(AnalysisEngineError::NoEligibleEvidence); + } + Ok(CanonicalBinding { + snapshot_id: request.snapshot_id.clone(), + knowledge_cutoff: cutoff.to_rfc3339(), + model: VALIDATION_CPU_F64_MODEL.to_owned(), + seed, + backend: VALIDATION_BACKEND.to_owned(), + precision: VALIDATION_PRECISION.to_owned(), + eligible_ids: eligible.into_iter().collect(), + }) +} + +fn compute_report( + observation: &RecoveryObservation, +) -> Result { + let rmse = root_mean_square_error(&observation.truth, &observation.recovered)?; + let rmse_standard_error = rmse_standard_error(&observation.truth, &observation.recovered)?; + let mean_bias = mean_bias(&observation.truth, &observation.recovered)?; + let bias_standard_error = bias_standard_error(&observation.truth, &observation.recovered)?; + let interval_coverage = interval_coverage( + &observation.truth, + &observation.interval_lower, + &observation.interval_upper, + )?; + let (coverage_wilson_lower, coverage_wilson_upper) = wilson_coverage_interval( + &observation.truth, + &observation.interval_lower, + &observation.interval_upper, + WILSON_Z, + )?; + let temporal_order_accuracy = + temporal_order_accuracy(&observation.truth_times, &observation.recovered_times)?; + let report = ValidationReport { + study_label: observation.study_label.clone(), + rmse, + rmse_standard_error, + mean_bias, + bias_standard_error, + interval_coverage, + coverage_wilson_lower, + coverage_wilson_upper, + temporal_order_accuracy, + monte_carlo_rmse: None, + }; + report.validate()?; + Ok(report) +} + +fn format_hex(digest: impl AsRef<[u8]>) -> String { + let bytes = digest.as_ref(); + let mut output = String::with_capacity(bytes.len() * 2); + for byte in bytes { + let _ = write!(output, "{byte:02x}"); + } + output +} + +#[cfg(test)] +mod tests { + use super::{ + RecoveryObservation, SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE, + SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION, VALIDATION_BACKEND, VALIDATION_CPU_F64_MODEL, + VALIDATION_PRECISION, VALIDATION_RUN_ID_PREFIX, WILSON_Z, complete_validation_run, + submit_validation_run, + }; + use crate::{ + AnalysisCorpus, AnalysisEngineError, AnalysisEvidenceUnit, MAX_ANALYSIS_IDENTIFIER_BYTES, + }; + use temporal_core::{AvailableTime, EventTime}; + use tepp_api::AnalysisRunRequest; + use validation_core::ValidationError; + + fn request() -> AnalysisRunRequest { + AnalysisRunRequest { + contract_version: 1, + idempotency_key: "idem-validation-1".into(), + tenant_workspace_id: "tenant-workspace-1".into(), + snapshot_id: "snapshot-1".into(), + knowledge_cutoff: "2026-08-01T00:00:00Z".into(), + model_contract_version: VALIDATION_CPU_F64_MODEL.into(), + output_profile: SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE.into(), + } + } + + fn unit(id: &str, available: &str) -> AnalysisEvidenceUnit { + AnalysisEvidenceUnit::new( + id, + EventTime::parse_rfc3339("2026-07-01T00:00:00Z").expect("event"), + AvailableTime::parse_rfc3339(available).expect("available"), + 1, + ) + .expect("unit") + } + + fn corpus(units: Vec) -> AnalysisCorpus { + AnalysisCorpus::new("snapshot-1", units).expect("corpus") + } + + fn recovery(truth: Vec, recovered: Vec, k: f64) -> RecoveryObservation { + let times = [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0]; + let times = times[..truth.len()].to_vec(); + let lower: Vec = truth.iter().map(|value| value - 0.5).collect(); + let upper: Vec = truth.iter().map(|value| value + 0.5).collect(); + RecoveryObservation::new( + "foundation-recovery", + truth, + recovered, + lower, + upper, + times.clone(), + times, + k, + false, + ) + .expect("observation") + } + + fn two_point( + label: impl Into, + k: f64, + authored_by_llm: bool, + ) -> Result { + RecoveryObservation::new( + label, + vec![1.0, 2.0], + vec![1.0, 2.0], + vec![0.0, 1.0], + vec![2.0, 3.0], + vec![1.0, 2.0], + vec![1.0, 2.0], + k, + authored_by_llm, + ) + } + + #[test] + fn submit_is_hash_stable_cutoff_safe_and_metric_free() { + let first = corpus(vec![ + unit("evidence-b", "2026-07-15T00:00:00Z"), + unit("evidence-a", "2026-07-10T00:00:00Z"), + unit("late", "2026-08-02T00:00:00Z"), + ]); + let second = corpus(vec![ + unit("evidence-a", "2026-07-10T00:00:00Z"), + unit("evidence-b", "2026-07-15T00:00:00Z"), + ]); + let receipt_a = submit_validation_run(&request(), &first, 7).expect("submit a"); + let receipt_b = submit_validation_run(&request(), &second, 7).expect("submit b"); + assert_eq!(receipt_a.run_id(), receipt_b.run_id()); + assert_eq!(receipt_a.binding_sha256(), receipt_b.binding_sha256()); + assert!(receipt_a.run_id().starts_with(VALIDATION_RUN_ID_PREFIX)); + assert_eq!( + receipt_a.run_id().len(), + VALIDATION_RUN_ID_PREFIX.len() + 32 + ); + assert_eq!(receipt_a.eligible_evidence_count(), 2); + assert_eq!(receipt_a.snapshot_id(), "snapshot-1"); + assert_eq!(receipt_a.knowledge_cutoff(), "2026-08-01T00:00:00Z"); + assert_eq!(receipt_a.model(), VALIDATION_CPU_F64_MODEL); + assert_eq!(receipt_a.seed(), 7); + assert_eq!(receipt_a.backend(), VALIDATION_BACKEND); + assert_eq!(receipt_a.precision(), VALIDATION_PRECISION); + assert_eq!( + receipt_a.output_profile(), + SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE + ); + let json = receipt_a.to_json().expect("json"); + assert!(!json.contains("rmse")); + assert!(!json.contains("bias")); + assert!(!json.contains("coverage")); + let other_seed = submit_validation_run(&request(), &first, 8).expect("seed"); + assert_ne!(receipt_a.run_id(), other_seed.run_id()); + } + + #[test] + fn complete_emits_operator_usable_scientific_acceptance_evidence() { + let offered = corpus(vec![ + unit("evidence-a", "2026-07-10T00:00:00Z"), + unit("evidence-b", "2026-07-15T00:00:00Z"), + ]); + let receipt = submit_validation_run(&request(), &offered, 11).expect("submit"); + let observation = recovery( + vec![0.70, 0.55, 0.40, -0.20, 0.85], + vec![0.70, 0.55, 0.40, -0.20, 0.85], + 3.0, + ); + assert_eq!(observation.study_label(), "foundation-recovery"); + assert_eq!(observation.truth().len(), 5); + assert_eq!(observation.recovered().len(), 5); + assert_eq!(observation.interval_lower().len(), 5); + assert_eq!(observation.interval_upper().len(), 5); + assert_eq!(observation.truth_times().len(), 5); + assert_eq!(observation.recovered_times().len(), 5); + assert!((observation.se_gate_k() - 3.0).abs() < f64::EPSILON); + assert!(!observation.authored_by_llm()); + let evidence = complete_validation_run(&receipt, &request(), &offered, &observation) + .expect("complete"); + assert_eq!( + evidence.schema_version(), + SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION + ); + assert_eq!(evidence.run_id(), receipt.run_id()); + assert!(evidence.se_gate_accepted()); + assert_eq!( + evidence.output_profile, + SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE + ); + assert_eq!(evidence.eligible_evidence_count, 2); + assert!((evidence.se_gate_k - 3.0).abs() < f64::EPSILON); + assert!((WILSON_Z - 1.96).abs() < f64::EPSILON); + let json = evidence.to_json().expect("json"); + assert!(json.contains(SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION)); + assert!(json.contains("rmse")); + assert_eq!(evidence.sha256().expect("digest").len(), 64); + assert!(evidence.to_human_summary().contains("foundation-recovery")); + let rejected = recovery( + vec![0.0, 1.0, 2.0, 3.0, 4.0], + vec![10.0, 11.0, 12.0, 13.0, 14.0], + 1.0, + ); + let refused = + complete_validation_run(&receipt, &request(), &offered, &rejected).expect("refused"); + assert!(!refused.se_gate_accepted()); + } + + #[test] + fn submit_and_complete_fail_closed_on_trust_and_shape_errors() { + let offered = corpus(vec![unit("evidence-a", "2026-07-10T00:00:00Z")]); + let mut wrong_profile = request(); + wrong_profile.output_profile = "validation-report".into(); + assert_eq!( + submit_validation_run(&wrong_profile, &offered, 1), + Err(AnalysisEngineError::InvalidValidationProfile) + ); + let mut wrong_model = request(); + wrong_model.model_contract_version = "temporal-evidence-v1".into(); + assert_eq!( + submit_validation_run(&wrong_model, &offered, 1), + Err(AnalysisEngineError::InvalidValidationProfile) + ); + let mismatched = AnalysisCorpus::new( + "snapshot-other", + vec![unit("evidence-a", "2026-07-10T00:00:00Z")], + ) + .expect("other"); + assert_eq!( + submit_validation_run(&request(), &mismatched, 1), + Err(AnalysisEngineError::SnapshotMismatch) + ); + let duplicate = corpus(vec![ + unit("same", "2026-07-10T00:00:00Z"), + unit("same", "2026-07-11T00:00:00Z"), + ]); + assert_eq!( + submit_validation_run(&request(), &duplicate, 1), + Err(AnalysisEngineError::DuplicateEvidence) + ); + let empty = AnalysisCorpus::new("snapshot-1", Vec::new()).expect("empty"); + assert_eq!( + submit_validation_run(&request(), &empty, 1), + Err(AnalysisEngineError::InvalidEvidence) + ); + let late = corpus(vec![unit("late", "2026-08-02T00:00:00Z")]); + assert_eq!( + submit_validation_run(&request(), &late, 1), + Err(AnalysisEngineError::NoEligibleEvidence) + ); + let mut invalid_request = request(); + invalid_request.idempotency_key.clear(); + assert!(matches!( + submit_validation_run(&invalid_request, &offered, 1), + Err(AnalysisEngineError::Api(_)) + )); + let mut invalid_cutoff = request(); + invalid_cutoff.knowledge_cutoff = "not-a-time".into(); + assert!(matches!( + submit_validation_run(&invalid_cutoff, &offered, 1), + Err(AnalysisEngineError::Api(_)) + )); + } + + #[test] + fn complete_refuses_llm_authored_recovery_and_tampered_receipt() { + let offered = corpus(vec![ + unit("evidence-a", "2026-07-10T00:00:00Z"), + unit("evidence-b", "2026-07-15T00:00:00Z"), + ]); + let receipt = submit_validation_run(&request(), &offered, 3).expect("submit"); + let llm = RecoveryObservation::new( + "llm-study", + vec![1.0, 2.0, 3.0], + vec![1.0, 2.0, 3.0], + vec![0.5, 1.5, 2.5], + vec![1.5, 2.5, 3.5], + vec![1.0, 2.0, 3.0], + vec![1.0, 2.0, 3.0], + 3.0, + true, + ) + .expect("llm"); + assert!(llm.authored_by_llm()); + assert_eq!( + complete_validation_run(&receipt, &request(), &offered, &llm), + Err(AnalysisEngineError::LlmAuthoredRecovery) + ); + let mut tampered = receipt.clone(); + tampered.run_id = "tepp-validation-deadbeefdeadbeefdeadbeefdeadbeef".into(); + let observation = recovery(vec![1.0, 2.0, 3.0], vec![1.0, 2.0, 3.0], 3.0); + assert_eq!( + complete_validation_run(&tampered, &request(), &offered, &observation), + Err(AnalysisEngineError::BindingMismatch) + ); + } + + #[test] + fn complete_refuses_nonfinite_recovery_and_invalid_observation() { + let offered = corpus(vec![ + unit("evidence-a", "2026-07-10T00:00:00Z"), + unit("evidence-b", "2026-07-15T00:00:00Z"), + ]); + let receipt = submit_validation_run(&request(), &offered, 3).expect("submit"); + let nan = RecoveryObservation::new( + "nan-study", + vec![1.0, 2.0, 3.0], + vec![f64::NAN, 2.0, 3.0], + vec![0.5, 1.5, 2.5], + vec![1.5, 2.5, 3.5], + vec![1.0, 2.0, 3.0], + vec![1.0, 2.0, 3.0], + 3.0, + false, + ) + .expect("nan observation"); + assert_eq!( + complete_validation_run(&receipt, &request(), &offered, &nan), + Err(AnalysisEngineError::Validation( + ValidationError::InvalidInput + )) + ); + assert_eq!( + two_point("", 3.0, false), + Err(AnalysisEngineError::InvalidEvidence) + ); + assert_eq!( + two_point("x".repeat(MAX_ANALYSIS_IDENTIFIER_BYTES + 1), 3.0, false), + Err(AnalysisEngineError::InvalidEvidence) + ); + assert_eq!( + two_point("nan-k", f64::NAN, false), + Err(AnalysisEngineError::Validation( + ValidationError::InvalidInput + )) + ); + assert_eq!( + two_point("neg-k", -1.0, false), + Err(AnalysisEngineError::Validation( + ValidationError::InvalidConfiguration + )) + ); + let converted: AnalysisEngineError = ValidationError::InvalidInput.into(); + assert_eq!(converted.to_string(), "invalid validation input"); + } +} diff --git a/crates/analysis_engine/tests/validation_run_contract.rs b/crates/analysis_engine/tests/validation_run_contract.rs new file mode 100644 index 000000000..1ebb164d5 --- /dev/null +++ b/crates/analysis_engine/tests/validation_run_contract.rs @@ -0,0 +1,103 @@ +//! GAP-003A scientific-acceptance validation-run contract. + +use analysis_engine::{ + AnalysisCorpus, AnalysisEngineError, AnalysisEvidenceUnit, RecoveryObservation, + SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE, SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION, + VALIDATION_CPU_F64_MODEL, complete_validation_run, submit_validation_run, +}; +use temporal_core::{AvailableTime, EventTime}; +use tepp_api::AnalysisRunRequest; + +fn request() -> AnalysisRunRequest { + AnalysisRunRequest { + contract_version: 1, + idempotency_key: "idem-validation-contract".into(), + tenant_workspace_id: "tenant-workspace-contract".into(), + snapshot_id: "snapshot-contract".into(), + knowledge_cutoff: "2026-08-01T00:00:00Z".into(), + model_contract_version: VALIDATION_CPU_F64_MODEL.into(), + output_profile: SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE.into(), + } +} + +fn unit(id: &str, available: &str) -> AnalysisEvidenceUnit { + AnalysisEvidenceUnit::new( + id, + EventTime::parse_rfc3339("2026-07-01T00:00:00Z").expect("event"), + AvailableTime::parse_rfc3339(available).expect("available"), + 1, + ) + .expect("unit") +} + +#[test] +fn durable_run_binds_cutoff_eligible_evidence_and_emits_acceptance_evidence() { + let corpus = AnalysisCorpus::new( + "snapshot-contract", + vec![ + unit("evidence-2", "2026-07-20T00:00:00Z"), + unit("evidence-1", "2026-07-10T00:00:00Z"), + unit("future", "2026-08-02T00:00:00Z"), + ], + ) + .expect("corpus"); + let receipt = submit_validation_run(&request(), &corpus, 42).expect("submit"); + assert_eq!(receipt.eligible_evidence_count(), 2); + assert!(!receipt.to_json().expect("json").contains("rmse")); + let observation = RecoveryObservation::new( + "contract-recovery", + vec![0.70, 0.55, 0.40, -0.20, 0.85], + vec![0.70, 0.55, 0.40, -0.20, 0.85], + vec![0.50, 0.35, 0.20, -0.40, 0.65], + vec![0.90, 0.75, 0.60, 0.00, 1.00], + vec![1.0, 2.0, 3.0, 4.0, 5.0], + vec![1.1, 1.9, 3.2, 3.8, 5.1], + 3.0, + false, + ) + .expect("observation"); + let evidence = + complete_validation_run(&receipt, &request(), &corpus, &observation).expect("complete"); + assert_eq!( + evidence.schema_version(), + SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION + ); + assert_eq!(evidence.run_id(), receipt.run_id()); + assert!(evidence.se_gate_accepted()); + assert!(evidence.to_json().expect("json").contains("rmse")); +} + +#[test] +fn llm_authored_recovery_and_cutoff_empty_corpora_fail_closed() { + let corpus = AnalysisCorpus::new( + "snapshot-contract", + vec![unit("only-future", "2026-08-02T00:00:00Z")], + ) + .expect("corpus"); + assert_eq!( + submit_validation_run(&request(), &corpus, 1), + Err(AnalysisEngineError::NoEligibleEvidence) + ); + let eligible = AnalysisCorpus::new( + "snapshot-contract", + vec![unit("evidence-1", "2026-07-10T00:00:00Z")], + ) + .expect("eligible"); + let receipt = submit_validation_run(&request(), &eligible, 1).expect("submit"); + let llm = RecoveryObservation::new( + "llm-contract", + vec![1.0, 2.0, 3.0], + vec![1.0, 2.0, 3.0], + vec![0.5, 1.5, 2.5], + vec![1.5, 2.5, 3.5], + vec![1.0, 2.0, 3.0], + vec![1.0, 2.0, 3.0], + 3.0, + true, + ) + .expect("llm"); + assert_eq!( + complete_validation_run(&receipt, &request(), &eligible, &llm), + Err(AnalysisEngineError::LlmAuthoredRecovery) + ); +} diff --git a/docs/TRACEABILITY.md b/docs/TRACEABILITY.md index 2b783c2ab..c2ab5d13f 100644 --- a/docs/TRACEABILITY.md +++ b/docs/TRACEABILITY.md @@ -50,6 +50,7 @@ The full APA 7th standards/literature register remains `docs/research/standards- | leakage-safe availability/cutoff snapshots | PRD; ADR 0002/0013 | `corpus_split` on protected main | implemented-main | | Unicode NFC/NFD canonical identity for split leakage | ADR 0004/0008/0013; UAX #15 | `corpus_split` canonical-equivalence links on active PR #59 | active-PR | | recovery metrics (RMSE, bias, coverage, graph, temporal order, Monte Carlo SE gates) | PRD; Test Strategy; ADR 0007/0014 | `validation_core` on protected main (PR #19); SE-aware Monte Carlo gates included | implemented-main | +| durable validation-run scientific acceptance evidence | ADR 0026; ADR 0002/0008/0014; GAP-003A; National Academies (2019); Wasserstein & Lazar (2016); Wilson (1927) | `analysis_engine` `submit_validation_run` / `complete_validation_run` on this PR: hash-stable cutoff-safe binding, metric-free receipt, `tepp.scientific_acceptance.v1` RMSE/bias/coverage/temporal-order/SE-aware evidence; LLM/NaN/empty/duplicate/snapshot/cutoff-empty fail closed; not implemented-main; Postgres persistence remains GAP-003B | active-PR | | PostgreSQL bitemporal/lineage persistence | ADR 0013; Architecture/ERD | `persistence_postgres` migration contracts, in-memory adapters, live SQL session/document SQL port, tenant RLS (`0002` + session GUC/role helpers), `DATABASE_URL` SQLx gate, optional session-affine `live-sqlx` `PgPool` driver, exact-head live PostgreSQL CI with isolation proof, append-only immutability triggers (`0004`), temporal interval ordering CHECKs (`0005`), typed membership assignment (`0006` implemented-main), event-relation/mention/instance SQL (#37–#39 implemented-main), source-artifact SQL (#40 implemented-main), audit-event SQL (#41 implemented-main), concurrent document-write stress (#43 implemented-main), backup/restore integrity revalidation (#44 implemented-main), `revision_order` later-revision system-time ordering implemented-main, entity/project target SQL on PR #131; remaining physical ERD constraints | partial | | known-truth temporal/event simulation manifests | PRD; TRD; Test Strategy | `tepp_simulation` on protected main; recovery metrics in `validation_core` | implemented-main | | versioned service/API contracts and exports | PRD; API contract; ADR 0011/0013 | `tepp_api` analysis-run/export/JSON-LD/GraphML contracts on protected main (PR #21); request-bound terminal result active in PR #157; HTTP service remains accepted-target; the `orchestrator_live` loopback interpretation listener is on this PR | partial | diff --git a/docs/adr/0026-validation-run-scientific-acceptance.md b/docs/adr/0026-validation-run-scientific-acceptance.md new file mode 100644 index 000000000..e769539f2 --- /dev/null +++ b/docs/adr/0026-validation-run-scientific-acceptance.md @@ -0,0 +1,88 @@ +# ADR 0026 — Durable validation-run scientific acceptance evidence + +**Decision status:** Accepted +**Implementation maturity:** active-PR — library-level `analysis_engine` binding on this PR; not implemented-main +**Date:** 2026-08-31 +**Supersedes:** None; complements ADR 0002, ADR 0008, ADR 0013, ADR 0014, and ADR 0022. +**Figma File ID:** N/A — this increment changes a Rust service crate and has no user-interface surface. +**Storybook inventory:** N/A — no reusable web object or interaction changed. + +## Context + +GAP-003A records that immutable evidence cannot yet be submitted to a durable +validation run that produces operator-usable scientific acceptance evidence. +`validation_core` already computes RMSE, bias, interval coverage, temporal-order +accuracy, and SE-aware gates, but those metrics remain library-level. An +accepted analysis run is a transport receipt; it must never carry scientific +results. Operators therefore have no hash-stable identity that binds cutoff-safe +evidence, model, seed, backend, and precision to one completion that emits +scientific acceptance evidence. + +Postgres persistence, restart/recovery, and Compose E2E remain GAP-003B. + +## Decision + +Add a bounded `analysis_engine` validation-run executor: + +- `submit_validation_run` binds sorted cutoff-eligible evidence identities, + snapshot, knowledge cutoff, model `validation_cpu_f64_v1`, seed, backend + `cpu`, and precision `f64` into a canonical SHA-256 digest. The durable + `run_id` is `tepp-validation-{32 hex}`. The receipt carries no RMSE, bias, + coverage, or gate fields. +- `complete_validation_run` rebinds the same scientific identity, refuses + LLM-authored recovery, computes `validation_core` recovery metrics, applies + the SE-aware gate `|RMSE − 0| ≤ k · SE(RMSE)`, and emits + `tepp.scientific_acceptance.v1` under output profile + `scientific_acceptance_v1`. +- Empty corpora, duplicate evidence identities, snapshot mismatch, invalid + profiles, non-finite inputs, and cutoff-empty eligibility fail closed. +- A computed recovery that fails the SE-aware gate still emits evidence with + `se_gate_accepted = false` so operators can read the metrics. Invalid or + LLM-authored recovery never emits evidence. +- The engine does not persist rows, claim implemented-main, or replace ADR 0014 + exact-head claim promotion. + +## Alternatives considered + +1. Keep metrics library-only — rejected because GAP-003A is the operator-visible + product-completion gap and has no live implementation PR. +2. Return scientific metrics on `AnalysisRunAccepted` — rejected because + accepted/running receipts must never carry scientific results. +3. Persist the run in PostgreSQL in this slice — rejected because durable + storage, restart, and Compose recovery belong to GAP-003B / issue #287. +4. Bind evidence, cutoff, model, seed, backend, and precision in + `analysis_engine` and emit scientific acceptance evidence through + `validation_core` — accepted because it is independently testable and does + not weaken fail-closed gates. + +## Consequences + +Operators can submit one immutable evidence snapshot and later complete it with +known-truth recovery to obtain a digest-bound scientific acceptance artifact. +The run identity is hash-stable for the scientific binding, independent of +evidence input order and of ineligible post-cutoff units. Persistence, HTTP +ingress, and release promotion remain later slices. LLM output still cannot +become scientific authority. + +## Verification + +The stacked PR includes unit and integration tests for hash-stable identity, +cutoff exclusion, metric-free receipts, SE-aware accept and refuse, and +fail-closed LLM, NaN, empty, duplicate, snapshot, profile, and cutoff-empty +paths. Run: + +```text +cargo fmt --all -- --check +cargo test -p analysis_engine +cargo clippy -p analysis_engine --all-targets -- -D warnings +``` + +Doctoring and APA 7th citations are in +`docs/research/validation-run-scientific-acceptance.md`. + +## Rollback and supersession + +Rollback removes `validation_run` from `analysis_engine` and stops exporting +the scientific-acceptance artifact. No persisted schema migration is +introduced. Supersession requires a new ADR if execution changes cutoff +semantics, binding identity, LLM-refusal, or scientific estimands. diff --git a/docs/adr/README.md b/docs/adr/README.md index 1254c8079..b068bad74 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -30,6 +30,7 @@ Read [`ADR_POLICY.md`](ADR_POLICY.md) first. **Decision status and implementatio | [0022](0022-deterministic-analysis-run-execution.md) | Deterministic cutoff-safe analysis-run execution | Accepted | active-PR | Closes the first executable product path from accepted run to digest-bound terminal result without claiming estimator authority. | | [0024](0024-lineage-pair-criterion-and-project-journey-posterior.md) | Independent Event Lineage pair criterion and posterior Project Journey | Proposed | active-PR | Strict artifacts preserve criterion/event-time draws, branches, ties, and CPU/GPU receipts without claiming the scientific estimator is complete. | | [0025](0025-macos-native-rust-mlx-metal-boundary.md) | macOS-native Rust-owned MLX Metal execution | Accepted | accepted-target | Compose authenticates to a native host service; Linux never claims Metal, and actual backend/parity receipts fail closed. | +| [0026](0026-validation-run-scientific-acceptance.md) | Durable validation-run scientific acceptance evidence | Accepted | active-PR | GAP-003A first slice binds cutoff-safe evidence to a hash-stable run that emits `tepp.scientific_acceptance.v1`; Postgres persistence remains GAP-003B. | | [0023](0023-lineage-criterion-anchor-contract.md) | TEPP-owned Event Lineage criterion anchor | Accepted | active-PR | PR #237 publishes the strict accepted/rejected artifact and identities; estimator execution remains fail-closed future work. | | [0024](0024-independent-topic-importance-anchor.md) | Posterior topic-context producer contract | Accepted | contract-only active-PR | Strict DTO/schema only; the current estimator does not emit it. fast-mlsirm owns case-deletion influence. | | [0001](0001-rust-first-modular-msa.md) | Rust-first numerical core and CPU `f64` reference | Accepted | partial | ADR 0011 owns cross-service/MSA authority; 0001 retains numerical/backend authority. | diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index 5ea0c95bf..d86947e63 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -248,7 +248,7 @@ the coordinated visual workspace, or a supported multi-tenant release. |---|---|---|---|---|---|---|---| | GAP-001 | Submission produces a durable accepted receipt, and the deterministic terminal-result lifecycle is now implemented-main. | `implemented-main` | closed on protected main | `340087494b0a` lineage merged through [PR #157](https://github.com/ContextualWisdomLab/TEPP/pull/157) (merged 2026-08-25T02:53Z); [#156](https://github.com/ContextualWisdomLab/TEPP/issues/156) CLOSED | — | — | Exact request/result/snapshot/cutoff/model/profile binding, typed terminal failures, deterministic retrieval, and cutoff-safe execution are protected-main behavior as of the #157 merge. | | GAP-002 | LineageWeave and other modular consumers can rely on the complete protected-main HTTP evidence/result boundary. | `partial` | consumer hardening remains | Terminal-result lifecycle implemented-main via #157; versioned API contract intact | [#156](https://github.com/ContextualWisdomLab/TEPP/issues/156) (closed) / [PR #155](https://github.com/ContextualWisdomLab/TEPP/pull/155) (merged) | — | Remaining work is consumer-side adoption evidence and any versioned-contract drift discovered during integration; core boundary is no longer the gap it was. | -| GAP-003A | Immutable evidence cannot yet be submitted to a durable validation run that produces operator-usable scientific acceptance evidence. | `accepted-target` | product-completion | `e65cd66` (validation metrics are library-level only) | [#166](https://github.com/ContextualWisdomLab/TEPP/issues/166) | `—` (issue program; no current implementation PR) | Compose/CLI/API execution must bind immutable evidence, cutoffs, model configuration, validation metrics, and reproducibility manifests to one idempotent run. | +| GAP-003A | Immutable evidence cannot yet be submitted to a durable validation run that produces operator-usable scientific acceptance evidence. | `accepted-target` | product-completion | `e65cd66` (validation metrics are library-level only) | [#166](https://github.com/ContextualWisdomLab/TEPP/issues/166) | this PR (`feat/validation-run-scientific-acceptance`; library-level binding; not implemented-main; not Compose persistence) | Compose/CLI/API execution must bind immutable evidence, cutoffs, model configuration, validation metrics, and reproducibility manifests to one idempotent run. | | GAP-003B | Scientific result artifacts cannot yet be persisted, restarted, and recovered as one supported operator workflow. | `accepted-target` | product-completion | `e65cd66` (persistence contracts lack E2E recovery) | [#166](https://github.com/ContextualWisdomLab/TEPP/issues/166) | `—` (issue program; no current implementation PR) | Durable storage, migration/rollback, restart/recovery, artifact digest verification, and terminal retrieval must pass against a real Compose deployment. | | GAP-003C | The persistence slice classifies concurrent-write SQLSTATEs, but has no measured hot-partition detection, routing, or mitigation for tenant/result workloads. | `accepted-target` | product-completion | `e65cd66` (conflict classification only; no measured partition control) | [#166](https://github.com/ContextualWisdomLab/TEPP/issues/166) | `—` (issue program; no current implementation PR) | A real Compose/PostgreSQL workload identifies hot keys and partition skew, applies bounded tenant/time or result routing without weakening 3NF or temporal authority, and proves conflict rate, latency, recovery, and migration/rollback behavior under load. | | GAP-004 | The central shared-latent temporal/relational topic estimator is absent. | `partial` | product vertical | CPU `f64` TRSL-TM reference estimator with ALR/ILR coordinates and refusal gates is implemented-main (v0.2.0 `topic_measurement`); fitted candidate-`K` scoring present | [#167](https://github.com/ContextualWisdomLab/TEPP/issues/167) | — | GPU, method effects, full Bayesian sampling, and topic birth/split/merge remain. This is not full #167 closure. | diff --git a/docs/research/validation-run-scientific-acceptance.md b/docs/research/validation-run-scientific-acceptance.md new file mode 100644 index 000000000..4bc06125c --- /dev/null +++ b/docs/research/validation-run-scientific-acceptance.md @@ -0,0 +1,32 @@ +# Validation-run scientific acceptance evidence + +## Scope + +This note doctors the first GAP-003A slice in `analysis_engine`: + +1. immutable evidence, knowledge cutoff, model, seed, backend, and precision bind to one hash-stable validation run; +2. the accepted receipt carries no scientific metrics; +3. completion emits `tepp.scientific_acceptance.v1` with RMSE, bias, interval coverage (Wilson bounds), temporal-order accuracy, and an SE-aware gate; +4. LLM-authored recovery, non-finite inputs, empty or duplicate evidence, snapshot mismatch, and cutoff-empty corpora fail closed. + +Postgres persistence, restart/recovery, and Compose execution remain GAP-003B. This slice is not implemented-main. + +## Authoritative sources + +National Academies of Sciences, Engineering, and Medicine. (2019). *Reproducibility and replicability in science*. The National Academies Press. https://doi.org/10.17226/25303 + +Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on *p*-values: Context, process, and purpose. *The American Statistician, 70*(2), 129–133. https://doi.org/10.1080/00031305.2016.1154108 + +Wilson, E. B. (1927). Probable inference, the law of succession, and statistical inference. *Journal of the American Statistical Association, 22*(158), 209–212. https://doi.org/10.1080/01621459.1927.10502953 + +## Application + +The National Academies (2019) separate computational reproducibility (same binding, same digest) from a scientific claim that recovery is correct. Wasserstein and Lazar (2016) refuse to treat a passing threshold as automatic scientific authority, so a failed SE-aware gate still reports metrics with `se_gate_accepted = false` rather than inventing a passing claim. Wilson (1927) supplies the coverage interval bounds already implemented in `validation_core`. TEPP therefore binds cutoff-safe evidence before any metric is computed, refuses LLM-authored recovery, and reports RMSE, bias, coverage, temporal order, and the SE-aware gate as operator-usable evidence (National Academies of Sciences, Engineering, and Medicine, 2019; Wasserstein & Lazar, 2016; Wilson, 1927). + +## Verification + +- identical eligible evidence in any order and with extra post-cutoff units yields the same `tepp-validation-{32 hex}` identity; +- receipts serialize without RMSE, bias, or coverage fields; +- known-truth recovery emits `tepp.scientific_acceptance.v1` with a digest-stable JSON body; +- a large residual vector remains operator-readable with `se_gate_accepted = false`; +- LLM authorship, NaN recovery, empty corpora, duplicates, snapshot mismatch, wrong profile/model, and cutoff-empty eligibility return dedicated fail-closed errors. From f51d0424b6f6e66db874304daa60bf62a814768f Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Mon, 31 Aug 2026 07:36:49 +0000 Subject: [PATCH 2/6] docs(gap): record GAP-003A current PR #356 Point the GAP-003A Current head SHA column at exact PR #356 f8a4dad. Library-level binding only; not implemented-main. --- docs/product-technical-gap-baseline.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index d86947e63..ee650024e 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -248,7 +248,7 @@ the coordinated visual workspace, or a supported multi-tenant release. |---|---|---|---|---|---|---|---| | GAP-001 | Submission produces a durable accepted receipt, and the deterministic terminal-result lifecycle is now implemented-main. | `implemented-main` | closed on protected main | `340087494b0a` lineage merged through [PR #157](https://github.com/ContextualWisdomLab/TEPP/pull/157) (merged 2026-08-25T02:53Z); [#156](https://github.com/ContextualWisdomLab/TEPP/issues/156) CLOSED | — | — | Exact request/result/snapshot/cutoff/model/profile binding, typed terminal failures, deterministic retrieval, and cutoff-safe execution are protected-main behavior as of the #157 merge. | | GAP-002 | LineageWeave and other modular consumers can rely on the complete protected-main HTTP evidence/result boundary. | `partial` | consumer hardening remains | Terminal-result lifecycle implemented-main via #157; versioned API contract intact | [#156](https://github.com/ContextualWisdomLab/TEPP/issues/156) (closed) / [PR #155](https://github.com/ContextualWisdomLab/TEPP/pull/155) (merged) | — | Remaining work is consumer-side adoption evidence and any versioned-contract drift discovered during integration; core boundary is no longer the gap it was. | -| GAP-003A | Immutable evidence cannot yet be submitted to a durable validation run that produces operator-usable scientific acceptance evidence. | `accepted-target` | product-completion | `e65cd66` (validation metrics are library-level only) | [#166](https://github.com/ContextualWisdomLab/TEPP/issues/166) | this PR (`feat/validation-run-scientific-acceptance`; library-level binding; not implemented-main; not Compose persistence) | Compose/CLI/API execution must bind immutable evidence, cutoffs, model configuration, validation metrics, and reproducibility manifests to one idempotent run. | +| GAP-003A | Immutable evidence cannot yet be submitted to a durable validation run that produces operator-usable scientific acceptance evidence. | `accepted-target` | product-completion | `e65cd66` (validation metrics are library-level only) | [#166](https://github.com/ContextualWisdomLab/TEPP/issues/166) | [#356](https://github.com/ContextualWisdomLab/TEPP/pull/356) `f8a4dad` (`feat/validation-run-scientific-acceptance`; library-level binding; not implemented-main; not Compose persistence) | Compose/CLI/API execution must bind immutable evidence, cutoffs, model configuration, validation metrics, and reproducibility manifests to one idempotent run. | | GAP-003B | Scientific result artifacts cannot yet be persisted, restarted, and recovered as one supported operator workflow. | `accepted-target` | product-completion | `e65cd66` (persistence contracts lack E2E recovery) | [#166](https://github.com/ContextualWisdomLab/TEPP/issues/166) | `—` (issue program; no current implementation PR) | Durable storage, migration/rollback, restart/recovery, artifact digest verification, and terminal retrieval must pass against a real Compose deployment. | | GAP-003C | The persistence slice classifies concurrent-write SQLSTATEs, but has no measured hot-partition detection, routing, or mitigation for tenant/result workloads. | `accepted-target` | product-completion | `e65cd66` (conflict classification only; no measured partition control) | [#166](https://github.com/ContextualWisdomLab/TEPP/issues/166) | `—` (issue program; no current implementation PR) | A real Compose/PostgreSQL workload identifies hot keys and partition skew, applies bounded tenant/time or result routing without weakening 3NF or temporal authority, and proves conflict rate, latency, recovery, and migration/rollback behavior under load. | | GAP-004 | The central shared-latent temporal/relational topic estimator is absent. | `partial` | product vertical | CPU `f64` TRSL-TM reference estimator with ALR/ILR coordinates and refusal gates is implemented-main (v0.2.0 `topic_measurement`); fitted candidate-`K` scoring present | [#167](https://github.com/ContextualWisdomLab/TEPP/issues/167) | — | GPU, method effects, full Bayesian sampling, and topic birth/split/merge remain. This is not full #167 closure. | From c5315e5b34f55dc87493f38a72c585e41fc00ef9 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Mon, 31 Aug 2026 07:52:35 +0000 Subject: [PATCH 3/6] fix(analysis): stamp recovery to the validation-run binding Refuse completion when recovery vectors belong to a different run, tenant, seed, or eligible evidence set. Include tenant workspace and output profile in the canonical digest so a tampered profile cannot pass. Bound recovery vector length and record a SHA-256 of the stamped vectors on tepp.scientific_acceptance.v1. Evidence fields stay private after completion. This is still the GAP-003A library slice (ADR 0026). It is not implemented-main and does not persist runs (GAP-003B). --- CHANGELOG.md | 2 +- crates/analysis_engine/src/lib.rs | 2 +- crates/analysis_engine/src/validation_run.rs | 370 ++++++++++++++---- .../tests/validation_run_contract.rs | 47 +++ docs/TRACEABILITY.md | 2 +- ...26-validation-run-scientific-acceptance.md | 22 +- .../validation-run-scientific-acceptance.md | 12 +- 7 files changed, 374 insertions(+), 83 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 31b32a03f..05a984036 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -38,7 +38,7 @@ All notable changes to TEPP are documented here. The format follows Keep a Chang ## [Unreleased] -- `analysis_engine` GAP-003A first slice (ADR 0026, active-PR, not implemented-main): `submit_validation_run` binds cutoff-eligible evidence identities, snapshot, knowledge cutoff, CPU `f64` model, seed, backend, and precision to a hash-stable `tepp-validation-{32 hex}` receipt that carries no scientific metrics. `complete_validation_run` emits `tepp.scientific_acceptance.v1` with RMSE, bias, Wilson coverage, temporal-order accuracy, and an SE-aware gate through `validation_core`. LLM-authored recovery, non-finite inputs, empty or duplicate evidence, snapshot mismatch, and cutoff-empty corpora fail closed. Postgres persistence remains GAP-003B. +- `analysis_engine` GAP-003A first slice (ADR 0026, active-PR, not implemented-main): `submit_validation_run` binds cutoff-eligible evidence identities, tenant workspace, snapshot, knowledge cutoff, CPU `f64` model, seed, backend, precision, and output profile to a hash-stable `tepp-validation-{32 hex}` receipt that carries no scientific metrics. `complete_validation_run` requires recovery vectors stamped to that receipt, records a SHA-256 of those vectors, and emits `tepp.scientific_acceptance.v1` with RMSE, bias, Wilson coverage, temporal-order accuracy, and an SE-aware gate through `validation_core`. LLM-authored recovery, a different run/tenant/seed/evidence set, a tampered output profile, oversized vectors, non-finite inputs, empty or duplicate evidence, snapshot mismatch, and cutoff-empty corpora fail closed. Postgres persistence remains GAP-003B. - `event_core` adds bounded Allen interval-consistency classification, atomic path-consistency closure, contradiction/resource refusals, and an explicit dependency-error fallback without claiming unrestricted global satisfiability. diff --git a/crates/analysis_engine/src/lib.rs b/crates/analysis_engine/src/lib.rs index 4c5e4536f..8fe96cd76 100644 --- a/crates/analysis_engine/src/lib.rs +++ b/crates/analysis_engine/src/lib.rs @@ -62,7 +62,7 @@ pub use topic_lineage_artifact::{ }; /// Durable scientific-acceptance validation-run contracts. pub use validation_run::{ - RecoveryObservation, SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE, + MAX_RECOVERY_VECTOR_LEN, RecoveryObservation, SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE, SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION, ScientificAcceptanceEvidence, VALIDATION_BACKEND, VALIDATION_CPU_F64_MODEL, VALIDATION_PRECISION, VALIDATION_RUN_ID_HEX_LEN, VALIDATION_RUN_ID_PREFIX, ValidationRunReceipt, WILSON_Z, complete_validation_run, diff --git a/crates/analysis_engine/src/validation_run.rs b/crates/analysis_engine/src/validation_run.rs index 1af31b6ea..c91d8f41f 100644 --- a/crates/analysis_engine/src/validation_run.rs +++ b/crates/analysis_engine/src/validation_run.rs @@ -1,11 +1,14 @@ //! Bind immutable evidence to a durable scientific-acceptance validation run. //! -//! This is the first GAP-003A slice: cutoff-safe evidence, model, seed, backend, -//! and precision hash to one durable run identity. The accepted receipt carries -//! no scientific metrics. Completion asks `validation_core` for RMSE, bias, -//! coverage, temporal-order accuracy, and an SE-aware gate, then emits -//! `tepp.scientific_acceptance.v1`. LLM-authored recovery, non-finite inputs, -//! empty or duplicate evidence, snapshot mismatch, and cutoff-empty corpora +//! This is the first GAP-003A slice: cutoff-safe evidence, tenant workspace, +//! output profile, model, seed, backend, and precision hash to one durable run +//! identity. The accepted receipt carries no scientific metrics. Completion +//! asks `validation_core` for RMSE, bias, coverage, temporal-order accuracy, +//! and an SE-aware gate, then emits `tepp.scientific_acceptance.v1`. Recovery +//! vectors must be stamped with that same run identity; a different run, +//! model, snapshot, seed, tenant, profile, or eligible evidence set fails +//! closed. LLM-authored recovery, non-finite inputs, empty or duplicate +//! evidence, snapshot mismatch, oversized recovery, and cutoff-empty corpora //! fail closed. Postgres persistence remains GAP-003B. use crate::{AnalysisCorpus, AnalysisEngineError, valid_identifier}; @@ -37,6 +40,8 @@ pub const VALIDATION_RUN_ID_PREFIX: &str = "tepp-validation-"; pub const VALIDATION_RUN_ID_HEX_LEN: usize = 32; /// Wilson critical value for nominal 95% coverage bounds. pub const WILSON_Z: f64 = 1.96; +/// Maximum length of one recovery, interval, or event-time vector. +pub const MAX_RECOVERY_VECTOR_LEN: usize = 10_000; /// Durable identity of one submitted validation run. Receipts never carry /// scientific metrics. @@ -46,6 +51,8 @@ pub struct ValidationRunReceipt { pub run_id: String, /// Lowercase SHA-256 of the canonical binding. pub binding_sha256: String, + /// Tenant workspace that owns the run. + pub tenant_workspace_id: String, /// Immutable source snapshot identity. pub snapshot_id: String, /// Historical cutoff applied to availability. @@ -87,6 +94,12 @@ impl ValidationRunReceipt { &self.binding_sha256 } + /// Return the bound tenant workspace identity. + #[must_use] + pub fn tenant_workspace_id(&self) -> &str { + &self.tenant_workspace_id + } + /// Return the bound snapshot identity. #[must_use] pub fn snapshot_id(&self) -> &str { @@ -134,11 +147,30 @@ impl ValidationRunReceipt { pub const fn eligible_evidence_count(&self) -> u64 { self.eligible_evidence_count } + + fn identity_record(&self) -> String { + format!( + "{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}", + self.run_id, + self.binding_sha256, + self.tenant_workspace_id, + self.snapshot_id, + self.knowledge_cutoff, + self.model, + self.seed, + self.backend, + self.precision, + self.output_profile, + self.eligible_evidence_count + ) + } } /// Known-truth recovery vectors offered to complete a validation run. #[derive(Clone, Debug, PartialEq)] pub struct RecoveryObservation { + run_id: String, + binding_sha256: String, study_label: String, truth: Vec, recovered: Vec, @@ -151,19 +183,24 @@ pub struct RecoveryObservation { } impl RecoveryObservation { - /// Construct a recovery observation. + /// Construct a recovery observation stamped to one submitted receipt. /// /// LLM authorship is recorded here and refused at completion. Non-finite - /// `se_gate_k` or a negative multiplier fail closed immediately. + /// `se_gate_k`, a negative multiplier, or an oversized vector fail closed + /// immediately. Completing a different run, model, snapshot, seed, tenant, + /// profile, or eligible evidence set fails later as a binding mismatch. /// /// # Errors /// /// Returns [`AnalysisEngineError::InvalidEvidence`] for an empty, oversized, - /// or control-bearing study label, and + /// or control-bearing study label, + /// [`AnalysisEngineError::LimitExceeded`] when any vector exceeds + /// [`MAX_RECOVERY_VECTOR_LEN`], and /// [`AnalysisEngineError::Validation`] for a non-finite or negative SE /// multiplier. #[allow(clippy::too_many_arguments)] pub fn new( + receipt: &ValidationRunReceipt, study_label: impl Into, truth: Vec, recovered: Vec, @@ -178,6 +215,19 @@ impl RecoveryObservation { if !valid_identifier(&study_label) { return Err(AnalysisEngineError::InvalidEvidence); } + if [ + truth.len(), + recovered.len(), + interval_lower.len(), + interval_upper.len(), + truth_times.len(), + recovered_times.len(), + ] + .into_iter() + .any(|len| len > MAX_RECOVERY_VECTOR_LEN) + { + return Err(AnalysisEngineError::LimitExceeded); + } if !se_gate_k.is_finite() { return Err(AnalysisEngineError::Validation( ValidationError::InvalidInput, @@ -189,6 +239,8 @@ impl RecoveryObservation { )); } Ok(Self { + run_id: receipt.run_id.clone(), + binding_sha256: receipt.binding_sha256.clone(), study_label, truth, recovered, @@ -201,6 +253,18 @@ impl RecoveryObservation { }) } + /// Return the stamped run identity. + #[must_use] + pub fn run_id(&self) -> &str { + &self.run_id + } + + /// Return the stamped binding digest. + #[must_use] + pub fn binding_sha256(&self) -> &str { + &self.binding_sha256 + } + /// Return the study label. #[must_use] pub fn study_label(&self) -> &str { @@ -254,39 +318,43 @@ impl RecoveryObservation { pub const fn authored_by_llm(&self) -> bool { self.authored_by_llm } + + fn digest_hex(&self) -> String { + let mut canonical = String::from("tepp.recovery_observation.v1\n"); + let _ = writeln!(canonical, "run_id={}", self.run_id); + let _ = writeln!(canonical, "binding={}", self.binding_sha256); + let _ = writeln!(canonical, "study={}", self.study_label); + let _ = writeln!(canonical, "se_gate_k={:016x}", self.se_gate_k.to_bits()); + let _ = writeln!(canonical, "authored_by_llm={}", self.authored_by_llm); + append_f64_vector(&mut canonical, "truth", &self.truth); + append_f64_vector(&mut canonical, "recovered", &self.recovered); + append_f64_vector(&mut canonical, "interval_lower", &self.interval_lower); + append_f64_vector(&mut canonical, "interval_upper", &self.interval_upper); + append_f64_vector(&mut canonical, "truth_times", &self.truth_times); + append_f64_vector(&mut canonical, "recovered_times", &self.recovered_times); + format_hex(Sha256::digest(canonical.into_bytes())) + } } /// Operator-usable scientific acceptance evidence for one completed run. #[derive(Clone, Debug, PartialEq, Serialize)] pub struct ScientificAcceptanceEvidence { - /// Versioned artifact schema. - pub schema_version: String, - /// Durable run identity. - pub run_id: String, - /// Canonical binding digest. - pub binding_sha256: String, - /// Immutable source snapshot identity. - pub snapshot_id: String, - /// Historical cutoff applied to availability. - pub knowledge_cutoff: String, - /// Bound model identity. - pub model: String, - /// Bound numeric seed. - pub seed: u64, - /// Bound compute backend. - pub backend: String, - /// Bound numeric precision. - pub precision: String, - /// Output profile. - pub output_profile: String, - /// Number of cutoff-eligible evidence identities. - pub eligible_evidence_count: u64, - /// Whether RMSE toward 0 passed the SE-aware gate. - pub se_gate_accepted: bool, - /// SE-gate multiplier used for acceptance. - pub se_gate_k: f64, - /// Machine-readable recovery report. - pub report: ValidationReport, + schema_version: String, + run_id: String, + binding_sha256: String, + recovery_sha256: String, + tenant_workspace_id: String, + snapshot_id: String, + knowledge_cutoff: String, + model: String, + seed: u64, + backend: String, + precision: String, + output_profile: String, + eligible_evidence_count: u64, + se_gate_accepted: bool, + se_gate_k: f64, + report: ValidationReport, } impl ScientificAcceptanceEvidence { @@ -330,20 +398,94 @@ impl ScientificAcceptanceEvidence { &self.run_id } + /// Return the scientific binding digest. + #[must_use] + pub fn binding_sha256(&self) -> &str { + &self.binding_sha256 + } + + /// Return the recovery-vector digest stamped into this artifact. + #[must_use] + pub fn recovery_sha256(&self) -> &str { + &self.recovery_sha256 + } + + /// Return the tenant workspace identity. + #[must_use] + pub fn tenant_workspace_id(&self) -> &str { + &self.tenant_workspace_id + } + + /// Return the snapshot identity. + #[must_use] + pub fn snapshot_id(&self) -> &str { + &self.snapshot_id + } + + /// Return the knowledge cutoff. + #[must_use] + pub fn knowledge_cutoff(&self) -> &str { + &self.knowledge_cutoff + } + + /// Return the bound model identity. + #[must_use] + pub fn model(&self) -> &str { + &self.model + } + + /// Return the bound seed. + #[must_use] + pub const fn seed(&self) -> u64 { + self.seed + } + + /// Return the bound backend. + #[must_use] + pub fn backend(&self) -> &str { + &self.backend + } + + /// Return the bound precision. + #[must_use] + pub fn precision(&self) -> &str { + &self.precision + } + + /// Return the output profile. + #[must_use] + pub fn output_profile(&self) -> &str { + &self.output_profile + } + + /// Return the eligible evidence count. + #[must_use] + pub const fn eligible_evidence_count(&self) -> u64 { + self.eligible_evidence_count + } + /// Return whether the SE-aware gate accepted RMSE toward 0. #[must_use] pub const fn se_gate_accepted(&self) -> bool { self.se_gate_accepted } + + /// Return the SE-gate multiplier used for acceptance. + #[must_use] + pub const fn se_gate_k(&self) -> f64 { + self.se_gate_k + } } struct CanonicalBinding { + tenant_workspace_id: String, snapshot_id: String, knowledge_cutoff: String, model: String, seed: u64, backend: String, precision: String, + output_profile: String, eligible_ids: Vec, } @@ -366,12 +508,14 @@ impl CanonicalBinding { fn canonical_bytes(&self) -> String { let mut canonical = String::from("tepp.validation_binding.v1\n"); + let _ = writeln!(canonical, "tenant={}", self.tenant_workspace_id); let _ = writeln!(canonical, "snapshot={}", self.snapshot_id); let _ = writeln!(canonical, "cutoff={}", self.knowledge_cutoff); let _ = writeln!(canonical, "model={}", self.model); let _ = writeln!(canonical, "seed={}", self.seed); let _ = writeln!(canonical, "backend={}", self.backend); let _ = writeln!(canonical, "precision={}", self.precision); + let _ = writeln!(canonical, "profile={}", self.output_profile); for identity in &self.eligible_ids { let _ = writeln!(canonical, "evidence={identity}"); } @@ -380,35 +524,20 @@ impl CanonicalBinding { fn identity_record(&self, digest_hex: &str, run_id: &str) -> String { format!( - "{run_id}\n{digest_hex}\n{}\n{}\n{}\n{}\n{}\n{}\n{}", + "{run_id}\n{digest_hex}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}", + self.tenant_workspace_id, self.snapshot_id, self.knowledge_cutoff, self.model, self.seed, self.backend, self.precision, + self.output_profile, self.eligible_count() ) } } -impl ValidationRunReceipt { - fn identity_record(&self) -> String { - format!( - "{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}", - self.run_id, - self.binding_sha256, - self.snapshot_id, - self.knowledge_cutoff, - self.model, - self.seed, - self.backend, - self.precision, - self.eligible_evidence_count - ) - } -} - /// Submit immutable evidence to a durable validation run. /// /// The receipt is hash-stable for one scientific binding and never includes @@ -431,13 +560,14 @@ pub fn submit_validation_run( Ok(ValidationRunReceipt { run_id, binding_sha256, + tenant_workspace_id: binding.tenant_workspace_id, snapshot_id: binding.snapshot_id, knowledge_cutoff: binding.knowledge_cutoff, model: binding.model, seed: binding.seed, backend: binding.backend, precision: binding.precision, - output_profile: SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE.to_owned(), + output_profile: binding.output_profile, eligible_evidence_count, }) } @@ -447,8 +577,9 @@ pub fn submit_validation_run( /// # Errors /// /// Returns a fail-closed engine error when recovery was LLM-authored, the -/// receipt does not match the rebound scientific identity, metric inputs are -/// invalid, or report validation fails. +/// observation is stamped to a different run identity, the receipt does not +/// match the rebound scientific identity, metric inputs are invalid, or +/// report validation fails. pub fn complete_validation_run( receipt: &ValidationRunReceipt, request: &AnalysisRunRequest, @@ -458,6 +589,12 @@ pub fn complete_validation_run( if observation.authored_by_llm { return Err(AnalysisEngineError::LlmAuthoredRecovery); } + if observation.run_id != receipt.run_id + || observation.binding_sha256 != receipt.binding_sha256 + || receipt.output_profile != SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE + { + return Err(AnalysisEngineError::BindingMismatch); + } let binding = bind_validation_run(request, corpus, receipt.seed)?; let binding_sha256 = binding.digest_hex(); let run_id = CanonicalBinding::run_id(&binding_sha256); @@ -476,6 +613,8 @@ pub fn complete_validation_run( schema_version: SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION.to_owned(), run_id, binding_sha256, + recovery_sha256: observation.digest_hex(), + tenant_workspace_id: binding.tenant_workspace_id, snapshot_id: binding.snapshot_id, knowledge_cutoff: binding.knowledge_cutoff, model: binding.model, @@ -528,12 +667,14 @@ fn bind_validation_run( return Err(AnalysisEngineError::NoEligibleEvidence); } Ok(CanonicalBinding { + tenant_workspace_id: request.tenant_workspace_id.clone(), snapshot_id: request.snapshot_id.clone(), knowledge_cutoff: cutoff.to_rfc3339(), model: VALIDATION_CPU_F64_MODEL.to_owned(), seed, backend: VALIDATION_BACKEND.to_owned(), precision: VALIDATION_PRECISION.to_owned(), + output_profile: SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE.to_owned(), eligible_ids: eligible.into_iter().collect(), }) } @@ -574,6 +715,13 @@ fn compute_report( Ok(report) } +fn append_f64_vector(canonical: &mut String, label: &str, values: &[f64]) { + let _ = writeln!(canonical, "{label}_len={}", values.len()); + for value in values { + let _ = writeln!(canonical, "{label}={:016x}", value.to_bits()); + } +} + fn format_hex(digest: impl AsRef<[u8]>) -> String { let bytes = digest.as_ref(); let mut output = String::with_capacity(bytes.len() * 2); @@ -586,7 +734,7 @@ fn format_hex(digest: impl AsRef<[u8]>) -> String { #[cfg(test)] mod tests { use super::{ - RecoveryObservation, SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE, + MAX_RECOVERY_VECTOR_LEN, RecoveryObservation, SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE, SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION, VALIDATION_BACKEND, VALIDATION_CPU_F64_MODEL, VALIDATION_PRECISION, VALIDATION_RUN_ID_PREFIX, WILSON_Z, complete_validation_run, submit_validation_run, @@ -624,12 +772,18 @@ mod tests { AnalysisCorpus::new("snapshot-1", units).expect("corpus") } - fn recovery(truth: Vec, recovered: Vec, k: f64) -> RecoveryObservation { + fn recovery( + receipt: &super::ValidationRunReceipt, + truth: Vec, + recovered: Vec, + k: f64, + ) -> RecoveryObservation { let times = [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0]; let times = times[..truth.len()].to_vec(); let lower: Vec = truth.iter().map(|value| value - 0.5).collect(); let upper: Vec = truth.iter().map(|value| value + 0.5).collect(); RecoveryObservation::new( + receipt, "foundation-recovery", truth, recovered, @@ -644,11 +798,13 @@ mod tests { } fn two_point( + receipt: &super::ValidationRunReceipt, label: impl Into, k: f64, authored_by_llm: bool, ) -> Result { RecoveryObservation::new( + receipt, label, vec![1.0, 2.0], vec![1.0, 2.0], @@ -682,6 +838,7 @@ mod tests { VALIDATION_RUN_ID_PREFIX.len() + 32 ); assert_eq!(receipt_a.eligible_evidence_count(), 2); + assert_eq!(receipt_a.tenant_workspace_id(), "tenant-workspace-1"); assert_eq!(receipt_a.snapshot_id(), "snapshot-1"); assert_eq!(receipt_a.knowledge_cutoff(), "2026-08-01T00:00:00Z"); assert_eq!(receipt_a.model(), VALIDATION_CPU_F64_MODEL); @@ -698,6 +855,10 @@ mod tests { assert!(!json.contains("coverage")); let other_seed = submit_validation_run(&request(), &first, 8).expect("seed"); assert_ne!(receipt_a.run_id(), other_seed.run_id()); + let mut other_tenant = request(); + other_tenant.tenant_workspace_id = "tenant-workspace-2".into(); + let other_tenant_receipt = submit_validation_run(&other_tenant, &first, 7).expect("tenant"); + assert_ne!(receipt_a.run_id(), other_tenant_receipt.run_id()); } #[test] @@ -708,10 +869,13 @@ mod tests { ]); let receipt = submit_validation_run(&request(), &offered, 11).expect("submit"); let observation = recovery( + &receipt, vec![0.70, 0.55, 0.40, -0.20, 0.85], vec![0.70, 0.55, 0.40, -0.20, 0.85], 3.0, ); + assert_eq!(observation.run_id(), receipt.run_id()); + assert_eq!(observation.binding_sha256(), receipt.binding_sha256()); assert_eq!(observation.study_label(), "foundation-recovery"); assert_eq!(observation.truth().len(), 5); assert_eq!(observation.recovered().len(), 5); @@ -728,20 +892,31 @@ mod tests { SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION ); assert_eq!(evidence.run_id(), receipt.run_id()); + assert_eq!(evidence.binding_sha256(), receipt.binding_sha256()); + assert_eq!(evidence.recovery_sha256().len(), 64); + assert_eq!(evidence.tenant_workspace_id(), "tenant-workspace-1"); assert!(evidence.se_gate_accepted()); assert_eq!( - evidence.output_profile, + evidence.output_profile(), SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE ); - assert_eq!(evidence.eligible_evidence_count, 2); - assert!((evidence.se_gate_k - 3.0).abs() < f64::EPSILON); + assert_eq!(evidence.eligible_evidence_count(), 2); + assert!((evidence.se_gate_k() - 3.0).abs() < f64::EPSILON); + assert_eq!(evidence.snapshot_id(), "snapshot-1"); + assert_eq!(evidence.knowledge_cutoff(), "2026-08-01T00:00:00Z"); + assert_eq!(evidence.model(), VALIDATION_CPU_F64_MODEL); + assert_eq!(evidence.seed(), 11); + assert_eq!(evidence.backend(), VALIDATION_BACKEND); + assert_eq!(evidence.precision(), VALIDATION_PRECISION); assert!((WILSON_Z - 1.96).abs() < f64::EPSILON); let json = evidence.to_json().expect("json"); assert!(json.contains(SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION)); assert!(json.contains("rmse")); + assert!(json.contains("recovery_sha256")); assert_eq!(evidence.sha256().expect("digest").len(), 64); assert!(evidence.to_human_summary().contains("foundation-recovery")); let rejected = recovery( + &receipt, vec![0.0, 1.0, 2.0, 3.0, 4.0], vec![10.0, 11.0, 12.0, 13.0, 14.0], 1.0, @@ -749,6 +924,7 @@ mod tests { let refused = complete_validation_run(&receipt, &request(), &offered, &rejected).expect("refused"); assert!(!refused.se_gate_accepted()); + assert_ne!(refused.recovery_sha256(), evidence.recovery_sha256()); } #[test] @@ -815,6 +991,7 @@ mod tests { ]); let receipt = submit_validation_run(&request(), &offered, 3).expect("submit"); let llm = RecoveryObservation::new( + &receipt, "llm-study", vec![1.0, 2.0, 3.0], vec![1.0, 2.0, 3.0], @@ -833,11 +1010,46 @@ mod tests { ); let mut tampered = receipt.clone(); tampered.run_id = "tepp-validation-deadbeefdeadbeefdeadbeefdeadbeef".into(); - let observation = recovery(vec![1.0, 2.0, 3.0], vec![1.0, 2.0, 3.0], 3.0); + let observation = recovery(&receipt, vec![1.0, 2.0, 3.0], vec![1.0, 2.0, 3.0], 3.0); assert_eq!( complete_validation_run(&tampered, &request(), &offered, &observation), Err(AnalysisEngineError::BindingMismatch) ); + let mut profile_tampered = receipt.clone(); + profile_tampered.output_profile = "tampered-profile".into(); + assert_eq!( + complete_validation_run(&profile_tampered, &request(), &offered, &observation), + Err(AnalysisEngineError::BindingMismatch) + ); + } + + #[test] + fn complete_refuses_recovery_stamped_to_a_different_run() { + let first = corpus(vec![unit("evidence-a", "2026-07-10T00:00:00Z")]); + let second = corpus(vec![ + unit("evidence-a", "2026-07-10T00:00:00Z"), + unit("evidence-b", "2026-07-15T00:00:00Z"), + ]); + let receipt_a = submit_validation_run(&request(), &first, 3).expect("a"); + let receipt_b = submit_validation_run(&request(), &second, 3).expect("b"); + assert_ne!(receipt_a.run_id(), receipt_b.run_id()); + let foreign = recovery(&receipt_a, vec![1.0, 2.0, 3.0], vec![1.0, 2.0, 3.0], 3.0); + assert_eq!( + complete_validation_run(&receipt_b, &request(), &second, &foreign), + Err(AnalysisEngineError::BindingMismatch) + ); + let other_seed = submit_validation_run(&request(), &first, 9).expect("seed"); + assert_eq!( + complete_validation_run(&other_seed, &request(), &first, &foreign), + Err(AnalysisEngineError::BindingMismatch) + ); + let mut other_tenant = request(); + other_tenant.tenant_workspace_id = "tenant-workspace-other".into(); + let other_tenant_receipt = submit_validation_run(&other_tenant, &first, 3).expect("tenant"); + assert_eq!( + complete_validation_run(&other_tenant_receipt, &other_tenant, &first, &foreign), + Err(AnalysisEngineError::BindingMismatch) + ); } #[test] @@ -848,6 +1060,7 @@ mod tests { ]); let receipt = submit_validation_run(&request(), &offered, 3).expect("submit"); let nan = RecoveryObservation::new( + &receipt, "nan-study", vec![1.0, 2.0, 3.0], vec![f64::NAN, 2.0, 3.0], @@ -866,25 +1079,46 @@ mod tests { )) ); assert_eq!( - two_point("", 3.0, false), + two_point(&receipt, "", 3.0, false), Err(AnalysisEngineError::InvalidEvidence) ); assert_eq!( - two_point("x".repeat(MAX_ANALYSIS_IDENTIFIER_BYTES + 1), 3.0, false), + two_point( + &receipt, + "x".repeat(MAX_ANALYSIS_IDENTIFIER_BYTES + 1), + 3.0, + false + ), Err(AnalysisEngineError::InvalidEvidence) ); assert_eq!( - two_point("nan-k", f64::NAN, false), + two_point(&receipt, "nan-k", f64::NAN, false), Err(AnalysisEngineError::Validation( ValidationError::InvalidInput )) ); assert_eq!( - two_point("neg-k", -1.0, false), + two_point(&receipt, "neg-k", -1.0, false), Err(AnalysisEngineError::Validation( ValidationError::InvalidConfiguration )) ); + let oversized = vec![0.0; MAX_RECOVERY_VECTOR_LEN + 1]; + assert_eq!( + RecoveryObservation::new( + &receipt, + "oversized", + oversized.clone(), + oversized.clone(), + oversized.clone(), + oversized.clone(), + oversized.clone(), + oversized, + 3.0, + false, + ), + Err(AnalysisEngineError::LimitExceeded) + ); let converted: AnalysisEngineError = ValidationError::InvalidInput.into(); assert_eq!(converted.to_string(), "invalid validation input"); } diff --git a/crates/analysis_engine/tests/validation_run_contract.rs b/crates/analysis_engine/tests/validation_run_contract.rs index 1ebb164d5..82d97896d 100644 --- a/crates/analysis_engine/tests/validation_run_contract.rs +++ b/crates/analysis_engine/tests/validation_run_contract.rs @@ -45,6 +45,7 @@ fn durable_run_binds_cutoff_eligible_evidence_and_emits_acceptance_evidence() { assert_eq!(receipt.eligible_evidence_count(), 2); assert!(!receipt.to_json().expect("json").contains("rmse")); let observation = RecoveryObservation::new( + &receipt, "contract-recovery", vec![0.70, 0.55, 0.40, -0.20, 0.85], vec![0.70, 0.55, 0.40, -0.20, 0.85], @@ -63,6 +64,7 @@ fn durable_run_binds_cutoff_eligible_evidence_and_emits_acceptance_evidence() { SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION ); assert_eq!(evidence.run_id(), receipt.run_id()); + assert_eq!(evidence.recovery_sha256().len(), 64); assert!(evidence.se_gate_accepted()); assert!(evidence.to_json().expect("json").contains("rmse")); } @@ -85,6 +87,7 @@ fn llm_authored_recovery_and_cutoff_empty_corpora_fail_closed() { .expect("eligible"); let receipt = submit_validation_run(&request(), &eligible, 1).expect("submit"); let llm = RecoveryObservation::new( + &receipt, "llm-contract", vec![1.0, 2.0, 3.0], vec![1.0, 2.0, 3.0], @@ -101,3 +104,47 @@ fn llm_authored_recovery_and_cutoff_empty_corpora_fail_closed() { Err(AnalysisEngineError::LlmAuthoredRecovery) ); } + +#[test] +fn recovery_from_a_different_run_or_tenant_fails_closed() { + let first = AnalysisCorpus::new( + "snapshot-contract", + vec![unit("evidence-1", "2026-07-10T00:00:00Z")], + ) + .expect("first"); + let second = AnalysisCorpus::new( + "snapshot-contract", + vec![ + unit("evidence-1", "2026-07-10T00:00:00Z"), + unit("evidence-2", "2026-07-20T00:00:00Z"), + ], + ) + .expect("second"); + let receipt_a = submit_validation_run(&request(), &first, 1).expect("a"); + let receipt_b = submit_validation_run(&request(), &second, 1).expect("b"); + let foreign = RecoveryObservation::new( + &receipt_a, + "foreign-recovery", + vec![1.0, 2.0, 3.0], + vec![1.0, 2.0, 3.0], + vec![0.5, 1.5, 2.5], + vec![1.5, 2.5, 3.5], + vec![1.0, 2.0, 3.0], + vec![1.0, 2.0, 3.0], + 3.0, + false, + ) + .expect("foreign"); + assert_eq!( + complete_validation_run(&receipt_b, &request(), &second, &foreign), + Err(AnalysisEngineError::BindingMismatch) + ); + let mut other_tenant = request(); + other_tenant.tenant_workspace_id = "tenant-workspace-other".into(); + let other_receipt = submit_validation_run(&other_tenant, &first, 1).expect("tenant"); + assert_ne!(receipt_a.run_id(), other_receipt.run_id()); + assert_eq!( + complete_validation_run(&other_receipt, &other_tenant, &first, &foreign), + Err(AnalysisEngineError::BindingMismatch) + ); +} diff --git a/docs/TRACEABILITY.md b/docs/TRACEABILITY.md index c2ab5d13f..6768092ec 100644 --- a/docs/TRACEABILITY.md +++ b/docs/TRACEABILITY.md @@ -50,7 +50,7 @@ The full APA 7th standards/literature register remains `docs/research/standards- | leakage-safe availability/cutoff snapshots | PRD; ADR 0002/0013 | `corpus_split` on protected main | implemented-main | | Unicode NFC/NFD canonical identity for split leakage | ADR 0004/0008/0013; UAX #15 | `corpus_split` canonical-equivalence links on active PR #59 | active-PR | | recovery metrics (RMSE, bias, coverage, graph, temporal order, Monte Carlo SE gates) | PRD; Test Strategy; ADR 0007/0014 | `validation_core` on protected main (PR #19); SE-aware Monte Carlo gates included | implemented-main | -| durable validation-run scientific acceptance evidence | ADR 0026; ADR 0002/0008/0014; GAP-003A; National Academies (2019); Wasserstein & Lazar (2016); Wilson (1927) | `analysis_engine` `submit_validation_run` / `complete_validation_run` on this PR: hash-stable cutoff-safe binding, metric-free receipt, `tepp.scientific_acceptance.v1` RMSE/bias/coverage/temporal-order/SE-aware evidence; LLM/NaN/empty/duplicate/snapshot/cutoff-empty fail closed; not implemented-main; Postgres persistence remains GAP-003B | active-PR | +| durable validation-run scientific acceptance evidence | ADR 0026; ADR 0002/0008/0014; GAP-003A; National Academies (2019); Wasserstein & Lazar (2016); Wilson (1927) | `analysis_engine` `submit_validation_run` / `complete_validation_run` on this PR: hash-stable cutoff-safe tenant/profile binding, metric-free receipt, recovery stamped to that `run_id`, `tepp.scientific_acceptance.v1` RMSE/bias/coverage/temporal-order/SE-aware evidence with recovery digest; LLM/NaN/empty/duplicate/snapshot/foreign-run/tampered-profile/oversized/cutoff-empty fail closed; not implemented-main; Postgres persistence remains GAP-003B | active-PR | | PostgreSQL bitemporal/lineage persistence | ADR 0013; Architecture/ERD | `persistence_postgres` migration contracts, in-memory adapters, live SQL session/document SQL port, tenant RLS (`0002` + session GUC/role helpers), `DATABASE_URL` SQLx gate, optional session-affine `live-sqlx` `PgPool` driver, exact-head live PostgreSQL CI with isolation proof, append-only immutability triggers (`0004`), temporal interval ordering CHECKs (`0005`), typed membership assignment (`0006` implemented-main), event-relation/mention/instance SQL (#37–#39 implemented-main), source-artifact SQL (#40 implemented-main), audit-event SQL (#41 implemented-main), concurrent document-write stress (#43 implemented-main), backup/restore integrity revalidation (#44 implemented-main), `revision_order` later-revision system-time ordering implemented-main, entity/project target SQL on PR #131; remaining physical ERD constraints | partial | | known-truth temporal/event simulation manifests | PRD; TRD; Test Strategy | `tepp_simulation` on protected main; recovery metrics in `validation_core` | implemented-main | | versioned service/API contracts and exports | PRD; API contract; ADR 0011/0013 | `tepp_api` analysis-run/export/JSON-LD/GraphML contracts on protected main (PR #21); request-bound terminal result active in PR #157; HTTP service remains accepted-target; the `orchestrator_live` loopback interpretation listener is on this PR | partial | diff --git a/docs/adr/0026-validation-run-scientific-acceptance.md b/docs/adr/0026-validation-run-scientific-acceptance.md index e769539f2..c692eb29a 100644 --- a/docs/adr/0026-validation-run-scientific-acceptance.md +++ b/docs/adr/0026-validation-run-scientific-acceptance.md @@ -25,17 +25,22 @@ Postgres persistence, restart/recovery, and Compose E2E remain GAP-003B. Add a bounded `analysis_engine` validation-run executor: - `submit_validation_run` binds sorted cutoff-eligible evidence identities, - snapshot, knowledge cutoff, model `validation_cpu_f64_v1`, seed, backend - `cpu`, and precision `f64` into a canonical SHA-256 digest. The durable + tenant workspace, snapshot, knowledge cutoff, model `validation_cpu_f64_v1`, + seed, backend `cpu`, precision `f64`, and output profile + `scientific_acceptance_v1` into a canonical SHA-256 digest. The durable `run_id` is `tepp-validation-{32 hex}`. The receipt carries no RMSE, bias, coverage, or gate fields. -- `complete_validation_run` rebinds the same scientific identity, refuses +- `complete_validation_run` rebinds the same scientific identity, requires + recovery vectors stamped to that `run_id` and binding digest, refuses LLM-authored recovery, computes `validation_core` recovery metrics, applies the SE-aware gate `|RMSE − 0| ≤ k · SE(RMSE)`, and emits `tepp.scientific_acceptance.v1` under output profile - `scientific_acceptance_v1`. + `scientific_acceptance_v1`. The artifact records a SHA-256 of the stamped + recovery vectors. Evidence fields are private after completion. - Empty corpora, duplicate evidence identities, snapshot mismatch, invalid - profiles, non-finite inputs, and cutoff-empty eligibility fail closed. + profiles, non-finite inputs, oversized recovery vectors, a different run / + tenant / seed / eligible evidence set, a tampered output profile, and + cutoff-empty eligibility fail closed. - A computed recovery that fails the SE-aware gate still emits evidence with `se_gate_accepted = false` so operators can read the metrics. Invalid or LLM-authored recovery never emits evidence. @@ -67,9 +72,10 @@ become scientific authority. ## Verification The stacked PR includes unit and integration tests for hash-stable identity, -cutoff exclusion, metric-free receipts, SE-aware accept and refuse, and -fail-closed LLM, NaN, empty, duplicate, snapshot, profile, and cutoff-empty -paths. Run: +cutoff exclusion, metric-free receipts, SE-aware accept and refuse, recovery +stamped to a foreign run or tenant, a tampered output profile, oversized +vectors, and fail-closed LLM, NaN, empty, duplicate, snapshot, profile, and +cutoff-empty paths. Run: ```text cargo fmt --all -- --check diff --git a/docs/research/validation-run-scientific-acceptance.md b/docs/research/validation-run-scientific-acceptance.md index 4bc06125c..287c0bfdc 100644 --- a/docs/research/validation-run-scientific-acceptance.md +++ b/docs/research/validation-run-scientific-acceptance.md @@ -4,10 +4,11 @@ This note doctors the first GAP-003A slice in `analysis_engine`: -1. immutable evidence, knowledge cutoff, model, seed, backend, and precision bind to one hash-stable validation run; +1. immutable evidence, tenant workspace, knowledge cutoff, model, seed, backend, precision, and output profile bind to one hash-stable validation run; 2. the accepted receipt carries no scientific metrics; -3. completion emits `tepp.scientific_acceptance.v1` with RMSE, bias, interval coverage (Wilson bounds), temporal-order accuracy, and an SE-aware gate; -4. LLM-authored recovery, non-finite inputs, empty or duplicate evidence, snapshot mismatch, and cutoff-empty corpora fail closed. +3. completion emits `tepp.scientific_acceptance.v1` with RMSE, bias, interval coverage (Wilson bounds), temporal-order accuracy, an SE-aware gate, and a SHA-256 of the stamped recovery vectors; +4. recovery vectors must be constructed against that receipt; a different run, tenant, seed, snapshot, profile, or eligible evidence set fails closed; +5. LLM-authored recovery, non-finite inputs, empty or duplicate evidence, snapshot mismatch, oversized recovery, and cutoff-empty corpora fail closed. Postgres persistence, restart/recovery, and Compose execution remain GAP-003B. This slice is not implemented-main. @@ -21,7 +22,7 @@ Wilson, E. B. (1927). Probable inference, the law of succession, and statistical ## Application -The National Academies (2019) separate computational reproducibility (same binding, same digest) from a scientific claim that recovery is correct. Wasserstein and Lazar (2016) refuse to treat a passing threshold as automatic scientific authority, so a failed SE-aware gate still reports metrics with `se_gate_accepted = false` rather than inventing a passing claim. Wilson (1927) supplies the coverage interval bounds already implemented in `validation_core`. TEPP therefore binds cutoff-safe evidence before any metric is computed, refuses LLM-authored recovery, and reports RMSE, bias, coverage, temporal order, and the SE-aware gate as operator-usable evidence (National Academies of Sciences, Engineering, and Medicine, 2019; Wasserstein & Lazar, 2016; Wilson, 1927). +The National Academies (2019) separate computational reproducibility (same binding, same digest) from a scientific claim that recovery is correct. Wasserstein and Lazar (2016) refuse to treat a passing threshold as automatic scientific authority, so a failed SE-aware gate still reports metrics with `se_gate_accepted = false` rather than inventing a passing claim. Wilson (1927) supplies the coverage interval bounds already implemented in `validation_core`. TEPP therefore binds cutoff-safe evidence and tenant/profile identity before any metric is computed, stamps recovery vectors to that binding, refuses LLM-authored recovery, and reports RMSE, bias, coverage, temporal order, and the SE-aware gate as operator-usable evidence (National Academies of Sciences, Engineering, and Medicine, 2019; Wasserstein & Lazar, 2016; Wilson, 1927). Meredith (1993) remains unread (Unpaywall/OpenAlex 2026-08-31T07:45Z: `is_oa: false`, 0 locations). Mislevy (1991, *Psychometrika, 56*, 177–196) remains unread on the same terms (DOI `10.1007/bf02294457`). ## Verification @@ -29,4 +30,7 @@ The National Academies (2019) separate computational reproducibility (same bindi - receipts serialize without RMSE, bias, or coverage fields; - known-truth recovery emits `tepp.scientific_acceptance.v1` with a digest-stable JSON body; - a large residual vector remains operator-readable with `se_gate_accepted = false`; +- recovery stamped to a different run, tenant, or seed returns `BindingMismatch`; +- a tampered receipt `output_profile` returns `BindingMismatch`; +- an oversized recovery vector returns `LimitExceeded`; - LLM authorship, NaN recovery, empty corpora, duplicates, snapshot mismatch, wrong profile/model, and cutoff-empty eligibility return dedicated fail-closed errors. From 791660336458cf99176e6bd3019d5fcb03dabf7c Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Mon, 31 Aug 2026 08:06:26 +0000 Subject: [PATCH 4/6] fix(analysis): pre-register the validation SE-gate multiplier GAP-003A Devin repair on #356. se_gate_k is part of the submitted scientific binding, not a completion-time choice. Changing k changes run identity. k must be finite, non-negative, and at most MAX_SE_GATE_K = 8. RecoveryObservation must match the receipt k; post-hoc or oversized multipliers fail closed. Receipt fields are private. Empty and length-mismatched recovery vectors fail at construction. Wasserstein and Lazar (2016) refuse post-hoc threshold shopping. Not implemented-main. Postgres persistence remains GAP-003B. ADR 0026. --- CHANGELOG.md | 2 +- crates/analysis_engine/src/lib.rs | 10 +- crates/analysis_engine/src/validation_run.rs | 272 ++++++++++++------ .../tests/validation_run_contract.rs | 30 +- docs/TRACEABILITY.md | 2 +- ...26-validation-run-scientific-acceptance.md | 27 +- .../validation-run-scientific-acceptance.md | 16 +- 7 files changed, 241 insertions(+), 118 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 05a984036..ccf1bacdb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -38,7 +38,7 @@ All notable changes to TEPP are documented here. The format follows Keep a Chang ## [Unreleased] -- `analysis_engine` GAP-003A first slice (ADR 0026, active-PR, not implemented-main): `submit_validation_run` binds cutoff-eligible evidence identities, tenant workspace, snapshot, knowledge cutoff, CPU `f64` model, seed, backend, precision, and output profile to a hash-stable `tepp-validation-{32 hex}` receipt that carries no scientific metrics. `complete_validation_run` requires recovery vectors stamped to that receipt, records a SHA-256 of those vectors, and emits `tepp.scientific_acceptance.v1` with RMSE, bias, Wilson coverage, temporal-order accuracy, and an SE-aware gate through `validation_core`. LLM-authored recovery, a different run/tenant/seed/evidence set, a tampered output profile, oversized vectors, non-finite inputs, empty or duplicate evidence, snapshot mismatch, and cutoff-empty corpora fail closed. Postgres persistence remains GAP-003B. +- `analysis_engine` GAP-003A first slice (ADR 0026, active-PR, not implemented-main): `submit_validation_run` binds cutoff-eligible evidence identities, tenant workspace, snapshot, knowledge cutoff, CPU `f64` model, seed, backend, precision, output profile, and a pre-registered SE-gate multiplier `k` (`0 ≤ k ≤ MAX_SE_GATE_K` = 8) to a hash-stable `tepp-validation-{32 hex}` receipt that carries no scientific metrics. Receipt fields are private. `complete_validation_run` requires recovery vectors stamped to that receipt with the same `k`, records a SHA-256 of those vectors, and emits `tepp.scientific_acceptance.v1` with RMSE, bias, Wilson coverage, temporal-order accuracy, and an SE-aware gate through `validation_core` using the submitted `k`. LLM-authored recovery, a different run/tenant/seed/evidence set, a post-hoc or oversized `k`, a tampered output profile, oversized/empty/mismatched vectors, non-finite inputs, empty or duplicate evidence, snapshot mismatch, and cutoff-empty corpora fail closed. Postgres persistence remains GAP-003B. - `event_core` adds bounded Allen interval-consistency classification, atomic path-consistency closure, contradiction/resource refusals, and an explicit dependency-error fallback without claiming unrestricted global satisfiability. diff --git a/crates/analysis_engine/src/lib.rs b/crates/analysis_engine/src/lib.rs index 8fe96cd76..d0d544ba4 100644 --- a/crates/analysis_engine/src/lib.rs +++ b/crates/analysis_engine/src/lib.rs @@ -62,11 +62,11 @@ pub use topic_lineage_artifact::{ }; /// Durable scientific-acceptance validation-run contracts. pub use validation_run::{ - MAX_RECOVERY_VECTOR_LEN, RecoveryObservation, SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE, - SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION, ScientificAcceptanceEvidence, VALIDATION_BACKEND, - VALIDATION_CPU_F64_MODEL, VALIDATION_PRECISION, VALIDATION_RUN_ID_HEX_LEN, - VALIDATION_RUN_ID_PREFIX, ValidationRunReceipt, WILSON_Z, complete_validation_run, - submit_validation_run, + MAX_RECOVERY_VECTOR_LEN, MAX_SE_GATE_K, RecoveryObservation, + SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE, SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION, + ScientificAcceptanceEvidence, VALIDATION_BACKEND, VALIDATION_CPU_F64_MODEL, + VALIDATION_PRECISION, VALIDATION_RUN_ID_HEX_LEN, VALIDATION_RUN_ID_PREFIX, + ValidationRunReceipt, WILSON_Z, complete_validation_run, submit_validation_run, }; /// Versioned artifact schema emitted by this engine. diff --git a/crates/analysis_engine/src/validation_run.rs b/crates/analysis_engine/src/validation_run.rs index c91d8f41f..2a31ae445 100644 --- a/crates/analysis_engine/src/validation_run.rs +++ b/crates/analysis_engine/src/validation_run.rs @@ -1,15 +1,16 @@ //! Bind immutable evidence to a durable scientific-acceptance validation run. //! //! This is the first GAP-003A slice: cutoff-safe evidence, tenant workspace, -//! output profile, model, seed, backend, and precision hash to one durable run -//! identity. The accepted receipt carries no scientific metrics. Completion -//! asks `validation_core` for RMSE, bias, coverage, temporal-order accuracy, -//! and an SE-aware gate, then emits `tepp.scientific_acceptance.v1`. Recovery -//! vectors must be stamped with that same run identity; a different run, -//! model, snapshot, seed, tenant, profile, or eligible evidence set fails -//! closed. LLM-authored recovery, non-finite inputs, empty or duplicate -//! evidence, snapshot mismatch, oversized recovery, and cutoff-empty corpora -//! fail closed. Postgres persistence remains GAP-003B. +//! output profile, model, seed, backend, precision, and the SE-gate multiplier +//! hash to one durable run identity. The accepted receipt carries no scientific +//! metrics. Completion asks `validation_core` for RMSE, bias, coverage, +//! temporal-order accuracy, and an SE-aware gate, then emits +//! `tepp.scientific_acceptance.v1`. Recovery vectors must be stamped with that +//! same run identity and the pre-registered multiplier; a different run, model, +//! snapshot, seed, tenant, profile, eligible evidence set, or post-hoc `k` +//! fails closed. LLM-authored recovery, non-finite inputs, empty or duplicate +//! evidence, snapshot mismatch, oversized recovery, oversized `k`, and +//! cutoff-empty corpora fail closed. Postgres persistence remains GAP-003B. use crate::{AnalysisCorpus, AnalysisEngineError, valid_identifier}; use serde::Serialize; @@ -42,33 +43,29 @@ pub const VALIDATION_RUN_ID_HEX_LEN: usize = 32; pub const WILSON_Z: f64 = 1.96; /// Maximum length of one recovery, interval, or event-time vector. pub const MAX_RECOVERY_VECTOR_LEN: usize = 10_000; +/// Largest finite SE-gate multiplier that may be pre-registered on a run. +/// +/// Conventional three-SE gates sit inside this bound. A larger `k` would make +/// `|RMSE| ≤ k · SE(RMSE)` an effectively unlimited post-hoc acceptance rule. +pub const MAX_SE_GATE_K: f64 = 8.0; /// Durable identity of one submitted validation run. Receipts never carry -/// scientific metrics. -#[derive(Clone, Debug, Eq, PartialEq, Serialize)] +/// scientific metrics. Fields are private so callers cannot rewrite the +/// binding or the pre-registered SE-gate multiplier after submit. +#[derive(Clone, Debug, PartialEq, Serialize)] pub struct ValidationRunReceipt { - /// Hash-stable run identity `tepp-validation-{32 hex}`. - pub run_id: String, - /// Lowercase SHA-256 of the canonical binding. - pub binding_sha256: String, - /// Tenant workspace that owns the run. - pub tenant_workspace_id: String, - /// Immutable source snapshot identity. - pub snapshot_id: String, - /// Historical cutoff applied to availability. - pub knowledge_cutoff: String, - /// Bound model identity. - pub model: String, - /// Bound numeric seed. - pub seed: u64, - /// Bound compute backend. - pub backend: String, - /// Bound numeric precision. - pub precision: String, - /// Requested output profile. - pub output_profile: String, - /// Number of cutoff-eligible evidence identities. - pub eligible_evidence_count: u64, + run_id: String, + binding_sha256: String, + tenant_workspace_id: String, + snapshot_id: String, + knowledge_cutoff: String, + model: String, + seed: u64, + backend: String, + precision: String, + output_profile: String, + eligible_evidence_count: u64, + se_gate_k: f64, } impl ValidationRunReceipt { @@ -148,9 +145,15 @@ impl ValidationRunReceipt { self.eligible_evidence_count } + /// Return the pre-registered SE-gate multiplier. + #[must_use] + pub const fn se_gate_k(&self) -> f64 { + self.se_gate_k + } + fn identity_record(&self) -> String { format!( - "{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}", + "{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{:016x}", self.run_id, self.binding_sha256, self.tenant_workspace_id, @@ -161,7 +164,8 @@ impl ValidationRunReceipt { self.backend, self.precision, self.output_profile, - self.eligible_evidence_count + self.eligible_evidence_count, + self.se_gate_k.to_bits() ) } } @@ -185,18 +189,19 @@ pub struct RecoveryObservation { impl RecoveryObservation { /// Construct a recovery observation stamped to one submitted receipt. /// - /// LLM authorship is recorded here and refused at completion. Non-finite - /// `se_gate_k`, a negative multiplier, or an oversized vector fail closed - /// immediately. Completing a different run, model, snapshot, seed, tenant, - /// profile, or eligible evidence set fails later as a binding mismatch. + /// LLM authorship is recorded here and refused at completion. The SE-gate + /// multiplier must equal the pre-registered receipt value; a post-hoc `k` + /// is a binding mismatch. Empty, length-mismatched, or oversized vectors + /// fail closed immediately. /// /// # Errors /// /// Returns [`AnalysisEngineError::InvalidEvidence`] for an empty, oversized, - /// or control-bearing study label, + /// or control-bearing study label or for empty/mismatched vectors, /// [`AnalysisEngineError::LimitExceeded`] when any vector exceeds - /// [`MAX_RECOVERY_VECTOR_LEN`], and - /// [`AnalysisEngineError::Validation`] for a non-finite or negative SE + /// [`MAX_RECOVERY_VECTOR_LEN`], [`AnalysisEngineError::BindingMismatch`] + /// when `se_gate_k` differs from the receipt, and + /// [`AnalysisEngineError::Validation`] for a non-finite or out-of-policy SE /// multiplier. #[allow(clippy::too_many_arguments)] pub fn new( @@ -215,28 +220,24 @@ impl RecoveryObservation { if !valid_identifier(&study_label) { return Err(AnalysisEngineError::InvalidEvidence); } - if [ + let lengths = [ truth.len(), recovered.len(), interval_lower.len(), interval_upper.len(), truth_times.len(), recovered_times.len(), - ] - .into_iter() - .any(|len| len > MAX_RECOVERY_VECTOR_LEN) - { + ]; + if lengths.iter().any(|len| *len > MAX_RECOVERY_VECTOR_LEN) { return Err(AnalysisEngineError::LimitExceeded); } - if !se_gate_k.is_finite() { - return Err(AnalysisEngineError::Validation( - ValidationError::InvalidInput, - )); + let n = truth.len(); + if n == 0 || lengths.iter().any(|len| *len != n) { + return Err(AnalysisEngineError::InvalidEvidence); } - if se_gate_k < 0.0 { - return Err(AnalysisEngineError::Validation( - ValidationError::InvalidConfiguration, - )); + let se_gate_k = require_se_gate_k(se_gate_k)?; + if se_gate_k.to_bits() != receipt.se_gate_k.to_bits() { + return Err(AnalysisEngineError::BindingMismatch); } Ok(Self { run_id: receipt.run_id.clone(), @@ -486,6 +487,7 @@ struct CanonicalBinding { backend: String, precision: String, output_profile: String, + se_gate_k: f64, eligible_ids: Vec, } @@ -516,6 +518,7 @@ impl CanonicalBinding { let _ = writeln!(canonical, "backend={}", self.backend); let _ = writeln!(canonical, "precision={}", self.precision); let _ = writeln!(canonical, "profile={}", self.output_profile); + let _ = writeln!(canonical, "se_gate_k={:016x}", self.se_gate_k.to_bits()); for identity in &self.eligible_ids { let _ = writeln!(canonical, "evidence={identity}"); } @@ -524,7 +527,7 @@ impl CanonicalBinding { fn identity_record(&self, digest_hex: &str, run_id: &str) -> String { format!( - "{run_id}\n{digest_hex}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}", + "{run_id}\n{digest_hex}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{}\n{:016x}", self.tenant_workspace_id, self.snapshot_id, self.knowledge_cutoff, @@ -533,7 +536,8 @@ impl CanonicalBinding { self.backend, self.precision, self.output_profile, - self.eligible_count() + self.eligible_count(), + self.se_gate_k.to_bits() ) } } @@ -546,14 +550,15 @@ impl CanonicalBinding { /// # Errors /// /// Returns a fail-closed engine error for an invalid request, wrong output -/// profile or model, snapshot mismatch, duplicate or empty evidence, or a -/// cutoff that admits no evidence. +/// profile or model, an out-of-policy SE-gate multiplier, snapshot mismatch, +/// duplicate or empty evidence, or a cutoff that admits no evidence. pub fn submit_validation_run( request: &AnalysisRunRequest, corpus: &AnalysisCorpus, seed: u64, + se_gate_k: f64, ) -> Result { - let binding = bind_validation_run(request, corpus, seed)?; + let binding = bind_validation_run(request, corpus, seed, se_gate_k)?; let binding_sha256 = binding.digest_hex(); let run_id = CanonicalBinding::run_id(&binding_sha256); let eligible_evidence_count = binding.eligible_count(); @@ -569,6 +574,7 @@ pub fn submit_validation_run( precision: binding.precision, output_profile: binding.output_profile, eligible_evidence_count, + se_gate_k: binding.se_gate_k, }) } @@ -591,11 +597,12 @@ pub fn complete_validation_run( } if observation.run_id != receipt.run_id || observation.binding_sha256 != receipt.binding_sha256 + || observation.se_gate_k.to_bits() != receipt.se_gate_k.to_bits() || receipt.output_profile != SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE { return Err(AnalysisEngineError::BindingMismatch); } - let binding = bind_validation_run(request, corpus, receipt.seed)?; + let binding = bind_validation_run(request, corpus, receipt.seed, receipt.se_gate_k)?; let binding_sha256 = binding.digest_hex(); let run_id = CanonicalBinding::run_id(&binding_sha256); if receipt.identity_record() != binding.identity_record(&binding_sha256, &run_id) { @@ -606,7 +613,7 @@ pub fn complete_validation_run( report.rmse, 0.0, report.rmse_standard_error, - observation.se_gate_k, + receipt.se_gate_k, )?; let eligible_evidence_count = binding.eligible_count(); let evidence = ScientificAcceptanceEvidence { @@ -624,19 +631,35 @@ pub fn complete_validation_run( output_profile: SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE.to_owned(), eligible_evidence_count, se_gate_accepted, - se_gate_k: observation.se_gate_k, + se_gate_k: receipt.se_gate_k, report, }; evidence.to_json()?; Ok(evidence) } +fn require_se_gate_k(se_gate_k: f64) -> Result { + if !se_gate_k.is_finite() { + return Err(AnalysisEngineError::Validation( + ValidationError::InvalidInput, + )); + } + if se_gate_k < 0.0 || se_gate_k > MAX_SE_GATE_K { + return Err(AnalysisEngineError::Validation( + ValidationError::InvalidConfiguration, + )); + } + Ok(if se_gate_k == 0.0 { 0.0 } else { se_gate_k }) +} + fn bind_validation_run( request: &AnalysisRunRequest, corpus: &AnalysisCorpus, seed: u64, + se_gate_k: f64, ) -> Result { request.to_json()?; + let se_gate_k = require_se_gate_k(se_gate_k)?; let requested = format!( "{}\n{}", request.output_profile, request.model_contract_version @@ -675,6 +698,7 @@ fn bind_validation_run( backend: VALIDATION_BACKEND.to_owned(), precision: VALIDATION_PRECISION.to_owned(), output_profile: SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE.to_owned(), + se_gate_k, eligible_ids: eligible.into_iter().collect(), }) } @@ -734,10 +758,10 @@ fn format_hex(digest: impl AsRef<[u8]>) -> String { #[cfg(test)] mod tests { use super::{ - MAX_RECOVERY_VECTOR_LEN, RecoveryObservation, SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE, - SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION, VALIDATION_BACKEND, VALIDATION_CPU_F64_MODEL, - VALIDATION_PRECISION, VALIDATION_RUN_ID_PREFIX, WILSON_Z, complete_validation_run, - submit_validation_run, + MAX_RECOVERY_VECTOR_LEN, MAX_SE_GATE_K, RecoveryObservation, + SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE, SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION, + VALIDATION_BACKEND, VALIDATION_CPU_F64_MODEL, VALIDATION_PRECISION, + VALIDATION_RUN_ID_PREFIX, WILSON_Z, complete_validation_run, submit_validation_run, }; use crate::{ AnalysisCorpus, AnalysisEngineError, AnalysisEvidenceUnit, MAX_ANALYSIS_IDENTIFIER_BYTES, @@ -828,8 +852,8 @@ mod tests { unit("evidence-a", "2026-07-10T00:00:00Z"), unit("evidence-b", "2026-07-15T00:00:00Z"), ]); - let receipt_a = submit_validation_run(&request(), &first, 7).expect("submit a"); - let receipt_b = submit_validation_run(&request(), &second, 7).expect("submit b"); + let receipt_a = submit_validation_run(&request(), &first, 7, 3.0).expect("submit a"); + let receipt_b = submit_validation_run(&request(), &second, 7, 3.0).expect("submit b"); assert_eq!(receipt_a.run_id(), receipt_b.run_id()); assert_eq!(receipt_a.binding_sha256(), receipt_b.binding_sha256()); assert!(receipt_a.run_id().starts_with(VALIDATION_RUN_ID_PREFIX)); @@ -849,16 +873,21 @@ mod tests { receipt_a.output_profile(), SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE ); + assert!((receipt_a.se_gate_k() - 3.0).abs() < f64::EPSILON); let json = receipt_a.to_json().expect("json"); assert!(!json.contains("rmse")); assert!(!json.contains("bias")); assert!(!json.contains("coverage")); - let other_seed = submit_validation_run(&request(), &first, 8).expect("seed"); + let other_seed = submit_validation_run(&request(), &first, 8, 3.0).expect("seed"); assert_ne!(receipt_a.run_id(), other_seed.run_id()); let mut other_tenant = request(); other_tenant.tenant_workspace_id = "tenant-workspace-2".into(); - let other_tenant_receipt = submit_validation_run(&other_tenant, &first, 7).expect("tenant"); + let other_tenant_receipt = + submit_validation_run(&other_tenant, &first, 7, 3.0).expect("tenant"); assert_ne!(receipt_a.run_id(), other_tenant_receipt.run_id()); + let other_k = submit_validation_run(&request(), &first, 7, 4.0).expect("other k"); + assert_ne!(receipt_a.run_id(), other_k.run_id()); + assert!((other_k.se_gate_k() - 4.0).abs() < f64::EPSILON); } #[test] @@ -867,7 +896,7 @@ mod tests { unit("evidence-a", "2026-07-10T00:00:00Z"), unit("evidence-b", "2026-07-15T00:00:00Z"), ]); - let receipt = submit_validation_run(&request(), &offered, 11).expect("submit"); + let receipt = submit_validation_run(&request(), &offered, 11, 3.0).expect("submit"); let observation = recovery( &receipt, vec![0.70, 0.55, 0.40, -0.20, 0.85], @@ -919,7 +948,7 @@ mod tests { &receipt, vec![0.0, 1.0, 2.0, 3.0, 4.0], vec![10.0, 11.0, 12.0, 13.0, 14.0], - 1.0, + 3.0, ); let refused = complete_validation_run(&receipt, &request(), &offered, &rejected).expect("refused"); @@ -933,13 +962,13 @@ mod tests { let mut wrong_profile = request(); wrong_profile.output_profile = "validation-report".into(); assert_eq!( - submit_validation_run(&wrong_profile, &offered, 1), + submit_validation_run(&wrong_profile, &offered, 1, 3.0), Err(AnalysisEngineError::InvalidValidationProfile) ); let mut wrong_model = request(); wrong_model.model_contract_version = "temporal-evidence-v1".into(); assert_eq!( - submit_validation_run(&wrong_model, &offered, 1), + submit_validation_run(&wrong_model, &offered, 1, 3.0), Err(AnalysisEngineError::InvalidValidationProfile) ); let mismatched = AnalysisCorpus::new( @@ -948,7 +977,7 @@ mod tests { ) .expect("other"); assert_eq!( - submit_validation_run(&request(), &mismatched, 1), + submit_validation_run(&request(), &mismatched, 1, 3.0), Err(AnalysisEngineError::SnapshotMismatch) ); let duplicate = corpus(vec![ @@ -956,29 +985,29 @@ mod tests { unit("same", "2026-07-11T00:00:00Z"), ]); assert_eq!( - submit_validation_run(&request(), &duplicate, 1), + submit_validation_run(&request(), &duplicate, 1, 3.0), Err(AnalysisEngineError::DuplicateEvidence) ); let empty = AnalysisCorpus::new("snapshot-1", Vec::new()).expect("empty"); assert_eq!( - submit_validation_run(&request(), &empty, 1), + submit_validation_run(&request(), &empty, 1, 3.0), Err(AnalysisEngineError::InvalidEvidence) ); let late = corpus(vec![unit("late", "2026-08-02T00:00:00Z")]); assert_eq!( - submit_validation_run(&request(), &late, 1), + submit_validation_run(&request(), &late, 1, 3.0), Err(AnalysisEngineError::NoEligibleEvidence) ); let mut invalid_request = request(); invalid_request.idempotency_key.clear(); assert!(matches!( - submit_validation_run(&invalid_request, &offered, 1), + submit_validation_run(&invalid_request, &offered, 1, 3.0), Err(AnalysisEngineError::Api(_)) )); let mut invalid_cutoff = request(); invalid_cutoff.knowledge_cutoff = "not-a-time".into(); assert!(matches!( - submit_validation_run(&invalid_cutoff, &offered, 1), + submit_validation_run(&invalid_cutoff, &offered, 1, 3.0), Err(AnalysisEngineError::Api(_)) )); } @@ -989,7 +1018,7 @@ mod tests { unit("evidence-a", "2026-07-10T00:00:00Z"), unit("evidence-b", "2026-07-15T00:00:00Z"), ]); - let receipt = submit_validation_run(&request(), &offered, 3).expect("submit"); + let receipt = submit_validation_run(&request(), &offered, 3, 3.0).expect("submit"); let llm = RecoveryObservation::new( &receipt, "llm-study", @@ -1021,6 +1050,12 @@ mod tests { complete_validation_run(&profile_tampered, &request(), &offered, &observation), Err(AnalysisEngineError::BindingMismatch) ); + let mut k_tampered = observation.clone(); + k_tampered.se_gate_k = 8.0; + assert_eq!( + complete_validation_run(&receipt, &request(), &offered, &k_tampered), + Err(AnalysisEngineError::BindingMismatch) + ); } #[test] @@ -1030,22 +1065,23 @@ mod tests { unit("evidence-a", "2026-07-10T00:00:00Z"), unit("evidence-b", "2026-07-15T00:00:00Z"), ]); - let receipt_a = submit_validation_run(&request(), &first, 3).expect("a"); - let receipt_b = submit_validation_run(&request(), &second, 3).expect("b"); + let receipt_a = submit_validation_run(&request(), &first, 3, 3.0).expect("a"); + let receipt_b = submit_validation_run(&request(), &second, 3, 3.0).expect("b"); assert_ne!(receipt_a.run_id(), receipt_b.run_id()); let foreign = recovery(&receipt_a, vec![1.0, 2.0, 3.0], vec![1.0, 2.0, 3.0], 3.0); assert_eq!( complete_validation_run(&receipt_b, &request(), &second, &foreign), Err(AnalysisEngineError::BindingMismatch) ); - let other_seed = submit_validation_run(&request(), &first, 9).expect("seed"); + let other_seed = submit_validation_run(&request(), &first, 9, 3.0).expect("seed"); assert_eq!( complete_validation_run(&other_seed, &request(), &first, &foreign), Err(AnalysisEngineError::BindingMismatch) ); let mut other_tenant = request(); other_tenant.tenant_workspace_id = "tenant-workspace-other".into(); - let other_tenant_receipt = submit_validation_run(&other_tenant, &first, 3).expect("tenant"); + let other_tenant_receipt = + submit_validation_run(&other_tenant, &first, 3, 3.0).expect("tenant"); assert_eq!( complete_validation_run(&other_tenant_receipt, &other_tenant, &first, &foreign), Err(AnalysisEngineError::BindingMismatch) @@ -1058,7 +1094,7 @@ mod tests { unit("evidence-a", "2026-07-10T00:00:00Z"), unit("evidence-b", "2026-07-15T00:00:00Z"), ]); - let receipt = submit_validation_run(&request(), &offered, 3).expect("submit"); + let receipt = submit_validation_run(&request(), &offered, 3, 3.0).expect("submit"); let nan = RecoveryObservation::new( &receipt, "nan-study", @@ -1122,4 +1158,66 @@ mod tests { let converted: AnalysisEngineError = ValidationError::InvalidInput.into(); assert_eq!(converted.to_string(), "invalid validation input"); } + + #[test] + fn se_gate_k_is_pre_registered_and_empty_vectors_fail_closed() { + let offered = corpus(vec![ + unit("evidence-a", "2026-07-10T00:00:00Z"), + unit("evidence-b", "2026-07-15T00:00:00Z"), + ]); + let receipt = submit_validation_run(&request(), &offered, 3, 3.0).expect("submit"); + assert_eq!( + two_point(&receipt, "huge-k", MAX_SE_GATE_K + 0.01, false), + Err(AnalysisEngineError::Validation( + ValidationError::InvalidConfiguration + )) + ); + assert_eq!( + two_point(&receipt, "post-hoc-k", 4.0, false), + Err(AnalysisEngineError::BindingMismatch) + ); + assert_eq!( + submit_validation_run(&request(), &offered, 3, MAX_SE_GATE_K + 0.01), + Err(AnalysisEngineError::Validation( + ValidationError::InvalidConfiguration + )) + ); + assert_eq!( + submit_validation_run(&request(), &offered, 3, f64::NAN), + Err(AnalysisEngineError::Validation( + ValidationError::InvalidInput + )) + ); + assert!(submit_validation_run(&request(), &offered, 3, MAX_SE_GATE_K).is_ok()); + assert_eq!( + RecoveryObservation::new( + &receipt, + "empty", + Vec::new(), + Vec::new(), + Vec::new(), + Vec::new(), + Vec::new(), + Vec::new(), + 3.0, + false, + ), + Err(AnalysisEngineError::InvalidEvidence) + ); + assert_eq!( + RecoveryObservation::new( + &receipt, + "mismatch", + vec![1.0, 2.0], + vec![1.0], + vec![0.0, 1.0], + vec![2.0, 3.0], + vec![1.0, 2.0], + vec![1.0, 2.0], + 3.0, + false, + ), + Err(AnalysisEngineError::InvalidEvidence) + ); + } } diff --git a/crates/analysis_engine/tests/validation_run_contract.rs b/crates/analysis_engine/tests/validation_run_contract.rs index 82d97896d..f7a7bc7be 100644 --- a/crates/analysis_engine/tests/validation_run_contract.rs +++ b/crates/analysis_engine/tests/validation_run_contract.rs @@ -41,8 +41,9 @@ fn durable_run_binds_cutoff_eligible_evidence_and_emits_acceptance_evidence() { ], ) .expect("corpus"); - let receipt = submit_validation_run(&request(), &corpus, 42).expect("submit"); + let receipt = submit_validation_run(&request(), &corpus, 42, 3.0).expect("submit"); assert_eq!(receipt.eligible_evidence_count(), 2); + assert!((receipt.se_gate_k() - 3.0).abs() < f64::EPSILON); assert!(!receipt.to_json().expect("json").contains("rmse")); let observation = RecoveryObservation::new( &receipt, @@ -77,7 +78,7 @@ fn llm_authored_recovery_and_cutoff_empty_corpora_fail_closed() { ) .expect("corpus"); assert_eq!( - submit_validation_run(&request(), &corpus, 1), + submit_validation_run(&request(), &corpus, 1, 3.0), Err(AnalysisEngineError::NoEligibleEvidence) ); let eligible = AnalysisCorpus::new( @@ -85,7 +86,7 @@ fn llm_authored_recovery_and_cutoff_empty_corpora_fail_closed() { vec![unit("evidence-1", "2026-07-10T00:00:00Z")], ) .expect("eligible"); - let receipt = submit_validation_run(&request(), &eligible, 1).expect("submit"); + let receipt = submit_validation_run(&request(), &eligible, 1, 3.0).expect("submit"); let llm = RecoveryObservation::new( &receipt, "llm-contract", @@ -120,8 +121,8 @@ fn recovery_from_a_different_run_or_tenant_fails_closed() { ], ) .expect("second"); - let receipt_a = submit_validation_run(&request(), &first, 1).expect("a"); - let receipt_b = submit_validation_run(&request(), &second, 1).expect("b"); + let receipt_a = submit_validation_run(&request(), &first, 1, 3.0).expect("a"); + let receipt_b = submit_validation_run(&request(), &second, 1, 3.0).expect("b"); let foreign = RecoveryObservation::new( &receipt_a, "foreign-recovery", @@ -141,10 +142,27 @@ fn recovery_from_a_different_run_or_tenant_fails_closed() { ); let mut other_tenant = request(); other_tenant.tenant_workspace_id = "tenant-workspace-other".into(); - let other_receipt = submit_validation_run(&other_tenant, &first, 1).expect("tenant"); + let other_receipt = submit_validation_run(&other_tenant, &first, 1, 3.0).expect("tenant"); assert_ne!(receipt_a.run_id(), other_receipt.run_id()); assert_eq!( complete_validation_run(&other_receipt, &other_tenant, &first, &foreign), Err(AnalysisEngineError::BindingMismatch) ); + let other_k = submit_validation_run(&request(), &first, 1, 4.0).expect("other k"); + assert_ne!(receipt_a.run_id(), other_k.run_id()); + assert_eq!( + RecoveryObservation::new( + &receipt_a, + "post-hoc-k", + vec![1.0, 2.0, 3.0], + vec![1.0, 2.0, 3.0], + vec![0.5, 1.5, 2.5], + vec![1.5, 2.5, 3.5], + vec![1.0, 2.0, 3.0], + vec![1.0, 2.0, 3.0], + 4.0, + false, + ), + Err(AnalysisEngineError::BindingMismatch) + ); } diff --git a/docs/TRACEABILITY.md b/docs/TRACEABILITY.md index 6768092ec..2bd5f446b 100644 --- a/docs/TRACEABILITY.md +++ b/docs/TRACEABILITY.md @@ -50,7 +50,7 @@ The full APA 7th standards/literature register remains `docs/research/standards- | leakage-safe availability/cutoff snapshots | PRD; ADR 0002/0013 | `corpus_split` on protected main | implemented-main | | Unicode NFC/NFD canonical identity for split leakage | ADR 0004/0008/0013; UAX #15 | `corpus_split` canonical-equivalence links on active PR #59 | active-PR | | recovery metrics (RMSE, bias, coverage, graph, temporal order, Monte Carlo SE gates) | PRD; Test Strategy; ADR 0007/0014 | `validation_core` on protected main (PR #19); SE-aware Monte Carlo gates included | implemented-main | -| durable validation-run scientific acceptance evidence | ADR 0026; ADR 0002/0008/0014; GAP-003A; National Academies (2019); Wasserstein & Lazar (2016); Wilson (1927) | `analysis_engine` `submit_validation_run` / `complete_validation_run` on this PR: hash-stable cutoff-safe tenant/profile binding, metric-free receipt, recovery stamped to that `run_id`, `tepp.scientific_acceptance.v1` RMSE/bias/coverage/temporal-order/SE-aware evidence with recovery digest; LLM/NaN/empty/duplicate/snapshot/foreign-run/tampered-profile/oversized/cutoff-empty fail closed; not implemented-main; Postgres persistence remains GAP-003B | active-PR | +| durable validation-run scientific acceptance evidence | ADR 0026; ADR 0002/0008/0014; GAP-003A; National Academies (2019); Wasserstein & Lazar (2016); Wilson (1927) | `analysis_engine` `submit_validation_run` / `complete_validation_run` on this PR: hash-stable cutoff-safe tenant/profile/`k` binding, metric-free private-field receipt, recovery stamped to that `run_id` with pre-registered `k` (`MAX_SE_GATE_K` = 8), `tepp.scientific_acceptance.v1` RMSE/bias/coverage/temporal-order/SE-aware evidence with recovery digest; LLM/NaN/empty/duplicate/snapshot/foreign-run/post-hoc-k/tampered-profile/oversized/cutoff-empty fail closed; not implemented-main; Postgres persistence remains GAP-003B | active-PR | | PostgreSQL bitemporal/lineage persistence | ADR 0013; Architecture/ERD | `persistence_postgres` migration contracts, in-memory adapters, live SQL session/document SQL port, tenant RLS (`0002` + session GUC/role helpers), `DATABASE_URL` SQLx gate, optional session-affine `live-sqlx` `PgPool` driver, exact-head live PostgreSQL CI with isolation proof, append-only immutability triggers (`0004`), temporal interval ordering CHECKs (`0005`), typed membership assignment (`0006` implemented-main), event-relation/mention/instance SQL (#37–#39 implemented-main), source-artifact SQL (#40 implemented-main), audit-event SQL (#41 implemented-main), concurrent document-write stress (#43 implemented-main), backup/restore integrity revalidation (#44 implemented-main), `revision_order` later-revision system-time ordering implemented-main, entity/project target SQL on PR #131; remaining physical ERD constraints | partial | | known-truth temporal/event simulation manifests | PRD; TRD; Test Strategy | `tepp_simulation` on protected main; recovery metrics in `validation_core` | implemented-main | | versioned service/API contracts and exports | PRD; API contract; ADR 0011/0013 | `tepp_api` analysis-run/export/JSON-LD/GraphML contracts on protected main (PR #21); request-bound terminal result active in PR #157; HTTP service remains accepted-target; the `orchestrator_live` loopback interpretation listener is on this PR | partial | diff --git a/docs/adr/0026-validation-run-scientific-acceptance.md b/docs/adr/0026-validation-run-scientific-acceptance.md index c692eb29a..8e9290a67 100644 --- a/docs/adr/0026-validation-run-scientific-acceptance.md +++ b/docs/adr/0026-validation-run-scientific-acceptance.md @@ -26,19 +26,23 @@ Add a bounded `analysis_engine` validation-run executor: - `submit_validation_run` binds sorted cutoff-eligible evidence identities, tenant workspace, snapshot, knowledge cutoff, model `validation_cpu_f64_v1`, - seed, backend `cpu`, precision `f64`, and output profile - `scientific_acceptance_v1` into a canonical SHA-256 digest. The durable + seed, backend `cpu`, precision `f64`, output profile + `scientific_acceptance_v1`, and the SE-gate multiplier `k` into a canonical + SHA-256 digest. `k` must be finite, non-negative, and at most + `MAX_SE_GATE_K` (8). Changing `k` changes the run identity. The durable `run_id` is `tepp-validation-{32 hex}`. The receipt carries no RMSE, bias, - coverage, or gate fields. + coverage, or gate-result fields. Receipt fields are private. - `complete_validation_run` rebinds the same scientific identity, requires - recovery vectors stamped to that `run_id` and binding digest, refuses - LLM-authored recovery, computes `validation_core` recovery metrics, applies - the SE-aware gate `|RMSE − 0| ≤ k · SE(RMSE)`, and emits + recovery vectors stamped to that `run_id` and binding digest with the same + pre-registered `k`, refuses LLM-authored recovery, computes + `validation_core` recovery metrics, applies the SE-aware gate + `|RMSE − 0| ≤ k · SE(RMSE)` using the submitted `k`, and emits `tepp.scientific_acceptance.v1` under output profile `scientific_acceptance_v1`. The artifact records a SHA-256 of the stamped recovery vectors. Evidence fields are private after completion. - Empty corpora, duplicate evidence identities, snapshot mismatch, invalid - profiles, non-finite inputs, oversized recovery vectors, a different run / + profiles, non-finite inputs, oversized recovery vectors, empty or + length-mismatched recovery, a post-hoc or oversized `k`, a different run / tenant / seed / eligible evidence set, a tampered output profile, and cutoff-empty eligibility fail closed. - A computed recovery that fails the SE-aware gate still emits evidence with @@ -72,10 +76,11 @@ become scientific authority. ## Verification The stacked PR includes unit and integration tests for hash-stable identity, -cutoff exclusion, metric-free receipts, SE-aware accept and refuse, recovery -stamped to a foreign run or tenant, a tampered output profile, oversized -vectors, and fail-closed LLM, NaN, empty, duplicate, snapshot, profile, and -cutoff-empty paths. Run: +pre-registered SE-gate `k`, cutoff exclusion, metric-free private receipts, +SE-aware accept and refuse, recovery stamped to a foreign run or tenant, a +post-hoc `k`, a tampered output profile, oversized or empty vectors, and +fail-closed LLM, NaN, empty, duplicate, snapshot, profile, and cutoff-empty +paths. Run: ```text cargo fmt --all -- --check diff --git a/docs/research/validation-run-scientific-acceptance.md b/docs/research/validation-run-scientific-acceptance.md index 287c0bfdc..2ce820517 100644 --- a/docs/research/validation-run-scientific-acceptance.md +++ b/docs/research/validation-run-scientific-acceptance.md @@ -4,11 +4,11 @@ This note doctors the first GAP-003A slice in `analysis_engine`: -1. immutable evidence, tenant workspace, knowledge cutoff, model, seed, backend, precision, and output profile bind to one hash-stable validation run; -2. the accepted receipt carries no scientific metrics; -3. completion emits `tepp.scientific_acceptance.v1` with RMSE, bias, interval coverage (Wilson bounds), temporal-order accuracy, an SE-aware gate, and a SHA-256 of the stamped recovery vectors; -4. recovery vectors must be constructed against that receipt; a different run, tenant, seed, snapshot, profile, or eligible evidence set fails closed; -5. LLM-authored recovery, non-finite inputs, empty or duplicate evidence, snapshot mismatch, oversized recovery, and cutoff-empty corpora fail closed. +1. immutable evidence, tenant workspace, knowledge cutoff, model, seed, backend, precision, output profile, and the SE-gate multiplier `k` bind to one hash-stable validation run; +2. the accepted receipt carries no scientific metrics and cannot be rewritten after submit; +3. completion emits `tepp.scientific_acceptance.v1` with RMSE, bias, interval coverage (Wilson bounds), temporal-order accuracy, an SE-aware gate using that pre-registered `k`, and a SHA-256 of the stamped recovery vectors; +4. recovery vectors must be constructed against that receipt with the same `k`; a different run, tenant, seed, snapshot, profile, eligible evidence set, or post-hoc `k` fails closed; +5. LLM-authored recovery, non-finite inputs, empty or duplicate evidence, snapshot mismatch, oversized recovery, oversized `k` (`MAX_SE_GATE_K` = 8), and cutoff-empty corpora fail closed. Postgres persistence, restart/recovery, and Compose execution remain GAP-003B. This slice is not implemented-main. @@ -22,15 +22,17 @@ Wilson, E. B. (1927). Probable inference, the law of succession, and statistical ## Application -The National Academies (2019) separate computational reproducibility (same binding, same digest) from a scientific claim that recovery is correct. Wasserstein and Lazar (2016) refuse to treat a passing threshold as automatic scientific authority, so a failed SE-aware gate still reports metrics with `se_gate_accepted = false` rather than inventing a passing claim. Wilson (1927) supplies the coverage interval bounds already implemented in `validation_core`. TEPP therefore binds cutoff-safe evidence and tenant/profile identity before any metric is computed, stamps recovery vectors to that binding, refuses LLM-authored recovery, and reports RMSE, bias, coverage, temporal order, and the SE-aware gate as operator-usable evidence (National Academies of Sciences, Engineering, and Medicine, 2019; Wasserstein & Lazar, 2016; Wilson, 1927). Meredith (1993) remains unread (Unpaywall/OpenAlex 2026-08-31T07:45Z: `is_oa: false`, 0 locations). Mislevy (1991, *Psychometrika, 56*, 177–196) remains unread on the same terms (DOI `10.1007/bf02294457`). +The National Academies (2019) separate computational reproducibility (same binding, same digest) from a scientific claim that recovery is correct. Wasserstein and Lazar (2016) refuse to treat a passing threshold as automatic scientific authority and refuse post-hoc threshold shopping, so `k` is pre-registered in the run binding and a failed SE-aware gate still reports metrics with `se_gate_accepted = false` rather than inventing a passing claim. Wilson (1927) supplies the coverage interval bounds already implemented in `validation_core`. TEPP therefore binds cutoff-safe evidence, tenant/profile identity, and `k` before any metric is computed, stamps recovery vectors to that binding, refuses LLM-authored or post-hoc-`k` recovery, and reports RMSE, bias, coverage, temporal order, and the SE-aware gate as operator-usable evidence (National Academies of Sciences, Engineering, and Medicine, 2019; Wasserstein & Lazar, 2016; Wilson, 1927). Meredith (1993) remains unread (Unpaywall/OpenAlex 2026-08-31T07:45Z: `is_oa: false`, 0 locations). Mislevy (1991, *Psychometrika, 56*, 177–196) remains unread on the same terms (DOI `10.1007/bf02294457`). ## Verification - identical eligible evidence in any order and with extra post-cutoff units yields the same `tepp-validation-{32 hex}` identity; +- distinct SE-gate multipliers yield distinct run identities; `k > 8` fails closed; - receipts serialize without RMSE, bias, or coverage fields; - known-truth recovery emits `tepp.scientific_acceptance.v1` with a digest-stable JSON body; - a large residual vector remains operator-readable with `se_gate_accepted = false`; -- recovery stamped to a different run, tenant, or seed returns `BindingMismatch`; +- recovery stamped to a different run, tenant, seed, or post-hoc `k` returns `BindingMismatch`; - a tampered receipt `output_profile` returns `BindingMismatch`; - an oversized recovery vector returns `LimitExceeded`; +- empty or length-mismatched recovery vectors return `InvalidEvidence`; - LLM authorship, NaN recovery, empty corpora, duplicates, snapshot mismatch, wrong profile/model, and cutoff-empty eligibility return dedicated fail-closed errors. From df33bfa3e61ae4de3dbfae16df0deac12d2f4003 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Mon, 31 Aug 2026 08:06:54 +0000 Subject: [PATCH 5/6] docs(gap): record GAP-003A pre-registered k head on #356 Point the GAP-003A Current head SHA column at exact PR #356 7916603. Pre-registered SE-gate k; not implemented-main. --- docs/product-technical-gap-baseline.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index ee650024e..c3e5a0780 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -248,7 +248,7 @@ the coordinated visual workspace, or a supported multi-tenant release. |---|---|---|---|---|---|---|---| | GAP-001 | Submission produces a durable accepted receipt, and the deterministic terminal-result lifecycle is now implemented-main. | `implemented-main` | closed on protected main | `340087494b0a` lineage merged through [PR #157](https://github.com/ContextualWisdomLab/TEPP/pull/157) (merged 2026-08-25T02:53Z); [#156](https://github.com/ContextualWisdomLab/TEPP/issues/156) CLOSED | — | — | Exact request/result/snapshot/cutoff/model/profile binding, typed terminal failures, deterministic retrieval, and cutoff-safe execution are protected-main behavior as of the #157 merge. | | GAP-002 | LineageWeave and other modular consumers can rely on the complete protected-main HTTP evidence/result boundary. | `partial` | consumer hardening remains | Terminal-result lifecycle implemented-main via #157; versioned API contract intact | [#156](https://github.com/ContextualWisdomLab/TEPP/issues/156) (closed) / [PR #155](https://github.com/ContextualWisdomLab/TEPP/pull/155) (merged) | — | Remaining work is consumer-side adoption evidence and any versioned-contract drift discovered during integration; core boundary is no longer the gap it was. | -| GAP-003A | Immutable evidence cannot yet be submitted to a durable validation run that produces operator-usable scientific acceptance evidence. | `accepted-target` | product-completion | `e65cd66` (validation metrics are library-level only) | [#166](https://github.com/ContextualWisdomLab/TEPP/issues/166) | [#356](https://github.com/ContextualWisdomLab/TEPP/pull/356) `f8a4dad` (`feat/validation-run-scientific-acceptance`; library-level binding; not implemented-main; not Compose persistence) | Compose/CLI/API execution must bind immutable evidence, cutoffs, model configuration, validation metrics, and reproducibility manifests to one idempotent run. | +| GAP-003A | Immutable evidence cannot yet be submitted to a durable validation run that produces operator-usable scientific acceptance evidence. | `accepted-target` | product-completion | `e65cd66` (validation metrics are library-level only) | [#166](https://github.com/ContextualWisdomLab/TEPP/issues/166) | [#356](https://github.com/ContextualWisdomLab/TEPP/pull/356) `7916603` (`feat/validation-run-scientific-acceptance`; pre-registered SE-gate `k`; not implemented-main; not Compose persistence) | Compose/CLI/API execution must bind immutable evidence, cutoffs, model configuration, validation metrics, and reproducibility manifests to one idempotent run. | | GAP-003B | Scientific result artifacts cannot yet be persisted, restarted, and recovered as one supported operator workflow. | `accepted-target` | product-completion | `e65cd66` (persistence contracts lack E2E recovery) | [#166](https://github.com/ContextualWisdomLab/TEPP/issues/166) | `—` (issue program; no current implementation PR) | Durable storage, migration/rollback, restart/recovery, artifact digest verification, and terminal retrieval must pass against a real Compose deployment. | | GAP-003C | The persistence slice classifies concurrent-write SQLSTATEs, but has no measured hot-partition detection, routing, or mitigation for tenant/result workloads. | `accepted-target` | product-completion | `e65cd66` (conflict classification only; no measured partition control) | [#166](https://github.com/ContextualWisdomLab/TEPP/issues/166) | `—` (issue program; no current implementation PR) | A real Compose/PostgreSQL workload identifies hot keys and partition skew, applies bounded tenant/time or result routing without weakening 3NF or temporal authority, and proves conflict rate, latency, recovery, and migration/rollback behavior under load. | | GAP-004 | The central shared-latent temporal/relational topic estimator is absent. | `partial` | product vertical | CPU `f64` TRSL-TM reference estimator with ALR/ILR coordinates and refusal gates is implemented-main (v0.2.0 `topic_measurement`); fitted candidate-`K` scoring present | [#167](https://github.com/ContextualWisdomLab/TEPP/issues/167) | — | GPU, method effects, full Bayesian sampling, and topic birth/split/merge remain. This is not full #167 closure. | From 3f9e2d8b39d2d92ee5d057a8c568087b36dbb74e Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Mon, 31 Aug 2026 10:32:42 +0000 Subject: [PATCH 6/6] feat(analysis): execute scientific acceptance on loopback lifecycle GAP-003A engine-on-loopback slice for issue #166. POST /v1/analysis-runs stays a metric-free receipt. POST /v1/analysis-runs/{run_id}/execute on ScientificAcceptanceLoopbackService runs submit_validation_run and complete_validation_run, records running then terminal, and GET returns tepp.scientific_acceptance.v1 without a caller-supplied artifact. The execute body carries corpus, recovery, seed, and pre-registered SE-gate k and refuses scientific_acceptance_json plus result-metric keys. Wrong profile, LLM recovery, unknown run, consumer mismatch, already-terminal status, and digest mismatch fail closed. tepp_api recognizes /execute and refuses it so the raw listener cannot pretend to execute. This does not duplicate the engine library, terminal-result DTO, GET, lifecycle POST, cancel HTTP, loopback CLI, or collection GET. Persistence remains GAP-003B. --- ARCHITECTURE.md | 4 +- .../scientific-acceptance-loopback-engine.md | 3 + CHANGELOG.md | 2 + DOCUMENTATION.md | 2 + crates/analysis_engine/src/lib.rs | 11 +- .../analysis_engine/src/loopback_execute.rs | 333 ++++++++++++++++++ .../tests/loopback_execute_contract.rs | 268 ++++++++++++++ crates/tepp_api/src/analysis_run_live.rs | 110 +++++- .../tepp_api/src/analysis_run_status_http.rs | 59 +++- crates/tepp_api/src/lib.rs | 12 +- crates/tepp_api/src/live_http.rs | 36 +- docs/API_CONTRACT.md | 7 +- docs/TRACEABILITY.md | 1 + ...2-scientific-acceptance-loopback-engine.md | 115 ++++++ docs/adr/README.md | 2 + docs/connectors/naruon-artifact-consumer.md | 1 + .../scientific-acceptance-loopback-engine.md | 31 ++ 17 files changed, 979 insertions(+), 18 deletions(-) create mode 100644 CHANGELOG.d/scientific-acceptance-loopback-engine.md create mode 100644 crates/analysis_engine/src/loopback_execute.rs create mode 100644 crates/analysis_engine/tests/loopback_execute_contract.rs create mode 100644 docs/adr/0032-scientific-acceptance-loopback-engine.md create mode 100644 docs/research/scientific-acceptance-loopback-engine.md diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 479f84d12..a027f4e3e 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -70,7 +70,7 @@ boundaries above remain the target modular MSA architecture. | `tepp_simulation` | known-truth temporal/event data generation | | `validation_core` | RMSE, bias, coverage, graph, Monte Carlo, and exact-head claim-promotion metrics | | `tepp_api` | versioned DTO, schema, terminal-result, and export contracts | -| `analysis_engine` | bounded cutoff-safe temporal evidence readiness execution, digest-bound terminal artifacts, and GAP-003A scientific-acceptance validation runs (`tepp.scientific_acceptance.v1`; not implemented-main) | +| `analysis_engine` | bounded cutoff-safe temporal evidence readiness execution, digest-bound terminal artifacts, GAP-003A scientific-acceptance validation runs (`tepp.scientific_acceptance.v1`; not implemented-main), and loopback `POST /v1/analysis-runs/{run_id}/execute` that produces that artifact without a caller-supplied payload | | `episode_membership` | event-time episode membership containment gate | | `prompt_source` | prompt boilerplate is not unique latent content and not stopword deletion | | `corpus_background` | corpus-background wording is not unique latent content and not stopword deletion | @@ -113,7 +113,7 @@ boundaries above remain the target modular MSA architecture. | `episode_membership` | episode membership cannot escape the episode event-time interval | | `membership_target` | language, episode, template, department, and opportunity-pool targets cannot collapse into entity or project | | `topic_measurement` | logistic-normal ALR/ILR coordinates and the CPU `f64` TRSL-TM reference estimator | -| `analysis_engine` | bounded cutoff-safe temporal evidence readiness execution, digest-bound terminal artifacts, and GAP-003A scientific-acceptance validation runs (`tepp.scientific_acceptance.v1`; not implemented-main) | +| `analysis_engine` | bounded cutoff-safe temporal evidence readiness execution, digest-bound terminal artifacts, GAP-003A scientific-acceptance validation runs (`tepp.scientific_acceptance.v1`; not implemented-main), and loopback `POST /v1/analysis-runs/{run_id}/execute` that produces that artifact without a caller-supplied payload | | `psychometric_core` | posterior-aware structural input gates, CWC within/between OLS plus the contextual effect, event-time log-rate, unequal-interval discrete-lag remapping, constant-predictor discrete effect, time-varying-predictor discrete effect (Eq. 14), exact scalar discrete process noise (Driver et al., 2017, Eq. 3), lagged latent covariance and unconditional latent variance (Driver et al., 2017, Eq. 3–4), stationary within-subject variance (Driver et al., 2017, Eq. 4 as `Δt → ∞`; `asymDIFFUSION`), trait-plus-state variance (Driver et al., 2017, §4.3 `TRAITVAR`; not process noise), observed-indicator variance and lagged observed covariance (Driver et al., 2017, Eq. 5; Table 2 `MANIFESTVAR` is `Θ`, not `Var(y)`; `MANIFESTTRAITVAR` is not `MANIFESTVAR`; `Θ` does not enter lagged observed covariance; observed-indicator mean is `τ + λ μ`; `MANIFESTMEANS` is not `E(y)`; `CINT` is not `MANIFESTMEANS`; discrete latent mean is `exp(a Δt) μ_0 + (exp(a Δt) − 1)/a κ`; `T0MEANS` is not `μ_t`; `CINT` is not the discrete increment; evolved observed mean is `τ + λ μ_t`; `τ + λ μ_0` is not `E(y_t)`; contemporaneous `TDPREDEFFECT` impulse is `m x`, not `CINT`, not `TIPREDEFFECT`, and not Voelkle Eq. 14; Eq. 5 of that contemporaneous impulse is `τ + λ(μ_t + m x)`, and `τ + λ μ_t` is not that observed mean; time-independent `TIPREDEFFECT` increment is `A^{-1}[e^{A Δt} − I] B z`, not `CINT`, not `M x`, not Voelkle Eq. 14, and not the coefficient `B`; Eq. 5 of that increment is `τ + λ(μ_t + A^{-1}[e^{A Δt} − I] B z)`, and `τ + λ μ_t` is not that observed mean; `τ + λ(μ_t + m x)` is not that observed mean; `τ + λ(μ_t + e^{a(t−u)} m x)` is not that observed mean when `u ≠ t`; within-interval `TDPREDEFFECT` carry is `e^{A(t−u)} M x` for `t0 < u < t`, not the contemporaneous Dirac, not `CINT`, not `TIPREDEFFECT`, and not Voelkle Eq. 14; Eq. 5 of that carry is `τ + λ(μ_t + e^{a(t−u)} m x)`, and `τ + λ μ_t` is not that observed mean; `τ + λ(μ_t + m x)` is not that carried observed mean when `u ≠ t`; first-occasion `T0TIPREDEFFECT` shift is `t0_b z` and Eq. 3 first-summand carry is `e^{A Δt} t0_b z` (`T0TIPREDEFFECT` is not `TIPREDEFFECT` `B`; `t0_b z` is not `A^{-1}[e^{A Δt} − I] B z`; `e^{A Δt} t0_b z` is not `t0_b z`; Eq. 5 of that carry is `τ + λ(μ_t + e^{a Δt} t0_b z)`, and `τ + λ μ_t` is not that observed mean; `τ + λ(μ_t + A^{-1}[e^{A Δt} − I] B z)` is not that observed mean), first-occasion `T0TDPREDEFFECT` shift is `t0_m x0` and Eq. 3 first-summand carry is `e^{A Δt} t0_m x0` (`T0TDPREDEFFECT` is not `TDPREDEFFECT` `M`; `t0_m x0` is not `M x`; `e^{A Δt} t0_m x0` is not `t0_m x0`; `e^{A Δt} t0_m x0` is not `e^{A(t−u)} M x` for `t0 < u < t`; `t0_m x0` is not `t0_b z`; an impulse at `u ≤ t0` that used `M` is already in `η(t0)` as `TDPREDEFFECT`, not as `T0TDPREDEFFECT`; Eq. 5 of that carry is `τ + λ(μ_t + e^{a Δt} t0_m x0)`, and `τ + λ μ_t` is not that observed mean; `τ + λ(μ_t + A^{-1}[e^{A Δt} − I] B z)` is not that observed mean; `τ + λ(μ_t + e^{a Δt} t0_b z)` is not that observed mean; §7.2 level-change `CINT` is `κ = −a m x` with `a < 0` so `−κ / a = m x` (`−a m x` is not the dissipating Dirac, not a free `CINT`, not `TIPREDEFFECT`, and not the extra near-zero-drift latent process also named in §7.2; Eq. 3 of that setting is `(1 − e^{a Δt}) m x`, which is not `m x`, not `κ`, and not `TIPREDEFFECT`; §7.2 extra-process contribution is `a_{ηξ} x (e^{ε Δt} − e^{a Δt}) / (ε − a)` (`ε = a` is `a_{ηξ} x Δt e^{a Δt}`; identification `TDPREDEFFECT` on the extra process is 1; printed extra `DRIFT` is `−0.000001`; not `κ = −a m x`, not `(1 − e^{a Δt}) m x`, and not the dissipating Dirac `m x`; `ε ≥ 0` fails closed; Eq. 5 of that contribution is `τ + λ(μ_t + a_{ηξ} x (e^{ε Δt} − e^{a Δt}) / (ε − a)`; the extra process has `LAMBDA` 0 and is not an observed indicator; `τ + λ μ_t` is not that observed mean; `τ + λ(μ_t + m x)` is not that observed mean; the contribution is not `E(y_t)`; the evolved-plus-contribution latent mean is not `E(y_t)`; after-t0 extra-process `TDPREDEFFECT` is `a_{ηξ} x (e^{ε(t−u)} − e^{a(t−u)}) / (ε − a)` for `t0 < u < t` while `μ_t` uses `Δt`; Eq. 5 of that after-t0 contribution is `τ + λ(μ_t + a_{ηξ} x (e^{ε(t−u)} − e^{a(t−u)}) / (ε − a)`; the first-occasion extra-process observed mean is not that observed mean when `u ≠ t0`; `e^{a(t−u)} m x` is a Dirac on the original process, not this `DRIFT` drive; §7.2 `asymTIPREDEFFECT` is `-B z / a` for `a < 0` (`-B z / a` is not the coefficient `B`, not `A^{-1}[e^{A Δt} − I] B z`, not `CINT`, and not `M x`; §7.2 `addedTIPREDVAR` is `(B / a)² v`, not `TRAITVAR`, not `asymDIFFUSION`, and not `-B z / a`; Table 2 `asymCINT` is `-κ / a` for `a < 0` and is not `κ`, not `A^{-1}[e^{A Δt} − I] κ`, not `T0MEANS`, and not `-B z / a`; p. 16 stationary `T0MEANS` is `-κ / a + −B z / a` and is not free `T0MEANS`, not `asymCINT` alone, not `asymTIPREDEFFECT` alone, and not the finite-interval discrete latent mean; Eq. 5 of that constrained mean is `τ + λ(−κ / a + −B z / a)`; `τ + λ μ_0` is not that observed mean; `τ + λ(−κ / a)` is not that observed mean when `B z ≠ 0`; `τ + λ μ_t` is not that observed mean; `MANIFESTMEANS` is not `E(y_0)`; the constrained latent mean is not `E(y_0)`; stationary `T0VAR` is `trait + −q / (2 a) + (B / a)² v` (not free `T0VAR`, not `asymDIFFUSION` alone, not `TRAITVAR` alone, not `addedTIPREDVAR` alone, and not the finite-interval discrete latent variance. Eq. 5 of that constrained variance is `λ²(trait + −q / (2 a) + (B / a)² v) + θ + ψ` (JSS PDF re-opened 2026-08-22T03:20Z; form the stationary latent variance first, then `λ² p + θ + ψ`; `λ² p_0` is not that observed variance; `λ²(−q / (2 a)) + θ` is not that observed variance when `TRAITVAR` or `addedTIPREDVAR` is nonzero; `MANIFESTVAR` is not `Var(y_0)`; the constrained latent variance is not `Var(y_0)`); lagged stationary `T0VAR` is `trait + e^{a Δt}(−q / (2 a)) + (B / a)² v` (trait and `addedTIPREDVAR` do not decay; contemporaneous `T0VAR` is not that lagged map; decaying the constrained total as if it were all state is not that lagged map; Eq. 5 of that lagged covariance is `λ²(trait + e^{a Δt}(−q / (2 a)) + (B / a)² v) + ψ`; `Θ` does not enter; contemporaneous `Var(y_0)` is not that lagged observed covariance; the lagged latent covariance is not that observed covariance); later-occasion stationary `T0VAR` is `trait + e^{2 a Δt}(−q / (2 a)) + Q_Δt + (B / a)² v` (trait and `addedTIPREDVAR` do not enter `Q_Δt`; under stationarity that composition equals contemporaneous `T0VAR`; evolving the constrained total as if it were all state is not that later map; the lagged covariance omits `Q_Δt`; `Q_Δt` is not that later map; Eq. 5 of that later-occasion variance is `λ²(trait + e^{2 a Δt}(−q / (2 a)) + Q_Δt + (B / a)² v) + θ + ψ`; lagged observed covariance omits `Q_Δt` and `θ`; `MANIFESTVAR` is not `Var(y_t)`; the later-occasion latent variance is not `Var(y_t)`))), irregular already-centered residual lag, Rubin `T` on OLS loadings, and strong-gated latent means (two-observation residual variance is identically `0` and caps at strong/scalar; Putnick & Bornstein, 2016) | | `validation_core` | RMSE, bias, coverage, graph, and Monte Carlo metrics | | `tepp_api` | versioned DTO, schema, and export contracts | diff --git a/CHANGELOG.d/scientific-acceptance-loopback-engine.md b/CHANGELOG.d/scientific-acceptance-loopback-engine.md new file mode 100644 index 000000000..1a70390a2 --- /dev/null +++ b/CHANGELOG.d/scientific-acceptance-loopback-engine.md @@ -0,0 +1,3 @@ +### Added + +- `analysis_engine` GAP-003A engine-on-loopback slice (ADR 0032, active-PR, not implemented-main): `ScientificAcceptanceLoopbackService` serves `POST /v1/analysis-runs/{run_id}/execute` so a `scientific_acceptance_v1` run produces `tepp.scientific_acceptance.v1` without a caller-supplied artifact. Persistence remains GAP-003B. diff --git a/CHANGELOG.md b/CHANGELOG.md index e4772696e..1dff8bc6e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -38,6 +38,8 @@ All notable changes to TEPP are documented here. The format follows Keep a Chang ## [Unreleased] +- `analysis_engine` GAP-003A engine-on-loopback slice (ADR 0032, active-PR, not implemented-main): `ScientificAcceptanceLoopbackService` serves `POST /v1/analysis-runs/{run_id}/execute` so an accepted `scientific_acceptance_v1` loopback run produces `tepp.scientific_acceptance.v1` without a caller-supplied artifact. The execute body carries corpus, recovery, seed, and pre-registered SE-gate `k` and refuses `scientific_acceptance_json` plus receipt metric keys. Wrong profile, LLM recovery, unknown run, consumer mismatch, already-terminal status, and digest mismatch fail closed. This does not duplicate the engine library (#356), terminal-result DTO (#358), GET (#359), lifecycle POST (#360), cancel HTTP (#361), loopback CLI (#362), or collection GET (#368); persistence remains GAP-003B. + - `analysis_engine` GAP-003A first slice (ADR 0026, active-PR, not implemented-main): `submit_validation_run` binds cutoff-eligible evidence identities, tenant workspace, snapshot, knowledge cutoff, CPU `f64` model, seed, backend, precision, output profile, and a pre-registered SE-gate multiplier `k` (`0 ≤ k ≤ MAX_SE_GATE_K` = 8) to a hash-stable `tepp-validation-{32 hex}` receipt that carries no scientific metrics. Receipt fields are private. `complete_validation_run` requires recovery vectors stamped to that receipt with the same `k`, records a SHA-256 of those vectors, and emits `tepp.scientific_acceptance.v1` with RMSE, bias, Wilson coverage, temporal-order accuracy, and an SE-aware gate through `validation_core` using the submitted `k`. LLM-authored recovery, a different run/tenant/seed/evidence set, a post-hoc or oversized `k`, a tampered output profile, oversized/empty/mismatched vectors, non-finite inputs, empty or duplicate evidence, snapshot mismatch, and cutoff-empty corpora fail closed. Postgres persistence remains GAP-003B. - `tepp_api` loopback `AnalysisRunLiveService` now serves production `POST /v1/analysis-runs/{run_id}/running` and `POST /v1/analysis-runs/{run_id}/terminal` so accepted/running stay metric-free and only a succeeded status with profile `scientific_acceptance_v1` may return `tepp.scientific_acceptance.v1` after a lifecycle POST. Canonical artifact bytes travel as `scientific_acceptance_json`. Reverse transitions, mutating a terminal run, failed-plus-artifact emission, receipt RMSE/bias/coverage/SE-gate keys, an unknown run, and consumer mismatch fail closed. This is the GAP-003A HTTP lifecycle slice for issue #166; it does not duplicate the `analysis_engine` library bind (#356), the terminal-result DTO wire (#358), or the GET status slice (#359); persistence remains GAP-003B. diff --git a/DOCUMENTATION.md b/DOCUMENTATION.md index 0d18e0e6f..a39eb96b0 100644 --- a/DOCUMENTATION.md +++ b/DOCUMENTATION.md @@ -59,6 +59,7 @@ TEPP's approved PRD v0.4 and implementation plan are the primary product baselin | Entity/project target SQL doctoring | [`docs/research/entity-project-sql.md`](docs/research/entity-project-sql.md) | | Scientific claim-promotion gate doctoring | [`docs/research/scientific-claim-promotion-gates.md`](docs/research/scientific-claim-promotion-gates.md) | | Validation-run scientific-acceptance doctoring | [`docs/research/validation-run-scientific-acceptance.md`](docs/research/validation-run-scientific-acceptance.md) | +| Scientific-acceptance loopback engine doctoring | [`docs/research/scientific-acceptance-loopback-engine.md`](docs/research/scientific-acceptance-loopback-engine.md) | | Retention/deletion/legal-hold doctoring | [`docs/research/retention-deletion-legal-hold.md`](docs/research/retention-deletion-legal-hold.md) | | Provider-payload minimization doctoring | [`docs/research/provider-payload-minimization.md`](docs/research/provider-payload-minimization.md) | | Relation absence is not negative evidence | [`docs/research/relation-absence-not-negative.md`](docs/research/relation-absence-not-negative.md) | @@ -149,6 +150,7 @@ TEPP's approved PRD v0.4 and implementation plan are the primary product baselin | Entity/project target SQL doctoring | [`docs/research/entity-project-sql.md`](docs/research/entity-project-sql.md) | | Scientific claim-promotion gate doctoring | [`docs/research/scientific-claim-promotion-gates.md`](docs/research/scientific-claim-promotion-gates.md) | | Validation-run scientific-acceptance doctoring | [`docs/research/validation-run-scientific-acceptance.md`](docs/research/validation-run-scientific-acceptance.md) | +| Scientific-acceptance loopback engine doctoring | [`docs/research/scientific-acceptance-loopback-engine.md`](docs/research/scientific-acceptance-loopback-engine.md) | | Retention/deletion/legal-hold doctoring | [`docs/research/retention-deletion-legal-hold.md`](docs/research/retention-deletion-legal-hold.md) | | Stopword-deletion doctoring | [`docs/research/stopword-deletion.md`](docs/research/stopword-deletion.md) | | Provider-payload minimization doctoring | [`docs/research/provider-payload-minimization.md`](docs/research/provider-payload-minimization.md) | diff --git a/crates/analysis_engine/src/lib.rs b/crates/analysis_engine/src/lib.rs index d0d544ba4..adebea503 100644 --- a/crates/analysis_engine/src/lib.rs +++ b/crates/analysis_engine/src/lib.rs @@ -8,10 +8,14 @@ //! through [`tepp_api`]. It deliberately does not claim latent-variable or topic //! estimation authority; those estimators remain separate scientific crates. //! estimation authority; it invokes estimators through their scientific crate -//! contracts and preserves their artifact meaning. +//! contracts and preserves their artifact meaning. Loopback +//! [`ScientificAcceptanceLoopbackService`] executes scientific acceptance on an +//! accepted analysis run so GET can return `tepp.scientific_acceptance.v1` +//! without a caller-supplied artifact. mod case_deletion_refit; mod lineage_criterion; +mod loopback_execute; mod topic_context_posterior; mod topic_lineage_artifact; mod validation_run; @@ -48,6 +52,11 @@ pub use lineage_criterion::{ LineageCriterionFit, LineageCriterionFitError, LineageCriterionObservation, fit_lineage_criterion_posteriors, }; +/// Loopback engine-execute contracts for scientific acceptance. +pub use loopback_execute::{ + ANALYSIS_RUN_EXECUTE_CONTRACT_VERSION, ANALYSIS_RUN_EXECUTE_PATH_SUFFIX, + ScientificAcceptanceLoopbackService, +}; /// Bounded posterior topic-context producer contract and record types. pub use topic_context_posterior::{ TOPIC_CONTEXT_POSTERIOR_BYTE_LIMIT, TOPIC_CONTEXT_POSTERIOR_SCHEMA_VERSION, diff --git a/crates/analysis_engine/src/loopback_execute.rs b/crates/analysis_engine/src/loopback_execute.rs new file mode 100644 index 000000000..14e8a655a --- /dev/null +++ b/crates/analysis_engine/src/loopback_execute.rs @@ -0,0 +1,333 @@ +//! Execute scientific-acceptance analysis on the loopback lifecycle path. +//! +//! GAP-003A engine-on-loopback slice: `POST /v1/analysis-runs/{run_id}/execute` +//! runs [`submit_validation_run`] and [`complete_validation_run`] against an +//! already accepted loopback run and records running then terminal status with +//! the produced `tepp.scientific_acceptance.v1` bytes. The execute body carries +//! corpus, recovery vectors, seed, and the pre-registered SE-gate multiplier. +//! It must not carry `scientific_acceptance_json`. GET then returns the artifact +//! without a caller-supplied terminal payload. Persistence remains GAP-003B. + +use crate::{ + AnalysisCorpus, AnalysisEngineError, AnalysisEvidenceUnit, RecoveryObservation, + SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE, SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION, + VALIDATION_CPU_F64_MODEL, complete_validation_run, submit_validation_run, +}; +use serde::Deserialize; +use temporal_core::{AvailableTime, EventTime}; +use tepp_api::{ + ANALYSIS_RUN_STATUS_PATH, AnalysisResultSummary, AnalysisRunStatus, AnalysisRunStatusState, + AnalysisRunTerminalResult, ApiError, DEFAULT_ANALYSIS_RUN_BYTE_LIMIT, + DEFAULT_PROJECT_HISTORY_BYTE_LIMIT, ErrorEnvelope, NaruonLiveResponse, + SCIENTIFIC_ACCEPTANCE_HTTP_PROFILE, SCIENTIFIC_ACCEPTANCE_HTTP_SCHEMA, + analysis_run_execute_path_run_id, parse_loopback_http_parts, +}; + +/// Result-metric keys that must not appear on an execute request object. +const EXECUTE_FORBIDDEN_RESULT_KEYS: [&str; 11] = [ + "rmse", + "rmse_standard_error", + "mean_bias", + "bias_standard_error", + "interval_coverage", + "coverage_wilson_lower", + "coverage_wilson_upper", + "temporal_order_accuracy", + "se_gate_accepted", + "scientific_acceptance", + "report", +]; + +/// Contract version for the loopback execute body. +pub const ANALYSIS_RUN_EXECUTE_CONTRACT_VERSION: u16 = 1; +/// Loopback path suffix owned by this engine glue. +pub const ANALYSIS_RUN_EXECUTE_PATH_SUFFIX: &str = "execute"; + +/// Loopback analysis-run service that can execute scientific acceptance. +/// +/// Create, GET, running, and terminal stay on [`tepp_api::AnalysisRunLiveService`]. +/// This wrapper intercepts `POST /v1/analysis-runs/{run_id}/execute` so the +/// engine produces the artifact. `tepp_api` cannot depend on this crate. +#[derive(Debug)] +pub struct ScientificAcceptanceLoopbackService { + live: tepp_api::AnalysisRunLiveService, + next_request_serial: u64, +} + +impl Default for ScientificAcceptanceLoopbackService { + fn default() -> Self { + Self::new() + } +} + +impl ScientificAcceptanceLoopbackService { + /// Construct an in-memory handler with no bound socket. + #[must_use] + pub fn new() -> Self { + Self { + live: tepp_api::AnalysisRunLiveService::new(), + next_request_serial: 1, + } + } + + /// Wrap an existing loopback listener. + #[must_use] + pub fn from_live(live: tepp_api::AnalysisRunLiveService) -> Self { + Self { + live, + next_request_serial: 1, + } + } + + /// Return the inner loopback service. + #[must_use] + pub fn live(&self) -> &tepp_api::AnalysisRunLiveService { + &self.live + } + + /// Return the inner loopback service mutably. + pub fn live_mut(&mut self) -> &mut tepp_api::AnalysisRunLiveService { + &mut self.live + } + + /// Parse and handle one complete HTTP/1.1 request already in memory. + #[must_use] + pub fn handle_http_request(&mut self, request: &str) -> NaruonLiveResponse { + match self.dispatch_http_request(request) { + Ok(response) => response, + Err(error) => self.response_from_error(error), + } + } + + fn dispatch_http_request(&mut self, request: &str) -> Result { + let Ok(parts) = parse_loopback_http_parts(request, DEFAULT_PROJECT_HISTORY_BYTE_LIMIT) + else { + return Ok(self.live.handle_http_request(request)); + }; + if parts.method == "POST" && is_execute_attempt(parts.path) { + let run_id = analysis_run_execute_path_run_id(parts.path)?; + return self.execute_scientific_acceptance(&run_id, &parts.headers, parts.body); + } + Ok(self.live.handle_http_request(request)) + } + + fn execute_scientific_acceptance( + &mut self, + path_run_id: &str, + headers: &std::collections::HashMap, + body: &str, + ) -> Result { + refuse_result_metrics_on_execute(body)?; + let consumer = self.live.authorize_loopback_headers(headers)?; + let idempotency_key = header_required(headers, "idempotency-key")?; + let execute = parse_execute_body(body)?; + if execute.run_id != path_run_id || execute.idempotency_key != idempotency_key { + return Err(ApiError::InvalidWirePayload); + } + let stored = self.live.loopback_run(path_run_id)?; + if stored.consumer != consumer || stored.accepted.idempotency_key != idempotency_key { + return Err(ApiError::InvalidWirePayload); + } + if stored.status.run_state != AnalysisRunStatusState::Accepted { + return Err(ApiError::InvalidWirePayload); + } + if stored.request.output_profile != SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE + || stored.request.output_profile != SCIENTIFIC_ACCEPTANCE_HTTP_PROFILE + || stored.request.model_contract_version != VALIDATION_CPU_F64_MODEL + { + return Err(ApiError::InvalidWirePayload); + } + let corpus = execute.corpus.to_corpus().map_err(map_engine_error)?; + let receipt = + submit_validation_run(&stored.request, &corpus, execute.seed, execute.se_gate_k) + .map_err(map_engine_error)?; + let observation = RecoveryObservation::new( + &receipt, + execute.study_label.clone(), + execute.truth.clone(), + execute.recovered.clone(), + execute.interval_lower.clone(), + execute.interval_upper.clone(), + execute.truth_times.clone(), + execute.recovered_times.clone(), + execute.se_gate_k, + execute.authored_by_llm, + ) + .map_err(map_engine_error)?; + let evidence = complete_validation_run(&receipt, &stored.request, &corpus, &observation) + .map_err(map_engine_error)?; + let artifact_json = evidence.to_json().map_err(map_engine_error)?; + let digest = evidence.sha256().map_err(map_engine_error)?; + if evidence.schema_version() != SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION + || evidence.schema_version() != SCIENTIFIC_ACCEPTANCE_HTTP_SCHEMA + || evidence.output_profile() != SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE + { + return Err(ApiError::InvalidWirePayload); + } + let summary = AnalysisResultSummary::new( + "scientific_acceptance", + evidence.eligible_evidence_count(), + 4, + "validated", + )?; + let terminal = AnalysisRunTerminalResult::succeeded( + &stored.request, + &stored.accepted, + evidence.run_id(), + digest, + SCIENTIFIC_ACCEPTANCE_HTTP_SCHEMA, + execute.completed_at.clone(), + summary, + )?; + let running = AnalysisRunStatus::running(&stored.accepted)?; + let terminal_status = + AnalysisRunStatus::terminal(&stored.request, &stored.accepted, terminal)?; + self.live + .record_loopback_status(path_run_id, running, None)?; + self.live + .record_loopback_status(path_run_id, terminal_status, Some(artifact_json))?; + let response_body = self.live.loopback_status_json(path_run_id)?; + Ok(NaruonLiveResponse { + status_code: 200, + reason_phrase: "OK", + body: response_body, + }) + } + + fn response_from_error(&mut self, error: ApiError) -> NaruonLiveResponse { + let request_id = format!("scientific-acceptance-execute-{}", self.next_request_serial); + self.next_request_serial += 1; + let (status_code, reason_phrase) = match error { + ApiError::AuthorizationDenied => (403, "Forbidden"), + ApiError::LimitExceeded => (413, "Payload Too Large"), + ApiError::UnsupportedContractVersion => (422, "Unprocessable Entity"), + _ => (400, "Bad Request"), + }; + let body = ErrorEnvelope::from_api_error(error, request_id) + .and_then(|envelope| envelope.to_json()) + .unwrap_or_else(|_| { + "{\"error_code\":\"invalid_wire_payload\",\"message\":\"invalid API wire payload\",\"request_id\":\"scientific-acceptance-execute-fallback\",\"retryable\":false}".to_owned() + }); + NaruonLiveResponse { + status_code, + reason_phrase, + body, + } + } +} + +/// Supported execute-body contract version. +#[derive(Clone, Debug, Deserialize, PartialEq)] +#[serde(deny_unknown_fields)] +struct ScientificAcceptanceExecuteRequest { + contract_version: u16, + run_id: String, + idempotency_key: String, + seed: u64, + se_gate_k: f64, + completed_at: String, + study_label: String, + authored_by_llm: bool, + corpus: ExecuteCorpus, + truth: Vec, + recovered: Vec, + interval_lower: Vec, + interval_upper: Vec, + truth_times: Vec, + recovered_times: Vec, +} + +#[derive(Clone, Debug, Deserialize, PartialEq, Eq)] +#[serde(deny_unknown_fields)] +struct ExecuteCorpus { + snapshot_id: String, + evidence_units: Vec, +} + +#[derive(Clone, Debug, Deserialize, PartialEq, Eq)] +#[serde(deny_unknown_fields)] +struct ExecuteEvidenceUnit { + evidence_id: String, + event_time: String, + available_time: String, + membership_count: u32, +} + +impl ExecuteCorpus { + fn to_corpus(&self) -> Result { + let mut units = Vec::with_capacity(self.evidence_units.len()); + for unit in &self.evidence_units { + let event_time = EventTime::parse_rfc3339(&unit.event_time) + .map_err(|_| AnalysisEngineError::InvalidEvidence)?; + let available_time = AvailableTime::parse_rfc3339(&unit.available_time) + .map_err(|_| AnalysisEngineError::InvalidEvidence)?; + units.push(AnalysisEvidenceUnit::new( + unit.evidence_id.clone(), + event_time, + available_time, + unit.membership_count, + )?); + } + AnalysisCorpus::new(self.snapshot_id.clone(), units) + } +} + +fn parse_execute_body(body: &str) -> Result { + if body.len() > DEFAULT_ANALYSIS_RUN_BYTE_LIMIT { + return Err(ApiError::LimitExceeded); + } + refuse_result_metrics_on_execute(body)?; + let execute: ScientificAcceptanceExecuteRequest = + serde_json::from_str(body).map_err(|_| ApiError::InvalidWirePayload)?; + if execute.contract_version != ANALYSIS_RUN_EXECUTE_CONTRACT_VERSION { + return Err(ApiError::UnsupportedContractVersion); + } + if execute.run_id.is_empty() + || execute.idempotency_key.is_empty() + || execute.completed_at.is_empty() + || execute.study_label.is_empty() + { + return Err(ApiError::InvalidWirePayload); + } + Ok(execute) +} + +fn refuse_result_metrics_on_execute(payload: &str) -> Result<(), ApiError> { + let Ok(value) = serde_json::from_str::(payload) else { + return Ok(()); + }; + let Some(object) = value.as_object() else { + return Err(ApiError::InvalidWirePayload); + }; + if EXECUTE_FORBIDDEN_RESULT_KEYS + .iter() + .any(|key| object.contains_key(*key)) + { + return Err(ApiError::InvalidWirePayload); + } + Ok(()) +} + +fn is_execute_attempt(path: &str) -> bool { + path.rsplit('/').next() == Some(ANALYSIS_RUN_EXECUTE_PATH_SUFFIX) + && path.starts_with(ANALYSIS_RUN_STATUS_PATH) +} + +fn header_required<'a>( + headers: &'a std::collections::HashMap, + name: &str, +) -> Result<&'a str, ApiError> { + let value = headers.get(name).ok_or(ApiError::InvalidWirePayload)?; + if value.is_empty() { + return Err(ApiError::InvalidWirePayload); + } + Ok(value.as_str()) +} + +fn map_engine_error(error: AnalysisEngineError) -> ApiError { + match error { + AnalysisEngineError::Api(api) => api, + AnalysisEngineError::LimitExceeded => ApiError::LimitExceeded, + _ => ApiError::InvalidWirePayload, + } +} diff --git a/crates/analysis_engine/tests/loopback_execute_contract.rs b/crates/analysis_engine/tests/loopback_execute_contract.rs new file mode 100644 index 000000000..4ea2333af --- /dev/null +++ b/crates/analysis_engine/tests/loopback_execute_contract.rs @@ -0,0 +1,268 @@ +//! GAP-003A engine-on-loopback scientific-acceptance contract. + +use analysis_engine::{ + ANALYSIS_RUN_EXECUTE_CONTRACT_VERSION, SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE, + SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION, ScientificAcceptanceLoopbackService, + VALIDATION_CPU_F64_MODEL, +}; +use tepp_api::{ + ANALYSIS_RUN_CONTRACT_VERSION, AnalysisRunAccepted, AnalysisRunRequest, + NARUON_ANALYSIS_RUN_PATH, NARUON_CONSUMER_CODE, SCIENTIFIC_ACCEPTANCE_HTTP_PROFILE, + SCIENTIFIC_ACCEPTANCE_HTTP_SCHEMA, +}; + +fn request(profile: &str, model: &str) -> AnalysisRunRequest { + AnalysisRunRequest { + contract_version: ANALYSIS_RUN_CONTRACT_VERSION, + idempotency_key: "idem-loopback-execute".into(), + tenant_workspace_id: "tenant-workspace-execute".into(), + snapshot_id: "snapshot-execute".into(), + knowledge_cutoff: "2026-08-01T00:00:00Z".into(), + model_contract_version: model.into(), + output_profile: profile.into(), + } +} + +fn http_post(path: &str, body: &str, consumer: &str, idempotency_key: &str) -> String { + format!( + "POST {path} HTTP/1.1\r\nHost: 127.0.0.1\r\ncontent-type: application/json\r\ntepp-consumer: {consumer}\r\ntepp-contract-version: 1\r\nidempotency-key: {idempotency_key}\r\ncontent-length: {}\r\n\r\n{body}", + body.len() + ) +} + +fn http_get(path: &str, consumer: &str, idempotency_key: &str) -> String { + format!( + "GET {path} HTTP/1.1\r\nHost: 127.0.0.1\r\ncontent-type: application/json\r\ntepp-consumer: {consumer}\r\ntepp-contract-version: 1\r\nidempotency-key: {idempotency_key}\r\ncontent-length: 0\r\n\r\n" + ) +} + +fn execute_body(run_id: &str, authored_by_llm: bool) -> String { + serde_json::json!({ + "contract_version": ANALYSIS_RUN_EXECUTE_CONTRACT_VERSION, + "run_id": run_id, + "idempotency_key": "idem-loopback-execute", + "seed": 42, + "se_gate_k": 3.0, + "completed_at": "2026-08-31T10:00:00Z", + "study_label": "loopback-recovery", + "authored_by_llm": authored_by_llm, + "corpus": { + "snapshot_id": "snapshot-execute", + "evidence_units": [ + { + "evidence_id": "evidence-1", + "event_time": "2026-07-01T00:00:00Z", + "available_time": "2026-07-10T00:00:00Z", + "membership_count": 1 + }, + { + "evidence_id": "evidence-2", + "event_time": "2026-07-01T00:00:00Z", + "available_time": "2026-07-20T00:00:00Z", + "membership_count": 1 + }, + { + "evidence_id": "future", + "event_time": "2026-07-01T00:00:00Z", + "available_time": "2026-08-02T00:00:00Z", + "membership_count": 1 + } + ] + }, + "truth": [0.70, 0.55, 0.40, -0.20, 0.85], + "recovered": [0.70, 0.55, 0.40, -0.20, 0.85], + "interval_lower": [0.50, 0.35, 0.20, -0.40, 0.65], + "interval_upper": [0.90, 0.75, 0.60, 0.00, 1.00], + "truth_times": [1.0, 2.0, 3.0, 4.0, 5.0], + "recovered_times": [1.1, 1.9, 3.2, 3.8, 5.1] + }) + .to_string() +} + +fn accept_run( + service: &mut ScientificAcceptanceLoopbackService, + profile: &str, + model: &str, +) -> AnalysisRunAccepted { + let run = request(profile, model); + let accepted = service.handle_http_request(&http_post( + NARUON_ANALYSIS_RUN_PATH, + &run.to_json().expect("create json"), + NARUON_CONSUMER_CODE, + run.idempotency_key.as_str(), + )); + assert_eq!(accepted.status_code, 202, "{}", accepted.body); + assert!(!accepted.body.contains("rmse")); + assert!(!accepted.body.contains("scientific_acceptance")); + AnalysisRunAccepted::from_json(&accepted.body).expect("accepted") +} + +#[test] +fn execute_produces_scientific_acceptance_without_caller_artifact() { + let mut service = ScientificAcceptanceLoopbackService::new(); + let accepted = accept_run( + &mut service, + SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE, + VALIDATION_CPU_F64_MODEL, + ); + let body = execute_body(&accepted.run_id, false); + assert!(!body.contains("scientific_acceptance_json")); + assert!(!body.contains("rmse")); + let execute = service.handle_http_request(&http_post( + &format!("{NARUON_ANALYSIS_RUN_PATH}/{}/execute", accepted.run_id), + &body, + NARUON_CONSUMER_CODE, + "idem-loopback-execute", + )); + assert_eq!(execute.status_code, 200, "{}", execute.body); + assert!(execute.body.contains(SCIENTIFIC_ACCEPTANCE_HTTP_SCHEMA)); + assert!(execute.body.contains("scientific_acceptance")); + assert!(execute.body.contains("\"succeeded\"")); + + let get = service.handle_http_request(&http_get( + &format!("{NARUON_ANALYSIS_RUN_PATH}/{}", accepted.run_id), + NARUON_CONSUMER_CODE, + "idem-loopback-execute", + )); + assert_eq!(get.status_code, 200, "{}", get.body); + assert!(get.body.contains(SCIENTIFIC_ACCEPTANCE_SCHEMA_VERSION)); + assert!(get.body.contains(SCIENTIFIC_ACCEPTANCE_HTTP_PROFILE)); + assert!(get.body.contains("scientific_acceptance")); + assert!(get.body.contains("rmse")); +} + +#[test] +fn execute_fail_closed_for_hostile_bodies_and_bindings() { + let mut service = ScientificAcceptanceLoopbackService::new(); + let accepted = accept_run( + &mut service, + SCIENTIFIC_ACCEPTANCE_OUTPUT_PROFILE, + VALIDATION_CPU_F64_MODEL, + ); + let body = execute_body(&accepted.run_id, false); + let path = format!("{NARUON_ANALYSIS_RUN_PATH}/{}/execute", accepted.run_id); + + let with_rmse = body.replacen('{', r#"{"rmse":0.1,"#, 1); + assert_eq!( + service + .handle_http_request(&http_post( + &path, + &with_rmse, + NARUON_CONSUMER_CODE, + "idem-loopback-execute", + )) + .status_code, + 400 + ); + + let with_artifact = body.replacen('{', r#"{"scientific_acceptance_json":"{}","#, 1); + assert_eq!( + service + .handle_http_request(&http_post( + &path, + &with_artifact, + NARUON_CONSUMER_CODE, + "idem-loopback-execute", + )) + .status_code, + 400 + ); + + let llm = execute_body(&accepted.run_id, true); + assert_eq!( + service + .handle_http_request(&http_post( + &path, + &llm, + NARUON_CONSUMER_CODE, + "idem-loopback-execute", + )) + .status_code, + 400 + ); + + assert_eq!( + service + .handle_http_request(&http_post( + &path, + &body, + "lineageweave", + "idem-loopback-execute", + )) + .status_code, + 400 + ); + + assert_eq!( + service + .handle_http_request(&http_post( + &format!("{NARUON_ANALYSIS_RUN_PATH}/tepp-run-999/execute"), + &execute_body("tepp-run-999", false), + NARUON_CONSUMER_CODE, + "idem-loopback-execute", + )) + .status_code, + 400 + ); + + let wrong_version = body.replace( + &format!("\"contract_version\":{ANALYSIS_RUN_EXECUTE_CONTRACT_VERSION}"), + "\"contract_version\":9", + ); + assert_eq!( + service + .handle_http_request(&http_post( + &path, + &wrong_version, + NARUON_CONSUMER_CODE, + "idem-loopback-execute", + )) + .status_code, + 422 + ); + + let ok = service.handle_http_request(&http_post( + &path, + &body, + NARUON_CONSUMER_CODE, + "idem-loopback-execute", + )); + assert_eq!(ok.status_code, 200, "{}", ok.body); + assert_eq!( + service + .handle_http_request(&http_post( + &path, + &body, + NARUON_CONSUMER_CODE, + "idem-loopback-execute", + )) + .status_code, + 400 + ); +} + +#[test] +fn execute_refuses_wrong_profile_and_leaves_accepted_metric_free() { + let mut service = ScientificAcceptanceLoopbackService::new(); + let accepted = accept_run( + &mut service, + "calibrated_event_measurement", + VALIDATION_CPU_F64_MODEL, + ); + let execute = service.handle_http_request(&http_post( + &format!("{NARUON_ANALYSIS_RUN_PATH}/{}/execute", accepted.run_id), + &execute_body(&accepted.run_id, false), + NARUON_CONSUMER_CODE, + "idem-loopback-execute", + )); + assert_eq!(execute.status_code, 400, "{}", execute.body); + let get = service.handle_http_request(&http_get( + &format!("{NARUON_ANALYSIS_RUN_PATH}/{}", accepted.run_id), + NARUON_CONSUMER_CODE, + "idem-loopback-execute", + )); + assert_eq!(get.status_code, 200, "{}", get.body); + assert!(get.body.contains("\"accepted\"")); + assert!(!get.body.contains("rmse")); + assert!(!get.body.contains("scientific_acceptance")); +} diff --git a/crates/tepp_api/src/analysis_run_live.rs b/crates/tepp_api/src/analysis_run_live.rs index 624d5ef46..1112ac7ce 100644 --- a/crates/tepp_api/src/analysis_run_live.rs +++ b/crates/tepp_api/src/analysis_run_live.rs @@ -8,8 +8,9 @@ //! `tepp.scientific_acceptance.v1` only on a succeeded status whose request //! profile is `scientific_acceptance_v1`. `POST /v1/analysis-runs/{run_id}/running` //! and `POST /v1/analysis-runs/{run_id}/terminal` are the production -//! status-update path. Completed psychometric estimation remains outside this -//! crate. +//! status-update path. `POST /v1/analysis-runs/{run_id}/execute` is recognized +//! and refused here; `analysis_engine` owns engine execution on this listener. +//! Completed psychometric estimation remains outside this crate. use std::collections::HashMap; use std::io::Write; @@ -49,6 +50,22 @@ struct LiveAnalysisRun { scientific_acceptance_json: Option, } +/// Public copy of one accepted loopback run for engine glue. +/// +/// Scientific-acceptance bytes stay private on the live service. Engine +/// execution records them through [`AnalysisRunLiveService::record_loopback_status`]. +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct LoopbackAnalysisRun { + /// Consumer identity that accepted the run. + pub consumer: String, + /// Original create request. + pub request: AnalysisRunRequest, + /// Accepted receipt. + pub accepted: AnalysisRunAccepted, + /// Current lifecycle status. + pub status: AnalysisRunStatus, +} + /// Loopback HTTP/1.1 analysis-run service shared by published CWL consumers. /// /// The service accepts only Naruon and `LineageWeave` consumer identities. Its @@ -197,7 +214,9 @@ impl AnalysisRunLiveService { AnalysisRunLiveRoute::Terminal { run_id } => { self.post_terminal_status(&run_id, &headers, body) } - AnalysisRunLiveRoute::Status { .. } => Err(ApiError::InvalidWirePayload), + AnalysisRunLiveRoute::Execute { .. } | AnalysisRunLiveRoute::Status { .. } => { + Err(ApiError::InvalidWirePayload) + } } } @@ -285,9 +304,17 @@ impl AnalysisRunLiveService { /// /// HTTP `POST /v1/analysis-runs/{run_id}/running` and /// `POST /v1/analysis-runs/{run_id}/terminal` call this after the request - /// is authenticated and parsed. Psychometric execution remains outside this - /// crate; this records the supplied status only. - pub(crate) fn record_loopback_status( + /// is authenticated and parsed. `analysis_engine` records a produced + /// scientific-acceptance artifact through the same method so GET can return + /// `tepp.scientific_acceptance.v1` without a caller-supplied artifact. + /// Psychometric execution remains outside this crate; this records the + /// supplied status only. + /// + /// # Errors + /// + /// Returns [`ApiError::InvalidWirePayload`] for an unknown run, a binding + /// mismatch, or an HTTP-layer scientific-acceptance gate failure. + pub fn record_loopback_status( &mut self, run_id: &str, status: AnalysisRunStatus, @@ -316,6 +343,66 @@ impl AnalysisRunLiveService { Ok(()) } + /// Return a copy of one accepted loopback run. + /// + /// # Errors + /// + /// Returns [`ApiError::InvalidWirePayload`] when the run identity is unknown + /// or internally inconsistent. + pub fn loopback_run(&self, run_id: &str) -> Result { + let replay_key = self + .runs_by_id + .get(run_id) + .ok_or(ApiError::InvalidWirePayload)?; + let stored = self + .accepted_runs + .get(replay_key) + .ok_or(ApiError::InvalidWirePayload)?; + if stored.accepted.run_id != run_id { + return Err(ApiError::InvalidWirePayload); + } + Ok(LoopbackAnalysisRun { + consumer: stored.consumer.clone(), + request: stored.request.clone(), + accepted: stored.accepted.clone(), + status: stored.status.clone(), + }) + } + + /// Serialize the current loopback status GET body for one run. + /// + /// # Errors + /// + /// Returns the same fail-closed errors as the GET status path. + pub fn loopback_status_json(&self, run_id: &str) -> Result { + let replay_key = self + .runs_by_id + .get(run_id) + .ok_or(ApiError::InvalidWirePayload)?; + let stored = self + .accepted_runs + .get(replay_key) + .ok_or(ApiError::InvalidWirePayload)?; + status_http_json( + &stored.status, + &stored.request, + stored.scientific_acceptance_json.as_deref(), + ) + } + + /// Authorize consumer-scoped loopback headers against this listener. + /// + /// # Errors + /// + /// Returns [`ApiError::InvalidWirePayload`] for missing contract headers + /// and [`ApiError::AuthorizationDenied`] for credential headers. + pub fn authorize_loopback_headers<'a>( + &self, + headers: &'a HashMap, + ) -> Result<&'a str, ApiError> { + require_headers(headers, self.bound_addr, true) + } + fn post_running_status( &mut self, run_id: &str, @@ -1484,6 +1571,17 @@ mod tests { accepted_dto.run_id, run.idempotency_key ); assert_eq!(service.handle_http_request(&put).status_code, 400); + assert_eq!( + service + .handle_http_request(&lifecycle_post( + &format!("{NARUON_ANALYSIS_RUN_PATH}/{}/execute", accepted_dto.run_id), + "{}", + NARUON_CONSUMER_CODE, + run.idempotency_key.as_str(), + )) + .status_code, + 400 + ); assert_eq!( service .handle_http_request(&lifecycle_post( diff --git a/crates/tepp_api/src/analysis_run_status_http.rs b/crates/tepp_api/src/analysis_run_status_http.rs index 08ea5d114..fc9f7e464 100644 --- a/crates/tepp_api/src/analysis_run_status_http.rs +++ b/crates/tepp_api/src/analysis_run_status_http.rs @@ -22,6 +22,8 @@ pub(crate) enum AnalysisRunLiveRoute { Running { run_id: String }, /// `POST /v1/analysis-runs/{run_id}/terminal` Terminal { run_id: String }, + /// `POST /v1/analysis-runs/{run_id}/execute` + Execute { run_id: String }, } /// Build a provider-owned `GET` analysis-run status exchange. @@ -137,7 +139,7 @@ fn require_run_id_length(run_id: &str) -> Result<(), ApiError> { } } -/// Parse `GET` status and `POST` running/terminal loopback routes. +/// Parse `GET` status and `POST` running/terminal/execute loopback routes. /// /// # Errors /// @@ -171,10 +173,35 @@ pub(crate) fn parse_analysis_run_live_route(path: &str) -> Result { + let run_id = decode_path_segment(encoded_id)?; + require_run_id_length(&run_id)?; + Ok(AnalysisRunLiveRoute::Execute { run_id }) + } Some(_) => Err(ApiError::InvalidWirePayload), } } +/// Extract the opaque run identity from `POST /v1/analysis-runs/{run_id}/execute`. +/// +/// The loopback listener in this crate refuses the execute suffix. Engine glue +/// in `analysis_engine` owns the execution path and uses this parser so hostile +/// encodings fail closed before any scientific work. +/// +/// # Errors +/// +/// Returns [`ApiError::InvalidWirePayload`] for a non-execute path or a hostile +/// encoding, and [`ApiError::LimitExceeded`] when the decoded identity exceeds +/// [`ANALYSIS_RUN_ID_MAX_LEN`]. +pub fn analysis_run_execute_path_run_id(path: &str) -> Result { + match parse_analysis_run_live_route(path)? { + AnalysisRunLiveRoute::Execute { run_id } => Ok(run_id), + AnalysisRunLiveRoute::Status { .. } + | AnalysisRunLiveRoute::Running { .. } + | AnalysisRunLiveRoute::Terminal { .. } => Err(ApiError::InvalidWirePayload), + } +} + /// Extract the opaque run identity from `GET /v1/analysis-runs/{run_id}`. /// /// # Errors @@ -185,9 +212,9 @@ pub(crate) fn parse_analysis_run_live_route(path: &str) -> Result Result { match parse_analysis_run_live_route(path)? { AnalysisRunLiveRoute::Status { run_id } => Ok(run_id), - AnalysisRunLiveRoute::Running { .. } | AnalysisRunLiveRoute::Terminal { .. } => { - Err(ApiError::InvalidWirePayload) - } + AnalysisRunLiveRoute::Running { .. } + | AnalysisRunLiveRoute::Terminal { .. } + | AnalysisRunLiveRoute::Execute { .. } => Err(ApiError::InvalidWirePayload), } } @@ -240,6 +267,7 @@ mod tests { } #[test] + #[allow(clippy::too_many_lines)] fn decodes_status_path_identities_and_refuses_hostile_segments() { assert_eq!( analysis_run_status_path_run_id("/v1/analysis-runs/tepp-run-1").expect("plain"), @@ -314,6 +342,29 @@ mod tests { run_id: "tepp-run-1".into() } ); + assert_eq!( + parse_analysis_run_live_route("/v1/analysis-runs/tepp-run-1/execute").expect("execute"), + AnalysisRunLiveRoute::Execute { + run_id: "tepp-run-1".into() + } + ); + assert_eq!( + analysis_run_execute_path_run_id("/v1/analysis-runs/tepp-run-1/execute") + .expect("execute id"), + "tepp-run-1" + ); + assert_eq!( + analysis_run_execute_path_run_id("/v1/analysis-runs/tepp-run-1"), + Err(ApiError::InvalidWirePayload) + ); + assert_eq!( + analysis_run_status_path_run_id("/v1/analysis-runs/tepp-run-1/execute"), + Err(ApiError::InvalidWirePayload) + ); + assert_eq!( + parse_analysis_run_live_route("/v1/analysis-runs/tepp-run-1/execute/extra"), + Err(ApiError::InvalidWirePayload) + ); assert_eq!( parse_analysis_run_live_route("/v1/analysis-runs/tepp-run-1/running/extra"), Err(ApiError::InvalidWirePayload) diff --git a/crates/tepp_api/src/lib.rs b/crates/tepp_api/src/lib.rs index facecf471..19774f10c 100644 --- a/crates/tepp_api/src/lib.rs +++ b/crates/tepp_api/src/lib.rs @@ -14,7 +14,9 @@ //! only a succeeded status with profile `scientific_acceptance_v1` may return //! `tepp.scientific_acceptance.v1`. `POST /v1/analysis-runs/{run_id}/running` //! and `POST /v1/analysis-runs/{run_id}/terminal` record those statuses on the -//! same loopback listener. +//! same loopback listener. `POST /v1/analysis-runs/{run_id}/execute` is refused +//! here; `analysis_engine` owns engine execution so a scientific-acceptance run +//! can produce `tepp.scientific_acceptance.v1` without a caller-supplied artifact. mod analysis_result; mod analysis_run; @@ -85,9 +87,11 @@ pub use analysis_run_lifecycle_http::naruon_analysis_run_running_exchange; /// Provider-owned terminal-status HTTP exchange builder. pub use analysis_run_lifecycle_http::naruon_analysis_run_terminal_exchange; /// Consumer-neutral loopback analysis-run service. -pub use analysis_run_live::AnalysisRunLiveService; +pub use analysis_run_live::{AnalysisRunLiveService, LoopbackAnalysisRun}; /// Analysis-run status HTTP exchange re-exports. -pub use analysis_run_status_http::{ANALYSIS_RUN_ID_MAX_LEN, naruon_analysis_run_status_exchange}; +pub use analysis_run_status_http::{ + ANALYSIS_RUN_ID_MAX_LEN, analysis_run_execute_path_run_id, naruon_analysis_run_status_exchange, +}; /// Corpus-split leakage-audit contract version. pub use corpus_split_manifest::CORPUS_SPLIT_MANIFEST_CONTRACT_VERSION; /// Versioned corpus-split leakage-audit manifest. @@ -110,6 +114,8 @@ pub use export::GraphMlExport; pub use export::JsonLdExport; /// Reproducibility manifest. pub use export::ReproducibilityManifest; +/// Parsed loopback HTTP/1.1 request parts for engine glue. +pub use live_http::{LoopbackHttpParts, parse_loopback_http_parts}; /// Output profile that authorizes scientific-acceptance on a loopback GET. pub use scientific_acceptance_http::SCIENTIFIC_ACCEPTANCE_HTTP_PROFILE; /// Schema identity returned on a succeeded scientific-acceptance GET. diff --git a/crates/tepp_api/src/live_http.rs b/crates/tepp_api/src/live_http.rs index ed8d7e564..274e650ba 100644 --- a/crates/tepp_api/src/live_http.rs +++ b/crates/tepp_api/src/live_http.rs @@ -53,7 +53,41 @@ pub(crate) fn read_http_request_with_limit( Ok(format!("{header_text}{body_text}")) } -/// Split one complete request into its header block and UTF-8 body. +/// Parsed loopback HTTP/1.1 request already in memory. +#[derive(Debug)] +pub struct LoopbackHttpParts<'a> { + /// HTTP method token from the request line. + pub method: &'a str, + /// Request-target path. Query strings are refused. + pub path: &'a str, + /// Lowercased unique headers. + pub headers: HashMap, + /// UTF-8 body whose length matches `Content-Length`. + pub body: &'a str, +} + +/// Parse one complete loopback HTTP/1.1 request already in memory. +/// +/// # Errors +/// +/// Returns [`ApiError::InvalidWirePayload`] for framing, version, or header +/// violations and [`ApiError::LimitExceeded`] when the header block or body +/// exceeds the configured bound. +pub fn parse_loopback_http_parts( + request: &str, + maximum_body_bytes: usize, +) -> Result, ApiError> { + let (header_block, body) = split_request_with_limit(request, maximum_body_bytes)?; + let mut lines = header_block.split("\r\n"); + let (method, path) = parse_request_line(lines.next().unwrap_or(""))?; + let headers = parse_headers(&mut lines)?; + Ok(LoopbackHttpParts { + method, + path, + headers, + body, + }) +} pub(crate) fn split_request(request: &str) -> Result<(&str, &str), ApiError> { split_request_with_limit(request, DEFAULT_ANALYSIS_RUN_BYTE_LIMIT) } diff --git a/docs/API_CONTRACT.md b/docs/API_CONTRACT.md index 87a7a4a87..81762d265 100644 --- a/docs/API_CONTRACT.md +++ b/docs/API_CONTRACT.md @@ -105,7 +105,12 @@ bodies stay metric-free, and only a succeeded status with profile `POST /v1/analysis-runs/{run_id}/running` and `POST /v1/analysis-runs/{run_id}/terminal` are the production loopback status-update path that records those statuses; they do not persist and do not -execute psychometric estimation. Production TLS remains a later adapter. +execute psychometric estimation. `POST /v1/analysis-runs/{run_id}/execute` on +`analysis_engine::ScientificAcceptanceLoopbackService` is the engine-owned +loopback path that produces `tepp.scientific_acceptance.v1` for a +`scientific_acceptance_v1` run without a caller-supplied artifact. The execute +body carries corpus, recovery, seed, and the pre-registered SE-gate multiplier +and refuses `scientific_acceptance_json`. Production TLS remains a later adapter. The stacked `analysis_engine` slice provides the first executable service-side path behind these DTOs. It consumes a bounded identity-free snapshot, excludes diff --git a/docs/TRACEABILITY.md b/docs/TRACEABILITY.md index fc102639d..f92f42e2d 100644 --- a/docs/TRACEABILITY.md +++ b/docs/TRACEABILITY.md @@ -56,6 +56,7 @@ The full APA 7th standards/literature register remains `docs/research/standards- | versioned service/API contracts and exports | PRD; API contract; ADR 0011/0013 | `tepp_api` analysis-run/export/JSON-LD/GraphML contracts on protected main (PR #21); request-bound terminal result active in PR #157; HTTP service remains accepted-target; the `orchestrator_live` loopback interpretation listener is on this PR | partial | | loopback analysis-run scientific-acceptance GET | ADR 0027; API contract; RFC 9110; FIPS 180-4 | `tepp_api` `GET /v1/analysis-runs/{run_id}` on `AnalysisRunLiveService` (#359): accepted/running stay metric-free; `tepp.scientific_acceptance.v1` only on succeeded `scientific_acceptance_v1`; not implemented-main | active-PR | | loopback analysis-run scientific-acceptance lifecycle POST | ADR 0028; API contract; RFC 9110; FIPS 180-4 | `tepp_api` `POST /v1/analysis-runs/{run_id}/running` and `/terminal` on `AnalysisRunLiveService` (this PR): production status-update path; accepted/running stay metric-free; `tepp.scientific_acceptance.v1` only after succeeded `scientific_acceptance_v1`; not implemented-main | active-PR | +| loopback scientific-acceptance engine execute | ADR 0032; ADR 0026/0027/0028; API contract; National Academies (2019); Wasserstein & Lazar (2016); Wilson (1927) | `analysis_engine` `POST /v1/analysis-runs/{run_id}/execute` on `ScientificAcceptanceLoopbackService` (this PR): engine produces `tepp.scientific_acceptance.v1` without caller-supplied artifact; metric keys/LLM/wrong-profile/unknown-run/consumer-mismatch/already-terminal fail closed; not implemented-main; Postgres persistence remains GAP-003B | active-PR | | executable cutoff-safe analysis-run readiness | ADR 0021; temporal research; API terminal-result contract | stacked `analysis_engine` PR on #157: availability cutoff, snapshot binding, multiple-membership aggregation, digest-bound artifact, realistic end-to-end tests | active-PR | | delayed-reporting cutoff eligibility in truth corpora | ADR 0002; research | `tepp_simulation` eligible-at-cutoff filter on the active PR | active-PR | | versioned service/API contracts and exports | PRD; API contract; ADR 0011/0013 | `tepp_api` analysis-run/export/JSON-LD/GraphML contracts on protected main (PR #21); HTTP service remaining accepted-target | partial | diff --git a/docs/adr/0032-scientific-acceptance-loopback-engine.md b/docs/adr/0032-scientific-acceptance-loopback-engine.md new file mode 100644 index 000000000..fea1e086e --- /dev/null +++ b/docs/adr/0032-scientific-acceptance-loopback-engine.md @@ -0,0 +1,115 @@ +# ADR 0032 — Scientific-acceptance loopback engine execute + +**Decision status:** Accepted +**Implementation maturity:** active-PR +**Date:** 2026-08-31 +**Supersedes:** None; complements ADR 0026 (engine library), ADR 0027 (GET), and ADR 0028 (lifecycle POST). Does not reuse ADR 0030 or ADR 0031. Does not supersede ADR 0014 claim-promotion authority. + +## Context + +ADR 0026 binds cutoff-safe evidence to a hash-stable validation run that emits +`tepp.scientific_acceptance.v1`. ADR 0027 serves GET status. ADR 0028 records +running and terminal status, but the terminal POST still requires +caller-supplied `scientific_acceptance_json`. Operators therefore cannot obtain +scientific-acceptance evidence from a `scientific_acceptance_v1` loopback run +without already possessing the artifact. Duplicating GET, lifecycle POST, the +engine library, collection GET, cancel HTTP, or the loopback CLI would collide +with live PRs. `analysis_engine` already depends on `tepp_api`; the reverse +dependency would cycle. + +## Decision + +`analysis_engine` owns `POST /v1/analysis-runs/{run_id}/execute` on a wrapper +around `AnalysisRunLiveService`: + +- The execute body carries corpus, recovery vectors, seed, pre-registered + SE-gate `k`, study label, completion time, and an explicit LLM-authorship + flag. It must not carry `scientific_acceptance_json` or receipt metric keys. +- The wrapper calls `submit_validation_run` then `complete_validation_run`, + records running then terminal through the public loopback recorder, and + returns the same status body GET would return. +- Only an accepted run whose request profile is `scientific_acceptance_v1` and + whose model is `validation_cpu_f64_v1` may execute. GET then returns + `tepp.scientific_acceptance.v1` without a caller-supplied artifact. +- Wrong profile, LLM recovery, metric keys, unknown run, consumer mismatch, + already-terminal status, digest mismatch, and unknown execute fields fail + closed. +- `tepp_api` recognizes the `/execute` suffix and refuses it so the raw + listener cannot pretend to execute. Persistence remains GAP-003B. + +## Non-goals + +- Production TLS, public bind, or durable status storage. +- Leiden community detection, Driver p.16 std-family restoration, or Figma/export work. +- Promoting an ADR 0014 scientific claim from HTTP success. +- Collection GET, cancel HTTP, or loopback CLI. + +## Alternatives considered + +1. **Keep caller-supplied terminal artifacts** — rejected because GAP-003A is + the operator-visible product-completion gap and the engine already emits the + schema. +2. **Add `analysis_engine` as a `tepp_api` dependency** — rejected as a crate + cycle. +3. **Persist execute rows in PostgreSQL** — rejected as GAP-003B / live draft + #287. +4. **Engine wrapper that records produced bytes on the existing lifecycle + path** — accepted. + +## Consequences + +- Operators can POST create, POST execute, and GET `tepp.scientific_acceptance.v1` + without supplying the artifact. +- Lifecycle POST remains the generic status-update path (ADR 0028). Execute is + the engine-owned production of those bytes. +- HTTP 200 on execute is not release evidence. + +## Failure and recovery + +Unknown run identities, extra path segments, metric keys, LLM recovery, wrong +profile, already-terminal runs, and consumer mismatch return a redacted `400` +envelope. Unsupported execute contract versions return `422`. Credential +headers remain `403`. The in-memory registry is not durable. + +## Security, privacy, scientific-integrity, and governance impact + +- No credential headers cross the consumer boundary. +- Execute remains loopback-only, size-bounded, and content-redacting. +- SHA-256 digest agreement is a byte-identity check, not a validity claim. +- LLM-authored recovery cannot become scientific authority. + +## Compatibility and migration + +GET status, POST create, POST running/terminal, temporal-context, and +project-history paths are unchanged. Production adapters may replace loopback +while preserving metric-free receipts and engine-produced scientific +acceptance. + +## Verification + +Falsifiable evidence: + +- POST create stays metric-free; +- POST execute without `scientific_acceptance_json` then GET returns + `tepp.scientific_acceptance.v1`; +- wrong profile, LLM recovery, metric keys, unknown run, consumer mismatch, + and a second execute on a terminal run fail closed; +- Clippy `-D warnings`, `analysis_engine` and `tepp_api` tests, rustdoc, and + exact-head review remain required. + +## Rollback and supersession + +Rollback removes the execute wrapper; GET, lifecycle POST, and the engine +library remain valid. A superseding ADR is required to persist status, bind a +public address, or treat HTTP success as an ADR 0014 claim. + +## Related authority + +- ADR 0026 owns the validation-run library bind. +- ADR 0027 owns the GET status read. +- ADR 0028 owns POST running/terminal. +- ADR 0018 owns consumer-scoped ingress and metric-free `202 Accepted`. +- ADR 0022 owns deterministic execution to a digest-bound terminal result. +- ADR 0014 owns scientific claim promotion. +- ADR 0008 owns SHA-256 identity. +- ADR 0011 owns standalone/modular HTTP boundaries. diff --git a/docs/adr/README.md b/docs/adr/README.md index d063a07b2..4bff0761f 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -33,6 +33,7 @@ Read [`ADR_POLICY.md`](ADR_POLICY.md) first. **Decision status and implementatio | [0026](0026-validation-run-scientific-acceptance.md) | Durable validation-run scientific acceptance evidence | Accepted | active-PR | GAP-003A first slice binds cutoff-safe evidence to a hash-stable run that emits `tepp.scientific_acceptance.v1`; Postgres persistence remains GAP-003B. | | [0027](0027-scientific-acceptance-http-status.md) | Scientific-acceptance loopback HTTP status path | Accepted | active-PR | GET `/v1/analysis-runs/{run_id}` stays metric-free on accepted/running; `tepp.scientific_acceptance.v1` only on succeeded `scientific_acceptance_v1`. | | [0028](0028-scientific-acceptance-http-lifecycle.md) | Scientific-acceptance loopback HTTP lifecycle POST | Accepted | active-PR | POST `/running` and `/terminal` are the production status-update path; GET remains ADR 0027. Persistence remains GAP-003B. | +| [0032](0032-scientific-acceptance-loopback-engine.md) | Scientific-acceptance loopback engine execute | Accepted | active-PR | POST `/execute` runs `analysis_engine` on the loopback lifecycle so GET returns `tepp.scientific_acceptance.v1` without a caller-supplied artifact. Persistence remains GAP-003B. | | [0023](0023-lineage-criterion-anchor-contract.md) | TEPP-owned Event Lineage criterion anchor | Accepted | active-PR | PR #237 publishes the strict accepted/rejected artifact and identities; estimator execution remains fail-closed future work. | | [0024](0024-independent-topic-importance-anchor.md) | Posterior topic-context producer contract | Accepted | contract-only active-PR | Strict DTO/schema only; the current estimator does not emit it. fast-mlsirm owns case-deletion influence. | | [0001](0001-rust-first-modular-msa.md) | Rust-first numerical core and CPU `f64` reference | Accepted | partial | ADR 0011 owns cross-service/MSA authority; 0001 retains numerical/backend authority. | @@ -141,6 +142,7 @@ Use the narrowest owning ADR when decisions overlap: - **project-history wire-size symmetry:** ADR 0019. - **LineageWeave project-history service boundary:** ADR 0021. - **accepted-run execution and terminal artifact production:** ADR 0022. +- **scientific-acceptance loopback engine execute:** ADR 0032. - **independent lineage criterion and posterior Project Journey:** ADR 0023. - **macOS-native Rust-owned MLX Metal execution:** ADR 0024. diff --git a/docs/connectors/naruon-artifact-consumer.md b/docs/connectors/naruon-artifact-consumer.md index 9c6f6d185..b28eec968 100644 --- a/docs/connectors/naruon-artifact-consumer.md +++ b/docs/connectors/naruon-artifact-consumer.md @@ -28,6 +28,7 @@ TEPP remains the scientific authority for estimation, recovery metrics, temporal | HTTP analysis-run create | `tepp_api` `naruon_analysis_run_exchange` → `POST /v1/analysis-runs` | naruon → TEPP | | HTTP export authorize | `tepp_api` `naruon_export_exchange` → `POST /v1/exports` | naruon → TEPP | | Live loopback POST | `tepp_api` `NaruonLiveService` → `POST /v1/analysis-runs` and `/v1/exports` | naruon → TEPP | +| Live loopback scientific-acceptance execute | `analysis_engine` `ScientificAcceptanceLoopbackService` → `POST /v1/analysis-runs/{run_id}/execute` | naruon → TEPP | Committed examples live under `examples/`. Schemas for analysis-run requests and corpus-split manifests live under `schemas/`. diff --git a/docs/research/scientific-acceptance-loopback-engine.md b/docs/research/scientific-acceptance-loopback-engine.md new file mode 100644 index 000000000..eadf07e90 --- /dev/null +++ b/docs/research/scientific-acceptance-loopback-engine.md @@ -0,0 +1,31 @@ +# Scientific-acceptance loopback engine execute + +## Scope + +This note doctors the GAP-003A engine-on-loopback slice: + +1. `POST /v1/analysis-runs/{run_id}/execute` runs `submit_validation_run` and `complete_validation_run` against an accepted loopback run; +2. the execute body carries corpus, recovery, seed, and pre-registered SE-gate `k` and must not carry `scientific_acceptance_json`; +3. the engine records running then terminal so GET returns `tepp.scientific_acceptance.v1` without a caller-supplied artifact; +4. wrong profile, LLM-authored recovery, receipt metric keys, unknown run, consumer mismatch, already-terminal status, and digest mismatch fail closed. + +Postgres persistence, restart/recovery, and Compose execution remain GAP-003B. This slice is not implemented-main. It does not duplicate the engine library (#356), terminal-result DTO (#358), GET (#359), lifecycle POST (#360), cancel HTTP (#361), loopback CLI (#362), or collection GET (#368). + +## Authoritative sources + +National Academies of Sciences, Engineering, and Medicine. (2019). *Reproducibility and replicability in science*. The National Academies Press. https://doi.org/10.17226/25303 + +Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on *p*-values: Context, process, and purpose. *The American Statistician, 70*(2), 129–133. https://doi.org/10.1080/00031305.2016.1154108 + +Wilson, E. B. (1927). Probable inference, the law of succession, and statistical inference. *Journal of the American Statistical Association, 22*(158), 209–212. https://doi.org/10.1080/01621459.1927.10502953 + +## Application + +The National Academies (2019) separate computational reproducibility (same binding, same digest) from a scientific claim that recovery is correct. Wasserstein and Lazar (2016) refuse to treat a passing threshold as automatic scientific authority, so execute produces the same `tepp.scientific_acceptance.v1` evidence as the library bind, including `se_gate_accepted = false` when the pre-registered gate fails, and never treats HTTP `200` as ADR 0014 promotion. Wilson (1927) supplies the coverage interval already implemented in `validation_core`. TEPP therefore executes the engine on the loopback lifecycle path, refuses caller-supplied artifacts and LLM-authored recovery, and reports RMSE, bias, coverage, temporal order, and the SE-aware gate only after engine completion (National Academies of Sciences, Engineering, and Medicine, 2019; Wasserstein & Lazar, 2016; Wilson, 1927). Meredith (1993) remains unread (Unpaywall/OpenAlex 2026-08-31T10:00Z: `is_oa: false`, 0 locations). Mislevy (1991, *Psychometrika, 56*, 177–196) remains unread on the same terms (DOI `10.1007/bf02294457`). + +## Verification + +- POST create then POST execute without `scientific_acceptance_json` then GET returns `tepp.scientific_acceptance.v1`; +- a `calibrated_event_measurement` run stays accepted and metric-free after execute is refused; +- LLM authorship, receipt metric keys, unknown-field `scientific_acceptance_json`, unknown run, consumer mismatch, and a second execute on a terminal run fail closed; +- the raw `AnalysisRunLiveService` still returns `400` for `/execute`.