diff --git a/CHANGELOG.md b/CHANGELOG.md index 062a69412..1d429c3f4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -38,6 +38,8 @@ All notable changes to TEPP are documented here. The format follows Keep a Chang ## [Unreleased] +- **Composed fitted-`K` topic-lineage analysis-run profile**: cutoff-safe `composed_fitted_lineage_v1` binds `select_fitted_candidate_k` then `execute_topic_lineage_run` at the selected `K` (`analysis_engine`). Not a Schwarz-only bind, not a Pareto-front bind, not a Bayesian sampler, and not implemented-main. + - `event_core` adds bounded Allen interval-consistency classification, atomic path-consistency closure, contradiction/resource refusals, and an explicit dependency-error fallback without claiming unrestricted global satisfiability. - `psychometric_core` recovers the Driver, Oud, and Voelkle (2017, Table 2, p. 12 `MANIFESTTRAITVAR`; §7.1, p. 19; p. 16 `MANIFESTTRAITVARstd`; footnote 4; 2017-era ctsem `summary.ctsemFit.R`; JSS PDF re-opened 2026-08-27T14:20Z from https://www.jstatsoft.org/index.php/jss/article/download/v077i05/1104) scalar standardised manifest-trait variance on current main after `0ce16e8` dropped the pre-consolidation code while research notes already named the map (register items 83–84). Table 2 names `MANIFESTTRAITVAR` `Ψ_τ` the additional time-invariant variance-covariance on the measurement level and sets it `NULL` when there is no manifest trait. Equation 5 writes `Γ ~ N(τ, Ψ)` and names that covariance the manifest traits. Section 7.1 names manifest traits stable individual differences in indicator levels, distinct from process-level `TRAITVAR` `φ_ξ`. Page 16 prints standardised matrices with the suffix `std` when appropriate. The printed example on p. 16 is `discreteDRIFTstd`, not `MANIFESTTRAITVARstd`. Footnote 4 standardises using only the relevant variance, not the total. The relevant variance for that named indicator-level correlation is `MANIFESTTRAITVAR`, not process-level `TRAITVAR` and not residual `MANIFESTVAR` `θ`. The 2017-era source forms `MANIFESTTRAITVARstd` only when `MANIFESTTRAITVAR != 0`, as `solve(sqrt(diag(MANIFESTTRAITVAR) + ridging)) %&% MANIFESTTRAITVAR` when `verbose = TRUE`. OpenMx `%&%` is `t(A) %*% B %*% A`. Unlike `TRAITVARstd`, that formation adds `diag(c(ridging), n.manifest)`. The default `ridging = FALSE` adds 0, not `0.0001`; that ridge is a numerical hack and is not this exact map. The scalar correlation is `ψ / ψ = 1` after strictly positive `MANIFESTTRAITVAR`. Form strictly positive `ψ` first, then `1 / √ψ`, then `(1 / √ψ) ψ (1 / √ψ)`. Unstandardised `MANIFESTTRAITVAR` is defined for a zero trait; standardised `MANIFESTTRAITVAR` is not. Zero `MANIFESTTRAITVAR` skips forming `MANIFESTTRAITVARstd` in the 2017-era source and fails closed here. Indicator-level trait variance is an event-time structural quantity, so a non-event clock fails closed. `MANIFESTTRAITVAR` does not require stable `a < 0`. Distinct positive `ψ` recover the same 1. `trait / trait = 1` is `TRAITVARstd` and recovers the same number and remains a distinct named quantity. `θ` is `MANIFESTVAR` and is measurement error, not this correlation. Meredith (1993) remains unread (web search 2026-08-27T14:20Z: Springer/Cambridge Core paywalled; Unpaywall historically `is_oa: false`; Springer `content/pdf` is an HTML stub). Mislevy (1991, *Psychometrika, 56*, 177–196) remains unread on the same terms (DOI `10.1007/bf02294457`). Still not a Kalman filter, not a matrix `expm`, not ESEM estimation, not DSEM, and not ctsem estimation. diff --git a/Cargo.lock b/Cargo.lock index 454a7d612..0949729f1 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -74,6 +74,7 @@ dependencies = [ "corpus_split", "event_core", "membership_core", + "model_selection", "relation_graph", "serde", "serde_json", diff --git a/crates/analysis_engine/Cargo.toml b/crates/analysis_engine/Cargo.toml index 7322212b2..c45e2eed7 100644 --- a/crates/analysis_engine/Cargo.toml +++ b/crates/analysis_engine/Cargo.toml @@ -15,6 +15,7 @@ publish = false [dependencies] event_core = { path = "../event_core", version = "0.2.0" } +model_selection = { path = "../model_selection", version = "0.2.0" } serde = { workspace = true } serde_json = { workspace = true } sha2 = { workspace = true } diff --git a/crates/analysis_engine/src/composed_fitted_lineage_artifact.rs b/crates/analysis_engine/src/composed_fitted_lineage_artifact.rs new file mode 100644 index 000000000..9e14ec577 --- /dev/null +++ b/crates/analysis_engine/src/composed_fitted_lineage_artifact.rs @@ -0,0 +1,385 @@ +//! Digest-bound fitted candidate-`K` selection composed with topic lineage. + +use model_selection::{FittedCandidateKConfig, select_fitted_candidate_model}; +use serde::{Deserialize, Serialize}; +use sha2::{Digest, Sha256}; +use temporal_core::KnowledgeCutoff; +use tepp_api::{ + AnalysisResultSummary, AnalysisRunAccepted, AnalysisRunRequest, AnalysisRunTerminalResult, +}; +use topic_measurement::ReferenceTopicInput; + +use crate::topic_lineage_artifact::{ + TOPIC_LINEAGE_MODEL_CONTRACT_VERSION, TOPIC_LINEAGE_OUTPUT_PROFILE, + topic_lineage_execution_from_model, +}; +use crate::{AnalysisEngineError, format_digest, require_receipt_identity, valid_identifier}; + +/// Versioned schema for a completed composed fitted-lineage artifact. +pub const COMPOSED_FITTED_LINEAGE_ARTIFACT_SCHEMA_VERSION: &str = "tepp.composed_fitted_lineage.v1"; +/// Model contract required by the composed fitted-lineage execution path. +pub const COMPOSED_FITTED_LINEAGE_MODEL_CONTRACT_VERSION: &str = "composed_fitted_lineage_v1"; +/// Analysis-run output profile required for a composed fitted-lineage artifact. +pub const COMPOSED_FITTED_LINEAGE_OUTPUT_PROFILE: &str = "composed_fitted_lineage_v1"; +/// Maximum canonical artifact JSON size. +pub const COMPOSED_FITTED_LINEAGE_ARTIFACT_BYTE_LIMIT: usize = 256 * 1024; +const COMPOSED_FITTED_LINEAGE_INFERENCE_STATUS: &str = + "fitted_k_composed_lineage_not_bayesian_sampler"; + +/// Cutoff-safe composition payload: fitted selection plus production lineage. +#[derive(Clone, Debug)] +pub struct ComposedFittedLineageInput<'a> { + input: &'a ReferenceTopicInput, + selection: &'a FittedCandidateKConfig, + method_name: &'a str, + llm_votes: &'a [u32], +} + +impl<'a> ComposedFittedLineageInput<'a> { + /// Construct a composition payload from existing scientific-crate values. + #[must_use] + pub const fn new( + input: &'a ReferenceTopicInput, + selection: &'a FittedCandidateKConfig, + method_name: &'a str, + llm_votes: &'a [u32], + ) -> Self { + Self { + input, + selection, + method_name, + llm_votes, + } + } + + /// Borrow the cutoff-safe reference-topic input. + #[must_use] + pub const fn input(&self) -> &'a ReferenceTopicInput { + self.input + } + + /// Borrow the fitted candidate-`K` configuration. + #[must_use] + pub const fn selection(&self) -> &'a FittedCandidateKConfig { + self.selection + } + + /// Return the declared statistical method identity. + #[must_use] + pub const fn method_name(&self) -> &'a str { + self.method_name + } + + /// Borrow optional LLM votes. They cannot define the numerical optimum. + #[must_use] + pub const fn llm_votes(&self) -> &'a [u32] { + self.llm_votes + } +} + +/// Completed, bounded fitted-`K` plus topic-lineage composition. +#[derive(Clone, Debug, Deserialize, PartialEq, Serialize)] +#[serde(deny_unknown_fields)] +pub struct ComposedFittedLineageArtifact { + /// Exact versioned schema identity. + pub schema_version: String, + /// Opaque accepted-run identity. + pub run_id: String, + /// Immutable source snapshot identity. + pub snapshot_id: String, + /// Historical evidence cutoff used by selection and the lineage fit. + pub knowledge_cutoff: String, + /// Statistically selected topic count `K`. + pub selected_k: u64, + /// Number of candidate topic counts offered to fitted selection. + pub candidate_count: u64, + /// Number of modeled evidence documents. + pub evidence_count: u64, + /// Topic count of the production lineage fit at selected `K`. + pub lineage_topic_count: u64, + /// Number of fitted same-topic sequence edges. + pub lineage_edge_count: u64, + /// Documents incident to at least one fitted sequence edge. + pub connected_post_count: u64, + /// SHA-256 digest of the inner topic-lineage artifact. + pub lineage_artifact_sha256: String, + /// Fixed claim boundary for consumer copy. + pub inference_status: String, +} + +impl ComposedFittedLineageArtifact { + /// Parse and fully validate a bounded artifact JSON payload. + /// + /// # Errors + /// + /// Returns [`AnalysisEngineError::InvalidComposedFittedLineageArtifact`] + /// when the schema, identifiers, counts, digest, or claim boundary fail. + pub fn from_json(payload: &str) -> Result { + if payload.len() > COMPOSED_FITTED_LINEAGE_ARTIFACT_BYTE_LIMIT { + return Err(AnalysisEngineError::LimitExceeded); + } + let artifact: Self = serde_json::from_str(payload) + .map_err(|_| AnalysisEngineError::InvalidComposedFittedLineageArtifact)?; + artifact.validate()?; + Ok(artifact) + } + + /// Serialize canonical validated artifact JSON. + /// + /// # Errors + /// + /// Returns a typed validation, serialization, or size failure. + pub fn to_json(&self) -> Result { + self.validate()?; + let payload = + serde_json::to_string(self).map_err(|_| AnalysisEngineError::SerializationFailure)?; + Ok(payload) + } + + /// Return the lowercase SHA-256 digest of canonical artifact JSON. + /// + /// # Errors + /// + /// Returns a typed validation or serialization failure. + pub fn sha256(&self) -> Result { + self.to_json() + .map(|json| format_digest(Sha256::digest(json.into_bytes()))) + } + + fn validate(&self) -> Result<(), AnalysisEngineError> { + if self.schema_version != COMPOSED_FITTED_LINEAGE_ARTIFACT_SCHEMA_VERSION + || !valid_identifier(&self.run_id) + || !valid_identifier(&self.snapshot_id) + || KnowledgeCutoff::parse_rfc3339(&self.knowledge_cutoff).is_err() + || self.selected_k < 2 + || self.candidate_count == 0 + || self.evidence_count < 2 + || self.lineage_topic_count != self.selected_k + || self.connected_post_count > self.evidence_count + || self.lineage_edge_count > self.evidence_count.saturating_mul(self.evidence_count - 1) + || self.lineage_artifact_sha256.len() != 64 + || !self + .lineage_artifact_sha256 + .bytes() + .all(|byte| matches!(byte, b'0'..=b'9' | b'a'..=b'f')) + || self.inference_status != COMPOSED_FITTED_LINEAGE_INFERENCE_STATUS + { + return Err(AnalysisEngineError::InvalidComposedFittedLineageArtifact); + } + Ok(()) + } +} + +/// One completed composed fitted-lineage artifact and its terminal result. +#[derive(Clone, Debug, PartialEq)] +pub struct ComposedFittedLineageExecution { + /// Digest-bound composed selection-plus-lineage artifact. + pub artifact: ComposedFittedLineageArtifact, + /// Terminal result carrying the composed artifact identity, digest, and schema. + pub terminal_result: AnalysisRunTerminalResult, +} + +/// Execute fitted candidate-`K` selection then the CPU `f64` topic-lineage fit. +/// +/// The executor invokes [`select_fitted_candidate_model`] and reuses its exact +/// winning fit to build topic lineage; it does not reimplement Schwarz scoring or +/// lineage edges. LLM votes cannot define the numerical optimum. This is not +/// a standalone fitted-`K` profile, not a Pareto-front profile, and not a +/// Bayesian sampler. +/// +/// # Errors +/// +/// Returns a request/receipt/snapshot/cutoff/profile error, model-selection +/// failure, estimator failure, or invalid artifact error. +pub fn execute_composed_fitted_lineage_run( + request: &AnalysisRunRequest, + accepted: &AnalysisRunAccepted, + snapshot_id: &str, + knowledge_cutoff: KnowledgeCutoff, + composition: &ComposedFittedLineageInput<'_>, + completed_at: impl Into, +) -> Result { + request.to_json()?; + accepted.to_json()?; + require_receipt_identity(request, accepted)?; + if request.snapshot_id != snapshot_id { + return Err(AnalysisEngineError::SnapshotMismatch); + } + if request.knowledge_cutoff != knowledge_cutoff.to_rfc3339() + || request.model_contract_version != COMPOSED_FITTED_LINEAGE_MODEL_CONTRACT_VERSION + || request.output_profile != COMPOSED_FITTED_LINEAGE_OUTPUT_PROFILE + || !valid_identifier(composition.method_name()) + || !composition.input().is_eligible_at(&knowledge_cutoff) + { + return Err(AnalysisEngineError::InvalidEvidence); + } + + let (selected_k, selected_model) = select_fitted_candidate_model( + composition.input(), + composition.selection(), + composition.method_name(), + composition.llm_votes(), + )?; + let mut lineage_request = request.clone(); + lineage_request.model_contract_version = TOPIC_LINEAGE_MODEL_CONTRACT_VERSION.into(); + lineage_request.output_profile = TOPIC_LINEAGE_OUTPUT_PROFILE.into(); + let completed_at = completed_at.into(); + #[rustfmt::skip] + let lineage = topic_lineage_execution_from_model(&lineage_request, accepted, snapshot_id, knowledge_cutoff, composition.input(), &selected_model, completed_at.clone())?; + let connected_post_count = lineage.artifact.connected_post_count; + let artifact = ComposedFittedLineageArtifact { + schema_version: COMPOSED_FITTED_LINEAGE_ARTIFACT_SCHEMA_VERSION.into(), + run_id: accepted.run_id.clone(), + snapshot_id: snapshot_id.to_owned(), + knowledge_cutoff: knowledge_cutoff.to_rfc3339(), + selected_k: u64::from(selected_k), + candidate_count: u64::try_from(composition.selection().candidate_topic_counts().len()) + .map_err(|_| AnalysisEngineError::ArithmeticOverflow)?, + evidence_count: lineage.artifact.evidence_count, + lineage_topic_count: lineage.artifact.topic_count, + lineage_edge_count: u64::try_from(lineage.artifact.sequence_edges.len()) + .map_err(|_| AnalysisEngineError::ArithmeticOverflow)?, + connected_post_count, + lineage_artifact_sha256: lineage.artifact.sha256()?, + inference_status: COMPOSED_FITTED_LINEAGE_INFERENCE_STATUS.into(), + }; + let digest = artifact.sha256()?; + #[rustfmt::skip] + let summary = AnalysisResultSummary::new("composed_fitted_lineage", artifact.evidence_count, 4, COMPOSED_FITTED_LINEAGE_INFERENCE_STATUS)?; + #[rustfmt::skip] + let terminal_result = AnalysisRunTerminalResult::succeeded(request, accepted, format!("composed_fitted_lineage_artifact_{}", &digest[..16]), digest, COMPOSED_FITTED_LINEAGE_ARTIFACT_SCHEMA_VERSION, completed_at, summary)?; + Ok(ComposedFittedLineageExecution { + artifact, + terminal_result, + }) +} + +#[cfg(test)] +mod tests { + use super::{ + COMPOSED_FITTED_LINEAGE_ARTIFACT_BYTE_LIMIT, + COMPOSED_FITTED_LINEAGE_ARTIFACT_SCHEMA_VERSION, COMPOSED_FITTED_LINEAGE_INFERENCE_STATUS, + ComposedFittedLineageArtifact, + }; + use crate::AnalysisEngineError; + + fn artifact() -> ComposedFittedLineageArtifact { + ComposedFittedLineageArtifact { + schema_version: COMPOSED_FITTED_LINEAGE_ARTIFACT_SCHEMA_VERSION.into(), + run_id: "run-1".into(), + snapshot_id: "snapshot-1".into(), + knowledge_cutoff: "2026-08-01T00:00:00Z".into(), + selected_k: 2, + candidate_count: 2, + evidence_count: 4, + lineage_topic_count: 2, + lineage_edge_count: 2, + connected_post_count: 4, + lineage_artifact_sha256: "ab".repeat(32), + inference_status: COMPOSED_FITTED_LINEAGE_INFERENCE_STATUS.into(), + } + } + + fn assert_invalid(artifact: &ComposedFittedLineageArtifact) { + assert_eq!( + artifact.to_json(), + Err(AnalysisEngineError::InvalidComposedFittedLineageArtifact) + ); + } + + #[test] + fn artifact_round_trip_and_size_bounds_fail_closed() { + let artifact = artifact(); + let payload = artifact.to_json().expect("json"); + assert_eq!( + ComposedFittedLineageArtifact::from_json(&payload), + Ok(artifact.clone()) + ); + assert_eq!(artifact.sha256().expect("digest").len(), 64); + assert_eq!( + ComposedFittedLineageArtifact::from_json("{}"), + Err(AnalysisEngineError::InvalidComposedFittedLineageArtifact) + ); + assert_eq!( + ComposedFittedLineageArtifact::from_json( + &"x".repeat(COMPOSED_FITTED_LINEAGE_ARTIFACT_BYTE_LIMIT + 1) + ), + Err(AnalysisEngineError::LimitExceeded) + ); + } + + #[test] + fn artifact_metadata_tampering_fails_closed() { + let artifact = artifact(); + let invalid_artifacts = [ + { + let mut value = artifact.clone(); + value.schema_version.clear(); + value + }, + { + let mut value = artifact.clone(); + value.run_id.clear(); + value + }, + { + let mut value = artifact.clone(); + value.snapshot_id.clear(); + value + }, + { + let mut value = artifact.clone(); + value.knowledge_cutoff = "invalid".into(); + value + }, + { + let mut value = artifact.clone(); + value.selected_k = 1; + value + }, + { + let mut value = artifact.clone(); + value.candidate_count = 0; + value + }, + { + let mut value = artifact.clone(); + value.evidence_count = 1; + value + }, + { + let mut value = artifact.clone(); + value.lineage_topic_count = 3; + value + }, + { + let mut value = artifact.clone(); + value.connected_post_count = 5; + value + }, + { + let mut value = artifact.clone(); + value.lineage_edge_count = 13; + value + }, + { + let mut value = artifact.clone(); + value.lineage_artifact_sha256.clear(); + value + }, + { + let mut value = artifact.clone(); + value.lineage_artifact_sha256 = "GG".repeat(32); + value + }, + { + let mut value = artifact.clone(); + value.inference_status.clear(); + value + }, + ]; + for invalid in invalid_artifacts { + assert_invalid(&invalid); + } + } +} diff --git a/crates/analysis_engine/src/lib.rs b/crates/analysis_engine/src/lib.rs index 72bd5854c..6504234cd 100644 --- a/crates/analysis_engine/src/lib.rs +++ b/crates/analysis_engine/src/lib.rs @@ -8,13 +8,18 @@ //! through [`tepp_api`]. It deliberately does not claim latent-variable or topic //! estimation authority; those estimators remain separate scientific crates. //! estimation authority; it invokes estimators through their scientific crate -//! contracts and preserves their artifact meaning. +//! contracts and preserves their artifact meaning. Fitted candidate-`K` +//! composed with topic lineage is invoked through [`model_selection`] and +//! [`execute_topic_lineage_run`] and is not a Bayesian sampler. mod case_deletion_refit; +mod composed_fitted_lineage_artifact; mod lineage_criterion; mod topic_context_posterior; mod topic_lineage_artifact; +use model_selection::ModelSelectionError; + use serde::Serialize; use sha2::{Digest, Sha256}; use std::collections::BTreeSet; @@ -41,6 +46,13 @@ pub use case_deletion_refit::ExhaustiveCaseDeletionError; pub use case_deletion_refit::ExhaustiveCaseDeletionFits; /// Fit the full corpus and every actual one-document deletion. pub use case_deletion_refit::fit_exhaustive_case_deletion; +/// Composed fitted-`K` plus topic-lineage artifact and execution contracts. +pub use composed_fitted_lineage_artifact::{ + COMPOSED_FITTED_LINEAGE_ARTIFACT_BYTE_LIMIT, COMPOSED_FITTED_LINEAGE_ARTIFACT_SCHEMA_VERSION, + COMPOSED_FITTED_LINEAGE_MODEL_CONTRACT_VERSION, COMPOSED_FITTED_LINEAGE_OUTPUT_PROFILE, + ComposedFittedLineageArtifact, ComposedFittedLineageExecution, ComposedFittedLineageInput, + execute_composed_fitted_lineage_run, +}; /// Rust-owned independent TDT link-criterion posterior fitting contracts. pub use lineage_criterion::{ LineageCriterionFit, LineageCriterionFitError, LineageCriterionObservation, @@ -248,6 +260,10 @@ pub enum AnalysisEngineError { TopicMeasurement(TopicMeasurementError), /// A topic-lineage artifact violated its bounded schema or count invariants. InvalidTopicLineageArtifact, + /// A model-selection gate rejected the offered candidates or method. + ModelSelection(ModelSelectionError), + /// A composed fitted-lineage artifact violated its bounded schema or counts. + InvalidComposedFittedLineageArtifact, } impl fmt::Display for AnalysisEngineError { @@ -262,6 +278,10 @@ impl fmt::Display for AnalysisEngineError { Self::LimitExceeded => "analysis corpus exceeded its execution bound", Self::TopicMeasurement(error) => return error.fmt(formatter), Self::InvalidTopicLineageArtifact => "invalid topic lineage artifact", + Self::ModelSelection(error) => return error.fmt(formatter), + Self::InvalidComposedFittedLineageArtifact => { + "invalid composed fitted-lineage artifact" + } }; formatter.write_str(message) } @@ -281,6 +301,12 @@ impl From for AnalysisEngineError { } } +impl From for AnalysisEngineError { + fn from(error: ModelSelectionError) -> Self { + Self::ModelSelection(error) + } +} + /// Execute the cutoff-safe temporal evidence readiness analysis. /// /// Evidence whose `available_time` is later than the request cutoff is excluded @@ -413,7 +439,8 @@ mod tests { use super::{ ANALYSIS_ARTIFACT_SCHEMA_VERSION, ANALYSIS_STATISTIC_COUNT, AnalysisCorpus, AnalysisEngineError, AnalysisEvidenceUnit, MAX_ANALYSIS_IDENTIFIER_BYTES, - MAX_EVIDENCE_UNITS, TopicMeasurementError, add_membership_count, execute_analysis_run, + MAX_EVIDENCE_UNITS, ModelSelectionError, TopicMeasurementError, add_membership_count, + execute_analysis_run, }; use temporal_core::{AvailableTime, EventTime}; use tepp_api::{AnalysisRunAccepted, AnalysisRunRequest, AnalysisRunTerminalState, ApiError}; @@ -681,6 +708,14 @@ mod tests { AnalysisEngineError::InvalidTopicLineageArtifact, "invalid topic lineage artifact", ), + ( + AnalysisEngineError::InvalidComposedFittedLineageArtifact, + "invalid composed fitted-lineage artifact", + ), + ( + AnalysisEngineError::ModelSelection(ModelSelectionError::EmptyCandidateSet), + "empty model-selection candidate set", + ), ]; for (error, message) in messages { assert_eq!(error.to_string(), message); @@ -689,6 +724,12 @@ mod tests { assert_eq!(converted.to_string(), "invalid API wire payload"); let from_topic: AnalysisEngineError = TopicMeasurementError::DidNotConverge.into(); assert_eq!(from_topic.to_string(), "topic estimator did not converge"); + let from_selection: AnalysisEngineError = + ModelSelectionError::LlmVoteIsNotStatisticalAuthority.into(); + assert_eq!( + from_selection.to_string(), + "llm vote is not statistical authority" + ); assert_eq!( add_membership_count(u64::MAX, 1), Err(AnalysisEngineError::ArithmeticOverflow) diff --git a/crates/analysis_engine/src/topic_lineage_artifact.rs b/crates/analysis_engine/src/topic_lineage_artifact.rs index 9b33ce179..6e27466bb 100644 --- a/crates/analysis_engine/src/topic_lineage_artifact.rs +++ b/crates/analysis_engine/src/topic_lineage_artifact.rs @@ -9,7 +9,7 @@ use tepp_api::{ AnalysisResultSummary, AnalysisRunAccepted, AnalysisRunRequest, AnalysisRunTerminalResult, }; use topic_measurement::{ - ReferenceTopicInput, ReferenceTopicModelConfig, fit_reference_topic_model, + ReferenceTopicInput, ReferenceTopicModel, ReferenceTopicModelConfig, fit_reference_topic_model, }; use uuid::Uuid; @@ -197,11 +197,32 @@ pub fn execute_topic_lineage_run( if request.knowledge_cutoff != knowledge_cutoff.to_rfc3339() || request.model_contract_version != TOPIC_LINEAGE_MODEL_CONTRACT_VERSION || request.output_profile != TOPIC_LINEAGE_OUTPUT_PROFILE + || !input.is_eligible_at(&knowledge_cutoff) { return Err(AnalysisEngineError::InvalidEvidence); } let model = fit_reference_topic_model(input, config)?; + topic_lineage_execution_from_model( + request, + accepted, + snapshot_id, + knowledge_cutoff, + input, + &model, + completed_at, + ) +} + +pub(crate) fn topic_lineage_execution_from_model( + request: &AnalysisRunRequest, + accepted: &AnalysisRunAccepted, + snapshot_id: &str, + knowledge_cutoff: KnowledgeCutoff, + input: &ReferenceTopicInput, + model: &ReferenceTopicModel, + completed_at: impl Into, +) -> Result { let topic_count = u64::try_from(model.topic_term_probabilities.len()) .map_err(|_| AnalysisEngineError::ArithmeticOverflow)?; let evidence_count = u64::try_from(input.document_count()) diff --git a/crates/analysis_engine/tests/composed_fitted_lineage_execution_contract.rs b/crates/analysis_engine/tests/composed_fitted_lineage_execution_contract.rs new file mode 100644 index 000000000..0458f5408 --- /dev/null +++ b/crates/analysis_engine/tests/composed_fitted_lineage_execution_contract.rs @@ -0,0 +1,334 @@ +//! End-to-end contract for fitted candidate-`K` composed with topic lineage. + +use analysis_engine::{ + AnalysisEngineError, COMPOSED_FITTED_LINEAGE_ARTIFACT_SCHEMA_VERSION, + COMPOSED_FITTED_LINEAGE_MODEL_CONTRACT_VERSION, COMPOSED_FITTED_LINEAGE_OUTPUT_PROFILE, + ComposedFittedLineageInput, TOPIC_LINEAGE_MODEL_CONTRACT_VERSION, TOPIC_LINEAGE_OUTPUT_PROFILE, + execute_composed_fitted_lineage_run, execute_topic_lineage_run, +}; +use corpus_split::{CorpusDocument, CorpusSnapshot}; +use membership_core::{ + GroupId, MemberId, MembershipAssignment, MembershipNetwork, MembershipRole, MembershipWeight, +}; +use model_selection::{FittedCandidateKConfig, ModelSelectionError}; +use relation_graph::{ + RelationEdge, RelationEndpointId, RelationEvidenceStatus, RelationGraph, RelationKind, +}; +use temporal_core::{ + AvailableTime, EventTime, KnowledgeCutoff, TemporalBoundary, TemporalInterval, + TemporalPrecision, +}; +use tepp_api::{AnalysisRunAccepted, AnalysisRunRequest, AnalysisRunTerminalState}; +use topic_measurement::{ReferenceTopicInput, SparseMatrix}; +use uuid::Uuid; + +fn event_time(day: u8) -> EventTime { + EventTime::parse_rfc3339(&format!("2026-07-{day:02}T00:00:00Z")).expect("event time") +} + +fn cutoff() -> KnowledgeCutoff { + KnowledgeCutoff::parse_rfc3339("2026-08-01T00:00:00Z").expect("cutoff") +} + +fn fixture() -> ReferenceTopicInput { + fixture_available_at("2026-07-01T00:00:00Z", cutoff()) +} + +fn fixture_available_at( + available_at: &str, + admission_cutoff: KnowledgeCutoff, +) -> ReferenceTopicInput { + let ids: Vec<_> = (1_u128..=4).map(Uuid::from_u128).collect(); + let times: Vec<_> = (1_u8..=4).map(event_time).collect(); + let available = AvailableTime::parse_rfc3339(available_at).expect("available"); + let mut snapshot = CorpusSnapshot::new(); + let mut memberships = MembershipNetwork::new(); + for id in &ids { + snapshot + .insert_if_eligible(CorpusDocument::new(*id, available), &admission_cutoff) + .expect("eligible"); + memberships + .insert( + MembershipAssignment::new( + MemberId::from_uuid(*id), + GroupId::from_uuid(Uuid::from_u128(100)), + MembershipRole::Project, + MembershipWeight::full().expect("weight"), + event_time(1), + event_time(9), + ) + .expect("membership"), + ) + .expect("insert"); + } + let mut relations = RelationGraph::new(); + for (source, target, source_day, target_day) in [(0, 1, 1, 2), (1, 2, 2, 3), (2, 3, 3, 4)] { + let interval = |day| { + TemporalInterval::bounded( + TemporalBoundary::Included(event_time(day)), + TemporalBoundary::Included( + EventTime::parse_rfc3339(&format!("2026-07-{day:02}T12:00:00Z")).expect("end"), + ), + TemporalPrecision::Second, + ) + .expect("interval") + }; + relations + .insert( + RelationEdge::new( + RelationKind::TransitionsTo, + RelationEndpointId::from_uuid(ids[source]), + RelationEndpointId::from_uuid(ids[target]), + RelationEvidenceStatus::Observed, + interval(source_day), + interval(target_day), + ) + .expect("relation"), + ) + .expect("insert relation"); + } + let counts = SparseMatrix::from_csr( + 4, + 4, + vec![0, 2, 4, 6, 8], + vec![0, 1, 0, 1, 2, 3, 2, 3], + vec![90.0, 10.0, 85.0, 15.0, 10.0, 90.0, 15.0, 85.0], + ) + .expect("counts"); + ReferenceTopicInput::new( + &snapshot, + ids, + &counts, + ×, + None, + &memberships, + &relations, + ) + .expect("input") +} + +fn request() -> AnalysisRunRequest { + AnalysisRunRequest { + contract_version: 1, + idempotency_key: "composed-fitted-lineage-idem".into(), + tenant_workspace_id: "tenant-workspace".into(), + snapshot_id: "snapshot-composed-fitted-lineage".into(), + knowledge_cutoff: "2026-08-01T00:00:00Z".into(), + model_contract_version: COMPOSED_FITTED_LINEAGE_MODEL_CONTRACT_VERSION.into(), + output_profile: COMPOSED_FITTED_LINEAGE_OUTPUT_PROFILE.into(), + } +} + +fn accepted(request: &AnalysisRunRequest) -> AnalysisRunAccepted { + AnalysisRunAccepted::new( + "run-composed-fitted-lineage", + "accepted", + &request.idempotency_key, + ) + .expect("accepted") +} + +fn selection() -> FittedCandidateKConfig { + FittedCandidateKConfig::new(vec![2, 3], vec![7, 11], 2_000, 1e-5).expect("selection") +} + +fn composition<'a>( + input: &'a ReferenceTopicInput, + selection: &'a FittedCandidateKConfig, + method_name: &'a str, + llm_votes: &'a [u32], +) -> ComposedFittedLineageInput<'a> { + ComposedFittedLineageInput::new(input, selection, method_name, llm_votes) +} + +fn execute( + request: &AnalysisRunRequest, +) -> Result { + let input = fixture(); + let selection = selection(); + execute_composed_fitted_lineage_run( + request, + &accepted(request), + "snapshot-composed-fitted-lineage", + cutoff(), + &composition(&input, &selection, "trsl_tm_reference", &[3]), + "2026-08-02T00:00:00Z", + ) +} + +#[test] +fn fitted_selection_then_lineage_emits_digest_bound_composition() { + let request = request(); + let execution = execute(&request).expect("execution"); + assert_eq!( + execution.artifact.schema_version, + COMPOSED_FITTED_LINEAGE_ARTIFACT_SCHEMA_VERSION + ); + assert!(execution.artifact.selected_k >= 2); + assert_eq!( + execution.artifact.lineage_topic_count, + execution.artifact.selected_k + ); + assert_eq!(execution.artifact.candidate_count, 2); + assert_eq!(execution.artifact.evidence_count, 4); + assert!(execution.artifact.lineage_edge_count >= 1); + assert_eq!(execution.artifact.lineage_artifact_sha256.len(), 64); + assert_eq!( + execution.artifact.inference_status, + "fitted_k_composed_lineage_not_bayesian_sampler" + ); + assert_eq!( + execution.terminal_result.run_state, + AnalysisRunTerminalState::Succeeded + ); + assert_eq!( + execution.terminal_result.result_sha256.as_deref(), + Some(execution.artifact.sha256().expect("digest").as_str()) + ); + assert_eq!( + execution.terminal_result.result_schema_version.as_deref(), + Some(COMPOSED_FITTED_LINEAGE_ARTIFACT_SCHEMA_VERSION) + ); +} + +#[test] +fn selected_fit_preserves_hyperparameters_and_cutoff_provenance() { + let request = request(); + let accepted = accepted(&request); + let input = fixture(); + let selection = selection() + .with_hyperparameters(1.7, 0.4, 0.03, 0.08, 0.12) + .expect("non-default selection"); + let execution = execute_composed_fitted_lineage_run( + &request, + &accepted, + "snapshot-composed-fitted-lineage", + cutoff(), + &composition(&input, &selection, "trsl_tm_reference", &[]), + "2026-08-02T00:00:00Z", + ) + .expect("composition"); + + let selected_k = u32::try_from(execution.artifact.selected_k).expect("selected K"); + let mut lineage_request = request.clone(); + lineage_request.model_contract_version = TOPIC_LINEAGE_MODEL_CONTRACT_VERSION.into(); + lineage_request.output_profile = TOPIC_LINEAGE_OUTPUT_PROFILE.into(); + let direct = execute_topic_lineage_run( + &lineage_request, + &accepted, + "snapshot-composed-fitted-lineage", + cutoff(), + &input, + &selection + .reference_config(selected_k) + .expect("exact config"), + "2026-08-02T00:00:00Z", + ) + .expect("direct lineage"); + assert_eq!( + execution.artifact.lineage_artifact_sha256, + direct.artifact.sha256().expect("lineage digest") + ); + + let late_input = fixture_available_at( + "2026-08-02T00:00:00Z", + KnowledgeCutoff::parse_rfc3339("2026-08-03T00:00:00Z").expect("later cutoff"), + ); + assert_eq!( + execute_composed_fitted_lineage_run( + &request, + &accepted, + "snapshot-composed-fitted-lineage", + cutoff(), + &composition(&late_input, &selection, "trsl_tm_reference", &[]), + "2026-08-02T00:00:00Z", + ), + Err(AnalysisEngineError::InvalidEvidence) + ); +} + +#[test] +fn lexical_method_and_empty_candidates_fail_closed() { + let request = request(); + let input = fixture(); + let selection = selection(); + assert_eq!( + execute_composed_fitted_lineage_run( + &request, + &accepted(&request), + "snapshot-composed-fitted-lineage", + cutoff(), + &composition(&input, &selection, "tfidf", &[]), + "2026-08-02T00:00:00Z", + ), + Err(AnalysisEngineError::ModelSelection( + ModelSelectionError::LexicalWeightForbidden + )) + ); + assert_eq!( + execute_composed_fitted_lineage_run( + &request, + &accepted(&request), + "snapshot-composed-fitted-lineage", + cutoff(), + &composition(&input, &selection, "", &[]), + "2026-08-02T00:00:00Z", + ), + Err(AnalysisEngineError::InvalidEvidence) + ); +} + +#[test] +fn execution_refuses_snapshot_profile_and_cutoff_mismatch() { + let request = request(); + let input = fixture(); + let selection = selection(); + assert_eq!( + execute_composed_fitted_lineage_run( + &request, + &accepted(&request), + "other-snapshot", + cutoff(), + &composition(&input, &selection, "trsl_tm_reference", &[]), + "2026-08-02T00:00:00Z", + ), + Err(AnalysisEngineError::SnapshotMismatch) + ); + for invalid_request in [ + { + let mut value = request.clone(); + value.knowledge_cutoff = "2026-08-02T00:00:00Z".into(); + value + }, + { + let mut value = request.clone(); + value.model_contract_version = "other-model".into(); + value + }, + { + let mut value = request.clone(); + value.output_profile = "fitted_candidate_k_v1".into(); + value + }, + { + let mut value = request.clone(); + value.output_profile = "pareto_candidate_k_v1".into(); + value + }, + { + let mut value = request.clone(); + value.output_profile = "trsl_topic_lineage_v1".into(); + value + }, + { + let mut value = request.clone(); + value.output_profile = "joint_posterior_draws_v1".into(); + value + }, + ] { + assert_eq!( + execute(&invalid_request), + Err(AnalysisEngineError::InvalidEvidence) + ); + } +} diff --git a/crates/analysis_engine/tests/topic_lineage_execution_contract.rs b/crates/analysis_engine/tests/topic_lineage_execution_contract.rs index 045a89465..3d7a7973a 100644 --- a/crates/analysis_engine/tests/topic_lineage_execution_contract.rs +++ b/crates/analysis_engine/tests/topic_lineage_execution_contract.rs @@ -158,13 +158,14 @@ fn fitted_topics_emit_digest_bound_predecessor_successor_counts() { } #[test] +#[allow(clippy::too_many_lines)] fn execution_refuses_binding_and_nonconvergence_without_an_artifact() { let (snapshot, ids, times, memberships, relations) = fixture(); let counts = SparseMatrix::from_csr(4, 2, vec![0, 1, 2, 3, 4], vec![0, 0, 1, 1], vec![1.0; 4]) .expect("counts"); let input = ReferenceTopicInput::new( &snapshot, - ids, + ids.clone(), &counts, ×, None, @@ -221,6 +222,38 @@ fn execution_refuses_binding_and_nonconvergence_without_an_artifact() { Err(AnalysisEngineError::InvalidEvidence) ); } + let late_available = + AvailableTime::parse_rfc3339("2026-08-02T00:00:00Z").expect("late available"); + let later_cutoff = + KnowledgeCutoff::parse_rfc3339("2026-08-03T00:00:00Z").expect("later cutoff"); + let mut late_snapshot = CorpusSnapshot::new(); + for id in &ids { + late_snapshot + .insert_if_eligible(CorpusDocument::new(*id, late_available), &later_cutoff) + .expect("later snapshot"); + } + let late_input = ReferenceTopicInput::new( + &late_snapshot, + ids, + &counts, + ×, + None, + &memberships, + &relations, + ) + .expect("late input"); + assert_eq!( + execute_topic_lineage_run( + &request, + &accepted, + "snapshot-topic-lineage", + cutoff, + &late_input, + &config, + "2026-08-02T00:00:00Z", + ), + Err(AnalysisEngineError::InvalidEvidence) + ); assert_eq!( execute_topic_lineage_run( &request, diff --git a/crates/corpus_split/src/snapshot.rs b/crates/corpus_split/src/snapshot.rs index 8e74c72ba..5e4e6623d 100644 --- a/crates/corpus_split/src/snapshot.rs +++ b/crates/corpus_split/src/snapshot.rs @@ -4,7 +4,7 @@ use crate::CorpusDocument; use crate::CorpusSplitError; use crate::cutoff_eligible; use std::collections::BTreeMap; -use temporal_core::KnowledgeCutoff; +use temporal_core::{AvailableTime, KnowledgeCutoff}; use uuid::Uuid; /// Immutable snapshot of documents eligible under a knowledge cutoff. @@ -46,6 +46,14 @@ impl CorpusSnapshot { self.documents.contains_key(&document_id) } + /// Return the availability time retained for one snapshot document. + #[must_use] + pub fn available_time(&self, document_id: Uuid) -> Option { + self.documents + .get(&document_id) + .map(|document| document.available_time) + } + /// Return the number of eligible documents. #[must_use] pub fn len(&self) -> usize { @@ -97,6 +105,11 @@ mod tests { assert_eq!(snapshot.len(), 1); assert!(!snapshot.is_empty()); assert!(snapshot.contains(early.document_id)); + assert_eq!( + snapshot.available_time(early.document_id), + Some(early.available_time) + ); + assert_eq!(snapshot.available_time(Uuid::nil()), None); assert_eq!(snapshot.document_ids().count(), 1); } } diff --git a/crates/model_selection/src/fitted.rs b/crates/model_selection/src/fitted.rs index 59673a991..389f3ec32 100644 --- a/crates/model_selection/src/fitted.rs +++ b/crates/model_selection/src/fitted.rs @@ -113,6 +113,40 @@ impl FittedCandidateKConfig { self.tolerance } + /// Build the exact reference-estimator configuration for one candidate. + /// + /// # Errors + /// + /// Returns [`ModelSelectionError::NonPositiveCandidateK`] when `candidate_k` + /// is not in this validated candidate set, or + /// [`ModelSelectionError::InvalidDiagnostic`] when conversion fails. + pub fn reference_config( + &self, + candidate_k: u32, + ) -> Result { + if !self.candidate_topic_counts.contains(&candidate_k) { + return Err(ModelSelectionError::NonPositiveCandidateK); + } + let topic_count = + usize::try_from(candidate_k).map_err(|_| ModelSelectionError::InvalidDiagnostic)?; + ReferenceTopicModelConfig::new( + topic_count, + self.seeds.clone(), + self.maximum_iterations, + self.tolerance, + ) + .and_then(|value| { + value.with_hyperparameters( + self.prior_variance, + self.relation_strength, + self.ridge, + self.topic_smoothing, + self.step_size, + ) + }) + .map_err(|_| ModelSelectionError::InvalidDiagnostic) + } + fn validate(&self) -> Result<(), ModelSelectionError> { if self.candidate_topic_counts.is_empty() { return Err(ModelSelectionError::EmptyCandidateSet); @@ -202,29 +236,29 @@ pub fn select_fitted_candidate_k( method_name: &str, llm_votes: &[u32], ) -> Result { + select_fitted_candidate_model(input, config, method_name, llm_votes) + .map(|(candidate_k, _)| candidate_k) +} + +/// Fit each candidate and return the selected `K` with its exact fitted model. +/// +/// # Errors +/// +/// Returns the same typed failures as [`select_fitted_candidate_k`]. +pub fn select_fitted_candidate_model( + input: &ReferenceTopicInput, + config: &FittedCandidateKConfig, + method_name: &str, + llm_votes: &[u32], +) -> Result<(u32, ReferenceTopicModel), ModelSelectionError> { refuse_nonstatistical_method(method_name)?; let mut candidates = Vec::new(); + let mut fitted = Vec::new(); for &candidate_k in config.candidate_topic_counts() { - #[allow(clippy::cast_possible_truncation)] - let topic_count = candidate_k as usize; - let fit_config = ReferenceTopicModelConfig::new( - topic_count, - config.seeds().to_vec(), - config.maximum_iterations(), - config.tolerance(), - ) - .and_then(|value| { - value.with_hyperparameters( - config.prior_variance, - config.relation_strength, - config.ridge, - config.topic_smoothing, - config.step_size, - ) - }) - .map_err(|_| ModelSelectionError::InvalidDiagnostic)?; + let fit_config = config.reference_config(candidate_k)?; if let Ok(model) = fit_reference_topic_model(input, &fit_config) { candidates.push(statistical_candidate_from_fit(input, candidate_k, &model)?); + fitted.push((candidate_k, model)); } } for &vote in llm_votes { @@ -233,7 +267,11 @@ pub fn select_fitted_candidate_k( if candidates.is_empty() { return Err(ModelSelectionError::NoSuccessfulFit); } - select_candidate_k(&candidates) + let selected_k = select_candidate_k(&candidates)?; + fitted + .into_iter() + .find(|(candidate_k, _)| *candidate_k == selected_k) + .ok_or(ModelSelectionError::NoSuccessfulFit) } fn refuse_nonstatistical_method(method: &str) -> Result<(), ModelSelectionError> { @@ -322,6 +360,11 @@ mod tests { assert_eq!(config.seeds(), &[7, 11]); assert_eq!(config.maximum_iterations(), 20); assert!((config.tolerance() - 1e-5).abs() < f64::EPSILON); + assert!(config.reference_config(2).is_ok()); + assert_eq!( + config.reference_config(4), + Err(ModelSelectionError::NonPositiveCandidateK) + ); refuse_nonstatistical_method("trsl_tm_reference").expect("allowed"); refuse_nonstatistical_method("logistic_normal").expect("allowed"); for method in [ diff --git a/crates/model_selection/src/lib.rs b/crates/model_selection/src/lib.rs index dbc67c8df..d24524dc6 100644 --- a/crates/model_selection/src/lib.rs +++ b/crates/model_selection/src/lib.rs @@ -23,6 +23,8 @@ pub use error::ModelSelectionError; pub use fitted::FittedCandidateKConfig; /// Fit each candidate `K` and select from the actual statistical diagnostics. pub use fitted::select_fitted_candidate_k; +/// Fit candidates and retain the exact selected reference model. +pub use fitted::select_fitted_candidate_model; /// Build a statistical candidate from one actual fitted model. pub use fitted::statistical_candidate_from_fit; /// Select the admissible candidate `K` from a Pareto-filtered statistical front. diff --git a/crates/topic_measurement/src/reference.rs b/crates/topic_measurement/src/reference.rs index a1e4665d0..0dec9ab9c 100644 --- a/crates/topic_measurement/src/reference.rs +++ b/crates/topic_measurement/src/reference.rs @@ -2,10 +2,10 @@ use std::collections::{BTreeMap, BTreeSet, HashMap}; -use corpus_split::CorpusSnapshot; +use corpus_split::{CorpusSnapshot, cutoff_eligible}; use membership_core::{GroupId, MemberId, MembershipNetwork, MembershipRole}; use relation_graph::RelationGraph; -use temporal_core::EventTime; +use temporal_core::{AvailableTime, EventTime, KnowledgeCutoff}; use uuid::Uuid; use crate::{SparseMatrix, TopicMeasurementError, from_additive_log_ratio}; @@ -41,6 +41,7 @@ pub enum PrevalenceFeature { #[derive(Clone, Debug)] pub struct ReferenceTopicInput { document_ids: Vec, + available_times: Vec, event_times: Vec, term_rows: Vec>, vocabulary_size: usize, @@ -104,8 +105,14 @@ impl ReferenceTopicInput { let (design, features) = build_design(&document_ids, event_times, covariates, memberships)?; let transition_pairs = collect_transition_pairs(&index_by_id, relations)?; + let available_times = document_ids + .iter() + .map(|id| snapshot.available_time(*id)) + .collect::>>() + .ok_or(TopicMeasurementError::InvalidModelInput)?; Ok(Self { document_ids, + available_times, event_times: event_times.to_vec(), term_rows, vocabulary_size: document_term.columns(), @@ -121,6 +128,14 @@ impl ReferenceTopicInput { self.document_ids.len() } + /// Return whether every modeled document was available by `knowledge_cutoff`. + #[must_use] + pub fn is_eligible_at(&self, knowledge_cutoff: &KnowledgeCutoff) -> bool { + self.available_times + .iter() + .all(|available| cutoff_eligible(available, knowledge_cutoff)) + } + /// Return the vocabulary size. #[must_use] pub const fn vocabulary_size(&self) -> usize { @@ -1091,13 +1106,17 @@ mod tests { validate_positive_definite, }; use crate::TopicMeasurementError; - use temporal_core::EventTime; + use temporal_core::{AvailableTime, EventTime}; use uuid::Uuid; fn event_time(day: u8) -> EventTime { EventTime::parse_rfc3339(&format!("2026-01-{day:02}T00:00:00Z")).expect("event time") } + fn available_time() -> AvailableTime { + AvailableTime::parse_rfc3339("2026-01-01T00:00:00Z").expect("available time") + } + #[test] fn numeric_helpers_are_deterministic_and_fail_closed() { let mut seed = 1; @@ -1130,6 +1149,7 @@ mod tests { fn impossible_numeric_states_and_mixed_topic_edges_fail_closed() { let input = ReferenceTopicInput { document_ids: vec![Uuid::from_u128(1), Uuid::from_u128(2)], + available_times: vec![available_time(); 2], event_times: vec![event_time(1), event_time(2)], term_rows: vec![vec![(0, f64::MAX)], vec![(0, f64::MAX)]], vocabulary_size: 2, @@ -1220,6 +1240,7 @@ mod tests { ) -> ReferenceTopicInput { ReferenceTopicInput { document_ids: vec![Uuid::from_u128(1), Uuid::from_u128(2)], + available_times: vec![available_time(); 2], event_times: vec![event_time(1), event_time(2)], term_rows, vocabulary_size, @@ -1335,6 +1356,7 @@ mod tests { } #[test] + #[allow(clippy::too_many_lines)] fn joint_precision_binds_basis_and_rejects_invalid_geometry() { let input = scoring_input(vec![vec![(0, 2.0)], vec![(1, 3.0)]], 3); let config = ReferenceTopicModelConfig::new(3, vec![1], 10, 1e-6).expect("config"); @@ -1387,6 +1409,7 @@ mod tests { let empty = ReferenceTopicInput { document_ids: Vec::new(), + available_times: Vec::new(), event_times: Vec::new(), term_rows: Vec::new(), vocabulary_size: 2, @@ -1408,6 +1431,7 @@ mod tests { ); let oversized = ReferenceTopicInput { document_ids: vec![Uuid::from_u128(1); 4_097], + available_times: vec![available_time(); 4_097], event_times: vec![event_time(1); 4_097], term_rows: vec![vec![(0, 1.0)]; 4_097], vocabulary_size: 2, diff --git a/docs/TRACEABILITY.md b/docs/TRACEABILITY.md index 2b783c2ab..c18031828 100644 --- a/docs/TRACEABILITY.md +++ b/docs/TRACEABILITY.md @@ -75,6 +75,7 @@ The full APA 7th standards/literature register remains `docs/research/standards- | report template/section/copied/style/modality method effects | ADR 0004/0012; PRD/TRD | simulation truth factors implemented; `corpus_background` background-versus-unique-content identity on the active PR; estimator-side method model remains future | partial | | report template/section/copied/style/modality method effects | ADR 0004/0012; PRD/TRD | simulation truth factors implemented; `prompt_source` prompt-versus-unique-content identity on the active PR; estimator-side method model remains future | partial | | candidate K statistical/Pareto gates | ADR 0012; research | `model_selection` fits each candidate `K` with the CPU `f64` reference and scores the actual mixture likelihood plus Schwarz's (1978) `ℓ − (p ln N)/2` penalty before the Pareto gate; candidate blinding, blinded LLM review, GPU, and backend comparison remain accepted-target | active-PR | +| composed fitted-K topic-lineage analysis-run | ADR 0012/0022/0055 | `analysis_engine` `composed_fitted_lineage_v1` binds `select_fitted_candidate_k` then `execute_topic_lineage_run` at selected `K`; refuses lexical methods; not a Schwarz-only bind, not a Pareto-front bind, not a Bayesian sampler, and not implemented-main | active-PR | | compositional topic correlation / stable clustering | ADR 0005/0012; research | future `network_analysis` | accepted-target | | posterior ESEM / longitudinal invariance / DSEM | ADR 0005 | `psychometric_core` construct/input gates, true-loading OLS recovery, posterior-draw point-estimate averaging, Rubin `T` on draw-level OLS loadings, CWC within/between OLS plus the contextual effect, event-time log-rate, constant- and time-varying-predictor discrete effects (Voelkle Eqs. 12 and 14), exact scalar discrete process noise (Driver et al., 2017, Eq. 3), lagged latent covariance and unconditional latent variance (Driver et al., 2017, Eq. 3–4), stationary within-subject variance (Driver et al., 2017, Eq. 4 as `Δt → ∞`; `asymDIFFUSION`), trait-plus-state variance (Driver et al., 2017, §4.3 `TRAITVAR`; not process noise), observed-indicator variance and lagged observed covariance (Driver et al., 2017, Eq. 5; Table 2 `MANIFESTVAR` is `Θ`, not `Var(y)`; `MANIFESTTRAITVAR` is not `MANIFESTVAR`; `Θ` does not enter lagged observed covariance; observed-indicator mean is `τ + λ μ`; `MANIFESTMEANS` is not `E(y)`; `CINT` is not `MANIFESTMEANS`; discrete latent mean is `exp(a Δt) μ_0 + (exp(a Δt) − 1)/a κ`; `T0MEANS` is not `μ_t`; evolved observed mean is `τ + λ μ_t`; `τ + λ μ_0` is not `E(y_t)`; contemporaneous `TDPREDEFFECT` impulse is `m x`, not `CINT`, not `TIPREDEFFECT`, and not Voelkle Eq. 14; Eq. 5 of that contemporaneous impulse is `τ + λ(μ_t + m x)`, and `τ + λ μ_t` is not that observed mean; time-independent `TIPREDEFFECT` increment is `A^{-1}[e^{A Δt} − I] B z`, not `CINT`, not `M x`, not Voelkle Eq. 14, and not the coefficient `B`; Eq. 5 of that increment is `τ + λ(μ_t + A^{-1}[e^{A Δt} − I] B z)`, and `τ + λ μ_t` is not that observed mean; `τ + λ(μ_t + m x)` is not that observed mean; `τ + λ(μ_t + e^{a(t−u)} m x)` is not that observed mean when `u ≠ t`; within-interval `TDPREDEFFECT` carry is `e^{A(t−u)} M x` for `t0 < u < t`, not the contemporaneous Dirac, not `CINT`, not `TIPREDEFFECT`, and not Voelkle Eq. 14; Eq. 5 of that carry is `τ + λ(μ_t + e^{a(t−u)} m x)`, and `τ + λ μ_t` is not that observed mean; `τ + λ(μ_t + m x)` is not that carried observed mean when `u ≠ t`; §7.2 level-change `CINT` is `κ = −a m x` (`a < 0`; not the dissipating Dirac, not a free `CINT`, not `TIPREDEFFECT`; Eq. 3 of that setting is `(1 − e^{a Δt}) m x`); §7.2 extra-process contribution is `a_{ηξ} x (e^{ε Δt} − e^{a Δt}) / (ε − a)` (not `κ = −a m x`, not `(1 − e^{a Δt}) m x`, not the dissipating Dirac; `ε ≥ 0` fails closed; Eq. 5 of that contribution is `τ + λ(μ_t + a_{ηξ} x (e^{ε Δt} − e^{a Δt}) / (ε − a)`; extra `LAMBDA` is 0; `τ + λ μ_t` is not that observed mean; after-t0 extra-process `TDPREDEFFECT` uses `t − u` with `t0 < u < t` while `μ_t` uses `Δt`; that after-t0 observed mean is not the first-occasion extra-process observed mean; §7.2 `asymTIPREDEFFECT` is `-B z / a` for `a < 0` and is not `B`, not `A^{-1}[e^{A Δt} − I] B z`, not `CINT`, and not `M x`; §7.2 `addedTIPREDVAR` is `(B / a)² v` and is not `TRAITVAR`, not `asymDIFFUSION`, and not `-B z / a`; Table 2 `asymCINT` is `-κ / a` for `a < 0` and is not `κ`, not `A^{-1}[e^{A Δt} − I] κ`, not `T0MEANS`, and not `-B z / a`; p. 16 stationary `T0MEANS` is `-κ / a + −B z / a` and is not free `T0MEANS`, not `asymCINT` alone, not `asymTIPREDEFFECT` alone, and not the finite-interval discrete latent mean; Eq. 5 of that constrained mean is `τ + λ(−κ / a + −B z / a)`; `τ + λ μ_0` is not that observed mean; `MANIFESTMEANS` is not `E(y_0)`; the constrained latent mean is not `E(y_0)`; stationary `T0VAR` is `trait + −q / (2 a) + (B / a)² v` (not free `T0VAR`, not `asymDIFFUSION` alone, not `TRAITVAR` alone, not `addedTIPREDVAR` alone, and not the finite-interval discrete latent variance. Eq. 5 of that constrained variance is `λ²(trait + −q / (2 a) + (B / a)² v) + θ + ψ` (JSS PDF re-opened 2026-08-22T03:20Z; form the stationary latent variance first, then `λ² p + θ + ψ`; `λ² p_0` is not that observed variance; `λ²(−q / (2 a)) + θ` is not that observed variance when `TRAITVAR` or `addedTIPREDVAR` is nonzero; `MANIFESTVAR` is not `Var(y_0)`; the constrained latent variance is not `Var(y_0)`)); lagged stationary `T0VAR` is `trait + e^{a Δt}(−q / (2 a)) + (B / a)² v` (trait and `addedTIPREDVAR` do not decay; contemporaneous `T0VAR` is not that lagged map; decaying the constrained total as if it were all state is not that lagged map; Eq. 5 of that lagged covariance is `λ²(trait + e^{a Δt}(−q / (2 a)) + (B / a)² v) + ψ`; `Θ` does not enter; contemporaneous `Var(y_0)` is not that lagged observed covariance; the lagged latent covariance is not that observed covariance); later-occasion stationary `T0VAR` is `trait + e^{2 a Δt}(−q / (2 a)) + Q_Δt + (B / a)² v` (trait and `addedTIPREDVAR` do not enter `Q_Δt`; under stationarity that composition equals contemporaneous `T0VAR`; evolving the constrained total as if it were all state is not that later map; the lagged covariance omits `Q_Δt`; `Q_Δt` is not that later map; Eq. 5 of that later-occasion variance is `λ²(trait + e^{2 a Δt}(−q / (2 a)) + Q_Δt + (B / a)² v) + θ + ψ`; lagged observed covariance omits `Q_Δt` and `θ`; `MANIFESTVAR` is not `Var(y_t)`; the later-occasion latent variance is not `Var(y_t)`); predetermined later-occasion `T0VAR` is `trait + e^{2 a Δt} p_0 + Q_Δt + (B / a)² v` (free `T0VAR` `p_0` is not that later map; setting `p_0 = −q / (2 a)` recovers the stationary later-occasion map; stationary later variance uses `−q / (2 a)` in place of `p_0` and is not that later map when `p_0` is free; evolving `trait + p_0 + (B / a)² v` as if it were all state is not that later map; Eq. 5 of that predetermined later-occasion variance is `λ²(trait + e^{2 a Δt} p_0 + Q_Δt + (B / a)² v) + θ + ψ`; `MANIFESTVAR` is not `Var(y_t)`; the predetermined later-occasion latent variance is not `Var(y_t)`; stationary later observed variance is not that observed variance when `p_0` is free); predetermined lagged `T0VAR` is `trait + e^{a Δt} p_0 + (B / a)² v` (free `T0VAR` `p_0` is not that lagged map; setting `p_0 = −q / (2 a)` recovers the stationary lagged map; stationary lagged covariance uses `−q / (2 a)` in place of `p_0` and is not that lagged map when `p_0` is free; evolving `trait + p_0 + (B / a)² v` as if it were all state is not that lagged map; later-occasion variance includes `Q_Δt` and is not that lagged map; Eq. 5 of that predetermined lagged covariance is `λ²(trait + e^{a Δt} p_0 + (B / a)² v) + ψ`; `MANIFESTVAR` does not enter; the predetermined lagged latent covariance is not that observed covariance; predetermined later observed variance includes `Q_Δt` and `θ` and is not that lagged observed covariance; stationary lagged observed covariance is not that observed covariance when `p_0` is free; the predetermined first-occasion variance of §4.3 predetermined `T0VAR` is `trait + p_0 + (B / a)² v`; free `p_0` is not that map; stationary first-occasion variance uses `−q / (2 a)` in place of `p_0` and is not that map when `p_0` is free; lagged covariance decays the state and is not that map; later-occasion variance includes `Q_Δt` and is not that map; Eq. 5 of that predetermined first-occasion variance is `λ²(trait + p_0 + (B / a)² v) + θ + ψ`; `MANIFESTVAR` is not that first-occasion observed variance; the predetermined first-occasion latent variance is not that observed variance; stationary first-occasion observed variance is not that observed variance when `p_0` is free; predetermined later observed variance includes `Q_Δt` and is not that first-occasion observed variance; later-start lagged covariance of predetermined `T0VAR` is `trait + e^{a s}(e^{2 a u} p_0 + Q_u) + (B / a)² v` (Driver et al., 2017, §4.3 `startoffset`; Eq. 4; JSS PDF re-opened 2026-08-23T10:27Z; first-occasion lagged omits `e^{a s} Q_u`; later-occasion variance does not lag; stationary lagged uses `−q / (2 a)`; decaying the later total is not that map; Eq. 5 of that later-start lagged covariance is `λ²` of it plus `ψ`; `Θ` does not enter; first-occasion lagged observed omits `e^{a s} Q_u`; later observed variance includes `Q_u` and `θ`; later-start later-occasion variance of predetermined `T0VAR` is `trait + e^{2 a s}(e^{2 a u} p_0 + Q_u) + Q_s + (B / a)² v` (Driver et al., 2017, §4.3 `startoffset`; Eq. 3–4 Chapman–Kolmogorov `Q_{u+s} = e^{2 a s} Q_u + Q_s`; JSS PDF re-opened 2026-08-23T11:05Z; later-occasion variance at `u` omits `Q_s`; later-start lagged covariance omits `Q_s`; stationary later uses `−q / (2 a)`; evolving the later total as if it were all state is not that map; ignoring `startoffset` omits `e^{2 a s} Q_u`; Eq. 5 of that later-start later-occasion variance is `λ²` of it plus `θ + ψ`; `MANIFESTVAR` is not that observed variance; p. 16 `discreteDRIFTstd` is `e^{a Δt}` after strictly positive `asymDIFFUSION` `-q / (2 a)` (footnote 4; unstandardised `e^{a Δt}` is defined for growing `a ≥ 0` and for zero diffusion and is not `discreteDRIFTstd`; the §7.1 trait-plus-state autocorrelation uses `TRAITVAR` and is not `discreteDRIFTstd`; p. 16 `discreteDIFFUSIONstd` is `Q_Δt / (−q / (2 a))` after strictly positive `asymDIFFUSION` `-q / (2 a)` (footnote 4; unstandardised `Q_Δt` is defined for growing `a ≥ 0` and for zero diffusion and is not `discreteDIFFUSIONstd`; the continuous standardisation `−2 a` is not `discreteDIFFUSIONstd`; `Q_Δt / (trait + p + added)` uses `TRAITVAR` and is not `discreteDIFFUSIONstd`; `TRAITVAR` is not the standardisation variance; p. 16 `DIFFUSIONstd` is `q / (−q / (2 a)) = −2 a` after strictly positive `asymDIFFUSION` `-q / (2 a)` (Driver et al., 2017, p. 16; Eq. 4; footnote 4; JSS PDF re-opened 2026-08-23T13:20Z; unstandardised `q` is defined for growing `a ≥ 0` and for zero diffusion and is not `DIFFUSIONstd`; the discrete standardisation `Q_Δt / (−q / (2 a))` depends on `Δt` and is not `DIFFUSIONstd`; `q / (trait + p + added)` uses `TRAITVAR` and is not `DIFFUSIONstd`; `TRAITVAR` is not the standardisation variance; p. 16 `DRIFTstd` is the continuous auto-effect after strictly positive `asymDIFFUSION` `-q / (2 a)` (Driver et al., 2017, p. 16; Eq. 1; footnote 4; JSS PDF re-opened 2026-08-23T13:28Z); unstandardised `a` is defined for growing `a ≥ 0` and for zero diffusion and is not `DRIFTstd`; the discrete standardisation `e^{a Δt}` depends on the event interval and is not `DRIFTstd`; `a p / (trait + p + added)` uses `TRAITVAR` and is not `DRIFTstd`; `TRAITVAR` is not the standardisation variance); p. 16 `asymTIPREDEFFECTstd` is `(-B / a) · √v / √(-q / (2 a))` after strictly positive `asymDIFFUSION` `-q / (2 a)` and strictly positive predictor variance `v` (Driver et al., 2017, p. 16; §7.2; footnote 4; JSS PDF re-opened 2026-08-23T14:25Z; unstandardised `-B / a` is defined for a zero coefficient and for zero predictor variance and is not `asymTIPREDEFFECTstd`; the finite-interval standardisation `A^{-1}[e^{A Δt} − I] B · √v / √p` depends on the event interval and is not `asymTIPREDEFFECTstd`; `(-B / a) · √v / √(trait + p + added)` uses `TRAITVAR` and is not `asymTIPREDEFFECTstd`; `TRAITVAR` is not the standardisation variance); p. 16 `TIPREDEFFECTstd` is `B · √v / √(-q / (2 a))` after strictly positive `asymDIFFUSION` `-q / (2 a)` and strictly positive predictor variance `v` (Driver et al., 2017, p. 16; §7.2; footnote 4; JSS PDF re-opened 2026-08-23T16:21Z; unstandardised `B` is defined for a zero coefficient and for zero predictor variance and is not `TIPREDEFFECTstd`; the asymptotic standardisation `(-B / a) · √v / √p` is the total change and is not `TIPREDEFFECTstd`; the finite-interval standardisation `A^{-1}[e^{A Δt} − I] B · √v / √p` depends on the event interval and is not `TIPREDEFFECTstd`; `B · √v / √(trait + p + added)` uses `TRAITVAR` and is not `TIPREDEFFECTstd`; `TRAITVAR` is not the standardisation variance); Table 3 `T0TIPREDEFFECTstd` is `t0_b · √v / √p_0` after strictly positive free `T0VAR` `p_0` and strictly positive predictor variance `v` (Driver et al., 2017, Table 3, p. 13; p. 16; footnote 4; 2017-era ctsem `summary.ctsemFit.R`; JSS PDF re-opened 2026-08-23T17:20Z; the affected variance is free `T0VAR`, not `asymDIFFUSION`; unstandardised `t0_b` is defined for a zero coefficient and for zero predictor variance and is not `T0TIPREDEFFECTstd`; `TIPREDEFFECTstd` `B · √v / √(-q / (2 a))` is the continuous coefficient and is not `T0TIPREDEFFECTstd`; `asymTIPREDEFFECTstd` `(-B / a) · √v / √p` is the total change and is not `T0TIPREDEFFECTstd`; `t0_b · √v / √(trait + p_0 + added)` uses `TRAITVAR` and is not `T0TIPREDEFFECTstd`; `TRAITVAR` is not the standardisation variance); 2017-era `addedT0TIPREDVAR` is `t0_b² v` (Driver et al., 2017, Table 3, p. 13; p. 16; §7.2; 2017-era ctsem `summary.ctsemFit.R`; JSS PDF re-opened 2026-08-23T18:20Z; `T0TIPREDEFFECT %*% TIPREDVAR %*% t(T0TIPREDEFFECT)` immediately after `T0TIPREDEFFECTstd`; form `t0_b` first, then square, then multiply by `v`; a zero coefficient or zero predictor variance is exactly zero; free `T0TIPREDEFFECT` does not require `a < 0`; `(B / a)² v` is `addedTIPREDVAR` and is not this first-occasion map; `t0_b · √v / √p_0` is `T0TIPREDEFFECTstd` and is not this variance; free `T0VAR` is not this extra TI variance; `TRAITVAR` is not this extra TI variance; Equation 5 of 2017-era `addedT0TIPREDVAR` is `λ² t0_b² v` (Driver et al., 2017, Eq. 5, p. 5; Table 3, p. 13; Table 2, p. 12; 2017-era ctsem `summary.ctsemFit.R`; JSS PDF re-opened 2026-08-23T19:10Z; form `t0_b² v` first, then `(λ extra) λ` with `θ = 0`; a zero loading or zero extra is exactly zero; `t0_b² v` is the latent extra, not the observed extra; `λ² p_0 + θ` is first-occasion observed variance, not this extra; `λ² (B / a)² v` is Eq. 5 of `addedTIPREDVAR`, not this first-occasion observed extra; `MANIFESTVAR` `θ` is not this extra; Equation 5 of §7.2 `addedTIPREDVAR` is `λ² (B / a)² v`; form `(B / a)² v` first, then `(λ extra) λ` with `θ = 0`; a zero loading or zero extra is exactly zero; lasting asymptotic extra requires `a < 0`; `(B / a)² v` is the latent extra, not the observed extra; `λ² t0_b² v` is first-occasion extra observed TI variance, not this extra; `λ² p + θ` is stationary observed variance, not this extra; `MANIFESTVAR` `θ` is not this extra; p. 16 `TDPREDEFFECTstd` is `m · √v / √(-q / (2 a))` after strictly positive `asymDIFFUSION` and strictly positive time-dependent predictor variance; unstandardised `M` is not `TDPREDEFFECTstd`; `TIPREDEFFECTstd` is not `TDPREDEFFECTstd` even when `M = B`; intercept-style `A^{-1}[e^{A Δt} − I] M · √v / √p` is not `TDPREDEFFECTstd`; `m · √v / √(trait + p + added)` uses `TRAITVAR` and is not `TDPREDEFFECTstd`; Table 3 / p. 16 `T0TDPREDEFFECTstd` is `t0_m · √v / √p_0` after strictly positive free `T0VAR` and strictly positive TD predictor variance; unstandardised `t0_m` is not `T0TDPREDEFFECTstd`; `TDPREDEFFECTstd` uses `asymDIFFUSION` and is not `T0TDPREDEFFECTstd`; `T0TIPREDEFFECTstd` is not `T0TDPREDEFFECTstd` even when `t0_m = t0_b`; `t0_m · √v / √(trait + p_0 + added)` uses `TRAITVAR` and is not `T0TDPREDEFFECTstd`; free `T0VAR` does not require `a < 0`; p. 16 `T0VARstd` is `p_0 / p_0 = 1` after strictly positive free `T0VAR` (`solve(sqrt(diag(T0VAR))) %&% T0VAR`; OpenMx `%&%` is `t(A) %*% B %*% A`; default ridge is 0); unstandardised `T0VAR` is not `T0VARstd`; `T0TDPREDEFFECTstd` is not `T0VARstd`; `addedT0TIPREDVAR` is not `T0VARstd`; p. 16 `TRAITVARstd` is `trait / trait = 1` after strictly positive `TRAITVAR` (`solve(sqrt(diag(TRAITVAR))) %&% TRAITVAR`; OpenMx `%&%` is `t(A) %*% B %*% A`; no ridge addend); unstandardised `TRAITVAR` is not `TRAITVARstd`; `T0VARstd` is not `TRAITVARstd` even when both equal 1; `addedT0TIPREDVAR` is not `TRAITVARstd`; p. 16 `MANIFESTTRAITVARstd` is `ψ / ψ = 1` after strictly positive `MANIFESTTRAITVAR` (`solve(sqrt(diag(MANIFESTTRAITVAR))) %&% MANIFESTTRAITVAR`; OpenMx `%&%` is `t(A) %*% B %*% A`; 2017-era source adds ridging; default ridge is 0); unstandardised `MANIFESTTRAITVAR` is not `MANIFESTTRAITVARstd`; `TRAITVARstd` is not `MANIFESTTRAITVARstd` even when both equal 1; `MANIFESTVAR` is not `MANIFESTTRAITVARstd`; p. 16 `MANIFESTVARstd` is `θ / θ = 1` after strictly positive `MANIFESTVAR` (`solve(sqrt(diag(MANIFESTVAR))) %&% MANIFESTVAR`; OpenMx `%&%` is `t(A) %*% B %*% A`; 2017-era source adds ridging; default ridge is 0; 2017-era `dimnames` assignment to `latentNames` is a source bug); unstandardised `MANIFESTVAR` is not `MANIFESTVARstd`; `MANIFESTTRAITVARstd` is not `MANIFESTVARstd` even when both equal 1; Equation 5 `Var(y)` is not `MANIFESTVARstd`; p. 16 `TIPREDVARstd` is `v / v = 1` after strictly positive `TIPREDVAR` (`solve(sqrt(diag(TIPREDVAR))) %&% TIPREDVAR`; OpenMx `%&%` is `t(A) %*% B %*% A`; 2017-era source adds ridging; default ridge is 0; `dimnames` are `TIpredNames`); unstandardised `TIPREDVAR` is not `TIPREDVARstd`; `MANIFESTVARstd` is not `TIPREDVARstd` even when both equal 1; §7.2 `addedTIPREDVAR` is not `TIPREDVARstd`; p. 16 `asymDIFFUSIONstd` is `p / p = 1` after strictly positive `asymDIFFUSION` (`solve(sqrt(diag(asymDIFFUSION))) %&% asymDIFFUSION`; OpenMx `%&%` is `t(A) %*% B %*% A`; 2017-era source adds ridging; default ridge is 0; `dimnames` are `latentNames`); unstandardised `asymDIFFUSION` is not `asymDIFFUSIONstd`; `TIPREDVARstd` is not `asymDIFFUSIONstd` even when both equal 1; `DIFFUSIONstd` `−2 a` is not `asymDIFFUSIONstd`; p. 16 `discreteCINTstd` is `A^{-1}[e^{A Δt} − I] κ / √p` after strictly positive `asymDIFFUSION`; unstandardised `discreteCINT` is not `discreteCINTstd`; `κ / √p` is not `discreteCINTstd`; `(-κ / a) / √p` is not `discreteCINTstd`; `asymCINTstd` is `(-κ / a) / √p` after strictly positive `asymDIFFUSION`; unstandardised `asymCINT` is not `asymCINTstd`; `κ / √p` is not `asymCINTstd`; `discreteCINTstd` is not `asymCINTstd`; `T0MEANSstd` is `μ_0 / √p_0` after strictly positive free `T0VAR`; unstandardised `T0MEANS` is not `T0MEANSstd`; `T0VARstd` is not `T0MEANSstd`; `μ_0 / √asymDIFFUSION` is not `T0MEANSstd`; `MANIFESTMEANSstd` is `τ / √θ` after strictly positive `MANIFESTVAR`; unstandardised `MANIFESTMEANS` is not `MANIFESTMEANSstd`; `MANIFESTVARstd` is not `MANIFESTMEANSstd`; `τ / √(λ² Var(η) + θ)` is not `MANIFESTMEANSstd`; p. 16 `CINTstd` is `κ / √p` after strictly positive `asymDIFFUSION`; unstandardised `CINT` is not `CINTstd`; `asymCINTstd` is not `CINTstd`; `discreteCINTstd` is not `CINTstd`; `κ / √(trait + p + added)` is not `CINTstd`;))))), irregular already-centered residual lag, and strong/strict-gated latent means on the stacked psychometric PR (two-observation residual variance is identically `0` and caps at strong/scalar; Putnick & Bornstein, 2016, PMC5145197 opened 2026-08-19T22:15Z); full ESEM/DSEM remaining | partial | | CPU bounded multithreading + GPU/VRAM streaming/parity | ADR 0001/0006 | future `compute_backend` | accepted-target | diff --git a/docs/adr/0055-composed-fitted-lineage-analysis-run.md b/docs/adr/0055-composed-fitted-lineage-analysis-run.md new file mode 100644 index 000000000..03210c270 --- /dev/null +++ b/docs/adr/0055-composed-fitted-lineage-analysis-run.md @@ -0,0 +1,85 @@ +# ADR 0055 — Fitted candidate-`K` composed with topic lineage as an analysis-run profile + +**Decision status:** Accepted +**Implementation maturity:** active-PR — composed on this branch; not implemented-main +**Date:** 2026-08-31 +**Supersedes:** None; complements ADR 0012 (fitted candidate-`K` and the CPU `f64` reference) and ADR 0022 (cutoff-safe analysis-run execution). +**Figma File ID:** N/A — this increment changes a Rust service crate and has no user-interface surface. +**Storybook inventory:** N/A — no reusable web object or interaction changed. + +## Context + +Protected main already fits each candidate `K` through +`model_selection::select_fitted_candidate_k` and already emits digest-bound +topic-lineage artifacts through `execute_topic_lineage_run`. Operators still +cannot request those two existing functions as one cutoff-safe analysis-run +profile. Standalone fitted candidate-`K` selection, Pareto-front selection, +joint Laplace draws, and topic activity remain different profiles. Full +Bayesian sampling, GPU, and topic birth/split/merge remain later GAP-004 +work and are not this slice. + +An LLM vote must not define the numerical optimum. + +## Decision + +Add the `composed_fitted_lineage_v1` analysis-run output profile to +`analysis_engine`. The executor: + +- consumes an already-constructed `ReferenceTopicInput` plus + `FittedCandidateKConfig`, method name, and optional LLM votes; +- requires every modeled document's retained availability time to be at or + before the request knowledge cutoff; +- retains the exact winning fitted model, including all non-default numerical + hyperparameters, and builds lineage from that fit without a second fit; +- emits a canonical SHA-256-digested `tepp.composed_fitted_lineage.v1` + artifact with selected `K`, candidate/evidence counts, lineage topic/edge + counts, the inner lineage digest, and inference status + `fitted_k_composed_lineage_not_bayesian_sampler`; +- refuses reuse of `fitted_candidate_k_v1`, `pareto_candidate_k_v1`, + `trsl_topic_lineage_v1`, and `joint_posterior_draws_v1` as this profile; +- does not invent a Bayesian sampler, persist rows, select GPU backends, or + emit topic birth/split/merge. + +This is end-to-end composition of existing selection and lineage functions, +not a second Schwarz-only bind and not a posterior sampler. + +## Alternatives considered + +1. Bind another standalone `select_fitted_candidate_k` profile — rejected + because that bind is already live as a separate analysis-run profile. +2. Bind another standalone topic-lineage profile — rejected because that + executor is already on protected main. +3. Invent a Bayesian sampler or topic birth/split/merge engine — rejected + because those functions do not exist on protected main. +4. Compose fitted selection with the existing topic-lineage executor under + ADR 0022 — accepted. + +## Consequences + +Operators can request cutoff-safe fitted-`K` selection followed by a +production lineage fit as one digest-bound terminal result. The artifact +does not claim Schwarz-only selection, Pareto-front selection, Bayesian +sampling, GPU parity, or topic birth/split/merge. Snapshot/profile/cutoff +mismatch, late-available evidence, lexical methods, inconsistent counts, and +failed selection fail closed. + +## Verification + +The PR includes Rust unit and integration tests for successful composition, +non-default-hyperparameter preservation, lexical-method refusal, +snapshot/profile/cutoff mismatch including late-available evidence and reuse +of live sibling profiles, count invariants, and artifact tampering. Run: + +```text +cargo fmt --all -- --check +cargo test -p analysis_engine +cargo clippy -p analysis_engine --all-targets -- -D warnings +python3 scripts/validate_documentation.py +``` + +## Rollback and supersession + +Rollback removes the `composed_fitted_lineage_v1` profile. No persisted +schema migration is introduced. Supersede only with an ADR that keeps +fitted selection distinct from LLM votes, Pareto-front selection, and +Bayesian sampling, and keeps lineage association distinct from causation. diff --git a/docs/adr/README.md b/docs/adr/README.md index 1254c8079..b8f96fa71 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -28,6 +28,7 @@ Read [`ADR_POLICY.md`](ADR_POLICY.md) first. **Decision status and implementatio | [0020](0020-span-grounded-semantic-units.md) | Span-grounded semantic units; language tags are not identity | Accepted | active-PR | First ADR 0004 production slice; concept alignment, invariance, and topic estimation are not claimed. | | [0021](0021-lineageweave-project-history-boundary.md) | LineageWeave project-history service boundary | Accepted | active-PR | Credential-free bounded project-history API preserves LineageWeave authorization ownership. | | [0022](0022-deterministic-analysis-run-execution.md) | Deterministic cutoff-safe analysis-run execution | Accepted | active-PR | Closes the first executable product path from accepted run to digest-bound terminal result without claiming estimator authority. | +| [0055](0055-composed-fitted-lineage-analysis-run.md) | Fitted candidate-`K` composed with topic lineage | Accepted | active-PR | Complements ADR 0012/0022; selection then production lineage, not a Schwarz-only bind and not a Bayesian sampler. | | [0024](0024-lineage-pair-criterion-and-project-journey-posterior.md) | Independent Event Lineage pair criterion and posterior Project Journey | Proposed | active-PR | Strict artifacts preserve criterion/event-time draws, branches, ties, and CPU/GPU receipts without claiming the scientific estimator is complete. | | [0025](0025-macos-native-rust-mlx-metal-boundary.md) | macOS-native Rust-owned MLX Metal execution | Accepted | accepted-target | Compose authenticates to a native host service; Linux never claims Metal, and actual backend/parity receipts fail closed. | | [0023](0023-lineage-criterion-anchor-contract.md) | TEPP-owned Event Lineage criterion anchor | Accepted | active-PR | PR #237 publishes the strict accepted/rejected artifact and identities; estimator execution remains fail-closed future work. | @@ -138,6 +139,7 @@ Use the narrowest owning ADR when decisions overlap: - **project-history wire-size symmetry:** ADR 0019. - **LineageWeave project-history service boundary:** ADR 0021. - **accepted-run execution and terminal artifact production:** ADR 0022. +- **composed fitted-K topic-lineage analysis-run claim boundary:** ADR 0055. - **independent lineage criterion and posterior Project Journey:** ADR 0023. - **macOS-native Rust-owned MLX Metal execution:** ADR 0024. diff --git a/docs/doctoring/composed-fitted-lineage-analysis-run.md b/docs/doctoring/composed-fitted-lineage-analysis-run.md new file mode 100644 index 000000000..611ddf886 --- /dev/null +++ b/docs/doctoring/composed-fitted-lineage-analysis-run.md @@ -0,0 +1,17 @@ +# Composed fitted-`K` topic-lineage analysis-run composition + +**Active slice:** ADR 0055 / `composed_fitted_lineage_v1` +**Protected-main status:** not implemented-main + +`model_selection` already fits each candidate `K` with the CPU `f64` +reference, and `analysis_engine` already emits digest-bound topic-lineage +artifacts. This slice binds those two existing functions to one cutoff-safe +analysis-run profile so an operator can request selection then production +lineage as a single terminal result. + +The executor refuses lexical methods and LLM-vote authority. It is not a +standalone Schwarz bind, not a Pareto-front bind, not a Bayesian sampler, +not GPU execution, and not topic birth/split/merge. + +Exact-head Checks and two independent approvals are required before any +implemented-main claim.