Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,8 @@ All notable changes to TEPP are documented here. The format follows Keep a Chang

## [Unreleased]

- **Joint posterior Laplace draws analysis-run profile**: cutoff-safe `joint_posterior_draws_v1` binds `draw_joint_gaussian` (Philox/Box-Muller/Cholesky) and refuses to claim MCMC, GPU, or topic birth/split/merge (`analysis_engine`). Not implemented-main.

- `event_core` adds bounded Allen interval-consistency classification, atomic path-consistency closure, contradiction/resource refusals, and an explicit dependency-error fallback without claiming unrestricted global satisfiability.

- `psychometric_core` recovers the Driver, Oud, and Voelkle (2017, Table 2, p. 12 `MANIFESTTRAITVAR`; §7.1, p. 19; p. 16 `MANIFESTTRAITVARstd`; footnote 4; 2017-era ctsem `summary.ctsemFit.R`; JSS PDF re-opened 2026-08-27T14:20Z from https://www.jstatsoft.org/index.php/jss/article/download/v077i05/1104) scalar standardised manifest-trait variance on current main after `0ce16e8` dropped the pre-consolidation code while research notes already named the map (register items 83–84). Table 2 names `MANIFESTTRAITVAR` `Ψ_τ` the additional time-invariant variance-covariance on the measurement level and sets it `NULL` when there is no manifest trait. Equation 5 writes `Γ ~ N(τ, Ψ)` and names that covariance the manifest traits. Section 7.1 names manifest traits stable individual differences in indicator levels, distinct from process-level `TRAITVAR` `φ_ξ`. Page 16 prints standardised matrices with the suffix `std` when appropriate. The printed example on p. 16 is `discreteDRIFTstd`, not `MANIFESTTRAITVARstd`. Footnote 4 standardises using only the relevant variance, not the total. The relevant variance for that named indicator-level correlation is `MANIFESTTRAITVAR`, not process-level `TRAITVAR` and not residual `MANIFESTVAR` `θ`. The 2017-era source forms `MANIFESTTRAITVARstd` only when `MANIFESTTRAITVAR != 0`, as `solve(sqrt(diag(MANIFESTTRAITVAR) + ridging)) %&% MANIFESTTRAITVAR` when `verbose = TRUE`. OpenMx `%&%` is `t(A) %*% B %*% A`. Unlike `TRAITVARstd`, that formation adds `diag(c(ridging), n.manifest)`. The default `ridging = FALSE` adds 0, not `0.0001`; that ridge is a numerical hack and is not this exact map. The scalar correlation is `ψ / ψ = 1` after strictly positive `MANIFESTTRAITVAR`. Form strictly positive `ψ` first, then `1 / √ψ`, then `(1 / √ψ) ψ (1 / √ψ)`. Unstandardised `MANIFESTTRAITVAR` is defined for a zero trait; standardised `MANIFESTTRAITVAR` is not. Zero `MANIFESTTRAITVAR` skips forming `MANIFESTTRAITVARstd` in the 2017-era source and fails closed here. Indicator-level trait variance is an event-time structural quantity, so a non-event clock fails closed. `MANIFESTTRAITVAR` does not require stable `a < 0`. Distinct positive `ψ` recover the same 1. `trait / trait = 1` is `TRAITVARstd` and recovers the same number and remains a distinct named quantity. `θ` is `MANIFESTVAR` and is measurement error, not this correlation. Meredith (1993) remains unread (web search 2026-08-27T14:20Z: Springer/Cambridge Core paywalled; Unpaywall historically `is_oa: false`; Springer `content/pdf` is an HTML stub). Mislevy (1991, *Psychometrika, 56*, 177–196) remains unread on the same terms (DOI `10.1007/bf02294457`). Still not a Kalman filter, not a matrix `expm`, not ESEM estimation, not DSEM, and not ctsem estimation.
Expand Down
1 change: 1 addition & 0 deletions DOCUMENTATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ TEPP's approved PRD v0.4 and implementation plan are the primary product baselin
| Hourly NIM OpenCode doctoring | [`docs/doctoring/hourly-nim-opencode-development.md`](docs/doctoring/hourly-nim-opencode-development.md) |
| Analysis engine v1 doctoring | [`docs/doctoring/analysis-engine-v1.md`](docs/doctoring/analysis-engine-v1.md) |
| Analysis engine gap-closure doctoring | [`docs/doctoring/analysis-engine-gap-closure.md`](docs/doctoring/analysis-engine-gap-closure.md) |
| Joint posterior Laplace draws analysis-run doctoring | [`docs/doctoring/joint-posterior-draws-analysis-run.md`](docs/doctoring/joint-posterior-draws-analysis-run.md) |
| Corpus-split leakage-audit wire doctoring | [`docs/research/corpus-split-manifest-wire.md`](docs/research/corpus-split-manifest-wire.md) |
| Unicode canonical-identity doctoring | [`docs/research/unicode-canonical-identity.md`](docs/research/unicode-canonical-identity.md) |
| Change history | [`CHANGELOG.md`](CHANGELOG.md) |
Expand Down
330 changes: 330 additions & 0 deletions crates/analysis_engine/src/joint_posterior_draws_artifact.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,330 @@
//! Digest-bound joint Gaussian Laplace plausible-value draws as an analysis-run profile.

use serde::{Deserialize, Serialize};
use sha2::{Digest, Sha256};
use temporal_core::KnowledgeCutoff;
use tepp_api::{
AnalysisResultSummary, AnalysisRunAccepted, AnalysisRunRequest, AnalysisRunTerminalResult,
};
use topic_measurement::{
JOINT_POSTERIOR_DRAW_ALGORITHM_VERSION, ReferenceTopicInput, ReferenceTopicModelConfig,
fit_reference_topic_model,
};
use uuid::Uuid;

use crate::{AnalysisEngineError, format_digest, require_receipt_identity, valid_identifier};

/// Versioned schema for a completed joint-posterior-draw artifact.
pub const JOINT_POSTERIOR_DRAWS_ARTIFACT_SCHEMA_VERSION: &str = "tepp.joint_posterior_draws.v1";
/// Model contract required by the joint-posterior-draw execution path.
pub const JOINT_POSTERIOR_DRAWS_MODEL_CONTRACT_VERSION: &str = "joint_posterior_draws_v1";
/// Analysis-run output profile required for a joint-posterior-draw artifact.
pub const JOINT_POSTERIOR_DRAWS_OUTPUT_PROFILE: &str = "joint_posterior_draws_v1";
/// Maximum canonical artifact JSON size.
pub const JOINT_POSTERIOR_DRAWS_ARTIFACT_BYTE_LIMIT: usize = 256 * 1024;
const JOINT_POSTERIOR_DRAWS_INFERENCE_STATUS: &str =
"joint_gaussian_laplace_plausible_values_not_mcmc";
const JOINT_GAUSS_NEWTON_LAPLACE: &str = "joint_gauss_newton_laplace";

/// Completed, bounded joint posterior Laplace draws for analysis-run clients.
#[derive(Clone, Debug, Deserialize, PartialEq, Serialize)]
#[serde(deny_unknown_fields)]
pub struct JointPosteriorDrawsArtifact {
/// Exact versioned schema identity.
pub schema_version: String,
/// Opaque accepted-run identity.
pub run_id: String,
/// Immutable source snapshot identity.
pub snapshot_id: String,
/// Historical evidence cutoff used by the fit and draws.
pub knowledge_cutoff: String,
/// SHA-256 identity binding algorithm, seed, basis, fit, and draws.
pub draw_set_id: String,
/// Stable counter-based draw algorithm identity.
pub algorithm_version: String,
/// Explicit Philox seed used for the draw set.
pub seed: u64,
/// Number of joint Gaussian draws materialized.
pub draw_count: u64,
/// Number of modeled evidence documents.
pub document_count: u64,
/// Number of global topics in the fitted model.
pub topic_count: u64,
/// Laplace approximation identity (not MCMC).
pub approximation: String,
/// Fixed claim boundary for consumer copy.
pub inference_status: String,
}

impl JointPosteriorDrawsArtifact {
/// Parse and fully validate a bounded artifact JSON payload.
///
/// # Errors
///
/// Returns [`AnalysisEngineError::InvalidJointPosteriorDrawsArtifact`] when
/// the schema, identifiers, counts, algorithm, or claim boundary fail.
pub fn from_json(payload: &str) -> Result<Self, AnalysisEngineError> {
if payload.len() > JOINT_POSTERIOR_DRAWS_ARTIFACT_BYTE_LIMIT {
return Err(AnalysisEngineError::LimitExceeded);
}
let artifact: Self = serde_json::from_str(payload)
.map_err(|_| AnalysisEngineError::InvalidJointPosteriorDrawsArtifact)?;
artifact.validate()?;
Ok(artifact)
}

/// Serialize canonical validated artifact JSON.
///
/// # Errors
///
/// Returns a typed validation, serialization, or size failure.
pub fn to_json(&self) -> Result<String, AnalysisEngineError> {
self.validate()?;
let payload =
serde_json::to_string(self).map_err(|_| AnalysisEngineError::SerializationFailure)?;
if payload.len() > JOINT_POSTERIOR_DRAWS_ARTIFACT_BYTE_LIMIT {
return Err(AnalysisEngineError::LimitExceeded);
}
Ok(payload)
}

/// Return the lowercase SHA-256 digest of canonical artifact JSON.
///
/// # Errors
///
/// Returns a typed validation or serialization failure.
pub fn sha256(&self) -> Result<String, AnalysisEngineError> {
self.to_json()
.map(|json| format_digest(Sha256::digest(json.into_bytes())))
}

fn validate(&self) -> Result<(), AnalysisEngineError> {
if self.schema_version != JOINT_POSTERIOR_DRAWS_ARTIFACT_SCHEMA_VERSION
|| !valid_identifier(&self.run_id)
|| !valid_identifier(&self.snapshot_id)
|| KnowledgeCutoff::parse_rfc3339(&self.knowledge_cutoff).is_err()
|| !valid_identifier(&self.draw_set_id)
|| self.draw_set_id.len() != 64
Comment on lines +106 to +107

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Malformed draw digests pass validation

from_json accepts any 64-byte identifier as draw_set_id, including non-hex values. Corrupted artifacts therefore masquerade as SHA-256-bound draw sets.

Suggested change
|| !valid_identifier(&self.draw_set_id)
|| self.draw_set_id.len() != 64
|| !valid_identifier(&self.draw_set_id)
|| self.draw_set_id.len() != 64
|| !self
.draw_set_id
.bytes()
.all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte))
Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

|| self.algorithm_version != JOINT_POSTERIOR_DRAW_ALGORITHM_VERSION
|| self.draw_count == 0
|| self.document_count < 2
|| self.topic_count < 2
|| self.approximation != JOINT_GAUSS_NEWTON_LAPLACE
|| self.inference_status != JOINT_POSTERIOR_DRAWS_INFERENCE_STATUS
{
return Err(AnalysisEngineError::InvalidJointPosteriorDrawsArtifact);
}
Ok(())
}
}

/// One completed joint-posterior-draw artifact and its terminal result.
#[derive(Clone, Debug, PartialEq)]
pub struct JointPosteriorDrawsExecution {
/// Digest-bound completed draw-set artifact.
pub artifact: JointPosteriorDrawsArtifact,
/// Terminal result carrying the artifact identity, digest, and schema.
pub terminal_result: AnalysisRunTerminalResult,
}

/// Execute cutoff-safe joint Gaussian Laplace draws as one analysis-run profile.
///
/// The executor fits the CPU `f64` TRSL-TM reference, builds the identified
/// joint Gauss-Newton Laplace precision, and invokes
/// [`topic_measurement::JointCoordinatePrecision::draw_joint_gaussian`]. It
/// does not invent MCMC, select GPU backends, score candidate `K`, or emit
/// topic birth/split/merge events.
///
/// # Errors
///
/// Returns a request/receipt/snapshot/cutoff/profile error, estimator failure,
/// invalid draw request, or invalid artifact error.
#[allow(clippy::too_many_arguments)]
pub fn execute_joint_posterior_draws_run(
request: &AnalysisRunRequest,
accepted: &AnalysisRunAccepted,
snapshot_id: &str,
knowledge_cutoff: KnowledgeCutoff,
input: &ReferenceTopicInput,
config: &ReferenceTopicModelConfig,
topic_ids: Vec<Uuid>,
draw_count: usize,
completed_at: impl Into<String>,
) -> Result<JointPosteriorDrawsExecution, AnalysisEngineError> {
request.to_json()?;
accepted.to_json()?;
require_receipt_identity(request, accepted)?;
if request.snapshot_id != snapshot_id {
return Err(AnalysisEngineError::SnapshotMismatch);
}
if request.knowledge_cutoff != knowledge_cutoff.to_rfc3339()
Comment on lines +157 to +160

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Input provenance is self-asserted

An input built from another snapshot or later cutoff passes execute_joint_posterior_draws_run when separate metadata matches the request. The artifact can include future evidence.

Prompt for agents
The new execute_joint_posterior_draws_run API accepts ReferenceTopicInput separately from snapshot_id and knowledge_cutoff, but ReferenceTopicInput does not retain either provenance value after construction. The executor therefore validates only caller assertions and can bind input built from a different snapshot or a later cutoff to the request. Redesign the boundary so the model input, or a validated wrapper around it, carries immutable snapshot identity and construction cutoff. Validate those retained values against AnalysisRunRequest before fitting, and add tests that build input under a later cutoff or different snapshot and verify rejection.
Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

|| request.model_contract_version != JOINT_POSTERIOR_DRAWS_MODEL_CONTRACT_VERSION
|| request.output_profile != JOINT_POSTERIOR_DRAWS_OUTPUT_PROFILE
{
return Err(AnalysisEngineError::InvalidEvidence);
}

let model = fit_reference_topic_model(input, config)?;
let precision = input.build_joint_coordinate_precision(&model, config, topic_ids)?;
let draws = precision.draw_joint_gaussian(model.seed, draw_count)?;
let document_count = u64::try_from(draws.document_ids().len())
.map_err(|_| AnalysisEngineError::ArithmeticOverflow)?;
let topic_count = u64::try_from(draws.topic_ids().len())
.map_err(|_| AnalysisEngineError::ArithmeticOverflow)?;
let draw_count =
u64::try_from(draws.draws().len()).map_err(|_| AnalysisEngineError::ArithmeticOverflow)?;
let artifact = JointPosteriorDrawsArtifact {
schema_version: JOINT_POSTERIOR_DRAWS_ARTIFACT_SCHEMA_VERSION.into(),
run_id: accepted.run_id.clone(),
snapshot_id: snapshot_id.to_owned(),
knowledge_cutoff: knowledge_cutoff.to_rfc3339(),
draw_set_id: draws.draw_set_id().to_owned(),
algorithm_version: JOINT_POSTERIOR_DRAW_ALGORITHM_VERSION.into(),
seed: draws.seed(),
draw_count,
document_count,
topic_count,
approximation: JOINT_GAUSS_NEWTON_LAPLACE.into(),
inference_status: JOINT_POSTERIOR_DRAWS_INFERENCE_STATUS.into(),
};
let digest = artifact.sha256()?;
let summary = AnalysisResultSummary::new(
"joint_posterior_draws",
document_count,
4,
JOINT_POSTERIOR_DRAWS_INFERENCE_STATUS,
)?;
let terminal_result = AnalysisRunTerminalResult::succeeded(
request,
accepted,
format!("joint_posterior_draws_artifact_{}", &digest[..16]),
digest,
JOINT_POSTERIOR_DRAWS_ARTIFACT_SCHEMA_VERSION,
completed_at,
summary,
)?;
Ok(JointPosteriorDrawsExecution {
artifact,
terminal_result,
})
Comment on lines +206 to +209

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Generated draws are discarded

Every successful execute_joint_posterior_draws_run returns only draw metadata and a digest. Callers cannot retrieve or reconstruct the generated coordinates.

Prompt for agents
execute_joint_posterior_draws_run materializes a JointPosteriorDrawSet, extracts metadata, then drops the set and returns only JointPosteriorDrawsArtifact plus AnalysisRunTerminalResult. No persistence API or retrieval identifier backed by storage exists, so the requested coordinates are unavailable after success. Define the intended data boundary and either return the bounded JointPosteriorDrawSet with the execution, persist it through an explicit artifact store referenced by the terminal result, or include a reconstructible and available payload. Keep the digest binding and size limits, and test that a successful caller can obtain the exact generated coordinates.
Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

}

#[cfg(test)]
mod tests {
use super::{
JOINT_POSTERIOR_DRAWS_ARTIFACT_BYTE_LIMIT, JOINT_POSTERIOR_DRAWS_ARTIFACT_SCHEMA_VERSION,
JOINT_POSTERIOR_DRAWS_INFERENCE_STATUS, JointPosteriorDrawsArtifact,
};
use crate::AnalysisEngineError;
use topic_measurement::JOINT_POSTERIOR_DRAW_ALGORITHM_VERSION;

fn artifact() -> JointPosteriorDrawsArtifact {
JointPosteriorDrawsArtifact {
schema_version: JOINT_POSTERIOR_DRAWS_ARTIFACT_SCHEMA_VERSION.into(),
run_id: "run-1".into(),
snapshot_id: "snapshot-1".into(),
knowledge_cutoff: "2026-08-01T00:00:00Z".into(),
draw_set_id: "a".repeat(64),
algorithm_version: JOINT_POSTERIOR_DRAW_ALGORITHM_VERSION.into(),
seed: 7,
draw_count: 4,
document_count: 4,
topic_count: 2,
approximation: "joint_gauss_newton_laplace".into(),
inference_status: JOINT_POSTERIOR_DRAWS_INFERENCE_STATUS.into(),
}
}

fn assert_invalid(artifact: &JointPosteriorDrawsArtifact) {
assert_eq!(
artifact.to_json(),
Err(AnalysisEngineError::InvalidJointPosteriorDrawsArtifact)
);
}

#[test]
fn artifact_round_trip_and_size_bounds_fail_closed() {
let artifact = artifact();
let payload = artifact.to_json().expect("json");
assert_eq!(
JointPosteriorDrawsArtifact::from_json(&payload),
Ok(artifact.clone())
);
assert_eq!(artifact.sha256().expect("digest").len(), 64);
assert_eq!(
JointPosteriorDrawsArtifact::from_json("{}"),
Err(AnalysisEngineError::InvalidJointPosteriorDrawsArtifact)
);
assert_eq!(
JointPosteriorDrawsArtifact::from_json(
&"x".repeat(JOINT_POSTERIOR_DRAWS_ARTIFACT_BYTE_LIMIT + 1)
),
Err(AnalysisEngineError::LimitExceeded)
);
}

#[test]
fn artifact_metadata_tampering_fails_closed() {
let artifact = artifact();
let invalid_artifacts = [
{
let mut value = artifact.clone();
value.schema_version.clear();
value
},
{
let mut value = artifact.clone();
value.run_id.clear();
value
},
{
let mut value = artifact.clone();
value.snapshot_id.clear();
value
},
{
let mut value = artifact.clone();
value.knowledge_cutoff = "invalid".into();
value
},
{
let mut value = artifact.clone();
value.draw_set_id.clear();
value
},
{
let mut value = artifact.clone();
value.algorithm_version.clear();
value
},
{
let mut value = artifact.clone();
value.draw_count = 0;
value
},
{
let mut value = artifact.clone();
value.document_count = 1;
value
},
{
let mut value = artifact.clone();
value.topic_count = 1;
value
},
{
let mut value = artifact.clone();
value.approximation.clear();
value
},
{
let mut value = artifact.clone();
value.inference_status.clear();
value
},
];
for invalid in invalid_artifacts {
assert_invalid(&invalid);
}
}
}
Loading
Loading