Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/product-technical-gap-baseline.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
## 3. General Architecture Gaps
- **DB Architecture**: Ensure PostgreSQL is strictly used (no file DBs), 3rd normal form is maintained, and Hot Partitions are handled. DB locks must be managed (or use read/write replicas).
- **Zotero Integration**: Papers and standards referenced by TEPP must be synced via Local Zotero API (http://localhost:23119/api/) and cited using APA 7th edition in docstrings.
- **Testing**: We need actual testing of Psychometrics (Fast-MLSIRM parameter calibration, RMSE of estimates, Fixed-Item Parameter Calibration, CAT) against synthetic/demo data.
- **Testing**: Partially resolved -- `tests/test_fast_mlsirm_gpcm_recovery.py` (new) simulates polytomous responses from known true item parameters and person thetas under the GPCM (Muraki, 1993) formula (`fast_mlsirm` ships no polytomous-specific simulator, so the response-generation formula is implemented directly in the test, matching `PolytomousFit`'s own documented parameterization), fits them with `fast_mlsirm.fit_polytomous` under the GPCM model -- the same function/model option and iteration bound `period_report.py`'s production code uses -- and asserts the recovered EAP thetas are close to true by RMSE and correlation (GPCM: RMSE ~0.30, correlation ~0.95). This is real theta-recovery accuracy testing against synthetic data with known ground truth, not item-parameter calibration or an infra-only smoke test. Still open: item-parameter calibration; an equivalent GRM (Samejima, 1969) recovery test (`tests/test_fast_mlsirm_grm_recovery.py`) is in progress on a separate branch and is not yet part of this repository; Fixed-Item Parameter Calibration (Kim, 2006 FIPC -- `period_report.py` uses this for later periods, untested) and CAT remain unverified -- though `fast_mlsirm.cat_simulate_polytomous` (a real adaptive-test simulator over a fitted GRM/GPCM bank, Dodd, De Ayala & Koch, 1995) was found to exist and is a concrete next step, not yet exercised anywhere in this repo's tests.
- **Security & Compliance**: PII masking cannot break the system. Need SOC 2 and CSAP compliance alternatives to blind PII masking.
- **LLM Orchestration**: Ensure ALL LLM calls route through `contextual-orchestrator` utilizing API keys (BYTEZ, NVIDIA, OPENROUTER, OPENAI) with auto model discovery and optimal reasoning effort allocation (Fugu/Conductor/TRINITY research).

Expand Down
3 changes: 3 additions & 0 deletions frontend/src/App.test.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,9 @@ describe("App, unauthenticated", () => {
state: expect.objectContaining({ returnUrl: expect.stringMatching(/^\//) }),
}),
);
// Persisted as a fallback in case the OIDC state round-trip is dropped
// (see oidcReturnUrl.ts's restoreOidcReturnUrl, consumed in main.tsx).
expect(window.sessionStorage.getItem("lineageweave.oidc.returnUrl")).toMatch(/^\//);
});
});

Expand Down
4 changes: 2 additions & 2 deletions frontend/src/App.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -4610,7 +4610,8 @@ export default function App({ showLabPanels = false }: { showLabPanels?: boolean
</div>
<div className="login-controls">
<button className="btn-primary" onClick={() => {
const returnUrl = window.location.pathname + window.location.search;
const returnUrl = returnUrlFromLocation();
rememberOidcReturnUrl(returnUrl);
void auth.signinRedirect({ state: { returnUrl } });
}}>
{t("Log in")}
Expand All @@ -4620,7 +4621,6 @@ export default function App({ showLabPanels = false }: { showLabPanels?: boolean
<small>Enterprise SSO Authentication</small>
</div>
</div>
Comment thread
seonghobae marked this conversation as resolved.
{destination === "admin" ? <AdminPanel currentBrandName={brandName} onBrandNameChange={setBrandName} accessToken={accessToken} /> : null}
</main>
<footer className="app-footer" role="contentinfo">
<div className="app-footer-title">
Expand Down
69 changes: 69 additions & 0 deletions tests/test_fast_mlsirm_gpcm_recovery.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
"""Real theta-recovery test for the GPCM model period_report.py's
production code can also fit: simulate polytomous responses from known
true item parameters and person abilities using the generalized partial
credit model (Muraki, 1993), fit them with fast_mlsirm.fit_polytomous(...,
model="gpcm") -- the same function/model option period_report.py's
"pick the model with fixed_item_calibration_diagnostics" step can select
-- and assert the recovered EAP thetas are close to the true thetas by
RMSE and correlation.

fast_mlsirm ships no polytomous-specific simulator, so the GPCM
category-probability formula is implemented directly here, matching the
library's own documented parameterization (PolytomousFit's docstring:
"GPCM additive category intercepts"): cumulative step logits
z_k = sum_{v=1}^{k} a*(theta - b_v), z_0 = 0, softmax over z.
"""

from __future__ import annotations

import numpy as np
from fast_mlsirm import fit_polytomous, score_polytomous

N_PERSONS = 400
N_ITEMS = 12
N_CAT = 4
SEED = 20260101

# A real run with these exact parameters/seed measures RMSE ~0.30 and
# correlation ~0.95 -- stronger recovery than the GRM test's ~0.38/~0.92
# at the same sample size, consistent with GPCM's additive (vs. GRM's
# cumulative) category structure being easier to identify here. The
# margins below stay loose enough to tolerate a minor fast-mlsirm version
# bump while still catching an actual estimation regression.
MAX_THETA_RMSE = 0.55
MIN_THETA_CORRELATION = 0.8
Comment thread
seonghobae marked this conversation as resolved.


def _gpcm_category_probs(theta: float, discrimination: float, steps: np.ndarray) -> np.ndarray:
"""Muraki (1993) generalized partial credit model category
probabilities for one person/item pair, given known true parameters."""
cumulative_steps = np.cumsum(discrimination * (theta - steps))
z = np.concatenate(([0.0], cumulative_steps))
z = z - z.max()
unnormalized = np.exp(z)
return unnormalized / unnormalized.sum()


def test_gpcm_recovers_true_theta_within_expected_rmse() -> None:
rng = np.random.default_rng(SEED)
true_theta = rng.normal(0.0, 1.0, N_PERSONS)
true_discrimination = rng.uniform(0.8, 2.0, N_ITEMS)
true_steps = rng.normal(0.0, 1.0, (N_ITEMS, N_CAT - 1))

responses = np.zeros((N_PERSONS, N_ITEMS))
for item in range(N_ITEMS):
for person in range(N_PERSONS):
probs = _gpcm_category_probs(true_theta[person], true_discrimination[item], true_steps[item])
responses[person, item] = rng.choice(N_CAT, p=probs)
Comment thread
seonghobae marked this conversation as resolved.

fit = fit_polytomous(responses, n_cat=N_CAT, model="gpcm", max_iter=80)
assert fit.converged

scored = score_polytomous(responses, fit)
theta_eap = scored["theta_eap"]

rmse = float(np.sqrt(np.mean((theta_eap - true_theta) ** 2)))
correlation = float(np.corrcoef(theta_eap, true_theta)[0, 1])

assert rmse < MAX_THETA_RMSE, f"theta recovery RMSE {rmse:.3f} exceeded {MAX_THETA_RMSE}"
assert correlation > MIN_THETA_CORRELATION, f"theta recovery correlation {correlation:.3f} below {MIN_THETA_CORRELATION}"
Loading