Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .github/workflows/validity-analysis-quality.yml
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,6 @@ jobs:
run: python -m compileall -q packages/validity-analysis/src packages/validity-analysis/tests
- name: Test governed handoff with exact statement and branch coverage
env:
PYTHONPATH: packages/validity-analysis/src
COVERAGE_FILE: /tmp/orgmetra-validity-analysis.coverage
run: python -m pytest -c packages/validity-analysis/pyproject.toml packages/validity-analysis/tests
- name: Require clean checkout
Comment thread
seonghobae marked this conversation as resolved.
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -35,3 +35,4 @@ secrets/
artifacts/
reports/
*.log
/.codegraph/
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ All notable changes to Orgmetra will be documented in this file.

- Accepted ADRs 0001–0003 now include buyer-facing Context, Decision, and Consequences grounded in verified ISO 30400:2022, ISO 30414:2025, Uniform Guidelines (29 C.F.R. Part 1607), SIOP (2018), OpenAPI Specification v3.2.0, OpenID Connect Core 1.0 errata set 2, CloudEvents v1.0.2, Jensen and Snodgrass (1999), Snodgrass (1999), and Allen (1983) records already listed in `docs/doctoring/REFERENCES.md`. ADRs 0004 and 0005 gained APA 7th References pointers to that same bibliography without changing their Decision bodies.
- Active-PR governed Job Analysis persistence/API on the canonical `JobAnalysisSnapshot` model: migration `0013_job_analysis_snapshot.sql` stores immutable tenant-scoped snapshot, Task, KSAO, Task–KSAO, FJA and write-command evidence; `POST /v1/tenants/{tenant_record_id}/job-analysis-snapshots` and matching GET enforce purpose-bound Keyverse scope, authenticated-principal actor authority, bounded/strict JSON handling, transactional Idempotency-Key serialization, parent-scope fail-closed integrity, forced RLS, and atomic audit/outbox evidence. ADR 0014 records the persistence decision while ADR 0007 remains the domain/evidence authority; validated evidence still requires accountable human review and non-LLM provenance, and the service does not make a high-impact employment decision.
- Active-PR validity-analysis execution boundary: exact-revision read-only Rust CPU recovery smoke contracts preserve aggregate scientific evidence separately from criterion-related validity estimates, freeze completion timestamps and numeric values before canonicalization, and keep unsupported designs and high-impact employment decisions fail-closed.
- Active-PR `orgmetra_selection_review` packet for PII-minimized, evidence-bound human selection review: canonical operational tenant identity, UUID-backed opaque candidate/Job/sealed-evidence/reviewer references, explicit purpose/reason/evidence version, deterministic canonical JSON and SHA-256 correlation, mandatory human decision state, redacted packet repr, and provenance-paired model evidence that remains `untrusted_draft`, with exact 100% owned statement and branch coverage required by its quality gate.
- Active performance-criterion scope hardening: `criterion_observation_scope_guard` rejects criterion outcomes for a Job the worker did not effectively hold at the observation date, observations before the relevant assignment, and observations outside the referenced performance cycle while preserving valid multiple-assignment cases and existing bitemporal correction semantics. The guard evaluates current-recorded facts, derives the date coordinate from `observed_at` in UTC so session `TimeZone` cannot alter the result, uses a trusted function search path, and adds no PII or automated employment decision authority. The Foundation PostgreSQL contract also rejects a closed `recorded_to` on each time-coordinate lookup and proves UTC midnight plus non-UTC session `TimeZone` boundaries.
- Bitemporal tenant-scoped organization hierarchy validation that rejects visible indirect parent cycles and reuses single-valued recorded-time reconstruction before graph traversal.
Expand Down
2 changes: 1 addition & 1 deletion docs/TEST_STRATEGY.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ Orgmetra does not combine fast-mlsirm and TEPP into one dependency.
### fast-mlsirm

- Canonical repository: `ContextualWisdomLab/fast-mlsirm`.
- Reviewed immutable revision for this baseline: `fb67ced09d8ee00542c05d56374537a9a7239751`.
- Reviewed immutable revision for this baseline: `04d0bc21a2a20693bcf16108cd76d394fe844d23`.
- Orgmetra contract identifier: `orgmetra.fast_mlsirm.v1`.
- Owner: `workforce_validation`; the normal online path consumes a Psychometrics Commons immutable result snapshot rather than calling the kernel from a role workspace.
- Backend: Rust production arithmetic with bounded CPU multithreading and GPU parity for material kernels. NumPy is a reference/parity path only.
Expand Down
3 changes: 2 additions & 1 deletion docs/TRACEABILITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,8 @@
| Keyverse identity and authorization | API Gateway / purpose-bound authorization | Published OIDC/API identity and scope contract plus Orgmetra-owned `orgmetra_keyverse_adapter` policy evaluation | tenant/actor/resource agreement, exact opaque target-resource reference, purpose, operation-specific scope, requested-field minimization, opaque subject, no stored credentials or protected values in authorization evidence | ADR-0002, ADR-0008 | implemented_on_protected_main |
| naruon communication and calendar | Integration Hub | Published API/event adapter | idempotency, delivery audit, no direct table access | ADR-0002 | planned |
| Psychometrics Commons @ `cc5850a0d1eacbbf16d03075534fce460a8286e6` | Workforce Validation | Immutable response/result snapshot contract | pinned revision, model/version/provenance snapshot, immutable result linkage, no direct application-table access | ADR-0002 | accepted_architecture |
| fast-mlsirm @ `fb67ced09d8ee00542c05d56374537a9a7239751` | Workforce Validation | Published `orgmetra.fast_mlsirm.v1` result contract; direct calls only from approved offline validation worker | pinned revision, contract identifier, backend/result provenance, CPU/GPU parity evidence where material, no duplicated kernel | ADR-0002 | accepted_architecture |
| fast-mlsirm @ `04d0bc21a2a20693bcf16108cd76d394fe844d23` | Workforce Validation | Published `orgmetra.fast_mlsirm.v1` result contract; direct calls only from approved offline validation worker | pinned revision, contract identifier, backend/result provenance, CPU/GPU parity evidence where material, no duplicated kernel | ADR-0002 | accepted_architecture |
| Rust validity execution and recovery evidence | Workforce Validation | `RustExecutionRequest`, `RustRecoveryEvidence`, and exact-revision read-only runner | exact built-in governance text, pinned revision, detached UTC completion time, numeric snapshots, aggregate-only recovery receipt, explicit nonconvergence, and no mapping to criterion-related validity or employment decisions | ADR-0027 | implemented_on_active_pr |
| TEPP temporal evidence | Workforce Validation | Published package/API contract | temporal provenance and version binding | ADR-0002 | planned |
| MHTML ETL Gateway / mightyETL | Governed Migration | Published ETL contract | lineage, idempotency, reconciliation, rollback | ADR-0002 | planned |
| Semantic Data Portal / OriginWeave / LineageWeave | Evidence and lineage adapters | Published API/event contracts | provenance, tenant ACL, retention, export controls | ADR-0002 | planned |
Expand Down
2 changes: 1 addition & 1 deletion docs/TRD.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ These identifiers are canonical across deployment names, ACLs, metrics, generate
| `keyverse_adapter` | Keyverse OIDC/SCIM contract | `integration_hub` |
| `naruon_adapter` | Naruon communication-intent contract | `integration_hub` |
| `psychometrics_commons_adapter` | immutable response/result snapshot contract pinned to `cc5850a0d1eacbbf16d03075534fce460a8286e6` | `workforce_validation` |
| `fast_mlsirm_adapter` | `orgmetra.fast_mlsirm.v1`, repository `ContextualWisdomLab/fast-mlsirm` pinned to `fb67ced09d8ee00542c05d56374537a9a7239751`; online role workspaces consume it through Psychometrics Commons, while direct calls are limited to an approved offline validation worker | `workforce_validation` |
| `fast_mlsirm_adapter` | `orgmetra.fast_mlsirm.v1`, repository `ContextualWisdomLab/fast-mlsirm` pinned to `04d0bc21a2a20693bcf16108cd76d394fe844d23`; online role workspaces consume it through Psychometrics Commons, while direct calls are limited to an approved offline validation worker | `workforce_validation` |
| `tepp_adapter` | `orgmetra.tepp.v1`, repository `ContextualWisdomLab/TEPP` pinned to `40adac9a26a8af85147ffa2795fb548ea243e0e5` | `workforce_validation` |
| `semantic_data_portal_adapter` | versioned ontology and data-catalog contract | `job_architecture` |
| `contextual_orchestrator_adapter` | schema-bound draft and verification operations; no authoritative writes | `job_architecture` and `integration_hub` |
Expand Down
9 changes: 7 additions & 2 deletions docs/adr/0027-governed-selection-validity-analysis-handoff.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ Orgmetra adds a leaf `orgmetra_validity_analysis` package whose `ValidationAnaly

The same package also validates `ValidationAnalysisResult` envelopes returned by the approved offline worker. A result must link to the handoff digest and the same pinned revision, identify a Rust CPU or GPU backend and precision, provide finite effect and interval values, match its sample size to aggregate missingness counts, reject impossible complete-versus-missing count combinations, and include explicit convergence diagnostics. The canonicalization boundary accepts only the exact governed `MissingnessSummary` and `ConvergenceDiagnostics` runtime types so subclass method overrides cannot add unreviewed or person-level fields to immutable result evidence. A nonconverged result remains typed scientific evidence requiring human review; it cannot be treated as a valid selection procedure or an employment decision.

The package does not invoke fast-mlsirm. An approved offline worker is the later execution boundary. Before execution, the Orgmetra host must re-resolve every reference inside the tenant, verify exact study/Job membership and evidence provenance, and prove requester/reviewer identities are distinct authoritative actors.
The package additionally defines `RustExecutionRequest` and `RustRecoveryEvidence`. The request makes cross-sectional, nested multilevel, multiple-membership, and longitudinal input structure explicit, and its exact built-in governance text is validated before comparisons or canonical serialization. Only cross-sectional and nested multilevel designs are runnable in this slice; multiple-membership and longitudinal requests fail closed until a reviewed estimator exists. The optional read-only `run_fast_mlsirm_recovery_evidence.py` runner verifies a clean exact revision before and after execution, invokes the foreign package's public Rust API in its own uv environment with a bounded 180-second timeout, and converts only aggregate simulation-recovery output into a receipt. Recovery evidence freezes completion time as a UTC instant and snapshots numeric values before canonicalization; recovery RMSE is not mapped to `ValidationAnalysisResult.effect_estimate`. Before an approved validity execution, the Orgmetra host must still re-resolve every reference inside the tenant, verify exact study/Job membership and evidence provenance, and prove requester/reviewer identities are distinct authoritative actors.

## Consequences

Expand All @@ -39,18 +39,23 @@ The package does not invoke fast-mlsirm. An approved offline worker is the later
- Aggregate missingness evidence cannot claim all observations are complete while simultaneously reporting predictor- or criterion-missing observations.
- Result canonicalization cannot be extended by an unreviewed subclass to serialize extra decision-like or person-level fields.
- Human interpretation remains explicit and separate from numerical output.
- A real bounded Rust CPU smoke run now proves the pinned worker path and preserves `max_iter_reached` as an explicit nonconvergence state without promoting it to validity evidence.
- The dedicated-writer boundary remains intact: Orgmetra consumes only a pinned foreign revision/contract boundary and never mutates fast-mlsirm.

### Limitations

- This slice does not execute a statistical model, estimate validity, correct for measurement error/range restriction, evaluate adverse impact, or assert legal compliance.
- The package does not estimate criterion-related validity, correct for measurement error/range restriction, evaluate adverse impact, or assert legal compliance. The optional runner executes only a bounded synthetic recovery smoke.
- No GPU run or CPU/GPU numerical parity result is claimed until hardware-backed paired measurements are captured.
- Multiple-membership and longitudinal contracts are represented but have no numerical estimator and intentionally fail closed.
- Sampling design, estimator choice, missing-data treatment, reliability evidence, multiplicity, transportability, fairness analysis, and model diagnostics must be encoded in the referenced analysis plan and reviewed before execution.
- The package validates the result envelope, but a future execution adapter must still re-resolve the handoff references, verify the result provenance artifact, and attach evidence only after accountable human review.

## Verification

The package regression suite starts RED when the public handoff contract is absent and covers canonical operational tenant UUIDs, opaque UUIDv4 references, exact evidence digests, distinct human actors, exact dependency pinning, timezone-aware event time, immutable governance constants, value minimization, deterministic canonicalization, SHA-256 correlation, impossible aggregate missingness rejection, exact governed aggregate-evidence runtime types, and 100% owned production statement/branch coverage. The repository-wide ADR numbering regression also fails closed if integration reuses an existing decision number.

On 2026-08-21, the optional runner was executed against fast-mlsirm revision `04d0bc21a2a20693bcf16108cd76d394fe844d23` for 48 synthetic persons and 3 items with Rust CPU and four requested Rayon threads. Both cross-sectional and nested multilevel runs produced aggregate recovery receipts; both were explicitly `max_iter_reached` after one bounded iteration. This is smoke evidence for the pinned execution path, not estimator acceptance or protected-branch truth.

## References

See `docs/doctoring/validation-analysis-handoff-references.md`.
Loading