Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
c28fe1b
test(validity): require governed analysis handoff
seonghobae Aug 20, 2026
5b0a1e5
feat(validity): add governed fast-mlsirm handoff
seonghobae Aug 20, 2026
3f99928
docs(validity): record governed analysis boundary
seonghobae Aug 20, 2026
30ac02a
chore(validity): integrate protected develop after #41
seonghobae Aug 20, 2026
a9168a9
test(validity): reject duplicate ADR numbers after integration
seonghobae Aug 20, 2026
6814fd9
fix(validity): reserve ADR 0027 for analysis handoff
seonghobae Aug 20, 2026
a4876a8
fix(validity): track reserved ADR 0027 in quality gate
seonghobae Aug 20, 2026
7e9cd76
docs(validity): bind references to ADR 0027
seonghobae Aug 20, 2026
a872938
fix(validity): remove conflicting ADR 0025 allocation
seonghobae Aug 20, 2026
22fd374
test(validity): require resolved reviewer identity separation
seonghobae Aug 21, 2026
8432d1c
fix(validity): require authoritative actor identity separation
seonghobae Aug 21, 2026
68a15c4
docs(validity): trace authoritative actor separation
seonghobae Aug 21, 2026
f0c30c5
docs(validity): record identity-separation repair
seonghobae Aug 21, 2026
c381518
chore(validity): integrate protected develop after #43
seonghobae Aug 21, 2026
d0c06d2
feat(validity): validate pinned numerical result envelopes
seonghobae Aug 21, 2026
0a31737
test(validity): reject impossible complete missingness totals
seonghobae Aug 21, 2026
2419aa9
fix(validity): reject impossible missingness summaries
seonghobae Aug 21, 2026
c17936c
test(validity): reject subclassed result evidence
seonghobae Aug 21, 2026
f31422a
fix(validity): require exact result evidence types
seonghobae Aug 21, 2026
e11ae2a
docs(validity): record result evidence hardening
seonghobae Aug 21, 2026
e9a7a31
docs(validity): define hardened result envelope
seonghobae Aug 21, 2026
8b26520
docs(validity): trace result integrity regressions
seonghobae Aug 21, 2026
b33cbda
docs(validity): bind result integrity in ADR
seonghobae Aug 21, 2026
081942a
test(validity): cover criterion overlap guard
seonghobae Aug 21, 2026
d6bb5f2
test(validity): reject temporal evidence subclasses
seonghobae Aug 21, 2026
b616a13
fix(validity): require exact temporal evidence type
seonghobae Aug 21, 2026
219c2e3
fix(validity): remove unused kernel revision import
seonghobae Aug 21, 2026
89c9707
test(validity): require completed_at diagnostics
seonghobae Aug 22, 2026
b72c7a0
fix(validity): report field-correct timestamp errors
seonghobae Aug 22, 2026
5197338
fix(validity): bind result timestamp diagnostic
seonghobae Aug 22, 2026
e7c3404
test(validity): align temporal diagnostic regression
seonghobae Aug 22, 2026
a8da84c
fix(validity): freeze handoff evidence boundaries
seonghobae Aug 29, 2026
5081844
test(validity): reject overflowing worker numerics
seonghobae Aug 29, 2026
a55f95c
test(validity): require ADR-wide quality trigger
seonghobae Aug 29, 2026
06db366
fix(validity): normalize numeric conversion failures
seonghobae Aug 29, 2026
be52fc7
fix(validity): run ADR uniqueness check for all ADR changes
seonghobae Aug 29, 2026
376c2ca
docs(validity): pin reproducible selection-validation references
seonghobae Aug 29, 2026
2ff2582
docs(validity): trace repaired evidence boundaries
seonghobae Aug 29, 2026
4359cfb
docs(validity): record numeric and missingness ownership boundaries
seonghobae Aug 29, 2026
2f6d1dc
test(validity): require shared config quality triggers
seonghobae Aug 29, 2026
8815f73
fix(validity): retrigger quality on shared config
seonghobae Aug 29, 2026
3c2a1de
docs(validity): record shared-config gate integrity
seonghobae Aug 29, 2026
6ca5547
docs(validity): trace shared-config quality evidence
seonghobae Aug 29, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 67 additions & 0 deletions .github/workflows/validity-analysis-quality.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
name: Validity Analysis Handoff Quality

on:
pull_request:
branches:
- bootstrap
- develop
- main
paths:
- "packages/validity-analysis/**"
- "docs/adr/**"
Comment thread
seonghobae marked this conversation as resolved.
- "docs/doctoring/validation-analysis-handoff-references.md"
- "docs/traceability/validation-analysis-handoff.md"
- ".github/requirements/foundation-test.txt"
- ".github/workflows/validity-analysis-quality.yml"
- ".gitignore"
- ".python-version"
- "conftest.py"
- "packages/conftest.py"
- "pyproject.toml"
- "pytest.ini"
- "setup.cfg"
- "tox.ini"
workflow_dispatch:

permissions:
contents: read

concurrency:
group: validity-analysis-quality-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true

jobs:
unit:
name: Validity handoff and 100% coverage
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Checkout exact candidate
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
persist-credentials: false
- name: Prove exact candidate checkout
env:
ORGMETRA_EXPECTED_HEAD_SHA: ${{ github.event.pull_request.head.sha || github.sha }}
run: test "$(git rev-parse HEAD)" = "$ORGMETRA_EXPECTED_HEAD_SHA"
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
check-latest: false
- name: Install reviewed test toolchain
run: |
python -m pip install --require-hashes --no-deps --only-binary=:all: -r .github/requirements/foundation-test.txt
python -m pip check
- name: Compile validity-analysis handoff
run: python -m compileall -q packages/validity-analysis/src packages/validity-analysis/tests
- name: Test governed handoff with exact statement and branch coverage
env:
PYTHONPATH: packages/validity-analysis/src
COVERAGE_FILE: /tmp/orgmetra-validity-analysis.coverage
run: python -m pytest -c packages/validity-analysis/pyproject.toml packages/validity-analysis/tests
- name: Require clean checkout
run: |
git diff --exit-code
test -z "$(git status --porcelain)"
Comment thread
seonghobae marked this conversation as resolved.
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ All notable changes to Orgmetra will be documented in this file.
- Accepted ADRs 0001–0003 now include buyer-facing Context, Decision, and Consequences grounded in verified ISO 30400:2022, ISO 30414:2025, Uniform Guidelines (29 C.F.R. Part 1607), SIOP (2018), OpenAPI Specification v3.2.0, OpenID Connect Core 1.0 errata set 2, CloudEvents v1.0.2, Jensen and Snodgrass (1999), Snodgrass (1999), and Allen (1983) records already listed in `docs/doctoring/REFERENCES.md`. ADRs 0004 and 0005 gained APA 7th References pointers to that same bibliography without changing their Decision bodies.
- Active-PR governed Job Analysis persistence/API on the canonical `JobAnalysisSnapshot` model: migration `0013_job_analysis_snapshot.sql` stores immutable tenant-scoped snapshot, Task, KSAO, Task–KSAO, FJA and write-command evidence; `POST /v1/tenants/{tenant_record_id}/job-analysis-snapshots` and matching GET enforce purpose-bound Keyverse scope, authenticated-principal actor authority, bounded/strict JSON handling, transactional Idempotency-Key serialization, parent-scope fail-closed integrity, forced RLS, and atomic audit/outbox evidence. ADR 0014 records the persistence decision while ADR 0007 remains the domain/evidence authority; validated evidence still requires accountable human review and non-LLM provenance, and the service does not make a high-impact employment decision.
- Active-PR `orgmetra_selection_review` packet for PII-minimized, evidence-bound human selection review: canonical operational tenant identity, UUID-backed opaque candidate/Job/sealed-evidence/reviewer references, explicit purpose/reason/evidence version, deterministic canonical JSON and SHA-256 correlation, mandatory human decision state, redacted packet repr, and provenance-paired model evidence that remains `untrusted_draft`, with exact 100% owned statement and branch coverage required by its quality gate.
- Active-PR `orgmetra_validity_analysis` handoff/result boundary: exact tenant and evidence references, reviewed fast-mlsirm pin, distinct requester/reviewer actors, aggregate-only scientific evidence, construction-time UTC and finite-number snapshots, canonical JSON/SHA-256 correlation, and human-review-only result authority.
- Active performance-criterion scope hardening: `criterion_observation_scope_guard` rejects criterion outcomes for a Job the worker did not effectively hold at the observation date, observations before the relevant assignment, and observations outside the referenced performance cycle while preserving valid multiple-assignment cases and existing bitemporal correction semantics. The guard evaluates current-recorded facts, derives the date coordinate from `observed_at` in UTC so session `TimeZone` cannot alter the result, uses a trusted function search path, and adds no PII or automated employment decision authority. The Foundation PostgreSQL contract also rejects a closed `recorded_to` on each time-coordinate lookup and proves UTC midnight plus non-UTC session `TimeZone` boundaries.
- Bitemporal tenant-scoped organization hierarchy validation that rejects visible indirect parent cycles and reuses single-valued recorded-time reconstruction before graph traversal.
- Stacked governed job-analysis evidence contract via `JobAnalysisSnapshot`, `TaskEvidence`, `KSAORequirement`, `TaskKSAOLink`, `FunctionalJobAnalysisProfile`, and `EvidenceSource`: tenant/Job-scoped observable tasks, explicit Task-to-KSAO linkage, importance/difficulty/proficiency ratings, source/version/retrieval/SHA-256 provenance, deterministic canonical snapshot bytes, current O*NET evidence support, and historical DOT Data/People/Things compatibility. Validated snapshots require accountable human review and complete non-LLM evidence; LLM-origin material remains `analysis_draft`, and the snapshot is evidence input rather than a hiring, promotion, termination, compensation, or other high-impact employment decision.
Expand Down
60 changes: 60 additions & 0 deletions docs/adr/0027-governed-selection-validity-analysis-handoff.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# ADR 0027: Govern selection-validity numerical work through an immutable handoff

- Status: Proposed
- Maturity: Active PR only; not protected-branch truth
- Date: 2026-08-21
- Owners: Orgmetra Workforce Validation

## Context

Protected Orgmetra already preserves exact validation-study cases, sealed selection evidence, candidate-to-worker lineage, and Job/cycle/staffing-scoped criterion observations. The remaining boundary is dangerous if left implicit: a statistical worker could receive an underspecified study, silently use a different dependency revision, or turn a model result into an employment decision.

The Uniform Guidelines recognize criterion-related validity evidence as empirical evidence relating a selection procedure to important job-performance elements and require validity studies to be accurate, standardized, documented, and periodically reviewed for currency. SIOP's *Principles for the Validation and Use of Personnel Selection Procedures* likewise treats validation as an evidence-and-inference problem rather than a correlation-only shortcut.

`ContextualWisdomLab/fast-mlsirm` owns numerical psychometric/statistical kernels. Its protected `main` was freshly resolved to commit `04d0bc21a2a20693bcf16108cd76d394fe844d23` on 2026-08-21. Orgmetra must not copy that implementation or write the foreign repository.

## Decision

Orgmetra adds a leaf `orgmetra_validity_analysis` package whose `ValidationAnalysisHandoff`:

- binds the exact tenant, validation study, Job, predictor snapshot, criterion snapshot, population snapshot, decision policy, and analysis plan through opaque references plus SHA-256 evidence digests;
- binds distinct requester and reviewer actor references;
- pins fast-mlsirm to reviewed immutable commit `04d0bc21a2a20693bcf16108cd76d394fe844d23`;
- declares the numerical boundary `read_only_pinned_revision` and the initial strategy `criterion_related`;
- requires downstream result evidence for effect estimate, uncertainty interval, sample size, missingness summary, and convergence diagnostics;
- serializes no raw person-level predictor, criterion, candidate, or worker values;
- remains `not_executed`, `scientific_evidence_only`, and human-review-required;
- produces deterministic canonical JSON and a SHA-256 digest for audit/result correlation.

Both handoff and result envelopes detach exact timezone-aware timestamps to one built-in UTC instant at construction. Result numeric evidence is converted to finite built-in floats before storage, and conversion failures including numeric overflow are normalized to the package's fail-closed `ValueError` contract, so caller-controlled timezone or numeric runtime behavior cannot rewrite canonical evidence after validation or escape normal malformed-result handling.

The same package also validates `ValidationAnalysisResult` envelopes returned by the approved offline worker. A result must link to the handoff digest and the same pinned revision, identify a Rust CPU or GPU backend and precision, provide finite effect and interval values, match its sample size to aggregate missingness counts, reject impossible complete-versus-missing count combinations, and include explicit convergence diagnostics. The canonicalization boundary accepts only the exact governed `MissingnessSummary` and `ConvergenceDiagnostics` runtime types so subclass method overrides cannot add unreviewed or person-level fields to immutable result evidence. A nonconverged result remains typed scientific evidence requiring human review; it cannot be treated as a valid selection procedure or an employment decision.

The package does not invoke fast-mlsirm. An approved offline worker is the later execution boundary. Before execution, the Orgmetra host must re-resolve every reference inside the tenant, verify exact study/Job membership and evidence provenance, and prove requester/reviewer identities are distinct authoritative actors.

## Consequences

### Positive

- Statistical work cannot silently drift to an unreviewed fast-mlsirm revision.
- A buyer or auditor can identify exactly which governed study evidence a result was supposed to analyze without copying raw worker data into the handoff.
- Aggregate missingness evidence cannot claim all observations are complete while simultaneously reporting predictor- or criterion-missing observations.
- Result canonicalization cannot be extended by an unreviewed subclass to serialize extra decision-like or person-level fields.
- Caller-controlled timestamp and numeric runtime behavior cannot rewrite an accepted canonical digest after construction or turn malformed oversized worker output into an uncaught exception type.
- Human interpretation remains explicit and separate from numerical output.
- The dedicated-writer boundary remains intact: Orgmetra consumes only a pinned foreign revision/contract boundary and never mutates fast-mlsirm.

### Limitations

- This slice does not execute a statistical model, estimate validity, correct for measurement error/range restriction, evaluate adverse impact, or assert legal compliance.
- Sampling design, estimator choice, missing-data treatment, reliability evidence, multiplicity, transportability, fairness analysis, and model diagnostics must be encoded in the referenced analysis plan and reviewed before execution.
- The generic result envelope does not invent an estimator-specific minimum complete-case count. Whether a converged estimator is identified under a particular missing-data design belongs to the reviewed analysis plan and numerical-worker contract; Orgmetra fails closed on impossible aggregate counts without silently replacing that foreign scientific contract with complete-case analysis.
- The package validates the result envelope, but a future execution adapter must still re-resolve the handoff references, verify the result provenance artifact, and attach evidence only after accountable human review.

## Verification

The package regression suite starts RED when the public handoff contract is absent and covers canonical operational tenant UUIDs, opaque UUIDv4 references, exact evidence digests, distinct human actors, exact dependency pinning, timezone-aware event time, immutable governance constants, value minimization, deterministic canonicalization, SHA-256 correlation, impossible aggregate missingness rejection, oversized numeric conversion rejection, exact governed aggregate-evidence runtime types, and 100% owned production statement/branch coverage. The repository-wide ADR numbering regression also fails closed if integration reuses an existing decision number, and the validity quality workflow contract requires any `docs/adr/**` change to execute that regression.

## References

See `docs/doctoring/validation-analysis-handoff-references.md`.
23 changes: 23 additions & 0 deletions docs/doctoring/validation-analysis-handoff-references.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Validation-analysis handoff references

Material decisions for ADR 0027 were checked against the following primary/authoritative sources on 2026-08-21. Regulatory currency was rechecked on 2026-08-29; fixed publication identifiers are retained so an auditor can reproduce the cited text even when agency web pages change.

## APA 7 references

Equal Employment Opportunity Commission, Civil Service Commission, Department of Justice, & Department of Labor. (1978). *Uniform Guidelines on Employee Selection Procedures (1978)*, 43 Fed. Reg. 38,290 (August 25, 1978) (codified at 29 C.F.R. pt. 1607). The EEOC continues to list 29 C.F.R. pt. 1607 among its Title VII regulations: https://www.eeoc.gov/regulations-and-guidelines

Society for Industrial and Organizational Psychology. (2018). Principles for the validation and use of personnel selection procedures. *Industrial and Organizational Psychology, 11*(S1), 1–97. https://doi.org/10.1017/iop.2018.195

ContextualWisdomLab. (2026). *fast-mlsirm* (Commit 04d0bc21a2a20693bcf16108cd76d394fe844d23) [Computer software]. GitHub. https://github.com/ContextualWisdomLab/fast-mlsirm/tree/04d0bc21a2a20693bcf16108cd76d394fe844d23

Tabassi, E. (2023). *Artificial intelligence risk management framework (AI RMF 1.0)* (NIST AI 100-1). National Institute of Standards and Technology. https://doi.org/10.6028/NIST.AI.100-1

Office of Personnel Management. (2026). *Removal of references to the Uniform Guidelines on Employee Selection Procedures in federal personnel regulations*, 91 Fed. Reg. 48,234 (July 31, 2026) (interim final rule, RIN 3206-AP20).

## Decision notes

- 43 Fed. Reg. 38,290 and the still-listed EEOC 29 C.F.R. pt. 1607 source support keeping criterion-related validity evidence tied to an explicit study design, job relevance, accuracy, reporting, and documentation rather than treating a bare coefficient as sufficient evidence. The fixed Federal Register identifier, not a mutable `/current/` eCFR URL, is the reproducible source for the 1978 text cited by this ADR.
- The July 31, 2026 OPM interim final rule removed UGESP references from specified federal civil-service regulations. Orgmetra therefore does not present UGESP as an undifferentiated government-wide mandate; applicability must be evaluated for the employer, jurisdiction, decision, and governing law at use time.
- The SIOP Principles are the professional validation baseline used for the handoff's evidence-and-human-review posture. The journal citation above fixes volume 11, Supplement S1, pages 1–97, and DOI 10.1017/iop.2018.195.
- The fast-mlsirm commit is recorded as a read-only dependency coordinate only. This Orgmetra slice does not modify or duplicate its numerical implementation.
- NIST AI RMF's govern, map, measure, and manage functions support preserving backend, precision, provenance, convergence, and human-review fields as inspectable result evidence rather than treating a model response as an autonomous decision.
26 changes: 26 additions & 0 deletions docs/traceability/validation-analysis-handoff.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Selection-validity analysis handoff traceability

## Buyer question

Can an organization send one exact, reviewable validation study to its statistical engine without copying raw person-level values into a workflow envelope, silently changing the numerical dependency, or treating model output as an employment decision?

## Active-PR contract

| Concern | Orgmetra evidence | Verification |
|---|---|---|
| Exact study scope | tenant, validation-study, Job, predictor, criterion, population, decision-policy, and analysis-plan references plus digests | namespace/UUID/digest regressions |
| Dependency integrity | immutable fast-mlsirm commit `04d0bc21a2a20693bcf16108cd76d394fe844d23` | malformed and unreviewed revision rejection |
| Privacy minimization | no raw person-level values in canonical handoff or result; result canonicalization accepts only exact governed missingness/convergence runtime types | canonical-payload/redacted-repr regressions plus subclass-injection rejection |
| Human authority | requester/reviewer references must differ, and the host must re-resolve both within the tenant and prove they resolve to distinct authoritative actor identities before execution | direct-construction fail-closed regression plus `test_next_action_requires_resolved_actor_identity_separation` |
| Scientific evidence | effect estimate, uncertainty interval, sample size, internally possible aggregate missingness, convergence diagnostics | immutable required-result-evidence regression plus impossible-missingness rejection |
| Numerical result boundary | handoff digest, pinned fast-mlsirm revision, Rust CPU/GPU backend, precision, finite estimate/interval, aggregate missingness, explicit convergence state | `ValidationAnalysisResult` invariant/canonicalization regressions, exact-runtime-type checks, and oversized-numeric `ValueError` normalization |
| Execution boundary | `not_executed`, `scientific_evidence_only`, read-only pinned foreign dependency | immutable governance regressions |
| Reproducibility | construction-time UTC timestamp snapshots, finite numeric snapshots, canonical RFC 3339 time, canonical JSON, SHA-256 handoff digest | mutable timezone/numeric and UTC-boundary regressions plus deterministic serialization/digest tests |
| Decision-record integrity | ADR numbers remain unique repository-wide and any `docs/adr/**` change reaches the validity quality gate | ADR uniqueness regression plus workflow-trigger contract regression |
| Quality-evidence freshness | package quality reruns whenever shared repository Python/test/clean-checkout configuration can alter execution or tracked-tree cleanliness | `test_quality_workflow_retriggers_on_shared_test_configuration` plus `.github/workflows/validity-analysis-quality.yml`; this supplemental package gate does not replace central required workflows |

## Maturity

`implemented_on_active_pr`.

Protected `develop` does **not** gain numerical validity computation from this slice. The handoff is execution preparation only. The active package now validates the minimum returned numerical/provenance envelope, including missingness consistency and exact aggregate-evidence runtime types, but protected Orgmetra evidence still requires host re-resolution, result-artifact verification, terminal checks, independent review, and accountable human interpretation.
Loading
Loading