-
Notifications
You must be signed in to change notification settings - Fork 0
feat(validity): add governed selection-validity analysis handoff #57
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
seonghobae
wants to merge
43
commits into
develop
Choose a base branch
from
feat/validation-analysis-handoff
base: develop
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
43 commits
Select commit
Hold shift + click to select a range
c28fe1b
test(validity): require governed analysis handoff
seonghobae 5b0a1e5
feat(validity): add governed fast-mlsirm handoff
seonghobae 3f99928
docs(validity): record governed analysis boundary
seonghobae 30ac02a
chore(validity): integrate protected develop after #41
seonghobae a9168a9
test(validity): reject duplicate ADR numbers after integration
seonghobae 6814fd9
fix(validity): reserve ADR 0027 for analysis handoff
seonghobae a4876a8
fix(validity): track reserved ADR 0027 in quality gate
seonghobae 7e9cd76
docs(validity): bind references to ADR 0027
seonghobae a872938
fix(validity): remove conflicting ADR 0025 allocation
seonghobae 22fd374
test(validity): require resolved reviewer identity separation
seonghobae 8432d1c
fix(validity): require authoritative actor identity separation
seonghobae 68a15c4
docs(validity): trace authoritative actor separation
seonghobae f0c30c5
docs(validity): record identity-separation repair
seonghobae c381518
chore(validity): integrate protected develop after #43
seonghobae d0c06d2
feat(validity): validate pinned numerical result envelopes
seonghobae 0a31737
test(validity): reject impossible complete missingness totals
seonghobae 2419aa9
fix(validity): reject impossible missingness summaries
seonghobae c17936c
test(validity): reject subclassed result evidence
seonghobae f31422a
fix(validity): require exact result evidence types
seonghobae e11ae2a
docs(validity): record result evidence hardening
seonghobae e9a7a31
docs(validity): define hardened result envelope
seonghobae 8b26520
docs(validity): trace result integrity regressions
seonghobae b33cbda
docs(validity): bind result integrity in ADR
seonghobae 081942a
test(validity): cover criterion overlap guard
seonghobae d6bb5f2
test(validity): reject temporal evidence subclasses
seonghobae b616a13
fix(validity): require exact temporal evidence type
seonghobae 219c2e3
fix(validity): remove unused kernel revision import
seonghobae 89c9707
test(validity): require completed_at diagnostics
seonghobae b72c7a0
fix(validity): report field-correct timestamp errors
seonghobae 5197338
fix(validity): bind result timestamp diagnostic
seonghobae e7c3404
test(validity): align temporal diagnostic regression
seonghobae a8da84c
fix(validity): freeze handoff evidence boundaries
seonghobae 5081844
test(validity): reject overflowing worker numerics
seonghobae a55f95c
test(validity): require ADR-wide quality trigger
seonghobae 06db366
fix(validity): normalize numeric conversion failures
seonghobae be52fc7
fix(validity): run ADR uniqueness check for all ADR changes
seonghobae 376c2ca
docs(validity): pin reproducible selection-validation references
seonghobae 2ff2582
docs(validity): trace repaired evidence boundaries
seonghobae 4359cfb
docs(validity): record numeric and missingness ownership boundaries
seonghobae 2f6d1dc
test(validity): require shared config quality triggers
seonghobae 8815f73
fix(validity): retrigger quality on shared config
seonghobae 3c2a1de
docs(validity): record shared-config gate integrity
seonghobae 6ca5547
docs(validity): trace shared-config quality evidence
seonghobae File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,67 @@ | ||
| name: Validity Analysis Handoff Quality | ||
|
|
||
| on: | ||
| pull_request: | ||
| branches: | ||
| - bootstrap | ||
| - develop | ||
| - main | ||
| paths: | ||
| - "packages/validity-analysis/**" | ||
| - "docs/adr/**" | ||
| - "docs/doctoring/validation-analysis-handoff-references.md" | ||
| - "docs/traceability/validation-analysis-handoff.md" | ||
| - ".github/requirements/foundation-test.txt" | ||
| - ".github/workflows/validity-analysis-quality.yml" | ||
| - ".gitignore" | ||
| - ".python-version" | ||
| - "conftest.py" | ||
| - "packages/conftest.py" | ||
| - "pyproject.toml" | ||
| - "pytest.ini" | ||
| - "setup.cfg" | ||
| - "tox.ini" | ||
| workflow_dispatch: | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| concurrency: | ||
| group: validity-analysis-quality-${{ github.event.pull_request.number || github.ref }} | ||
| cancel-in-progress: true | ||
|
|
||
| jobs: | ||
| unit: | ||
| name: Validity handoff and 100% coverage | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 10 | ||
| steps: | ||
| - name: Checkout exact candidate | ||
| uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | ||
| with: | ||
| ref: ${{ github.event.pull_request.head.sha || github.sha }} | ||
| persist-credentials: false | ||
| - name: Prove exact candidate checkout | ||
| env: | ||
| ORGMETRA_EXPECTED_HEAD_SHA: ${{ github.event.pull_request.head.sha || github.sha }} | ||
| run: test "$(git rev-parse HEAD)" = "$ORGMETRA_EXPECTED_HEAD_SHA" | ||
| - name: Set up Python | ||
| uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | ||
| with: | ||
| python-version: "3.14" | ||
| check-latest: false | ||
| - name: Install reviewed test toolchain | ||
| run: | | ||
| python -m pip install --require-hashes --no-deps --only-binary=:all: -r .github/requirements/foundation-test.txt | ||
| python -m pip check | ||
| - name: Compile validity-analysis handoff | ||
| run: python -m compileall -q packages/validity-analysis/src packages/validity-analysis/tests | ||
| - name: Test governed handoff with exact statement and branch coverage | ||
| env: | ||
| PYTHONPATH: packages/validity-analysis/src | ||
| COVERAGE_FILE: /tmp/orgmetra-validity-analysis.coverage | ||
| run: python -m pytest -c packages/validity-analysis/pyproject.toml packages/validity-analysis/tests | ||
| - name: Require clean checkout | ||
| run: | | ||
| git diff --exit-code | ||
| test -z "$(git status --porcelain)" | ||
|
seonghobae marked this conversation as resolved.
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
60 changes: 60 additions & 0 deletions
60
docs/adr/0027-governed-selection-validity-analysis-handoff.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,60 @@ | ||
| # ADR 0027: Govern selection-validity numerical work through an immutable handoff | ||
|
|
||
| - Status: Proposed | ||
| - Maturity: Active PR only; not protected-branch truth | ||
| - Date: 2026-08-21 | ||
| - Owners: Orgmetra Workforce Validation | ||
|
|
||
| ## Context | ||
|
|
||
| Protected Orgmetra already preserves exact validation-study cases, sealed selection evidence, candidate-to-worker lineage, and Job/cycle/staffing-scoped criterion observations. The remaining boundary is dangerous if left implicit: a statistical worker could receive an underspecified study, silently use a different dependency revision, or turn a model result into an employment decision. | ||
|
|
||
| The Uniform Guidelines recognize criterion-related validity evidence as empirical evidence relating a selection procedure to important job-performance elements and require validity studies to be accurate, standardized, documented, and periodically reviewed for currency. SIOP's *Principles for the Validation and Use of Personnel Selection Procedures* likewise treats validation as an evidence-and-inference problem rather than a correlation-only shortcut. | ||
|
|
||
| `ContextualWisdomLab/fast-mlsirm` owns numerical psychometric/statistical kernels. Its protected `main` was freshly resolved to commit `04d0bc21a2a20693bcf16108cd76d394fe844d23` on 2026-08-21. Orgmetra must not copy that implementation or write the foreign repository. | ||
|
|
||
| ## Decision | ||
|
|
||
| Orgmetra adds a leaf `orgmetra_validity_analysis` package whose `ValidationAnalysisHandoff`: | ||
|
|
||
| - binds the exact tenant, validation study, Job, predictor snapshot, criterion snapshot, population snapshot, decision policy, and analysis plan through opaque references plus SHA-256 evidence digests; | ||
| - binds distinct requester and reviewer actor references; | ||
| - pins fast-mlsirm to reviewed immutable commit `04d0bc21a2a20693bcf16108cd76d394fe844d23`; | ||
| - declares the numerical boundary `read_only_pinned_revision` and the initial strategy `criterion_related`; | ||
| - requires downstream result evidence for effect estimate, uncertainty interval, sample size, missingness summary, and convergence diagnostics; | ||
| - serializes no raw person-level predictor, criterion, candidate, or worker values; | ||
| - remains `not_executed`, `scientific_evidence_only`, and human-review-required; | ||
| - produces deterministic canonical JSON and a SHA-256 digest for audit/result correlation. | ||
|
|
||
| Both handoff and result envelopes detach exact timezone-aware timestamps to one built-in UTC instant at construction. Result numeric evidence is converted to finite built-in floats before storage, and conversion failures including numeric overflow are normalized to the package's fail-closed `ValueError` contract, so caller-controlled timezone or numeric runtime behavior cannot rewrite canonical evidence after validation or escape normal malformed-result handling. | ||
|
|
||
| The same package also validates `ValidationAnalysisResult` envelopes returned by the approved offline worker. A result must link to the handoff digest and the same pinned revision, identify a Rust CPU or GPU backend and precision, provide finite effect and interval values, match its sample size to aggregate missingness counts, reject impossible complete-versus-missing count combinations, and include explicit convergence diagnostics. The canonicalization boundary accepts only the exact governed `MissingnessSummary` and `ConvergenceDiagnostics` runtime types so subclass method overrides cannot add unreviewed or person-level fields to immutable result evidence. A nonconverged result remains typed scientific evidence requiring human review; it cannot be treated as a valid selection procedure or an employment decision. | ||
|
|
||
| The package does not invoke fast-mlsirm. An approved offline worker is the later execution boundary. Before execution, the Orgmetra host must re-resolve every reference inside the tenant, verify exact study/Job membership and evidence provenance, and prove requester/reviewer identities are distinct authoritative actors. | ||
|
|
||
| ## Consequences | ||
|
|
||
| ### Positive | ||
|
|
||
| - Statistical work cannot silently drift to an unreviewed fast-mlsirm revision. | ||
| - A buyer or auditor can identify exactly which governed study evidence a result was supposed to analyze without copying raw worker data into the handoff. | ||
| - Aggregate missingness evidence cannot claim all observations are complete while simultaneously reporting predictor- or criterion-missing observations. | ||
| - Result canonicalization cannot be extended by an unreviewed subclass to serialize extra decision-like or person-level fields. | ||
| - Caller-controlled timestamp and numeric runtime behavior cannot rewrite an accepted canonical digest after construction or turn malformed oversized worker output into an uncaught exception type. | ||
| - Human interpretation remains explicit and separate from numerical output. | ||
| - The dedicated-writer boundary remains intact: Orgmetra consumes only a pinned foreign revision/contract boundary and never mutates fast-mlsirm. | ||
|
|
||
| ### Limitations | ||
|
|
||
| - This slice does not execute a statistical model, estimate validity, correct for measurement error/range restriction, evaluate adverse impact, or assert legal compliance. | ||
| - Sampling design, estimator choice, missing-data treatment, reliability evidence, multiplicity, transportability, fairness analysis, and model diagnostics must be encoded in the referenced analysis plan and reviewed before execution. | ||
| - The generic result envelope does not invent an estimator-specific minimum complete-case count. Whether a converged estimator is identified under a particular missing-data design belongs to the reviewed analysis plan and numerical-worker contract; Orgmetra fails closed on impossible aggregate counts without silently replacing that foreign scientific contract with complete-case analysis. | ||
| - The package validates the result envelope, but a future execution adapter must still re-resolve the handoff references, verify the result provenance artifact, and attach evidence only after accountable human review. | ||
|
|
||
| ## Verification | ||
|
|
||
| The package regression suite starts RED when the public handoff contract is absent and covers canonical operational tenant UUIDs, opaque UUIDv4 references, exact evidence digests, distinct human actors, exact dependency pinning, timezone-aware event time, immutable governance constants, value minimization, deterministic canonicalization, SHA-256 correlation, impossible aggregate missingness rejection, oversized numeric conversion rejection, exact governed aggregate-evidence runtime types, and 100% owned production statement/branch coverage. The repository-wide ADR numbering regression also fails closed if integration reuses an existing decision number, and the validity quality workflow contract requires any `docs/adr/**` change to execute that regression. | ||
|
|
||
| ## References | ||
|
|
||
| See `docs/doctoring/validation-analysis-handoff-references.md`. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| # Validation-analysis handoff references | ||
|
|
||
| Material decisions for ADR 0027 were checked against the following primary/authoritative sources on 2026-08-21. Regulatory currency was rechecked on 2026-08-29; fixed publication identifiers are retained so an auditor can reproduce the cited text even when agency web pages change. | ||
|
|
||
| ## APA 7 references | ||
|
|
||
| Equal Employment Opportunity Commission, Civil Service Commission, Department of Justice, & Department of Labor. (1978). *Uniform Guidelines on Employee Selection Procedures (1978)*, 43 Fed. Reg. 38,290 (August 25, 1978) (codified at 29 C.F.R. pt. 1607). The EEOC continues to list 29 C.F.R. pt. 1607 among its Title VII regulations: https://www.eeoc.gov/regulations-and-guidelines | ||
|
|
||
| Society for Industrial and Organizational Psychology. (2018). Principles for the validation and use of personnel selection procedures. *Industrial and Organizational Psychology, 11*(S1), 1–97. https://doi.org/10.1017/iop.2018.195 | ||
|
|
||
| ContextualWisdomLab. (2026). *fast-mlsirm* (Commit 04d0bc21a2a20693bcf16108cd76d394fe844d23) [Computer software]. GitHub. https://github.com/ContextualWisdomLab/fast-mlsirm/tree/04d0bc21a2a20693bcf16108cd76d394fe844d23 | ||
|
|
||
| Tabassi, E. (2023). *Artificial intelligence risk management framework (AI RMF 1.0)* (NIST AI 100-1). National Institute of Standards and Technology. https://doi.org/10.6028/NIST.AI.100-1 | ||
|
|
||
| Office of Personnel Management. (2026). *Removal of references to the Uniform Guidelines on Employee Selection Procedures in federal personnel regulations*, 91 Fed. Reg. 48,234 (July 31, 2026) (interim final rule, RIN 3206-AP20). | ||
|
|
||
| ## Decision notes | ||
|
|
||
| - 43 Fed. Reg. 38,290 and the still-listed EEOC 29 C.F.R. pt. 1607 source support keeping criterion-related validity evidence tied to an explicit study design, job relevance, accuracy, reporting, and documentation rather than treating a bare coefficient as sufficient evidence. The fixed Federal Register identifier, not a mutable `/current/` eCFR URL, is the reproducible source for the 1978 text cited by this ADR. | ||
| - The July 31, 2026 OPM interim final rule removed UGESP references from specified federal civil-service regulations. Orgmetra therefore does not present UGESP as an undifferentiated government-wide mandate; applicability must be evaluated for the employer, jurisdiction, decision, and governing law at use time. | ||
| - The SIOP Principles are the professional validation baseline used for the handoff's evidence-and-human-review posture. The journal citation above fixes volume 11, Supplement S1, pages 1–97, and DOI 10.1017/iop.2018.195. | ||
| - The fast-mlsirm commit is recorded as a read-only dependency coordinate only. This Orgmetra slice does not modify or duplicate its numerical implementation. | ||
| - NIST AI RMF's govern, map, measure, and manage functions support preserving backend, precision, provenance, convergence, and human-review fields as inspectable result evidence rather than treating a model response as an autonomous decision. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| # Selection-validity analysis handoff traceability | ||
|
|
||
| ## Buyer question | ||
|
|
||
| Can an organization send one exact, reviewable validation study to its statistical engine without copying raw person-level values into a workflow envelope, silently changing the numerical dependency, or treating model output as an employment decision? | ||
|
|
||
| ## Active-PR contract | ||
|
|
||
| | Concern | Orgmetra evidence | Verification | | ||
| |---|---|---| | ||
| | Exact study scope | tenant, validation-study, Job, predictor, criterion, population, decision-policy, and analysis-plan references plus digests | namespace/UUID/digest regressions | | ||
| | Dependency integrity | immutable fast-mlsirm commit `04d0bc21a2a20693bcf16108cd76d394fe844d23` | malformed and unreviewed revision rejection | | ||
| | Privacy minimization | no raw person-level values in canonical handoff or result; result canonicalization accepts only exact governed missingness/convergence runtime types | canonical-payload/redacted-repr regressions plus subclass-injection rejection | | ||
| | Human authority | requester/reviewer references must differ, and the host must re-resolve both within the tenant and prove they resolve to distinct authoritative actor identities before execution | direct-construction fail-closed regression plus `test_next_action_requires_resolved_actor_identity_separation` | | ||
| | Scientific evidence | effect estimate, uncertainty interval, sample size, internally possible aggregate missingness, convergence diagnostics | immutable required-result-evidence regression plus impossible-missingness rejection | | ||
| | Numerical result boundary | handoff digest, pinned fast-mlsirm revision, Rust CPU/GPU backend, precision, finite estimate/interval, aggregate missingness, explicit convergence state | `ValidationAnalysisResult` invariant/canonicalization regressions, exact-runtime-type checks, and oversized-numeric `ValueError` normalization | | ||
| | Execution boundary | `not_executed`, `scientific_evidence_only`, read-only pinned foreign dependency | immutable governance regressions | | ||
| | Reproducibility | construction-time UTC timestamp snapshots, finite numeric snapshots, canonical RFC 3339 time, canonical JSON, SHA-256 handoff digest | mutable timezone/numeric and UTC-boundary regressions plus deterministic serialization/digest tests | | ||
| | Decision-record integrity | ADR numbers remain unique repository-wide and any `docs/adr/**` change reaches the validity quality gate | ADR uniqueness regression plus workflow-trigger contract regression | | ||
| | Quality-evidence freshness | package quality reruns whenever shared repository Python/test/clean-checkout configuration can alter execution or tracked-tree cleanliness | `test_quality_workflow_retriggers_on_shared_test_configuration` plus `.github/workflows/validity-analysis-quality.yml`; this supplemental package gate does not replace central required workflows | | ||
|
|
||
| ## Maturity | ||
|
|
||
| `implemented_on_active_pr`. | ||
|
|
||
| Protected `develop` does **not** gain numerical validity computation from this slice. The handoff is execution preparation only. The active package now validates the minimum returned numerical/provenance envelope, including missingness consistency and exact aggregate-evidence runtime types, but protected Orgmetra evidence still requires host re-resolution, result-artifact verification, terminal checks, independent review, and accountable human interpretation. |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.