Skip to content

build(validity): make execution lane reproducible - #58

Draft
seonghobae wants to merge 8 commits into
feat/validation-analysis-handofffrom
codex/validity-rust-execution
Draft

build(validity): make execution lane reproducible#58
seonghobae wants to merge 8 commits into
feat/validation-analysis-handofffrom
codex/validity-rust-execution

Conversation

@seonghobae

@seonghobae seonghobae commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Make the validity-analysis quality workflow use package-local pytest source configuration rather than a manually supplied PYTHONPATH.
  • Add the canonical uv run verification command and checked uv.lock.
  • Fail closed when directly constructed Rust recovery evidence contradicts its structural design.

This remains a stacked preparation PR for #57. It does not claim statistical execution or validity estimation; the pinned fast-mlsirm handoff/result boundary remains human-reviewed scientific evidence only.

Latest integrity repair

Fresh current-source review verified Devin finding PRRT_kwDOT5D6WM6dZ9ON: direct RustRecoveryEvidence construction accepted cross-sectional receipts carrying cluster metadata, nested receipts with missing/one cluster, and nested receipts with more clusters than people.

A regression-only commit landed first at 14f0f08109a134518287024d14b49bb611d7e7fd. It adds direct-construction tests requiring cross-sectional cluster_count=None, nested integer cluster_count >= 2, and cluster_count <= sample_size. Because this PR targets a stale feature-branch parent, no hosted workflow materialized on that RED head; absent execution is not represented as terminal RED or GREEN evidence.

Root repair 852b088ee1fdde511ecbfdda6abcbe0dd0b7802e changes only the owning RustRecoveryEvidence.__post_init__ boundary so contradictory scientific receipts fail closed before canonical evidence can be issued. The addressed bug thread is resolved. Informational review notes remain separate and are not treated as defects.

Fresh stack truth

The child remains based on a predecessor snapshot of #57 and therefore does not include the current parent tree. Parent checks/reviews do not transfer. The exact-current-head pull-request workflow inventory is empty because this stacked base does not materialize the applicable hosted lane; absence is non-passing and no predecessor check is substituted.

Parent #57 remains unintegrated. Its own current gates and reviews must be evaluated independently before this child may adopt protected truth.

Stack discipline

Process #57 dependency-first. The older non-Draft wording is historical metadata and is superseded by the live Draft state. Do not merge this descendant based on mechanical mergeability or predecessor evidence, and do not restack or create no-op churn solely to manufacture descendant evidence while #57 remains unintegrated. After #57 actually integrates, retarget/reconcile this existing child branch against the then-current protected develop, refetch the resulting exact head/base/conflict state, rerun the full validity-analysis suite with exact owned statement/branch coverage plus every applicable global/central gate, and obtain qualifying independent non-author review without transferring predecessor or parent evidence.

Do not self-approve, use routine administrator bypass, weaken a gate, mutate fast-mlsirm or another dedicated-writer repository, or treat absent/cancelled/stale/predecessor evidence as passing.

@coderabbitai

coderabbitai Bot commented Aug 21, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 937fa0af-5246-40b0-9dba-8a19dff71d52

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@seonghobae

Copy link
Copy Markdown
Contributor Author

Validation update for head 3feaf271784ca10b263cafad67a1f7bd971f2440:

  • validity-analysis: 139 passed, 100% owned statement/branch coverage.
  • repository validator, root Node validation (55 tests), uv lock check, compileall, diff check, and CodeGraph status passed.
  • real bounded smoke: clean sparse checkout at fast-mlsirm 04d0bc21a2a20693bcf16108cd76d394fe844d23; Rust CPU, nested multilevel cluster_id, 48 synthetic persons x 3 items, RAYON_NUM_THREADS=4, 180s timeout, max_iter_reached after 1 bounded iteration; aggregate receipt emitted and checkout remained clean.
  • cross-sectional smoke also emitted an aggregate max_iter_reached receipt; multiple-membership and longitudinal contracts fail closed without worker invocation.
  • This is recovery-path evidence only, not criterion-related validity, GPU/CPU parity, estimator acceptance, deployment proof, or protected-branch truth.

PR remains draft and blocked; no independent hosted review is present and required hosted checks are queued. No merge performed.

@seonghobae

Copy link
Copy Markdown
Contributor Author

Current exact PR head is now c79a6ed49627e6a47947f171aeeed2bf02a8c152; this final commit only updates the execution-plan handoff after the 3feaf27 code/docs head.

Fresh verification at c79a6ed: validity-analysis 139 passed with 100% owned statement/branch coverage; repository validator, root Node 55-test validation, uv lock check, and CodeGraph all pass. PR #58 remains draft/blocked with no hosted review and queued required checks; no merge or protection bypass performed.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

Devin Review found 4 new potential issues.

Devin Review

Comment thread packages/validity-analysis/src/orgmetra_validity_analysis/execution.py Outdated
Comment on lines +251 to +264
try:
output = run_worker(
repository=repository,
design_code=args.design_code,
rust_device=args.rust_device,
persons=args.persons,
items_per_dim=args.items_per_dim,
clusters=args.clusters,
seed=args.seed,
worker_count=args.worker_count,
timeout_seconds=args.timeout_seconds,
)
finally:
resolve_revision(repository)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Info: Post-run checkout verification is unconditional

main repeats exact-revision and cleanliness checks after worker success or failure. Temporary build paths keep expected artifacts outside the foreign checkout.

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +245 to +252
occasion_count=2 if args.design_code == "longitudinal" else 1,
maximum_memberships=2 if args.design_code == "multiple_membership" else 1,
worker_count=args.worker_count,
rust_device=args.rust_device,
)
request.require_runnable()
try:
output = run_worker(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Info: Contract-only designs cannot reach worker

require_runnable rejects multiple-membership and longitudinal requests before run_worker. Their CLI exposure does not silently execute the current estimator.

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +361 to +366
def build_rust_recovery_evidence(
request: RustExecutionRequest,
worker_output: Mapping[str, object],
*,
completed_at: datetime,
) -> RustRecoveryEvidence:

@devin-ai-integration devin-ai-integration Bot Aug 29, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Info: Recovery metrics cannot populate validity results

build_rust_recovery_evidence returns only RustRecoveryEvidence. No path maps recovery errors into criterion-related effect estimates.

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 new potential issue.

Devin Review

Comment on lines +287 to +293
if self.design_code == "nested_multilevel":
if self.cluster_count is None or type(self.cluster_count) is not int or self.cluster_count < 2:
raise ValueError("nested_multilevel recovery evidence requires cluster_count >= 2")
if self.cluster_count > self.sample_size:
raise ValueError("nested_multilevel recovery evidence cluster_count cannot exceed sample_size")
elif self.cluster_count is not None:
raise ValueError("cross_sectional recovery evidence cannot carry cluster_count")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Info: Cluster invariants match worker output

Nested runs cannot produce more clusters than people, while cross-sectional runs report none. Valid builder output remains accepted under both designs.

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

@seonghobae
seonghobae marked this pull request as draft September 5, 2026 19:13

Copy link
Copy Markdown
Contributor Author

Lifecycle authority correction: #58 is now Draft because it remains based on predecessor #57 snapshot 081942a... rather than current parent 6ca5547..., and its exact current head has no authoritative PR-triggered hosted acceptance. The body’s earlier non-draft lifecycle line is historical. Process #57 first; after protected integration, non-force retarget/reconcile #58 to fresh develop, preserve the valid Rust recovery/reproducibility delta, and reacquire full exact-head scientific/coverage/security/review evidence. No parent/predecessor evidence transfers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant