Skip to content

Qualify schema 2 red-reproduction evidence in the trusted judge - #79

Draft
test1card wants to merge 6 commits into
masterfrom
codex/red-reproduction-v2-judge
Draft

test1card wants to merge 6 commits into
masterfrom
codex/red-reproduction-v2-judge

Conversation

@test1card

Copy link
Copy Markdown
Owner

Purpose

Protected CI loads the red-reproduction judge from the trusted base. The current judge accepts only exact schema 1 receipts.

Pull request #65 creates schema 2 receipts. It cannot safely change the judge and use that changed judge in one candidate.

This prerequisite changes the trusted judge first. It does not add a schema 2 receipt binding.

Decision

  • Add an append-only trusted expectation manifest.
  • Let a later candidate use only expectations that already exist in its trusted base.
  • Bind each schema 2 receipt to the trusted-base commit and manifest blob.
  • Reject an expectation and its first receipt when one candidate adds both.
  • Keep each current schema 1 receipt valid only at its exact locator and digest.
  • Store canonical environment placeholders instead of local paths and user data.
  • Accept the variable indentation in normal pytest assertion diagnostics.

Rejected alternatives

  1. Change the judge only in pull request fix: the Windows ONEDIR gate waits for the process, not for inherited pipe EOF #65. Protected CI uses the immutable judge from master, so that change cannot qualify its own receipt.
  2. Let the receipt author supply the expected failure. The author could then label an unrelated failure as expected.
  3. Add a schema 2 receipt in this prerequisite. The current trusted judge would reject it before pytest collection.

Evidence

The candidate is aa9ccce.

  • The red-reproduction test modules passed: 39 tests.
  • Registry, guard collection, and baseline tests passed: 9 tests.
  • Protected and sealed runner integration tests passed: 3 tests.
  • The complete docs collection passed in two exact subsets: 34 tests and 33 tests.
  • The two generated-pair freshness nodes passed together.
  • Ruff lint passed for all five changed Python files.
  • Ruff format check passed for all five changed Python files.
  • UTF-8 decoding and AST parsing passed for all five changed Python files.
  • The registry still binds six schema 1 receipts and zero schema 2 receipts.

The red-before control used the same new guard on commit 84fdf96. The guard failed because the old comparator did not raise. The candidate passes that guard.

Hosted CI is pending. This pull request stays in draft until Codex returns a clean exact-head verdict and no live findings remain.

Follow-up

After this prerequisite merges, refresh pull request #65 onto the new master. Regenerate its schema 2 receipts against that trusted base. Then repeat its reviews and merge gates.

@test1card

Copy link
Copy Markdown
Owner Author

@codex

Please review this draft at exact commit aa9ccce. Check the trusted-base boundary, same-candidate expectation laundering, backward compatibility, and receipt privacy.

@test1card

Copy link
Copy Markdown
Owner Author

Coordinator decision record for exact head aa9ccce

Decision: split the trust migration into this prerequisite and the later pull request #65 refresh.

Facts:

Rejected alternatives:

  1. Keep the judge change in fix: the Windows ONEDIR gate waits for the process, not for inherited pipe EOF #65. The protected workflow ignores the candidate judge, so the receipt fails before pytest collection.
  2. Let a receipt state its own expected failure. A candidate can then label an unrelated failure as expected.
  3. Add schema 2 receipt bindings here. The current trusted judge rejects those bindings.

This prerequisite adds trusted expectations and schema 2 validation only. The live registry still binds six schema 1 receipts and zero schema 2 receipts.

The old comparator at 84fdf96 fails the new self-authorization guard with DID NOT RAISE. This head passes the same guard.

The complete local evidence is in the pull request body. GitHub Codex and one cold built-in reviewer provide the two independent reviews. Pull request #65 will consume the new trusted expectations only after this prerequisite reaches master.

@test1card

Copy link
Copy Markdown
Owner Author

@codex review

Exact head: aa9ccce

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: aa9ccce744

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tools/red_reproduction.py Outdated
Comment thread tools/red_reproduction.py Outdated
Comment thread tools/ci_active_checkout_runner.py Outdated
@test1card

Copy link
Copy Markdown
Owner Author

@codex review

Review target: c03f1f9003b786b288954f4c9cd3c3c8055c195d.

This correction addresses all current findings at the prior head.

Decisions and mechanisms:

  • The producer loads the capture plugin by verified file path. A checkout-local module with the same name cannot shadow it.
  • The trusted manifest now binds both the expected failure message and the exact guard blob. The failed call must originate in that guard file.
  • Schema version 2 no longer stores raw stdout, stderr, or pytest long representations. It stores only the canonical failure signature and structured call location.
  • The expectation manifest must remain a regular non-executable Git blob. Executable-bit and symlink changes fail closed.
  • The candidate comparison checks the receipt guard blobs against the trusted-base manifest.

Rejected alternatives:

  • Base64 was rejected because it is reversible encoding, not redaction.
  • PYTHONPATH ordering was rejected because the defective checkout still precedes external plugin paths during normal import resolution.
  • Message-only binding was rejected because unrelated code can emit the same text.
  • Content-only manifest comparison was rejected because a symlink can retain the same blob id while changing resolution semantics.

Verification on this exact candidate:

  • Red-before: the prior implementation failed the raw-output privacy test, imported the colliding checkout plugin, and accepted a manifest type change.
  • 41 focused red-reproduction tests passed after correction.
  • 104 governance, registry, baseline, and active-guard tests passed with basetemp outside the repository. An earlier run with basetemp inside the repository was discarded because strict discovery treated temporary test files as candidate files.
  • The focused 41-test set passed again after manual formatting.
  • Both derived-pair freshness tests passed.
  • Ruff lint passed. Ruff format check reports all five changed Python files formatted.
  • The staged baseline blob contains LF bytes and no carriage returns.
  • CryoDAQ was not started.

One independent built-in cold breadth re-review is running on this exact commit.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c03f1f9003

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tools/red_reproduction.py
launcher_path.write_text(_PYTEST_LAUNCHER, encoding="utf-8", newline="\n")
report_path = plugin_dir / "pytest-report.json"
environment = _test_environment(worktree, python)
environment["CRYODAQ_RED_REPRODUCTION_REPORT"] = str(report_path)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Protect the capture report from candidate hooks

When the defective commit contains a conftest.py session-finish hook, that candidate code can read CRYODAQ_RED_REPRODUCTION_REPORT and overwrite the JSON after the trusted plugin writes it (for example, with a tryfirst hookwrapper whose teardown runs last). I reproduced this against the exact producer: an unrelated failure originating in helper.py was accepted after the hook forged the expected guard path and message. The report therefore needs an authenticated/protected channel that code under test cannot write, rather than a candidate-visible filesystem path.

AGENTS.md reference: AGENTS.md:L380-L386

Useful? React with 👍 / 👎.

Comment thread tools/red_reproduction.py
import sys
from pathlib import Path

import pytest

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Load pytest outside the defective checkout namespace

When the defective tree contains src/pytest.py (or an equivalent package), this launcher imports that checkout-controlled module because _test_environment places <worktree>/src on PYTHONPATH before the interpreter starts. Its fake main() can invoke or bypass the trusted plugin, emit the accepted failure summary/report, and return 1 without running real pytest. Fresh evidence after the file-bound plugin fix is that the launcher still resolves pytest through the defective checkout, so load and verify the real pytest package before exposing candidate import paths.

AGENTS.md reference: AGENTS.md:L380-L386

Useful? React with 👍 / 👎.

Comment on lines +664 to +666
_mode, kind, object_id = _tree_entry(root, candidate, path)
if kind != "blob":
raise RedReproductionComparisonError(f"candidate-added red-reproduction evidence is not a blob: {identity}")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Require a safe mode for newly added receipts

When a candidate adds a schema-v2 receipt with mode 100755, this branch discards _mode and accepts the executable JSON blob; the full registry validator also reads only its bytes, so both Linux and Windows checks can remain green. Once that candidate becomes the trusted base, the inherited-binding logic preserves this unsafe mode permanently and refuses later correction. Require candidate-added receipts to start as regular non-executable 100644 files, just as the manifest does.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant