Add reusable campaign admission contract - #457
Conversation
|
@codex review Fresh independent review requested for head
No providers, deployment, private data, or merge were used. Full repository checks pass locally. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3f2d2da5c7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const generator = resolve(options.smokeGenerator); | ||
| if (generator !== join(project, "generate_smoke.py")) throw new Error("smoke generator must be generate_smoke.py inside the locked project"); |
There was a problem hiding this comment.
When campaigns admit is run against a caller-supplied project, this verifies only that the script has the expected filename; neither the manifest nor another trusted artifact binds the script's contents. A modified generate_smoke.py can therefore skip ToolEnv execution, emit self-consistent trace/state/receipt files, and pass the subsequent byte comparison because the CLI reruns that same untrusted script. Hash or otherwise attest the generator before treating its output as proof of an actual pinned-Verifiers mutation.
Useful? React with 👍 / 👎.
| if (!pins.some((pin) => pin.name === "verifiers" && pin.version === "0.2.1" && typeof pin.git_revision === "string" && GIT_REVISION.test(pin.git_revision))) { | ||
| errors.push("environment.resolved_packages must pin verifiers 0.2.1 to a full git revision"); | ||
| } |
There was a problem hiding this comment.
For any non-Cedar workload whose lock, workload contract, receipt, and trace consistently attest a different exact Verifiers version, this unconditional environment check still rejects admission. That defeats the new generic version/commit matching at lines 291–295 and makes the reusable cross-workload contract effectively hard-coded to 0.2.1; apply this constraint only when the workload is identified as Cedar.
Useful? React with 👍 / 👎.
| const expectedCandidate = sha256Bytes(Buffer.from(canonicalize({ parent_candidate_sha256: lineage.parent_candidate_sha256, prompt_sha256: lineage.prompt_sha256, model_attestation_sha256: lineage.model_attestation_sha256, checkpoint_sha256: lineage.checkpoint_sha256, executable_bundle_sha256: endpoint.executable_bundle_sha256 }))); | ||
| if (lineage.candidate_sha256 !== expectedCandidate) errors.push("candidate mutation is not bound to parent, prompt, model, checkpoint, and executable bundle"); |
There was a problem hiding this comment.
If candidate_lineage.prompt_sha256 is changed to an unrelated hash and candidate_sha256 is recomputed from the fields here, admission still succeeds because the lineage prompt is never compared with workload_contract.benchmark_prompt_sha256. This permits a candidate generated for one prompt to be admitted against another workload while presenting internally consistent lineage, so the prompt lineage needs a cross-attestation to the admitted prompt.
Useful? React with 👍 / 👎.
| const installed = Array.isArray(receipt.installed_distributions) ? receipt.installed_distributions.map(object) : []; | ||
| const declared = Array.isArray(object(manifest.environment).resolved_packages) ? (object(manifest.environment).resolved_packages as unknown[]).map(object) : []; | ||
| const installedPairs = installed.map((pin) => `${pin.name}==${pin.version}`).sort(); | ||
| const declaredPairs = new Set(declared.map((pin) => `${pin.name}==${pin.version}`)); | ||
| if (installedPairs.some((pin) => !declaredPairs.has(pin))) errors.push("inside-project installed distributions contain a package outside the exact lock inventory"); |
There was a problem hiding this comment.
For direct callers of the exported validator, this one-way subset check accepts an inventory with arbitrary locked packages omitted: removing mcp (or even verifiers) from installed_distributions and updating the two inventory/receipt hashes leaves every remaining pair in declaredPairs, so admission can still succeed. Consequently the receipt does not attest the claimed exact installed environment; validate the complete expected environment inventory, including required workload distributions, rather than only rejecting extras.
Useful? React with 👍 / 👎.
| const delta = object(receipt.verified_state_delta); | ||
| if (delta.path !== "/records/alpha/status" || delta.before !== "pending" || delta.after !== "ready") errors.push("execution receipt does not verify the required state delta"); |
There was a problem hiding this comment.
When validating a receipt outside the fixed CLI fixture, verified_state_delta.independently_asserted can be false or absent and the receipt and manifest hashes can be updated without producing any admission error. The validator checks only the path and values, so evidence that explicitly says the delta was not independently asserted is still admitted; require this receipt field to be exactly true.
Useful? React with 👍 / 👎.
|
@codex review Fresh independent review requested for head
Adversarial tests cover campaign/workload substitution, arbitrary artifact hashes, omitted packages after rehash, missing Verifiers, runtime mismatch, and fabricated metrics. Both locked generators replay byte-for-byte and the full repository check passes locally. No providers, private data, deployment, merge, or holdout activity. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 923d2101dc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (declaredPins !== lockedPins) throw new Error("resolved package pins do not exactly match uv.lock"); | ||
| const generator = resolve(options.smokeGenerator); | ||
| if (generator !== join(project, "generate_smoke.py")) throw new Error("smoke generator must be generate_smoke.py inside the locked project"); | ||
| const comparisons = [["trace.json", options.trace], ["execution-receipt.json", options.executionReceipt], ["before-state.json", options.beforeState], ["after-state.json", options.afterState], ["overflow-receipt.json", options.overflowReceipt], ["campaign-evidence.json", options.campaignEvidence], ["applicable-lock.json", options.applicableLock]] as const; |
There was a problem hiding this comment.
When --project is caller-controlled, the fresh head adds campaign evidence and applicable-lock output to the byte comparisons, but both are still produced by the same unattested generate_smoke.py; the preceding check verifies only its path. Replacing that script can synthesize a fully self-consistent trace, receipts, inventory, and evidence bundle which then reproduces byte-for-byte, so bind the generator contents to a trusted digest outside the supplied project before treating this replay as execution proof.
Useful? React with 👍 / 👎.
| if (canonicalize(installed) !== canonicalize(applicable)) errors.push("installed distributions do not equal the exact applicable locked project inventory"); | ||
| if (canonicalize(applicable) !== canonicalize(applicableArtifact)) errors.push("applicable locked distributions do not match the supplied uv-export artifact"); |
There was a problem hiding this comment.
For direct callers of validateCampaignAdmission, equality between installed, applicable, and the supplied artifact does not prove that their versions came from environment.resolved_packages. For example, changing mcp or aiofiles to 9.9.9 in both arrays and rehashing the artifact, receipt, and environment still passes because only Verifiers is cross-checked against declared; compare every applicable name/version with the exact lock pins or derive this inventory in trusted code.
Useful? React with 👍 / 👎.
| const executableEvidence = object(evidence.executable_bundle); | ||
| const healthEvidence = object(evidence.health_receipt); | ||
| if (executableEvidence.environment_sha256 !== endpoint.environment_sha256 || executableEvidence.model_attestation_sha256 !== endpoint.model_attestation_sha256 || executableEvidence.checkpoint_sha256 !== lineage.checkpoint_sha256 || executableEvidence.frozen !== true) errors.push("executable artifact does not match endpoint environment/model/checkpoint attestation"); |
There was a problem hiding this comment.
The new executable-artifact check binds its environment, model, checkpoint, and frozen flag, but never binds executableEvidence.policy to workload_contract.candidate_tag_sha256 or the mutation receipt's candidate hash. A caller can replace the executable policy, recompute the executable/health/lineage/overflow hashes, and still admit evidence whose request and trace evaluated a different candidate from the endpoint bundle that would be used afterward.
Useful? React with 👍 / 👎.
| const healthEvidence = object(evidence.health_receipt); | ||
| if (executableEvidence.environment_sha256 !== endpoint.environment_sha256 || executableEvidence.model_attestation_sha256 !== endpoint.model_attestation_sha256 || executableEvidence.checkpoint_sha256 !== lineage.checkpoint_sha256 || executableEvidence.frozen !== true) errors.push("executable artifact does not match endpoint environment/model/checkpoint attestation"); | ||
| if (healthEvidence.status !== "healthy" || healthEvidence.executable_bundle_sha256 !== endpoint.executable_bundle_sha256 || healthEvidence.environment_sha256 !== endpoint.environment_sha256 || healthEvidence.model_attestation_sha256 !== endpoint.model_attestation_sha256) errors.push("health artifact does not attest the admitted executable endpoint"); | ||
| if (object(evidence.lineage).parent_candidate_sha256 !== lineage.parent_candidate_sha256 || object(evidence.lineage).prompt_sha256 !== lineage.prompt_sha256 || object(evidence.lineage).model_attestation_sha256 !== lineage.model_attestation_sha256 || object(evidence.lineage).checkpoint_sha256 !== lineage.checkpoint_sha256 || object(evidence.lineage).executable_bundle_sha256 !== endpoint.executable_bundle_sha256 || object(evidence.lineage).health_receipt_sha256 !== endpoint.health_receipt_sha256) errors.push("candidate lineage claims do not match supplied lineage artifact"); |
There was a problem hiding this comment.
Fresh evidence in this head only verifies that the supplied lineage artifact repeats candidate_lineage.prompt_sha256; neither value is compared with the request messages hash or workload_contract.benchmark_prompt_sha256. Changing both lineage prompt fields and recomputing the lineage and candidate hashes therefore still admits a candidate attributed to an unrelated prompt, so cross-attest this field to the manifest's defined admitted-prompt hash.
Useful? React with 👍 / 👎.
|
@codex review Fresh independent review requested at The workload-local Added a coherent-fabrication regression that rewrites the trace, manifest, execution receipt, metric, hashes, and transport fingerprints consistently; admission still fails because independent agent-tools derivation disagrees. Exact Cedar Verifiers 0.2.1 and second 0.2.2.dev77 locked fixtures remain covered. Validation: |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d348fe7031
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const smoke = object(manifest.mutation_smoke); | ||
| if (smoke.execution_receipt_sha256 !== sha256Bytes(artifacts.executionReceipt)) errors.push("mutation_smoke.execution_receipt_sha256 does not match supplied receipt"); | ||
| if (receipt.schema_version !== "understudy.synthetic_verifiers_execution.v1") errors.push("synthetic execution receipt schema is invalid"); | ||
| if (receipt.trusted_generator_version !== "understudy.agent_tools.campaign_admission_generator.v1" || receipt.trusted_generator_sha256 !== sha256Bytes(artifacts.trustedGenerator) || receipt.trusted_generator_sha256 !== object(manifest.environment).trusted_generator_sha256) errors.push("execution evidence is not derived by the trusted agent-tools generator"); |
There was a problem hiding this comment.
Fresh evidence at this head shows that direct callers of the newly exported validateCampaignAdmission still choose artifacts.trustedGenerator; this check merely compares that caller-controlled buffer with hashes in the caller-controlled manifest and receipt. A caller can substitute arbitrary generator bytes, update those two hashes, and retain an admitted result, so coherent fabricated evidence remains possible whenever admission uses the public validator rather than the CLI. Load or hash the packaged runtime asset inside the validator instead of accepting the trust root as a transport artifact.
Useful? React with 👍 / 👎.
| CAMPAIGN_ID = "public-synthetic-campaign" | ||
| WORKLOAD_ID = "cedar-public-synthetic" | ||
| REQUEST_ID = "request-public-1" | ||
| EXECUTION_ID = "execution-public-1" |
There was a problem hiding this comment.
When campaigns admit is used for any campaign other than the public fixture, the trusted generator always emits these fixed campaign, workload, request, and execution IDs. The CLI byte-compares that generated campaign evidence and trace with the supplied files, while validateIdentity requires those files to match the manifest and request, so no real campaign with different IDs can ever be admitted. Pass the admitted identity into the audited generator, or keep campaign-specific evidence outside the fixed synthetic replay.
Useful? React with 👍 / 👎.
|
@codex review Fresh independent review requested at
New direct-library adversarial coverage proves that substituted generator bytes, a substituted generator path, and a coherently rewritten metric + trace + manifest + execution receipt are rejected. CLI exact locked replay, npm package inclusion, Cedar Verifiers 0.2.1, and the independent 0.2.2.dev77 fixture remain green. Validation: |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f9bc9fe237
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| }); | ||
| if (workload.candidate_source !== "outer_system_transcript" || workload.policy_injection_count !== 1) errors.push("candidate policy must come from the outer system transcript and be injected exactly once"); | ||
| if (workload.benchmark_prompt_sha256 !== sha256Bytes(Buffer.from(canonicalize(fixedMessages)))) errors.push("fixed benchmark prompt slot is not hash-bound independently of the candidate policy"); | ||
| if (workload.tool_schema_sha256 !== toolsHash) errors.push("workload tool schema hash does not match tools artifact"); |
There was a problem hiding this comment.
For direct callers of validateCampaignAdmission, a response and trace can coherently rename set-record to an undeclared tool such as delete-all, update the tool-step, trace, execution-receipt, and manifest hashes, and still be admitted while the supplied tools artifact continues to declare only set-record. This line binds the catalog's hash but no code checks executed tool names or arguments against that catalog, so admission can attest a mutation performed through an interface the workload never authorized.
Useful? React with 👍 / 👎.
| const errors: string[] = publishedSchemaErrors(manifest); | ||
| if (value.schema_version !== CAMPAIGN_ADMISSION_SCHEMA_VERSION) errors.push(`schema_version must equal ${CAMPAIGN_ADMISSION_SCHEMA_VERSION}`); | ||
| validateEnvironment(value, errors); | ||
| const fingerprints = fingerprintTransport(artifacts); |
There was a problem hiding this comment.
When any caller-supplied request, response, tools, or trace buffer contains malformed JSON, fingerprintTransport throws here before the validator's error-collection paths run. Direct-library consumers therefore receive an undocumented synchronous exception instead of the promised AdmissionResult with admitted: false, which can crash a service that treats this exported function as the fail-closed validation boundary; catch transport parsing failures and report them as admission errors.
Useful? React with 👍 / 👎.
|
@codex review Fresh independent review requested at Added a versioned, workload-generic Prompt-bearing evaluation evidence and Adversarial coverage includes dense-only gain, scorer exception, missing/duplicate task, prompt mismatch, scorer/preimage substitution, row-preimage mismatch, and prompt-leakage classification. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a1da9822a0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const scorerHash = sha(artifacts.trustedScorerContract); | ||
| if (value.trusted_scorer_contract_sha256 !== scorerHash || scorerHash !== artifacts.expectedTrustedScorerContractSha256) errors.push("trusted scorer contract hash does not match independently expected scorer-contract preimage"); | ||
| const declared = object(value.named_preimages); | ||
| for (const [name, bytes] of Object.entries(artifacts.namedPreimages)) if (declared[name] !== sha(bytes) || sha(bytes) !== artifacts.expectedNamedPreimageSha256[name as keyof typeof artifacts.expectedNamedPreimageSha256]) errors.push(`named preimage ${name} hash mismatch`); |
There was a problem hiding this comment.
When a JavaScript caller omits harness, scorer, terminal, export, or promotion from both artifact maps, this loop never visits those required names, so schema-valid arbitrary hashes in evidence.named_preimages remain unchecked and the result can still be PROMOTE. The TypeScript record type is not a runtime guarantee for this public validator; iterate the fixed nine-name set and reject every missing buffer or expected digest.
Useful? React with 👍 / 👎.
| if (row.candidate_prompt_sha256 !== row.evaluated_prompt_sha256) errors.push(`prompt hash mismatch for ${id}/${task}`); | ||
| const rowPreimage = artifacts.rowPreimages[`${id}/${task}`]; | ||
| if (!rowPreimage || row.canonical_row_sha256 !== sha(rowPreimage.canonical) || row.physical_row_sha256 !== sha(rowPreimage.physical)) errors.push(`canonical/physical row hashes do not match supplied preimages for ${id}/${task}`); |
There was a problem hiding this comment.
When a row's strict_exact value is changed while its two declared hashes and supplied preimage buffers are left untouched, this check still passes because it verifies only the buffers' digests and never decodes either preimage to compare the candidate, task, prompt, score, or exception data. An originally losing candidate can therefore flip enough booleans to become a strict-exact winner and receive PROMOTE; derive the scored receipt fields from the canonical/physical row contents or validate their semantic equality.
Useful? React with 👍 / 👎.
1 participant
What changed
campaign_id, reject unknown properties, and test schema/runtime locksteppyproject.toml,uv.lock, liveuv lock --check, exact lockeduv run --project ... --lockedargv, uv/Python identity, complete resolved package inventory, git revisions, MCP/tool-schema hashes, and immutable container digestmax_tokensWhy
Campaigns need one reusable admission boundary before optimizer, endpoint, or training spend begins. This turns previously separate run conventions into an inspectable receipt and prevents compilation from hiding a malformed or non-mutating paid canary.
Safety and privacy
The implementation performs local validation only. It does not deploy, call providers, execute customer workloads, merge changes, or include private prompts, traces, credentials, or answer keys.
Validation
npm run check: 1,191 tests across 162 suites passed; typecheck, build, skill validation, and package dry-run passedgit diff --checkpassedCedar compatibility note
The fixture locks canonical Verifiers 0.2.1 at commit
ab65b6e8d34b03d162408d4bcb854430a86809e6withmcp==1.29.0. Historical Cedar artifacts that lack the generated execution receipt, exact environment inventory, per-step response-to-trace transport pairing, assertion fraction, or verified state delta will fail admission. They require private re-execution or a private adapter that produces equivalent hash-bound evidence; this public contract intentionally does not weaken admission for legacy rows.