Skip to content

Add fail-closed preference pair admission - #468

Merged
lluisinthedesert merged 4 commits into
mainfrom
devin/seed43-preference-admission
Aug 5, 2026
Merged

Add fail-closed preference pair admission#468
lluisinthedesert merged 4 commits into
mainfrom
devin/seed43-preference-admission

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Summary

Replace the workload-specific preference gate with a reusable, provider-free admission contract that cannot be satisfied by caller assertions or redacted payload metadata.

admitPreferencePairs(pairs, config) now requires:

  • exact lowercase SHA-256 values and a private verifyRejectedPayload resolver that proves the rejected preimage hash;
  • frozen TRAIN allowlist membership plus one source capability, renderer, tool-schema, and runtime binding;
  • unique pair/source-row identities, unless explicitly configured for multi-rejection rows;
  • normalized chosen/rejected tool/no-action semantics, independent of token inequality;
  • verifier-derived rejection classification with required-vs-desired coverage;
  • action-family and multi-effect continuation coverage with pair-scoped and global rejection reasons.

The regression suite covers lied payload availability, non-hex hashes, duplicate IDs, cross-split and mixed provenance, semantic identity with different tokens, mislabeled classes, and legitimate no-action sentinel handling. Raw rejected payloads remain private to the caller-supplied verifier.

Link to Devin session: https://app.devin.ai/sessions/23d3bbcf33b34ddda0fb4f3ebbf16607

@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@lluisinthedesert lluisinthedesert left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fail-closed direction is right, but this gate is not yet safe to reuse. P0: rejectedPayloadAvailable is caller-asserted and is not bound to a raw private preimage/hash, so a fabricated true value passes. Require a rejected payload SHA and verified private preimage hash result. Also validate SHA-256 as lowercase hex (not length only), bind rows to an explicit frozen TRAIN allowlist and one expected capability/runtime/renderer/tool-schema set, reject duplicate pair IDs, compare normalized chosen/rejected semantics rather than token inequality alone, and validate rejection classes from the actual rejected response rather than trusting labels. Please add adversarial tests for a lied availability flag, nonhex hashes, mixed runtime/split, duplicate IDs, semantically identical responses with different tokens, and mislabeled rejection classes. Keep payloads private and create a fresh admission lineage after changing this contract.

@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

Addressed the P0 review in commits 8971ff36 and cfab611a. The API is now generic (admitPreferencePairs / PreferencePair), requires exact lowercase-hex hashes and a caller-supplied private verifyRejectedPayload result, binds frozen TRAIN rows and expected capability/runtime/renderer/tool-schema hashes, rejects duplicate identities, compares normalized semantics, validates verifier-derived rejection classes, supports required vs desired coverage, and returns pair-scoped plus global reasons. Added the requested adversarial tests plus opposite-sign identical-token and legitimate no-action sentinel cases. npm run check passes under Node 22.19.0; the earlier Node 20 failure was the environment’s node:sqlite incompatibility.

@lluisinthedesert lluisinthedesert left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed through 899ce0b. The admission API is now generic, binds rejected payload hashes through the private verifier callback, enforces lowercase SHA-256 and frozen TRAIN/provenance identities, validates semantic/class mismatches, distinguishes legitimate no-action from action-row collapse, rejects every duplicate source occurrence after the first, and includes the requested adversarial regressions. The remaining CI rerun is for an unrelated known timing-sensitive outcome-executors test; no unrelated code change is requested.

@lluisinthedesert
lluisinthedesert merged commit 2da7907 into main Aug 5, 2026
8 checks passed
@lluisinthedesert
lluisinthedesert deleted the devin/seed43-preference-admission branch August 5, 2026 04:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant