Add fail-closed preference pair admission - #468
Conversation
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
lluisinthedesert
left a comment
There was a problem hiding this comment.
The fail-closed direction is right, but this gate is not yet safe to reuse. P0: rejectedPayloadAvailable is caller-asserted and is not bound to a raw private preimage/hash, so a fabricated true value passes. Require a rejected payload SHA and verified private preimage hash result. Also validate SHA-256 as lowercase hex (not length only), bind rows to an explicit frozen TRAIN allowlist and one expected capability/runtime/renderer/tool-schema set, reject duplicate pair IDs, compare normalized chosen/rejected semantics rather than token inequality alone, and validate rejection classes from the actual rejected response rather than trusting labels. Please add adversarial tests for a lied availability flag, nonhex hashes, mixed runtime/split, duplicate IDs, semantically identical responses with different tokens, and mislabeled rejection classes. Keep payloads private and create a fresh admission lineage after changing this contract.
|
Addressed the P0 review in commits |
lluisinthedesert
left a comment
There was a problem hiding this comment.
Re-reviewed through 899ce0b. The admission API is now generic, binds rejected payload hashes through the private verifier callback, enforces lowercase SHA-256 and frozen TRAIN/provenance identities, validates semantic/class mismatches, distinguishes legitimate no-action from action-row collapse, rejects every duplicate source occurrence after the first, and includes the requested adversarial regressions. The remaining CI rerun is for an unrelated known timing-sensitive outcome-executors test; no unrelated code change is requested.
Summary
Replace the workload-specific preference gate with a reusable, provider-free admission contract that cannot be satisfied by caller assertions or redacted payload metadata.
admitPreferencePairs(pairs, config)now requires:verifyRejectedPayloadresolver that proves the rejected preimage hash;The regression suite covers lied payload availability, non-hex hashes, duplicate IDs, cross-split and mixed provenance, semantic identity with different tokens, mislabeled classes, and legitimate no-action sentinel handling. Raw rejected payloads remain private to the caller-supplied verifier.
Link to Devin session: https://app.devin.ai/sessions/23d3bbcf33b34ddda0fb4f3ebbf16607