conformance: ratchet DEV-1149-class experimental gaps (22 ticket-linked entries) - #20
Merged
Merged
Conversation
…d expected failures Executes the DEV-1149 decision (gaps stay visible as ratchets, never reclassified): every raw experimental fail/timeout from nightly run 29384698254 gets a missing_feature/bug entry pointing at DEV-1149 (missing batched events/track route) or DEV-1148 (hot-path Class 1). 22 new entries; scenarios already ratcheted untouched. Unlocks honest auto-promotion of these scenarios fleet-wide (a ratcheted expected_fail is promotion-acceptable; a raw fail is not). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
rsd-raindrop
approved these changes
Jul 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Executes the DEV-1149 decision (option b: gaps stay VISIBLE as ticket-linked ratchets, never reclassified). Every raw experimental-scenario fail/timeout from nightly run 29384698254 becomes an expected-failure entry:
missing_feature→ DEV-1149 (missing batched events/track route — confirmed by codex source analysis: TrackAI/TrackEvent ship single-object events/track_partial synchronously, no batch sender exists),bug→ DEV-1148 (hot-path Class 1). 22 new entries; existing entries untouched; scenario files and drivers untouched.Verification: this PR's own conformance fault-lane CI is the proof — with these entries every raw fail becomes expected_fail and the lane must exit 0 with zero unexpected_pass.
Effect: unlocks honest auto-promotion (harness#69/#71) of these scenarios — a ratcheted expected_fail is promotion-acceptable, a raw fail is not. The gaps stay fully visible in MATRIX.md as ratchet rows until the DEV-1149/DEV-1148 implementations remove the entries (that removal = proof of fix).
Coordinator-authored (Devin has no go/rust access) — queued for Pavel's approval.
🤖 Generated with Claude Code
Note
Low Risk
Documentation-only conformance allowlist updates; no runtime, scenario, or driver code changes.
Overview
Adds 22 ticket-linked expected-failure rows to
conformance/failures.txtso nightly experimental fault scenarios that still fail raw become ratchetedmissing_featureorbugoutcomes instead of breaking the conformance fault lane.Most new rows are
missing_feature→ DEV-1149 (scenarios assert the ratified batchedevents/trackroute; the SDK only sends single-objectevents/track_partial).hot-path-sustained-track-ai@faultis classified asbug→ DEV-1148 (CONTRACT 1.1 hot-path blocking). Existing entries, scenarios, and drivers are unchanged—only the allowlist grows so CI can pass with zero unexpected passes while gaps stay visible until fixes remove the rows.Reviewed by Cursor Bugbot for commit 0c49795. Bugbot is set up for automated code reviews on this repo. Configure here.