Repository navigation
fix(vibe): Get the writer's rules to Codex and fix visual-cues eval fixtures - #297
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The four visual-cues Codex evals that failed on main
642d235(vibe 1.0.44 Codex baseline) were plugin gaps on the Codex path, plus two fixture bugs that showed up while rerunning them.Cause. Codex loads a skill by reading the file through shell output, which keeps the start and end of a long output and drops the middle. SKILL.md is about 14,400 tokens; in every recorded run the read cut out lines from section 2 to section 5 step 3. The Codex context doing the writer's work never opened
agents/vibe-change-worker.mdorreferences/quality-loop.md, where the "Acceptance criteria before ready" and "Visual references" rules live. So it rendered with no reference image, paraphrased the person's words with no source or result, and logged noReference readlines or token-mapped criteria. Told only to read in parts, it stopped after 200 lines, before those sections.Plugin changes
quality-loop.mdsections first. The writer's definition says the same for a Codex writer.quality-loop.md, "Visual references", now opens with "Recording it in the table": the line shapes for aReference read(every image on the session, a correction included), one criterion per reference property naming its token or component, the reference check after the last edit, and a quoted criterion with no render when there is no reference. Criterion lines sit above### Checks. It leads the section because a Codex writer reading from the section's first line stopped before its end.Fixture and grader fixes (shared with the Claude suite)
@importline as part of the first selector and dropped that rule, so a card that imported the tokens rendered with no shadow. It now strips@importlines and comments.aria-selected="true"oris-selected). Twice Codex switched toaria-pressed, which the renderer cannot draw.hover-stateandselected-staterequired{right after the state selector, so.meeting-card:hover, .meeting-card:focus-visible { box-shadow: var(--shadow-raised) }failed though it applies the token on hover. They now accept a grouped selector; a descendant-only or missing rule still fails.criteria-mappedsaid "FAIL if any criterion ... lacks a token", which also failed a correct extra criterion for something the design system has no token for (the plugin logs those as conflicts). It now names its four required criteria.Results (one run per case; Codex
gpt-6-sollow effort throughevals/codex/run.py, Claudeclaude plugin eval --tag visual-cueswith Opus)On Claude,
pm-two-screenshot-pagestill failscriteria-recorded. In all five Claude runs of that case (1.0.44 and this branch), the model opened no plugin file, so no plugin prose reaches it. In a real session the Claude writer runs with--agent vibe-change-worker, so its definition is its system prompt. The eval runs the writer's work in the root instead, which is a fixture fidelity gap to handle separately.Tests. New
test_vibe_codex_reading_contract.py: 8 cases, all failing on 1.0.44 and passing here.uv run pytest plugins/: 2307 passed, 3 skipped. Ruff and pyright clean.Bumps vibe to 1.0.46 (both manifests) with a CHANGELOG entry.