Skip to content

fix(vibe): Get the writer's rules to Codex and fix visual-cues eval fixtures - #297

Merged
shafty023 merged 3 commits into
mainfrom
fix/vibe-codex-eval-gaps
Oct 16, 2026
Merged

shafty023 merged 3 commits into
mainfrom
fix/vibe-codex-eval-gaps

Conversation

@shafty023

Copy link
Copy Markdown
Collaborator

The four visual-cues Codex evals that failed on main 642d235 (vibe 1.0.44 Codex baseline) were plugin gaps on the Codex path, plus two fixture bugs that showed up while rerunning them.

Cause. Codex loads a skill by reading the file through shell output, which keeps the start and end of a long output and drops the middle. SKILL.md is about 14,400 tokens; in every recorded run the read cut out lines from section 2 to section 5 step 3. The Codex context doing the writer's work never opened agents/vibe-change-worker.md or references/quality-loop.md, where the "Acceptance criteria before ready" and "Visual references" rules live. So it rendered with no reference image, paraphrased the person's words with no source or result, and logged no Reference read lines or token-mapped criteria. Told only to read in parts, it stopped after 200 lines, before those sections.

Plugin changes

  • SKILL.md harness notes, inside the part a single read keeps: Codex reads long files to their last line in parts of at most 200 lines. In both harnesses, whatever context does the writer's work reads the writer's definition and the two named quality-loop.md sections first. The writer's definition says the same for a Codex writer.
  • quality-loop.md, "Visual references", now opens with "Recording it in the table": the line shapes for a Reference read (every image on the session, a correction included), one criterion per reference property naming its token or component, the reference check after the last edit, and a quoted criterion with no render when there is no reference. Criterion lines sit above ### Checks. It leads the section because a Codex writer reading from the section's first line stopped before its end.

Fixture and grader fixes (shared with the Claude suite)

  • The stand-in screenshot tool treated an @import line as part of the first selector and dropped that rule, so a card that imported the tokens rendered with no shadow. It now strips @import lines and comments.
  • The stand-in catalog states the selected-state contract the renderer and grader already used (aria-selected="true" or is-selected). Twice Codex switched to aria-pressed, which the renderer cannot draw.
  • hover-state and selected-state required { right after the state selector, so .meeting-card:hover, .meeting-card:focus-visible { box-shadow: var(--shadow-raised) } failed though it applies the token on hover. They now accept a grouped selector; a descendant-only or missing rule still fails.
  • criteria-mapped said "FAIL if any criterion ... lacks a token", which also failed a correct extra criterion for something the design system has no token for (the plugin logs those as conflicts). It now names its four required criteria.

Results (one run per case; Codex gpt-6-sol low effort through evals/codex/run.py, Claude claude plugin eval --tag visual-cues with Opus)

Case Codex 1.0.44 Codex here Claude 1.0.44 Claude here
executive, no reference FAIL PASS FAIL PASS
product manager, reference card FAIL PASS FAIL PASS
product manager, outside-product cards FAIL PASS PASS PASS
product manager, two screenshots FAIL PASS FAIL FAIL

On Claude, pm-two-screenshot-page still fails criteria-recorded. In all five Claude runs of that case (1.0.44 and this branch), the model opened no plugin file, so no plugin prose reaches it. In a real session the Claude writer runs with --agent vibe-change-worker, so its definition is its system prompt. The eval runs the writer's work in the root instead, which is a fixture fidelity gap to handle separately.

Tests. New test_vibe_codex_reading_contract.py: 8 cases, all failing on 1.0.44 and passing here. uv run pytest plugins/: 2307 passed, 3 skipped. Ruff and pyright clean.

Bumps vibe to 1.0.46 (both manifests) with a CHANGELOG entry.

@shafty023
shafty023 merged commit a2c9505 into main Oct 16, 2026
10 checks passed
@shafty023
shafty023 deleted the fix/vibe-codex-eval-gaps branch October 16, 2026 02:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant