feat(vibe-tests): setup test — installing the system into an app that already has one - #5268
feat(vibe-tests): setup test — installing the system into an app that already has one#5268nynexman4464 wants to merge 2 commits into
Conversation
… already has one Every arm we run today hands the agent a project where the design system is already correctly wired: the nightly's `project-astryx-tailwind` environment ships a globals.css with the layer order declared and every import in place, and the adoption test installs the system in every condition on purpose. So nothing measures the install itself — the twenty minutes between "we're going to use Astryx" and "the app renders correctly with Astryx in it", in an app that already has a styling system, a token vocabulary and a year of CSS. That gap matters because of what a bad install looks like: it looks like a good one. Applying the recipe our own README prints to the fixture leaves tsc clean, the bundler happy, zero console errors, zero failed requests — and the page title at 1.10:1 contrast. The only signal is that the page is wrong, which is the one signal an agent, CI and a diff review all lack. - `fixture-app/` — the same ops console the adoption test uses, made runnable: a setup run has to be COMPILED AND RENDERED to be scored, so the fixture is a real Vite + React + Tailwind v4 build with a middleware data stub. It builds before every run, and that pristine build is the reference every number is a delta against. - `conditions.json` — four arms, floor (public docs only) to ceiling (handed the exact edits). The system is installed in NO condition; installing it is the task. - `prompts.json` — three feature-shaped requests. None says "don't break the app": a recipe that only works when the user thinks to ask for that is not a recipe. - `setup-measure.mjs` — build, serve, render, probe, in BOTH color schemes. The system paints through light-dark() and the fixture has no color-mode system for it to read, so an arm can be correct on a dark-mode laptop and unreadable in headless CI; a single-scheme measurement would inherit whoever ran it. - `setup-eval.ts` — the deterministic scorer: app regressions by category, legibility against the AA floor, mode dependence, variable capture. No LLM in this path, and the analyzer never sees the condition id. - `apply-recipe.mjs` — our documented recipes, transcribed, so the mechanism can be verified with no agent in the loop and the ceiling arm has something to hand an agent verbatim. No agent has run this yet: no agent findings, no scores, no recommendation. What the mechanism-verification run does show is that the three causes separate cleanly — a recipe written for an empty file, a Tailwind bridge that renames 13 tokens the app already uses, and a theme provider that follows the OS — and that the candidate recipe removes the legibility failure and the mode dependence while leaving a typography residue the ceiling arm cannot remove either. PLAN.md has the tables and the pre-registered decision rule.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
meta-cla
Bot
added
the
CLA Signed
PR Analysis Report📚 Storybook PreviewView Storybook for this PR 🧪 Sandbox PreviewView Sandbox for this PR No new or modified components detected. Bundle Size SummaryNo component packages changed. Accessibility AuditStatus: No accessibility violations detected. Generated by PR Enrichment workflow | Storybook | Sandbox | View full report |
… position is the fourth cause A claim we had heard and had no mechanism for: you "can't just put the imports in globals.css". You can — but the layer statement has to go FIRST, and putting it anywhere else is worse than not following the recipe at all. `@layer a, b, c;` only orders names that are not already registered, and the app's own `@import 'tailwindcss'` registers theme/base/utilities on its way past. A statement below that import can only append the system's layers AFTER Tailwind's utilities — the exact inverse of what the recipe exists to arrange — so the system's reset outranks every utility the app already uses and quietly strips borders, padding and the type scale off every control. Measured on the fixture: 38 changed properties against the documented recipe's 30, with the colour mode correct and nothing in the console. Source review cannot catch this, and neither can a class-string lint, because the `@layer` statement does not survive compilation: Tailwind rewrites it and the bundle carries only `@layer components;`, so the effective order is the order the layer BLOCKS first appear in the output. So the measurement reads the emitted CSS. That is the same guard agentcloud's own e2e spec makes after D114534048, where a duplicate astryx.css import demoted astryx-base to the bottom of the cascade and a button rendered square — generalized here into a measure with a verdict attached. - `apply-recipe.mjs` gains `appended-at-end`, the naive edit, as a fourth arm. - `setup-measure.mjs` records the emitted layer order. - `setup-eval.ts` gains `cascadeInverted`, and an inverted cascade now reads as silent-damage rather than drift — it is one edit from correct and damages more of the app than the recipe it was trying to follow. - PLAN.md carries the four causes, both colour schemes, and a new section on where each was already found and written down: the production incident behind cause 4, the comment refusing the bridge behind cause 2, the `dark:`-follows- the-OS sibling of cause 3, and the abandoned installer fix behind cause 1. Plus the two open upstream items the test depends on (#5205, #5165). - The residue every arm leaves now has a name: `--color-accent` and `--color-border` collide because `@scope` bounds selector matching, not inheritance, and the provider syncs its attribute onto <html>. The guidance says to re-assert those on the boundary element; the harness reports the collision but does not yet assert the fix, and PLAN says so. Verification: 24 tests pass (4 new). Four arms measured end to end in both colour schemes; the tables in PLAN.md are that run.
Update: a fourth cause, and where all four had already been foundSince the first push I read the full 1:1 transcript this came out of, and mined the diffs and sessions where this problem has already been hit. Two things changed. The layer statement's POSITION is a cause on its ownThe claim I went looking for was "you can't just inject them into globals.css, for whatever reason, it doesn't work there — you need a separate file." You can. It has to go first, and putting it anywhere else is worse than not following the recipe at all. New arm
The system's reset then outranks every utility the app already uses, and Source review cannot catch this and neither can a class-string lint, because the Every cause here has already been found once, by someone, somewhere unfindableThat is the argument for the harness, so
The residue now has a nameEvery arm, the candidate included, captures Two open upstream items this test depends on
24 tests (4 new), still all four detectors proven to go red when disabled. |
1 participant
The gap
Every vibe test we run hands the agent a project where the design system is already correctly wired. The nightly's
project-astryx-tailwindenvironment ships aglobals.csswith the layer order declared and every import in place. The adoption test (#4902) installs the system in every condition on purpose, because there the variable is whether the agent reaches for it.So nothing we run measures the install — the twenty minutes between "we're going to use Astryx" and "the app renders correctly with Astryx in it", in an app that already has a styling system, a token vocabulary, house components and a year of CSS.
Why that gap is expensive
A bad install looks exactly like a good one. Applying the recipe
packages/core/README.mdprints, to the fixture app, by script:tsc --noEmitvite buildThe toolchain says nothing. The only signal is that the page is wrong — the one signal an agent doesn't have, CI doesn't have, and a diff review doesn't have. A greenfield arm cannot produce this finding at all: there is no "before" to measure against.
What this PR is
The design (
PLAN.md) and a working harness. No agent has run it yet — no agent findings, no scores, no recommendation. Mechanism verified without agents first, same order ascli-discovery-test..tsxfilePieces
fixture-app/— the same ops console the adoption test uses, made runnable. A setup run has to be compiled and rendered to be scored, so this is a real Vite + React + Tailwind v4 build with a middleware data stub. It builds and renders before every run, and that pristine build is the reference every number is a delta against. (When feat(vibe-tests): adoption test — introducing Astryx into an app that already has a styling system #4902 lands the two fixtures should converge to one copy — this one is a superset.)conditions.json— four arms: public docs only (floor) →AGENTS.mdpoints at the CLI → an existing-app recipe exists → handed the exact edits (ceiling). The system is installed in no condition; installing it is the task.prompts.json— three feature-shaped requests. None of them says "don't break the app": a recipe that only works when the user thinks to ask for that is not a recipe.setup-measure.mjs— build, serve, render, probe — in both color schemes. Astryx paints throughlight-dark()and the fixture has no color-mode system for it to read, so an arm can be correct on a dark-mode laptop and unreadable in headless CI. A single-scheme measurement would inherit whichever machine ran it.setup-eval.ts— the deterministic scorer: app regressions by category, legibility against the AA floor, mode dependence, variable capture. No LLM in this path, and the analyzer never sees the condition id (Checker Protocol §1).apply-recipe.mjs— our own documented recipes, transcribed, so the mechanism can be verified with no agent in the loop and so the ceiling arm has something to hand an agent verbatim.setup-eval.test.ts— 20 tests, each check asserted on a sample that should trip it and one that shouldn't.Mechanism-verification run
Three recipes applied by script, no agent.
--scheme light(a light-mode machine, and every headless CI):docs-verbatimdocs-no-bridgelayered-in-placeSame three sandboxes under
--scheme dark: zero unreadable probes in every arm. That flip is the "works on my machine" failure, and it is why both schemes are measured.Three causes separate cleanly, and each is separately fixable:
globals.csswholesale, with Tailwind imported in three pieces so Astryx can sit between them. An existing app's file is not empty. Transcribed literally,--spacing,--text-xsand--font-sansstop resolving at:root, and the type scale moves under every existing utility —text-xsin this console goes 12px → 10px, across every table, badge and button.tailwind-theme.cssdeclares--color-card,--color-muted,--color-border,--color-accent,--color-primary,--color-secondary, the whole--radius-*and--shadow-*ladders, the--text-*scale and the base--spacingunit — 13 exact collisions with the default shadcn/Tailwind vocabulary, re-pointed app-wide by one import line. It is printed unconditionally in the README, inastryx docs styling-libraries, and in our ownenvironments/project-astryx-tailwind.<Theme>is a color boundary that follows the OS. It paintscolor: light-dark(#171717, #fafafa)on its wrapper; an app whose dark look is its own CSS variables has nodata-themefor it to read, so every inherited piece of text under the provider turns near-black on a near-black surface.modefixes it in one word and no setup path mentions it.The candidate recipe (4 CSS lines and a prop, against the documented 7-line replacement) removes the legibility failure and the mode dependence and cuts regressions from 30 to 11. What it cannot remove is the residue: 11 properties still move, all typography and inherited color. That is a product question — there is currently no way to adopt the components without adopting the type ramp for everything under the provider — and it is what the ceiling arm exists to expose.
Pre-registered decision rule
Set before any agent runs. A guidance change ships if, against the floor, it eliminates legibility failures and halves app regressions with non-overlapping CIs, without increasing console errors or breaking the build. Regression count alone never justifies shipping — it is a drift measure; legibility and mode dependence are the severity ones. Damage the ceiling arm still produces is a product finding filed against the packages, not evidence that guidance needs more words.
Verification
pnpm vitest run internal/vibe-tests/setup-test— 20 tests pass.silent-damage— every mutation turned the suite red, and green again on restore.run-setup.mjsbuilds both arms' sandboxes, each builds before the agent arrives, the guidance lands inAGENTS.md, the git baseline commits, andsetup-measure.mjs+setup-aggregate.tsproduce the tables above.eslintclean (0 errors, 0 warnings on the new files); pre-commitcheck:*suite clean.Notes for review
s2guidance text is a strawman, not a proposal — it is the candidate the test exists to judge.eslint.config.jsgains one ignore, andinternal/vibe-tests/.gitignoreone negation so the fixture's ownAGENTS.md— the independent variable — is not swallowed by the generated-docs rule.node_modulesfor the app's own dependencies, so registry friction on the system's packages is measured but install-time flakiness is not.Next step is a pilot — floor vs ceiling on
s1, K=3, one agent — enough to answer "do the arms separate at all" before anyone pays for the full grid. Not run yet.