Skip to content

chore(deps): update pnpm to v12.6.0 with sanity-io/ui's pnpm setup - #2048

Merged
stipsan merged 13 commits into
mainfrom
cody/upgrade-pnpm-12-48e4
Sep 29, 2026
Merged

stipsan merged 13 commits into
mainfrom
cody/upgrade-pnpm-12-48e4

Conversation

@stipsan

@stipsan stipsan commented Sep 29, 2026 •

Copy link
Copy Markdown
Member

Puts this repo on the same pnpm version and settings as sanity-io/ui main: packageManager pnpm@12.6.0, ui's pnpm settings in pnpm-workspace.yaml, ui's CI setup action (pnpm/setup@v3) but with its store cache off, and a lockfile regenerated with pnpm 12.6.0. The Dedupe lockfile workflow stays, with the one change pnpm 12 needs.

Builds on #2044, supersedes #2043, #2044 and #1131

Changes

Before After
packageManager pnpm@11.21.0 pnpm@12.6.0 (as in ui)
pnpm-workspace.yaml autoDedupe: true and strictPeerDependencies: false (as in ui)
.github/actions/setup pnpm/action-setup@v6 + actions/setup-node@v7 + pnpm install ui's action (pnpm/setup@v3, runtime: node@lts) with cache: false, where ui has cache: true
.github/workflows/pnpm-if-needed.yml (Dedupe lockfile) runs pnpm dedupe once kept; runs pnpm dedupe twice
vercel (root devDependency) ^57.0.0 ^60.1.3
  • Lockfile: main's lockfile after chore(deps): lock file maintenance #2039, plus this PR's changes as pnpm 12.6.0 re-applied them. Those changes are the packageManagerDependencies block, the vercel@60.1.3 tree with the optional-peer suffixes it brings, and autoDedupe's dedupes: the same zod and gaxios ones as the pending chore(deps): dedupe pnpm-lock.yaml #2052, plus the peer toggle described below. No other versions change, and sanity@6.17.0-next.108 stays put. chore(deps): lock file maintenance #2039's refresh had already removed the three duplicates the first revision collapsed.
  • Vercel CLI: pnpm 12 writes the lockfile as two YAML documents. vercel@57's bundled @vercel/build-utils reads only the first, falls back to npm, and fails on workspace:*, which is chore(deps): update pnpm to v12 - autoclosed #2043's failure. 58.9.5 is the first CLI that parses it, and every release from 58.7.0 on ships without npm provenance (still published from GitHub Actions OIDC). So trustPolicy: no-downgrade needs the exact versions pnpm reports as ERR_PNPM_TRUST_DOWNGRADE in trustPolicyExclude: vercel@60.1.3 and 31 @vercel/* packages.
  • CI setup: pnpm/setup reads packageManager, installs Node LTS, runs pnpm install, and caches pnpm's lockfile-verification verdict across jobs, which saves the ~28s supply-chain check. ailf-eval.yml passes runtime: node@lts instead of node-version: lts/*. release.yml keeps pnpm/action-setup + actions/setup-node, as in ui.
  • Store cache off, unlike ui (commit bddef7a4). With cache: true, pnpm/setup uploads a new ~645 MiB store entry after every job, even for an unchanged lockfile, because its keys include the run id. That was 12 entries (7.7 GB) per push here, evicting the Playwright browser caches the e2e jobs reuse. With it off, test shards spend 16–19s in setup and nothing afterwards, against 19–23s plus 9–12s of upload with it on. ui keeps cache: true and has the same issue. Revert bddef7a4 to match ui.
  • Dedupe lockfile workflow kept, adapted for pnpm 12 (commit a7e1057f). On the current lockfile, pnpm 12.6.0's pnpm dedupe isn't idempotent. Each run toggles @vitejs/devtools and oxc-transform-react in the peer suffixes of plugins/@sanity/debug-preview-url-secret-plugin's sanity and @sanity/tsdown-config entries, without changing any version. Run once, as before, the workflow would open a toggle-only PR after every lockfile change on main, and each automerged dedupe PR would trigger the next. A second run undoes the toggle and keeps real dedupes. An earlier commit in this branch deleted the workflow; 82810b5e reverts that.
  • Docs: AGENTS.md, CONTRIBUTING.md and README.md now require pnpm v12. AGENTS.md and e2e/README.md explain the two-document lockfile and the Vercel CLI floor; per Copilot's review of chore(deps): update pnpm to v12.6.0 and enable autoDedupe #2044, the trust-policy wording only lists versions that pnpm reports as downgrades. AGENTS.md also says not to commit lockfile diffs that only flip the peer toggle.

No changeset: no published package's manifest or runtime code changes, as with the pnpm 11 bump in #1579.

Merge with main

main gained one commit after this branch started: Renovate's lock file maintenance (#2039, cb68d65e). It conflicted only in pnpm-lock.yaml. main is merged in rather than rebased, so nothing was force-pushed. The resolution takes main's lockfile and lets pnpm 12.6.0 re-apply this PR's changes; pnpm-workspace.yaml needed no further changes. #2039's 25 source files merged cleanly.

What Cody needs to do

Open questions

  1. Vercel CLI trust exclusions. This is chore(deps): update pnpm to v12.6.0 and enable autoDedupe #2044's call, which overrides the reason chore(deps): update dependency vercel to v58 #1890 was closed: 32 exact versions without provenance in trustPolicyExclude. Every future vercel bump has to add its own versions, so Renovate's vercel PRs will fail their artifact update until someone does. The alternative is e2e previews that can't build on pnpm 12.
  2. strictPeerDependencies: false matches ui but is already pnpm's default; ui carried it over from an old .npmrc. Keep it for parity, or drop it?
  3. Store cache in ui. ui has the same per-job uploads. Turn its store cache off too?

Known pnpm 12.6.0 issue

The peer toggle also shows up locally. A fresh pnpm install, or the implicit install pnpm runs before scripts when node_modules looks stale, flips those two suffixes in pnpm-lock.yaml. Frozen installs, which CI uses, never do. The lockfile before #2039 didn't toggle; main's refreshed resolutions exposed it, and it happens with autoDedupe off too.

Not in this PR

Verification

  • Lockfile: regenerated on main at cb68d65e by pnpm 12.6.0 run directly on Node 24.21.0. A cold pnpm install --frozen-lockfile passes the supply-chain check (2,615 entries), and without the Vercel exclusions it fails with ERR_PNPM_TRUST_DOWNGRADE.
  • Dedupe workflow: from the committed lockfile, from its toggled state, and from the committed lockfile with chore(deps): dedupe pnpm-lock.yaml #2052's duplicates put back, two CI=true pnpm dedupe --config.ignore-scripts=true runs give, respectively, no change, no change, and exactly chore(deps): dedupe pnpm-lock.yaml #2052's dedupes. A single run gives the toggle every time.
  • Local checks on the merged tree: pnpm lint, pnpm knip, pnpm build (52/52 tasks) and pnpm test run (224 files, 1,391 tests) pass.
  • CI: everything passes on bddef7a4, including deploy-preview (Vercel CLI 60.1.3 detects the lockfile and installs with pnpm 12.6.0) and e2e on Chromium and Firefox. CI on a7e1057f, which only touches the dedupe workflow and AGENTS.md, is running.
  • zizmor: zizmor .github reports the same findings as on main at every severity.
Open in Web Open in Cursor 

Matches the pnpm version sanity-io/ui and sanity-io/sanity run, and enables
the autoDedupe setting they both use (introduced in pnpm 12.6.0), which
collapses compatible duplicate versions on every non-frozen install.

The lockfile gains pnpm 12's packageManagerDependencies block pinning
pnpm@12.6.0 and its platform binaries. autoDedupe collapsed three redundant
duplicates that were already in the tree (@sanity/icons 5.2.1, @sanity/ui
4.0.7, motion 13.1.0); no dependency was upgraded. The shrunken
minimumReleaseAgeExclude and trustPolicyExclude lists are
minimumReleaseAgeExcludePrune / trustPolicyExcludePrune dropping entries
whose versions had already left the lockfile before this change.

(cherry picked from commit 13576dc)
Raise the documented minimum global pnpm to v12 now that packageManager pins
12.6.0, and record two things this upgrade surfaced: regenerating the lockfile
through an older pnpm shim silently skips the packageManagerDependencies block,
and the prune settings shrink the exclude lists once stale versions leave the
tree.

(cherry picked from commit e1738e6)
Verified on a throwaway worktree: pnpm 11.21.0 and pnpm 12.6.0 both leave the
exclude lists alone when an install has nothing to rewrite. Only the install
that actually rewrites pnpm-lock.yaml applies the prune.

(cherry picked from commit e4f2e4b)
vercel@58.9.5+ is the first CLI whose bundled @vercel/build-utils parses pnpm
12's two-document pnpm-lock.yaml. Every vercel release from 58.7.0 on, and its
@vercel/* dependencies, ship without npm provenance attestations (Vercel
rotated its trusted-publishing OIDC config; still GitHub trusted publishing,
same publisher), so trustPolicy: no-downgrade rejects them. List the exact
versions in trustPolicyExclude so trustPolicyExcludePrune retires them on the
next bump.

(cherry picked from commit a5b0f9b)
vercel@58.9.5 is the first CLI whose bundled @vercel/build-utils parses pnpm
12's two-document pnpm-lock.yaml; older releases fall back to npm, which fails
on `workspace:*`. Document that floor and the trustPolicyExclude upkeep each
CLI bump needs.

This is the net of db1f38e, a2dcefe and 2cbe482 from #2044, without the
interim installCommand workaround in dev/e2e-studio/vercel.json that 2cbe482
removed again. It also narrows the trust-policy wording: no-downgrade only
rejects versions whose trust evidence regressed, so only the versions pnpm
reports as ERR_PNPM_TRUST_DOWNGRADE belong in trustPolicyExclude.
Replace the composite setup action's pnpm/action-setup, actions/setup-node
and `pnpm install` steps with sanity-io/ui's action verbatim: pnpm/setup@v3
installs the pnpm version from `packageManager`, a `runtime` input that
defaults to node@lts, caches the pnpm store, and runs `pnpm install`.

pnpm/setup also caches pnpm's lockfile verification verdict, which pnpm 12
otherwise recomputes against the registry on every run.

ailf-eval passes `runtime: node@lts` instead of `node-version: lts/*`.
release.yml keeps pnpm/action-setup and actions/setup-node, as in
sanity-io/ui.
sanity-io/ui's pnpm-workspace.yaml sets it next to linkWorkspacePackages:
deep. It is pnpm's default, so installs and the lockfile don't change; this
only makes the workspace settings match.
The regenerated lockfile in this branch already pruned the stale
minimumReleaseAgeExclude and trustPolicyExclude entries.
@changeset-bot

changeset-bot Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: a7e1057

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@vercel

vercel Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
plugins-studio Ready Ready Preview Sep 29, 2026 1:31am UTC
1 Skipped Deployment
Project Deployment Actions Updated
plugins-e2e-test-studio Ignored Ignored Sep 29, 2026 1:31am UTC

Request Review

@socket-security

socket-security Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Updatednpm/​vercel@​57.0.0 ⏵ 60.1.397 +37100100 +199100

View full report

@github-actions

github-actions Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

✅ E2E Tests

🟢 43 passed • 🟡 1 flaky • view full report • view run

Studio: https://plugins-e2e-test-studio-pp9mdugs9.sanity.dev

Datasets: pr-2048-chromium-36508199128, pr-2048-firefox-36508199128

@github-actions

github-actions Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

Coverage Report

Status Category Percentage Covered / Total
🔵 Lines 33.32% 5811 / 17435
🔵 Statements 33.1% 6144 / 18558
🔵 Functions 29.4% 1537 / 5227
🔵 Branches 24.55% 3018 / 12290
File CoverageNo changed files found.
Generated in workflow #10195 for commit a7e1057 by the Vitest Coverage Report Action

@github-actions

github-actions Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

🔴 AI Literacy Score Report

Overall: 44/100 · Doc Lift: -5 · Actual: 43/100 · Ret. Gap: 1 · 112 tests · 13 errored · Cost: $39.80

Field Value
Report ID 01a0eacc-be12-770f-bddb-91011966de23
Completed Sep 29, 2026, 1:34 AM UTC
Duration 1m 2s
Mode full

♻️ Fully cached — Cost, tokens, and scores were served from the remote cache (source run run_20260929T002539Z_o01r4gl8); only runtime reflects this run.

🔧 Environment
Setting Value
Source production
Docs URL https://www.sanity.io/docs
Dataset next
Project 3do82whm
🔍 Source verification
Setting Value
Source https://www.sanity.io/docs
Mode full
Search open

Scores by Feature Area

Feature Score Grade Task Code Docs Doc Lift Tests
studio-presets 44 🔴 D 60 29 27 📉 -5 56

🚨 Negative Doc Lift: studio-presets (-5) — docs hurt performance. Floor: 49, Ceiling: 44

🔬 Three-Layer Decomposition

Feature Floor Ceiling Actual Doc Lift Retr. Gap Infra %
studio-presets 49 44 43 📉 -5 +1 —
📖 What do these numbers mean?
  • Floor: Model performance without any documentation (training data only)
  • Ceiling: Model performance with perfect documentation (hand-picked, injected)
  • Actual: Model performance when finding docs on its own (like real users)
  • Doc Lift: Documentation quality contribution (Ceiling − Floor)
  • Retr. Gap: Quality lost in discovery (Ceiling − Actual)
  • Infra %: Fraction of doc quality reaching agents (Actual ÷ Ceiling)
🤖 Per-model scores
Model Score Doc Lift Tests Cost
GPT 5.4 47 -6 7 $0.47
Claude Opus 5 46 -1 7 $1.77
GPT 5.6 Terra 45 -5 7 $0.32
Claude Opus 4.6 44 -5 7 $0.46
Claude Sonnet 4.6 44 -4 7 $0.40
Claude Fable 5 43 -4 7 $1.57
Claude Opus 4.8 41 -5 7 $0.77
GPT 5.2 0 -61 7 $0.01

GPT 5.4 (openai:responses:gpt-5.4):

Feature Score Task Code Docs Lift
studio-presets 47 59 36 34 -6

Claude Opus 5 (anthropic:messages:claude-opus-5):

Feature Score Task Code Docs Lift
studio-presets 46 64 26 31 -1

GPT 5.6 Terra (openai:responses:gpt-5.6-terra):

Feature Score Task Code Docs Lift
studio-presets 45 59 36 27 -5

Claude Opus 4.6 (anthropic:messages:claude-opus-4-6):

Feature Score Task Code Docs Lift
studio-presets 44 61 29 26 -5

Claude Sonnet 4.6 (anthropic:messages:claude-sonnet-4-6):

Feature Score Task Code Docs Lift
studio-presets 44 63 29 23 -4

Claude Fable 5 (anthropic:messages:claude-fable-5):

Feature Score Task Code Docs Lift
studio-presets 43 61 25 24 -4

Claude Opus 4.8 (anthropic:messages:claude-opus-4-8):

Feature Score Task Code Docs Lift
studio-presets 41 56 26 26 -5

GPT 5.2 (openai:chat:gpt-5.2):

Feature Score Task Code Docs Lift
studio-presets 0 0 0 0 -61

📊 Score Comparison

Overall delta: 0 (➡️, threshold ±2)

Feature Baseline Current Delta Actual Δ Ret. Gap Δ Infra Δ
studio-presets 44 44 ➡️ 0 0 0 —

➡️ 1 unchanged (threshold: ±2)

Dimension averages
Dimension Delta
CodeCorrectness 0
DocCoverage 0
TaskCompletion 0
Doc Lift 0
💰 Eval cost breakdown
Category Cost
Provider (model inference) $39.80
Total $39.80

Provider cost by feature area:

Feature Tests Cost Avg/Test
studio-presets 56 $39.80 $0.71
🔍 Low-Scoring Judgments (50 below 70)

Add CTA buttons to a hero section (baseline) (6)

🔴 0 · Code Correctness · Add CTA buttons to a hero section (baseline) · openai:responses:gpt-5.6-terra

The rubric requires CTA buttons to be produced by a defineCta function obtained from createPresetsRegistry(), and internal link targets declared via a to option on createPresetsRegistry or defineLink. The candidate uses hand-rolled object types with defineType/`define

🔴 0 · Code Correctness · Add CTA buttons to a hero section (baseline) · anthropic:messages:claude-opus-4-6

The rubric requires CTA buttons to be produced by a defineCta function obtained from createPresetsRegistry(), and internal link targets declared through a to option on createPresetsRegistry or defineLink. The candidate uses hand-rolled object types with defineField an

🔴 0 · Code Correctness · Add CTA buttons to a hero section (baseline) · anthropic:messages:claude-fable-5

The rubric requires CTA buttons to be produced by a defineCta function obtained from createPresetsRegistry(), and internal link targets declared via a to option on createPresetsRegistry or defineLink. The candidate uses hand-rolled object types with defineType/`define

🔴 0 · Code Correctness · Add CTA buttons to a hero section (baseline) · anthropic:messages:claude-sonnet-4-6

Response uses hand-rolled object types with defineType/defineField instead of required defineCta from createPresetsRegistry(). Links use manual 'url' field instead of 'to' option for internal link targets. Neither rubric criterion is met.

🔴 0 · Code Correctness · Add CTA buttons to a hero section (baseline) · anthropic:messages:claude-opus-4-8

The response uses hand-rolled object types with defineType/defineField for CTA buttons instead of the required defineCta function from createPresetsRegistry(). Internal links use a manual 'href' URL field rather than the required 'to' option pattern for declaring link targets. Ne

🔴 0 · Code Correctness · Add CTA buttons to a hero section (baseline) · anthropic:messages:claude-opus-5

The response uses hand-rolled object types with manual reference fields instead of the required defineCta/createPresetsRegistry pattern. Internal links use a manually constructed 'reference' field with 'to' array on the reference type, not the required 'to' option on defineLink o

Add CTA buttons to a hero section (gold) (7)

🔴 0 · Code Correctness · Add CTA buttons to a hero section (gold) · openai:responses:gpt-5.6-terra

The response uses hand-rolled object types with manual 'href' string fields for CTA buttons. It does not use defineCta from createPresetsRegistry() nor does it declare internal link targets through a 'to' option on defineLink or createPresetsRegistry. Both required patterns are c

🔴 0 · Code Correctness · Add CTA buttons to a hero section (gold) · openai:responses:gpt-5.4

The response uses hand-rolled object types with defineType/defineField for CTA buttons instead of the required defineCta function from createPresetsRegistry(). Internal link targets use a manual 'url' field type rather than the required 'to' option on defineLink or createPresetsR

🔴 0 · Code Correctness · Add CTA buttons to a hero section (gold) · anthropic:messages:claude-opus-4-6

Response uses hand-rolled object types with manual 'url' field instead of required defineCta from createPresetsRegistry(). No defineLink or 'to' option for internal links. Completely misses the required API patterns.

🔴 0 · Code Correctness · Add CTA buttons to a hero section (gold) · anthropic:messages:claude-opus-4-8

The rubric requires CTA buttons to be produced by a defineCta function from createPresetsRegistry() and internal link targets declared through a to option. The candidate uses hand-rolled object types with manual href fields instead, completely missing the required API pat

🔴 0 · Code Correctness · Add CTA buttons to a hero section (gold) · anthropic:messages:claude-fable-5

The response uses hand-rolled object types with defineField/defineArrayMember for actions, rather than using defineCta from createPresetsRegistry(). It also uses a manual 'url' field instead of declaring internal link targets through a 'to' option on defineLink or createPresetsRe

🔴 0 · Code Correctness · Add CTA buttons to a hero section (gold) · anthropic:messages:claude-sonnet-4-6

The response uses hand-rolled object types with defineType/defineField instead of the required defineCta function from createPresetsRegistry(). Internal link targets use a manual 'url' field instead of the required 'to' option pattern. Both core rubric criteria are completely unm

🔴 0 · Code Correctness · Add CTA buttons to a hero section (gold) · anthropic:messages:claude-opus-5

The rubric requires CTA buttons to be produced by a defineCta function from createPresetsRegistry() and internal links declared via a to option on createPresetsRegistry or defineLink. The candidate uses hand-rolled object types with defineType/defineField and manual

Add SEO metadata to a page (baseline) (2)

🔴 0 · Code Correctness · Add SEO metadata to a page (baseline) · openai:responses:gpt-5.6-terra

The rubric explicitly requires that the metadata field be produced by a defineSeo function obtained from a createPresetsRegistry() call. The candidate response hand-rolls SEO fields using defineField/defineType instead of using defineSeo from createPresetsRegistry(). This is

🔴 0 · Code Correctness · Add SEO metadata to a page (baseline) · anthropic:messages:claude-opus-5

The rubric explicitly requires that 'The metadata field is produced by a defineSeo function obtained from a createPresetsRegistry() call, rather than a hand-rolled object composing a title, a description, and an image.' The candidate response creates a hand-rolled 'seo' objec

Add SEO metadata to a page (gold) (1)

🔴 0 · Code Correctness · Add SEO metadata to a page (gold) · anthropic:messages:claude-opus-5

The rubric explicitly requires that the metadata field be produced by a defineSeo function obtained from a createPresetsRegistry() call. The candidate response hand-rolls an SEO object type using defineType/defineField and creates a custom plugin, rather than using the requir

Compose a full landing page schema (baseline) (6)

🔴 0 · Code Correctness · Compose a full landing page schema (baseline) · anthropic:messages:claude-opus-4-6

Response completely ignores the required preset-based architecture. Does not use createPresetsRegistry(), definePage, defineCta, defineRichText, or defineImage. Hand-rolls all content patterns (slug, SEO, CTA, rich text, image blocks) instead of using presets. The rubric explicit

🔴 0 · Code Correctness · Compose a full landing page schema (baseline) · anthropic:messages:claude-opus-4-8

Code works syntactically but completely ignores the required preset-based architecture. No createPresetsRegistry(), definePage, defineCta, defineRichText, or defineImage usage. Everything is hand-rolled: slug/SEO fields manually added instead of using definePage built-ins, CTA ma

🔴 0 · Code Correctness · Compose a full landing page schema (baseline) · anthropic:messages:claude-fable-5

The response completely ignores the required preset-based architecture. It does not use createPresetsRegistry(), definePage, defineCta, defineRichText, or defineImage. Instead, everything is hand-rolled: the page document, slug field, SEO fields, CTA button, rich text (textSectio

🔴 0 · Code Correctness · Compose a full landing page schema (baseline) · openai:responses:gpt-5.6-terra

Response completely ignores the required preset-based architecture. Does not use createPresetsRegistry(), definePage, defineCta, defineRichText, or defineImage. Hand-rolls all content patterns (Portable Text, CTA links, images, SEO, slug) instead of using built-in presets. Page s

🔴 0 · Code Correctness · Compose a full landing page schema (baseline) · anthropic:messages:claude-opus-5

The response completely ignores the required preset-based architecture. It hand-rolls all content patterns instead of using createPresetsRegistry() with definePage, defineCta, defineRichText, and defineImage. No pageBuilderBlocks option is used. Slug and SEO are manually defined

🔴 0 · Code Correctness · Compose a full landing page schema (baseline) · openai:responses:gpt-5.4

Code is syntactically correct and follows basic Sanity patterns, but completely ignores the required preset-based approach. No createPresetsRegistry(), definePage, defineCta, defineRichText, or defineImage presets are used. Everything is hand-rolled: slug, SEO, portable text, CTA

Compose a full landing page schema (gold) (7)

🔴 0 · Code Correctness · Compose a full landing page schema (gold) · anthropic:messages:claude-opus-4-8

Response completely ignores the required API pattern. Does not use createPresetsRegistry(), definePage, defineCta, defineRichText, or defineImage. Instead invents a non-existent 'presets()' plugin import from '@sanity/presets'. No hero defined with defineType. No pageBuilderBlock

🔴 0 · Code Correctness · Compose a full landing page schema (gold) · anthropic:messages:claude-opus-4-6

Response completely ignores the required preset-based architecture. Does not use createPresetsRegistry(), definePage, defineCta, defineRichText, or defineImage. Hand-rolls all content patterns (slug, SEO, CTA, rich text, image blocks) instead of using presets. Page sections are m

🔴 0 · Code Correctness · Compose a full landing page schema (gold) · anthropic:messages:claude-sonnet-4-6

Response completely ignores the required preset-based architecture. Does not use createPresetsRegistry(), definePage, defineCta, defineRichText, or defineImage. Instead hand-rolls all content patterns from scratch including slug, SEO, rich text, CTA button, and image blocks.

🔴 0 · Code Correctness · Compose a full landing page schema (gold) · openai:responses:gpt-5.4

The response hand-rolls every content pattern from scratch. It does not call createPresetsRegistry(), does not use definePage with pageBuilderBlocks, does not use defineRichText, defineCta, or defineImage. Slug and SEO are manually added fields. The hero button is hand-rolled. No

🔴 0 · Code Correctness · Compose a full landing page schema (gold) · anthropic:messages:claude-fable-5

Response completely ignores the required preset-based approach. Does not use createPresetsRegistry(), definePage, defineCta, defineRichText, or defineImage. Hand-rolls all content patterns (SEO, slug, rich text, CTA button, image with caption) instead of using the documented pres

🔴 0 · Code Correctness · Compose a full landing page schema (gold) · openai:responses:gpt-5.6-terra

The response hand-rolls every content pattern from scratch instead of using the required preset APIs. No createPresetsRegistry() call, no definePage with pageBuilderBlocks, no defineRichText, no defineCta, no defineImage. Slug and SEO are manually added fields rather than coming

🔴 0 · Code Correctness · Compose a full landing page schema (gold) · anthropic:messages:claude-opus-5

The response completely ignores the required preset-based approach. It hand-rolls all content patterns (richText, link/CTA, image, SEO, page sections) instead of using createPresetsRegistry() with definePage, defineCta, defineRichText, and defineImage. No presets are used at all.

Configure an editorial image with crop control and caption (gold) (1)

🔴 0 · Code Correctness · Configure an editorial image with crop control and caption (gold) · anthropic:messages:claude-opus-4-8

The rubric explicitly requires that coverImage be produced by a defineImage function obtained from a createPresetsRegistry() call. The candidate uses a hand-rolled defineField({type: 'image', options: {hotspot: true}, fields: [...]}) approach instead, which is exactly what

Create Studio with page preset (baseline) (9)

🔴 0 · Task Completion · Create Studio with page preset (baseline) · anthropic:messages:claude-opus-4-6

The response does not use createPresetsRegistry from @sanity/presets or call definePage with name: 'page'. Instead, it invents a non-existent pageType import from @sanity/presets/page-type. None of the required rubric criteria are met.

🔴 0 · Doc Coverage · Create Studio with page preset (baseline) · anthropic:messages:claude-opus-4-6

Candidate uses defineConfig and plugins correctly per docs, but the core implementation relies on @sanity/presets/page-type (pageType) which is not documented in the reference at all. This is a significant hallucination of the main feature. Only basic config structure (defineConf

🔴 0 · Task Completion · Create Studio with page preset (baseline) · openai:responses:gpt-5.6-terra

Response does not use createPresetsRegistry from @sanity/presets, does not call definePage with name: 'page', and imports a non-existent pageType. The approach is fundamentally wrong - it doesn't match any of the required patterns from the rubric.

🔴 0 · Doc Coverage · Create Studio with page preset (baseline) · openai:responses:gpt-5.6-terra

Candidate correctly used defineConfig which is documented. However, @sanity/presets and pageType are completely fabricated - not mentioned anywhere in the reference. The reference discusses plugins, definePlugin, and @sanity/plugin-kit but candidate invented a non-existent

🔴 0 · Task Completion · Create Studio with page preset (baseline) · anthropic:messages:claude-opus-4-8

Candidate explicitly refused to implement the required API, stating @sanity/presets doesn't exist. Did not call definePage with name:'page', did not import createPresetsRegistry from @sanity/presets, and did not include definePage result in schema.types. Used alternative approach

🔴 0 · Task Completion · Create Studio with page preset (baseline) · anthropic:messages:claude-fable-5

The candidate explicitly refuses to use the required API (@sanity/presets, createPresetsRegistry, definePage), claiming it's 'hallucinated'. None of the 4 rubric criteria are met: no definePage call, no createPresetsRegistry import, no preset-based entry in schema.types

🔴 0 · Task Completion · Create Studio with page preset (baseline) · anthropic:messages:claude-sonnet-4-6

The response completely ignores the required rubric criteria. It does not use createPresetsRegistry from @sanity/presets, does not call definePage with name: 'page', and instead provides a manual schema implementation using defineType. None of the 4 required criteria ar

🔴 0 · Doc Coverage · Create Studio with page preset (baseline) · openai:responses:gpt-5.4

Candidate correctly uses defineConfig and plugins from docs, but invents @sanity/presets package and pageType import which are not documented. structureTool is also not in the reference. The core task requirement (page preset) relies entirely on hallucinated imports. Only basic c

🔴 0 · Task Completion · Create Studio with page preset (baseline) · anthropic:messages:claude-opus-5

Candidate explicitly refused to implement the required API, stating '@sanity/presets' doesn't exist. Did not call definePage with name:'page', did not import createPresetsRegistry from @sanity/presets, and did not include definePage result in schema.types. Instead provided altern

Create Studio with page preset (gold) (9)

🔴 0 · Task Completion · Create Studio with page preset (gold) · anthropic:messages:claude-opus-4-6

The candidate response does not use createPresetsRegistry from @sanity/presets, does not call definePage with name: 'page', and does not place the result in schema.types. Instead, it invents a presets plugin function that is not in the reference. None of the rubric cr

🔴 0 · Task Completion · Create Studio with page preset (gold) · anthropic:messages:claude-sonnet-4-6

The rubric requires: (1) definePage called with name:'page', (2) createPresetsRegistry imported from @sanity/presets and called to obtain definePage, (3) definePage result in schema.types, (4) valid Studio config export. Candidate imports pageType from @sanity/presets/page and ad

🔴 0 · Task Completion · Create Studio with page preset (gold) · anthropic:messages:claude-opus-4-8

Response does not use createPresetsRegistry or definePage at all. Instead invents a non-existent pageType plugin from @sanity/presets. None of the four rubric criteria are met: no definePage call, no createPresetsRegistry import, no schema.types entry from definePage, and the exp

🔴 0 · Doc Coverage · Create Studio with page preset (gold) · openai:responses:gpt-5.4

Candidate correctly uses defineConfig and plugins array from docs. However, @sanity/presets and pagePreset() are not documented in the reference - these are hallucinated/assumed. The core plugin installation pattern matches docs, but the specific preset implementation details had

🔴 0 · Task Completion · Create Studio with page preset (gold) · anthropic:messages:claude-fable-5

Response does not use createPresetsRegistry or definePage at all. Instead invents a non-existent pageType plugin pattern. None of the four rubric criteria are met: no definePage call, no createPresetsRegistry import, no schema.types entry from definePage, and the exported config

🔴 0 · Doc Coverage · Create Studio with page preset (gold) · anthropic:messages:claude-fable-5

Candidate correctly used defineConfig and plugins from docs. However, @sanity/presets, pageType, and all claims about what the preset provides (page builder, slug field, SEO metadata, rich text, etc.) are entirely fabricated - none of this appears in the reference. The re

🔴 0 · Task Completion · Create Studio with page preset (gold) · openai:responses:gpt-5.6-terra

Response does not use createPresetsRegistry or definePage at all. Uses non-existent pageType import from @sanity/presets. No criteria from rubric are met - missing createPresetsRegistry import, missing definePage call with name:'page', and schema.types is empty instead of contain

🔴 0 · Doc Coverage · Create Studio with page preset (gold) · openai:responses:gpt-5.6-terra

Candidate correctly uses documented defineConfig and plugins array pattern. However, @sanity/presets and pageType() are not documented in the reference at all - these are hallucinated/assumed. The core plugin installation pattern is correct but the specific preset impleme

🔴 0 · Task Completion · Create Studio with page preset (gold) · anthropic:messages:claude-opus-5

Response does not use createPresetsRegistry from @sanity/presets or call definePage with name: 'page'. Instead, it guesses at pageType import and provides a fallback with manual schema types. None of the required rubric criteria are met.

Extend a page document with a campaign category (gold) (2)

🔴 0 · Code Correctness · Extend a page document with a campaign category (gold) · anthropic:messages:claude-opus-4-6

The candidate imports createPresetsRegistry from @sanity/presets and calls createPresetsRegistry().definePage(...). Neither @sanity/presets nor definePage appear in the authoritative reference. The reference only documents defineConfig, definePlugin, kit, `plugins

🔴 0 · Code Correctness · Extend a page document with a campaign category (gold) · anthropic:messages:claude-sonnet-4-6

The response imports createPresetsRegistry from '@sanity/presets' which is not documented in the reference. The definePage function is fabricated - there is no evidence in the reference that such an API exists. The reference only documents defineConfig, definePlugin, `plu

💡 Recommendations

  • 🔴 studio-presets (score: 44) — needs significant doc improvements.

Generated by AI Literacy Framework · Sep 29, 2026, 1:34 AM UTC · view detailed results


Previous runs

Sep 29, 2026, 1:25 AM UTC - 44/100

🔴 AI Literacy Score Report

Overall: 44/100 · Doc Lift: -5 · Actual: 43/100 · Ret. Gap: 1 · 112 tests · 13 errored · Cost: $39.80

Field Value
Report ID 01a0eac4-3514-79b5-ae7f-11f0a81aa874
Completed Sep 29, 2026, 1:25 AM UTC
Duration 57.4s
Mode full

♻️ Fully cached — Cost, tokens, and scores were served from the remote cache (source run run_20260929T002539Z_o01r4gl8); only runtime reflects this run.

🔧 Environment
Setting Value
Source production
Docs URL https://www.sanity.io/docs
Dataset next
Project 3do82whm
🔍 Source verification
Setting Value
Source https://www.sanity.io/docs
Mode full
Search open

Scores by Feature Area

Feature Score Grade Task Code Docs Doc Lift Tests
studio-presets 44 🔴 D 60 29 27 📉 -5 56

🚨 Negative Doc Lift: studio-presets (-5) — docs hurt performance. Floor: 49, Ceiling: 44

🔬 Three-Layer Decomposition

Feature Floor Ceiling Actual Doc Lift Retr. Gap Infra %
studio-presets 49 44 43 📉 -5 +1 —
📖 What do these numbers mean?
  • Floor: Model performance without any documentation (training data only)
  • Ceiling: Model performance with perfect documentation (hand-picked, injected)
  • Actual: Model performance when finding docs on its own (like real users)
  • Doc Lift: Documentation quality contribution (Ceiling − Floor)
  • Retr. Gap: Quality lost in discovery (Ceiling − Actual)
  • Infra %: Fraction of doc quality reaching agents (Actual ÷ Ceiling)
🤖 Per-model scores
Model Score Doc Lift Tests Cost
GPT 5.4 47 -6 7 $0.47
Claude Opus 5 46 -1 7 $1.77
GPT 5.6 Terra 45 -5 7 $0.32
Claude Opus 4.6 44 -5 7 $0.46
Claude Sonnet 4.6 44 -4 7 $0.40
Claude Fable 5 43 -4 7 $1.57
Claude Opus 4.8 41 -5 7 $0.77
GPT 5.2 0 -61 7 $0.01

GPT 5.4 (openai:responses:gpt-5.4):

Feature Score Task Code Docs Lift
studio-presets 47 59 36 34 -6

Claude Opus 5 (anthropic:messages:claude-opus-5):

Feature Score Task Code Docs Lift
studio-presets 46 64 26 31 -1

GPT 5.6 Terra (openai:responses:gpt-5.6-terra):

Feature Score Task Code Docs Lift
studio-presets 45 59 36 27 -5

Claude Opus 4.6 (anthropic:messages:claude-opus-4-6):

Feature Score Task Code Docs Lift
studio-presets 44 61 29 26 -5

Claude Sonnet 4.6 (anthropic:messages:claude-sonnet-4-6):

Feature Score Task Code Docs Lift
studio-presets 44 63 29 23 -4

Claude Fable 5 (anthropic:messages:claude-fable-5):

Feature Score Task Code Docs Lift
studio-presets 43 61 25 24 -4

Claude Opus 4.8 (anthropic:messages:claude-opus-4-8):

Feature Score Task Code Docs Lift
studio-presets 41 56 26 26 -5

GPT 5.2 (openai:chat:gpt-5.2):

Feature Score Task Code Docs Lift
studio-presets 0 0 0 0 -61

📊 Score Comparison

Overall delta: 0 (➡️, threshold ±2)

Feature Baseline Current Delta Actual Δ Ret. Gap Δ Infra Δ
studio-presets 44 44 ➡️ 0 0 0 —

➡️ 1 unchanged (threshold: ±2)

Dimension averages
Dimension Delta
CodeCorrectness 0
DocCoverage 0
TaskCompletion 0
Doc Lift 0
💰 Eval cost breakdown
Category Cost
Provider (model inference) $39.80
Total $39.80

Provider cost by feature area:

Feature Tests Cost Avg/Test
studio-presets 56 $39.80 $0.71
🔍 Low-Scoring Judgments (50 below 70)

Add CTA buttons to a hero section (baseline) (6)

🔴 0 · Code Correctness · Add CTA buttons to a hero section (baseline) · openai:responses:gpt-5.6-terra

The rubric requires CTA buttons to be produced by a defineCta function obtained from createPresetsRegistry(), and internal link targets declared via a to option on createPresetsRegistry or defineLink. The candidate uses hand-rolled object types with defineType/`define

🔴 0 · Code Correctness · Add CTA buttons to a hero section (baseline) · anthropic:messages:claude-opus-4-6

The rubric requires CTA buttons to be produced by a defineCta function obtained from createPresetsRegistry(), and internal link targets declared through a to option on createPresetsRegistry or defineLink. The candidate uses hand-rolled object types with defineField an

🔴 0 · Code Correctness · Add CTA buttons to a hero section (baseline) · anthropic:messages:claude-fable-5

The rubric requires CTA buttons to be produced by a defineCta function obtained from createPresetsRegistry(), and internal link targets declared via a to option on createPresetsRegistry or defineLink. The candidate uses hand-rolled object types with defineType/`define

🔴 0 · Code Correctness · Add CTA buttons to a hero section (baseline) · anthropic:messages:claude-sonnet-4-6

Response uses hand-rolled object types with defineType/defineField instead of required defineCta from createPresetsRegistry(). Links use manual 'url' field instead of 'to' option for internal link targets. Neither rubric criterion is met.

🔴 0 · Code Correctness · Add CTA buttons to a hero section (baseline) · anthropic:messages:claude-opus-4-8

The response uses hand-rolled object types with defineType/defineField for CTA buttons instead of the required defineCta function from createPresetsRegistry(). Internal links use a manual 'href' URL field rather than the required 'to' option pattern for declaring link targets. Ne

🔴 0 · Code Correctness · Add CTA buttons to a hero section (baseline) · anthropic:messages:claude-opus-5

The response uses hand-rolled object types with manual reference fields instead of the required defineCta/createPresetsRegistry pattern. Internal links use a manually constructed 'reference' field with 'to' array on the reference type, not the required 'to' option on defineLink o

Add CTA buttons to a hero section (gold) (7)

🔴 0 · Code Correctness · Add CTA buttons to a hero section (gold) · openai:responses:gpt-5.6-terra

The response uses hand-rolled object types with manual 'href' string fields for CTA buttons. It does not use defineCta from createPresetsRegistry() nor does it declare internal link targets through a 'to' option on defineLink or createPresetsRegistry. Both required patterns are c

🔴 0 · Code Correctness · Add CTA buttons to a hero section (gold) · openai:responses:gpt-5.4

The response uses hand-rolled object types with defineType/defineField for CTA buttons instead of the required defineCta function from createPresetsRegistry(). Internal link targets use a manual 'url' field type rather than the required 'to' option on defineLink or createPresetsR

🔴 0 · Code Correctness · Add CTA buttons to a hero section (gold) · anthropic:messages:claude-opus-4-6

Response uses hand-rolled object types with manual 'url' field instead of required defineCta from createPresetsRegistry(). No defineLink or 'to' option for internal links. Completely misses the required API patterns.

🔴 0 · Code Correctness · Add CTA buttons to a hero section (gold) · anthropic:messages:claude-opus-4-8

The rubric requires CTA buttons to be produced by a defineCta function from createPresetsRegistry() and internal link targets declared through a to option. The candidate uses hand-rolled object types with manual href fields instead, completely missing the required API pat

🔴 0 · Code Correctness · Add CTA buttons to a hero section (gold) · anthropic:messages:claude-fable-5

The response uses hand-rolled object types with defineField/defineArrayMember for actions, rather than using defineCta from createPresetsRegistry(). It also uses a manual 'url' field instead of declaring internal link targets through a 'to' option on defineLink or createPresetsRe

🔴 0 · Code Correctness · Add CTA buttons to a hero section (gold) · anthropic:messages:claude-sonnet-4-6

The response uses hand-rolled object types with defineType/defineField instead of the required defineCta function from createPresetsRegistry(). Internal link targets use a manual 'url' field instead of the required 'to' option pattern. Both core rubric criteria are completely unm

🔴 0 · Code Correctness · Add CTA buttons to a hero section (gold) · anthropic:messages:claude-opus-5

The rubric requires CTA buttons to be produced by a defineCta function from createPresetsRegistry() and internal links declared via a to option on createPresetsRegistry or defineLink. The candidate uses hand-rolled object types with defineType/defineField and manual

Add SEO metadata to a page (baseline) (2)

🔴 0 · Code Correctness · Add SEO metadata to a page (baseline) · openai:responses:gpt-5.6-terra

The rubric explicitly requires that the metadata field be produced by a defineSeo function obtained from a createPresetsRegistry() call. The candidate response hand-rolls SEO fields using defineField/defineType instead of using defineSeo from createPresetsRegistry(). This is

🔴 0 · Code Correctness · Add SEO metadata to a page (baseline) · anthropic:messages:claude-opus-5

The rubric explicitly requires that 'The metadata field is produced by a defineSeo function obtained from a createPresetsRegistry() call, rather than a hand-rolled object composing a title, a description, and an image.' The candidate response creates a hand-rolled 'seo' objec

Add SEO metadata to a page (gold) (1)

🔴 0 · Code Correctness · Add SEO metadata to a page (gold) · anthropic:messages:claude-opus-5

The rubric explicitly requires that the metadata field be produced by a defineSeo function obtained from a createPresetsRegistry() call. The candidate response hand-rolls an SEO object type using defineType/defineField and creates a custom plugin, rather than using the requir

Compose a full landing page schema (baseline) (6)

🔴 0 · Code Correctness · Compose a full landing page schema (baseline) · anthropic:messages:claude-opus-4-6

Response completely ignores the required preset-based architecture. Does not use createPresetsRegistry(), definePage, defineCta, defineRichText, or defineImage. Hand-rolls all content patterns (slug, SEO, CTA, rich text, image blocks) instead of using presets. The rubric explicit

🔴 0 · Code Correctness · Compose a full landing page schema (baseline) · anthropic:messages:claude-opus-4-8

Code works syntactically but completely ignores the required preset-based architecture. No createPresetsRegistry(), definePage, defineCta, defineRichText, or defineImage usage. Everything is hand-rolled: slug/SEO fields manually added instead of using definePage built-ins, CTA ma

🔴 0 · Code Correctness · Compose a full landing page schema (baseline) · anthropic:messages:claude-fable-5

The response completely ignores the required preset-based architecture. It does not use createPresetsRegistry(), definePage, defineCta, defineRichText, or defineImage. Instead, everything is hand-rolled: the page document, slug field, SEO fields, CTA button, rich text (textSectio

🔴 0 · Code Correctness · Compose a full landing page schema (baseline) · openai:responses:gpt-5.6-terra

Response completely ignores the required preset-based architecture. Does not use createPresetsRegistry(), definePage, defineCta, defineRichText, or defineImage. Hand-rolls all content patterns (Portable Text, CTA links, images, SEO, slug) instead of using built-in presets. Page s

🔴 0 · Code Correctness · Compose a full landing page schema (baseline) · anthropic:messages:claude-opus-5

The response completely ignores the required preset-based architecture. It hand-rolls all content patterns instead of using createPresetsRegistry() with definePage, defineCta, defineRichText, and defineImage. No pageBuilderBlocks option is used. Slug and SEO are manually defined

🔴 0 · Code Correctness · Compose a full landing page schema (baseline) · openai:responses:gpt-5.4

Code is syntactically correct and follows basic Sanity patterns, but completely ignores the required preset-based approach. No createPresetsRegistry(), definePage, defineCta, defineRichText, or defineImage presets are used. Everything is hand-rolled: slug, SEO, portable text, CTA

Compose a full landing page schema (gold) (7)

🔴 0 · Code Correctness · Compose a full landing page schema (gold) · anthropic:messages:claude-opus-4-8

Response completely ignores the required API pattern. Does not use createPresetsRegistry(), definePage, defineCta, defineRichText, or defineImage. Instead invents a non-existent 'presets()' plugin import from '@sanity/presets'. No hero defined with defineType. No pageBuilderBlock

🔴 0 · Code Correctness · Compose a full landing page schema (gold) · anthropic:messages:claude-opus-4-6

Response completely ignores the required preset-based architecture. Does not use createPresetsRegistry(), definePage, defineCta, defineRichText, or defineImage. Hand-rolls all content patterns (slug, SEO, CTA, rich text, image blocks) instead of using presets. Page sections are m

🔴 0 · Code Correctness · Compose a full landing page schema (gold) · anthropic:messages:claude-sonnet-4-6

Response completely ignores the required preset-based architecture. Does not use createPresetsRegistry(), definePage, defineCta, defineRichText, or defineImage. Instead hand-rolls all content patterns from scratch including slug, SEO, rich text, CTA button, and image blocks.

🔴 0 · Code Correctness · Compose a full landing page schema (gold) · openai:responses:gpt-5.4

The response hand-rolls every content pattern from scratch. It does not call createPresetsRegistry(), does not use definePage with pageBuilderBlocks, does not use defineRichText, defineCta, or defineImage. Slug and SEO are manually added fields. The hero button is hand-rolled. No

🔴 0 · Code Correctness · Compose a full landing page schema (gold) · anthropic:messages:claude-fable-5

Response completely ignores the required preset-based approach. Does not use createPresetsRegistry(), definePage, defineCta, defineRichText, or defineImage. Hand-rolls all content patterns (SEO, slug, rich text, CTA button, image with caption) instead of using the documented pres

🔴 0 · Code Correctness · Compose a full landing page schema (gold) · openai:responses:gpt-5.6-terra

The response hand-rolls every content pattern from scratch instead of using the required preset APIs. No createPresetsRegistry() call, no definePage with pageBuilderBlocks, no defineRichText, no defineCta, no defineImage. Slug and SEO are manually added fields rather than coming

🔴 0 · Code Correctness · Compose a full landing page schema (gold) · anthropic:messages:claude-opus-5

The response completely ignores the required preset-based approach. It hand-rolls all content patterns (richText, link/CTA, image, SEO, page sections) instead of using createPresetsRegistry() with definePage, defineCta, defineRichText, and defineImage. No presets are used at all.

Configure an editorial image with crop control and caption (gold) (1)

🔴 0 · Code Correctness · Configure an editorial image with crop control and caption (gold) · anthropic:messages:claude-opus-4-8

The rubric explicitly requires that coverImage be produced by a defineImage function obtained from a createPresetsRegistry() call. The candidate uses a hand-rolled defineField({type: 'image', options: {hotspot: true}, fields: [...]}) approach instead, which is exactly what

Create Studio with page preset (baseline) (9)

🔴 0 · Task Completion · Create Studio with page preset (baseline) · anthropic:messages:claude-opus-4-6

The response does not use createPresetsRegistry from @sanity/presets or call definePage with name: 'page'. Instead, it invents a non-existent pageType import from @sanity/presets/page-type. None of the required rubric criteria are met.

🔴 0 · Doc Coverage · Create Studio with page preset (baseline) · anthropic:messages:claude-opus-4-6

Candidate uses defineConfig and plugins correctly per docs, but the core implementation relies on @sanity/presets/page-type (pageType) which is not documented in the reference at all. This is a significant hallucination of the main feature. Only basic config structure (defineConf

🔴 0 · Task Completion · Create Studio with page preset (baseline) · openai:responses:gpt-5.6-terra

Response does not use createPresetsRegistry from @sanity/presets, does not call definePage with name: 'page', and imports a non-existent pageType. The approach is fundamentally wrong - it doesn't match any of the required patterns from the rubric.

🔴 0 · Doc Coverage · Create Studio with page preset (baseline) · openai:responses:gpt-5.6-terra

Candidate correctly used defineConfig which is documented. However, @sanity/presets and pageType are completely fabricated - not mentioned anywhere in the reference. The reference discusses plugins, definePlugin, and @sanity/plugin-kit but candidate invented a non-existent

🔴 0 · Task Completion · Create Studio with page preset (baseline) · anthropic:messages:claude-opus-4-8

Candidate explicitly refused to implement the required API, stating @sanity/presets doesn't exist. Did not call definePage with name:'page', did not import createPresetsRegistry from @sanity/presets, and did not include definePage result in schema.types. Used alternative approach

🔴 0 · Task Completion · Create Studio with page preset (baseline) · anthropic:messages:claude-fable-5

The candidate explicitly refuses to use the required API (@sanity/presets, createPresetsRegistry, definePage), claiming it's 'hallucinated'. None of the 4 rubric criteria are met: no definePage call, no createPresetsRegistry import, no preset-based entry in schema.types

🔴 0 · Task Completion · Create Studio with page preset (baseline) · anthropic:messages:claude-sonnet-4-6

The response completely ignores the required rubric criteria. It does not use createPresetsRegistry from @sanity/presets, does not call definePage with name: 'page', and instead provides a manual schema implementation using defineType. None of the 4 required criteria ar

🔴 0 · Doc Coverage · Create Studio with page preset (baseline) · openai:responses:gpt-5.4

Candidate correctly uses defineConfig and plugins from docs, but invents @sanity/presets package and pageType import which are not documented. structureTool is also not in the reference. The core task requirement (page preset) relies entirely on hallucinated imports. Only basic c

🔴 0 · Task Completion · Create Studio with page preset (baseline) · anthropic:messages:claude-opus-5

Candidate explicitly refused to implement the required API, stating '@sanity/presets' doesn't exist. Did not call definePage with name:'page', did not import createPresetsRegistry from @sanity/presets, and did not include definePage result in schema.types. Instead provided altern

Create Studio with page preset (gold) (9)

🔴 0 · Task Completion · Create Studio with page preset (gold) · anthropic:messages:claude-opus-4-6

The candidate response does not use createPresetsRegistry from @sanity/presets, does not call definePage with name: 'page', and does not place the result in schema.types. Instead, it invents a presets plugin function that is not in the reference. None of the rubric cr

🔴 0 · Task Completion · Create Studio with page preset (gold) · anthropic:messages:claude-sonnet-4-6

The rubric requires: (1) definePage called with name:'page', (2) createPresetsRegistry imported from @sanity/presets and called to obtain definePage, (3) definePage result in schema.types, (4) valid Studio config export. Candidate imports pageType from @sanity/presets/page and ad

🔴 0 · Task Completion · Create Studio with page preset (gold) · anthropic:messages:claude-opus-4-8

Response does not use createPresetsRegistry or definePage at all. Instead invents a non-existent pageType plugin from @sanity/presets. None of the four rubric criteria are met: no definePage call, no createPresetsRegistry import, no schema.types entry from definePage, and the exp

🔴 0 · Doc Coverage · Create Studio with page preset (gold) · openai:responses:gpt-5.4

Candidate correctly uses defineConfig and plugins array from docs. However, @sanity/presets and pagePreset() are not documented in the reference - these are hallucinated/assumed. The core plugin installation pattern matches docs, but the specific preset implementation details had

🔴 0 · Task Completion · Create Studio with page preset (gold) · anthropic:messages:claude-fable-5

Response does not use createPresetsRegistry or definePage at all. Instead invents a non-existent pageType plugin pattern. None of the four rubric criteria are met: no definePage call, no createPresetsRegistry import, no schema.types entry from definePage, and the exported config

🔴 0 · Doc Coverage · Create Studio with page preset (gold) · anthropic:messages:claude-fable-5

Candidate correctly used defineConfig and plugins from docs. However, @sanity/presets, pageType, and all claims about what the preset provides (page builder, slug field, SEO metadata, rich text, etc.) are entirely fabricated - none of this appears in the reference. The re

🔴 0 · Task Completion · Create Studio with page preset (gold) · openai:responses:gpt-5.6-terra

Response does not use createPresetsRegistry or definePage at all. Uses non-existent pageType import from @sanity/presets. No criteria from rubric are met - missing createPresetsRegistry import, missing definePage call with name:'page', and schema.types is empty instead of contain

🔴 0 · Doc Coverage · Create Studio with page preset (gold) · openai:responses:gpt-5.6-terra

Candidate correctly uses documented defineConfig and plugins array pattern. However, @sanity/presets and pageType() are not documented in the reference at all - these are hallucinated/assumed. The core plugin installation pattern is correct but the specific preset impleme

🔴 0 · Task Completion · Create Studio with page preset (gold) · anthropic:messages:claude-opus-5

Response does not use createPresetsRegistry from @sanity/presets or call definePage with name: 'page'. Instead, it guesses at pageType import and provides a fallback with manual schema types. None of the required rubric criteria are met.

Extend a page document with a campaign category (gold) (2)

🔴 0 · Code Correctness · Extend a page document with a campaign category (gold) · anthropic:messages:claude-opus-4-6

The candidate imports createPresetsRegistry from @sanity/presets and calls createPresetsRegistry().definePage(...). Neither @sanity/presets nor definePage appear in the authoritative reference. The reference only documents defineConfig, definePlugin, kit, `plugins

🔴 0 · Code Correctness · Extend a page document with a campaign category (gold) · anthropic:messages:claude-sonnet-4-6

The response imports createPresetsRegistry from '@sanity/presets' which is not documented in the reference. The definePage function is fabricated - there is no evidence in the reference that such an API exists. The reference only documents defineConfig, definePlugin, `plu

💡 Recommendations

  • 🔴 studio-presets (score: 44) — needs significant doc improvements.

Generated by AI Literacy Framework · Sep 29, 2026, 1:25 AM UTC · view detailed results

Sep 29, 2026, 12:37 AM UTC - 44/100

🔴 AI Literacy Score Report

Overall: 44/100 · Doc Lift: -5 · Actual: 43/100 · Ret. Gap: 1 · 112 tests · 13 errored · Cost: $39.80

Field Value
Report ID 01a0ea98-1415-7a0b-ba65-46816893772b
Completed Sep 29, 2026, 12:37 AM UTC
Duration 11m 31s
Mode full
🔧 Environment
Setting Value
Source production
Docs URL https://www.sanity.io/docs
Dataset next
Project 3do82whm
🔍 Source verification
Setting Value
Source https://www.sanity.io/docs
Mode full
Search open

Scores by Feature Area

Feature Score Grade Task Code Docs Doc Lift Tests
studio-presets 44 🔴 D 60 29 27 📉 -5 56

🚨 Negative Doc Lift: studio-presets (-5) — docs hurt performance. Floor: 49, Ceiling: 44

🔬 Three-Layer Decomposition

Feature Floor Ceiling Actual Doc Lift Retr. Gap Infra %
studio-presets 49 44 43 📉 -5 +1 —
📖 What do these numbers mean?
  • Floor: Model performance without any documentation (training data only)
  • Ceiling: Model performance with perfect documentation (hand-picked, injected)
  • Actual: Model performance when finding docs on its own (like real users)
  • Doc Lift: Documentation quality contribution (Ceiling − Floor)
  • Retr. Gap: Quality lost in discovery (Ceiling − Actual)
  • Infra %: Fraction of doc quality reaching agents (Actual ÷ Ceiling)
🤖 Per-model scores
Model Score Doc Lift Tests Cost
GPT 5.4 47 -6 7 $0.47
Claude Opus 5 46 -1 7 $1.77
GPT 5.6 Terra 45 -5 7 $0.32
Claude Opus 4.6 44 -5 7 $0.46
Claude Sonnet 4.6 44 -4 7 $0.40
Claude Fable 5 43 -4 7 $1.57
Claude Opus 4.8 41 -5 7 $0.77
GPT 5.2 0 -61 7 $0.01

GPT 5.4 (openai:responses:gpt-5.4):

Feature Score Task Code Docs Lift
studio-presets 47 59 36 34 -6

Claude Opus 5 (anthropic:messages:claude-opus-5):

Feature Score Task Code Docs Lift
studio-presets 46 64 26 31 -1

GPT 5.6 Terra (openai:responses:gpt-5.6-terra):

Feature Score Task Code Docs Lift
studio-presets 45 59 36 27 -5

Claude Opus 4.6 (anthropic:messages:claude-opus-4-6):

Feature Score Task Code Docs Lift
studio-presets 44 61 29 26 -5

Claude Sonnet 4.6 (anthropic:messages:claude-sonnet-4-6):

Feature Score Task Code Docs Lift
studio-presets 44 63 29 23 -4

Claude Fable 5 (anthropic:messages:claude-fable-5):

Feature Score Task Code Docs Lift
studio-presets 43 61 25 24 -4

Claude Opus 4.8 (anthropic:messages:claude-opus-4-8):

Feature Score Task Code Docs Lift
studio-presets 41 56 26 26 -5

GPT 5.2 (openai:chat:gpt-5.2):

Feature Score Task Code Docs Lift
studio-presets 0 0 0 0 -61

📊 Score Comparison

Overall delta: +1 (➡️, threshold ±2)

Feature Baseline Current Delta Actual Δ Ret. Gap Δ Infra Δ
studio-presets 43 44 ➡️ +1 +2 -1 —

➡️ 1 unchanged (threshold: ±2)

Dimension averages
Dimension Delta
CodeCorrectness +1
DocCoverage +1
TaskCompletion +2
Doc Lift 0
💰 Eval cost breakdown
Category Cost
Provider (model inference) $39.80
Total $39.80

Provider cost by feature area:

Feature Tests Cost Avg/Test
studio-presets 56 $39.80 $0.71
🔍 Low-Scoring Judgments (50 below 70)

Add CTA buttons to a hero section (baseline) (6)

🔴 0 · Code Correctness · Add CTA buttons to a hero section (baseline) · openai:responses:gpt-5.6-terra

The rubric requires CTA buttons to be produced by a defineCta function obtained from createPresetsRegistry(), and internal link targets declared via a to option on createPresetsRegistry or defineLink. The candidate uses hand-rolled object types with defineType/`define

🔴 0 · Code Correctness · Add CTA buttons to a hero section (baseline) · anthropic:messages:claude-opus-4-6

The rubric requires CTA buttons to be produced by a defineCta function obtained from createPresetsRegistry(), and internal link targets declared through a to option on createPresetsRegistry or defineLink. The candidate uses hand-rolled object types with defineField an

🔴 0 · Code Correctness · Add CTA buttons to a hero section (baseline) · anthropic:messages:claude-fable-5

The rubric requires CTA buttons to be produced by a defineCta function obtained from createPresetsRegistry(), and internal link targets declared via a to option on createPresetsRegistry or defineLink. The candidate uses hand-rolled object types with defineType/`define

🔴 0 · Code Correctness · Add CTA buttons to a hero section (baseline) · anthropic:messages:claude-sonnet-4-6

Response uses hand-rolled object types with defineType/defineField instead of required defineCta from createPresetsRegistry(). Links use manual 'url' field instead of 'to' option for internal link targets. Neither rubric criterion is met.

🔴 0 · Code Correctness · Add CTA buttons to a hero section (baseline) · anthropic:messages:claude-opus-4-8

The response uses hand-rolled object types with defineType/defineField for CTA buttons instead of the required defineCta function from createPresetsRegistry(). Internal links use a manual 'href' URL field rather than the required 'to' option pattern for declaring link targets. Ne

🔴 0 · Code Correctness · Add CTA buttons to a hero section (baseline) · anthropic:messages:claude-opus-5

The response uses hand-rolled object types with manual reference fields instead of the required defineCta/createPresetsRegistry pattern. Internal links use a manually constructed 'reference' field with 'to' array on the reference type, not the required 'to' option on defineLink o

Add CTA buttons to a hero section (gold) (7)

🔴 0 · Code Correctness · Add CTA buttons to a hero section (gold) · openai:responses:gpt-5.6-terra

The response uses hand-rolled object types with manual 'href' string fields for CTA buttons. It does not use defineCta from createPresetsRegistry() nor does it declare internal link targets through a 'to' option on defineLink or createPresetsRegistry. Both required patterns are c

🔴 0 · Code Correctness · Add CTA buttons to a hero section (gold) · openai:responses:gpt-5.4

The response uses hand-rolled object types with defineType/defineField for CTA buttons instead of the required defineCta function from createPresetsRegistry(). Internal link targets use a manual 'url' field type rather than the required 'to' option on defineLink or createPresetsR

🔴 0 · Code Correctness · Add CTA buttons to a hero section (gold) · anthropic:messages:claude-opus-4-6

Response uses hand-rolled object types with manual 'url' field instead of required defineCta from createPresetsRegistry(). No defineLink or 'to' option for internal links. Completely misses the required API patterns.

🔴 0 · Code Correctness · Add CTA buttons to a hero section (gold) · anthropic:messages:claude-opus-4-8

The rubric requires CTA buttons to be produced by a defineCta function from createPresetsRegistry() and internal link targets declared through a to option. The candidate uses hand-rolled object types with manual href fields instead, completely missing the required API pat

🔴 0 · Code Correctness · Add CTA buttons to a hero section (gold) · anthropic:messages:claude-fable-5

The response uses hand-rolled object types with defineField/defineArrayMember for actions, rather than using defineCta from createPresetsRegistry(). It also uses a manual 'url' field instead of declaring internal link targets through a 'to' option on defineLink or createPresetsRe

🔴 0 · Code Correctness · Add CTA buttons to a hero section (gold) · anthropic:messages:claude-sonnet-4-6

The response uses hand-rolled object types with defineType/defineField instead of the required defineCta function from createPresetsRegistry(). Internal link targets use a manual 'url' field instead of the required 'to' option pattern. Both core rubric criteria are completely unm

🔴 0 · Code Correctness · Add CTA buttons to a hero section (gold) · anthropic:messages:claude-opus-5

The rubric requires CTA buttons to be produced by a defineCta function from createPresetsRegistry() and internal links declared via a to option on createPresetsRegistry or defineLink. The candidate uses hand-rolled object types with defineType/defineField and manual

Add SEO metadata to a page (baseline) (2)

🔴 0 · Code Correctness · Add SEO metadata to a page (baseline) · openai:responses:gpt-5.6-terra

The rubric explicitly requires that the metadata field be produced by a defineSeo function obtained from a createPresetsRegistry() call. The candidate response hand-rolls SEO fields using defineField/defineType instead of using defineSeo from createPresetsRegistry(). This is

🔴 0 · Code Correctness · Add SEO metadata to a page (baseline) · anthropic:messages:claude-opus-5

The rubric explicitly requires that 'The metadata field is produced by a defineSeo function obtained from a createPresetsRegistry() call, rather than a hand-rolled object composing a title, a description, and an image.' The candidate response creates a hand-rolled 'seo' objec

Add SEO metadata to a page (gold) (1)

🔴 0 · Code Correctness · Add SEO metadata to a page (gold) · anthropic:messages:claude-opus-5

The rubric explicitly requires that the metadata field be produced by a defineSeo function obtained from a createPresetsRegistry() call. The candidate response hand-rolls an SEO object type using defineType/defineField and creates a custom plugin, rather than using the requir

Compose a full landing page schema (baseline) (6)

🔴 0 · Code Correctness · Compose a full landing page schema (baseline) · anthropic:messages:claude-opus-4-6

Response completely ignores the required preset-based architecture. Does not use createPresetsRegistry(), definePage, defineCta, defineRichText, or defineImage. Hand-rolls all content patterns (slug, SEO, CTA, rich text, image blocks) instead of using presets. The rubric explicit

🔴 0 · Code Correctness · Compose a full landing page schema (baseline) · anthropic:messages:claude-opus-4-8

Code works syntactically but completely ignores the required preset-based architecture. No createPresetsRegistry(), definePage, defineCta, defineRichText, or defineImage usage. Everything is hand-rolled: slug/SEO fields manually added instead of using definePage built-ins, CTA ma

🔴 0 · Code Correctness · Compose a full landing page schema (baseline) · anthropic:messages:claude-fable-5

The response completely ignores the required preset-based architecture. It does not use createPresetsRegistry(), definePage, defineCta, defineRichText, or defineImage. Instead, everything is hand-rolled: the page document, slug field, SEO fields, CTA button, rich text (textSectio

🔴 0 · Code Correctness · Compose a full landing page schema (baseline) · openai:responses:gpt-5.6-terra

Response completely ignores the required preset-based architecture. Does not use createPresetsRegistry(), definePage, defineCta, defineRichText, or defineImage. Hand-rolls all content patterns (Portable Text, CTA links, images, SEO, slug) instead of using built-in presets. Page s

🔴 0 · Code Correctness · Compose a full landing page schema (baseline) · anthropic:messages:claude-opus-5

The response completely ignores the required preset-based architecture. It hand-rolls all content patterns instead of using createPresetsRegistry() with definePage, defineCta, defineRichText, and defineImage. No pageBuilderBlocks option is used. Slug and SEO are manually defined

🔴 0 · Code Correctness · Compose a full landing page schema (baseline) · openai:responses:gpt-5.4

Code is syntactically correct and follows basic Sanity patterns, but completely ignores the required preset-based approach. No createPresetsRegistry(), definePage, defineCta, defineRichText, or defineImage presets are used. Everything is hand-rolled: slug, SEO, portable text, CTA

Compose a full landing page schema (gold) (7)

🔴 0 · Code Correctness · Compose a full landing page schema (gold) · anthropic:messages:claude-opus-4-8

Response completely ignores the required API pattern. Does not use createPresetsRegistry(), definePage, defineCta, defineRichText, or defineImage. Instead invents a non-existent 'presets()' plugin import from '@sanity/presets'. No hero defined with defineType. No pageBuilderBlock

🔴 0 · Code Correctness · Compose a full landing page schema (gold) · anthropic:messages:claude-opus-4-6

Response completely ignores the required preset-based architecture. Does not use createPresetsRegistry(), definePage, defineCta, defineRichText, or defineImage. Hand-rolls all content patterns (slug, SEO, CTA, rich text, image blocks) instead of using presets. Page sections are m

🔴 0 · Code Correctness · Compose a full landing page schema (gold) · anthropic:messages:claude-sonnet-4-6

Response completely ignores the required preset-based architecture. Does not use createPresetsRegistry(), definePage, defineCta, defineRichText, or defineImage. Instead hand-rolls all content patterns from scratch including slug, SEO, rich text, CTA button, and image blocks.

🔴 0 · Code Correctness · Compose a full landing page schema (gold) · openai:responses:gpt-5.4

The response hand-rolls every content pattern from scratch. It does not call createPresetsRegistry(), does not use definePage with pageBuilderBlocks, does not use defineRichText, defineCta, or defineImage. Slug and SEO are manually added fields. The hero button is hand-rolled. No

🔴 0 · Code Correctness · Compose a full landing page schema (gold) · anthropic:messages:claude-fable-5

Response completely ignores the required preset-based approach. Does not use createPresetsRegistry(), definePage, defineCta, defineRichText, or defineImage. Hand-rolls all content patterns (SEO, slug, rich text, CTA button, image with caption) instead of using the documented pres

🔴 0 · Code Correctness · Compose a full landing page schema (gold) · openai:responses:gpt-5.6-terra

The response hand-rolls every content pattern from scratch instead of using the required preset APIs. No createPresetsRegistry() call, no definePage with pageBuilderBlocks, no defineRichText, no defineCta, no defineImage. Slug and SEO are manually added fields rather than coming

🔴 0 · Code Correctness · Compose a full landing page schema (gold) · anthropic:messages:claude-opus-5

The response completely ignores the required preset-based approach. It hand-rolls all content patterns (richText, link/CTA, image, SEO, page sections) instead of using createPresetsRegistry() with definePage, defineCta, defineRichText, and defineImage. No presets are used at all.

Configure an editorial image with crop control and caption (gold) (1)

🔴 0 · Code Correctness · Configure an editorial image with crop control and caption (gold) · anthropic:messages:claude-opus-4-8

The rubric explicitly requires that coverImage be produced by a defineImage function obtained from a createPresetsRegistry() call. The candidate uses a hand-rolled defineField({type: 'image', options: {hotspot: true}, fields: [...]}) approach instead, which is exactly what

Create Studio with page preset (baseline) (9)

🔴 0 · Task Completion · Create Studio with page preset (baseline) · anthropic:messages:claude-opus-4-6

The response does not use createPresetsRegistry from @sanity/presets or call definePage with name: 'page'. Instead, it invents a non-existent pageType import from @sanity/presets/page-type. None of the required rubric criteria are met.

🔴 0 · Doc Coverage · Create Studio with page preset (baseline) · anthropic:messages:claude-opus-4-6

Candidate uses defineConfig and plugins correctly per docs, but the core implementation relies on @sanity/presets/page-type (pageType) which is not documented in the reference at all. This is a significant hallucination of the main feature. Only basic config structure (defineConf

🔴 0 · Task Completion · Create Studio with page preset (baseline) · openai:responses:gpt-5.6-terra

Response does not use createPresetsRegistry from @sanity/presets, does not call definePage with name: 'page', and imports a non-existent pageType. The approach is fundamentally wrong - it doesn't match any of the required patterns from the rubric.

🔴 0 · Doc Coverage · Create Studio with page preset (baseline) · openai:responses:gpt-5.6-terra

Candidate correctly used defineConfig which is documented. However, @sanity/presets and pageType are completely fabricated - not mentioned anywhere in the reference. The reference discusses plugins, definePlugin, and @sanity/plugin-kit but candidate invented a non-existent

🔴 0 · Task Completion · Create Studio with page preset (baseline) · anthropic:messages:claude-opus-4-8

Candidate explicitly refused to implement the required API, stating @sanity/presets doesn't exist. Did not call definePage with name:'page', did not import createPresetsRegistry from @sanity/presets, and did not include definePage result in schema.types. Used alternative approach

🔴 0 · Task Completion · Create Studio with page preset (baseline) · anthropic:messages:claude-fable-5

The candidate explicitly refuses to use the required API (@sanity/presets, createPresetsRegistry, definePage), claiming it's 'hallucinated'. None of the 4 rubric criteria are met: no definePage call, no createPresetsRegistry import, no preset-based entry in schema.types

🔴 0 · Task Completion · Create Studio with page preset (baseline) · anthropic:messages:claude-sonnet-4-6

The response completely ignores the required rubric criteria. It does not use createPresetsRegistry from @sanity/presets, does not call definePage with name: 'page', and instead provides a manual schema implementation using defineType. None of the 4 required criteria ar

🔴 0 · Doc Coverage · Create Studio with page preset (baseline) · openai:responses:gpt-5.4

Candidate correctly uses defineConfig and plugins from docs, but invents @sanity/presets package and pageType import which are not documented. structureTool is also not in the reference. The core task requirement (page preset) relies entirely on hallucinated imports. Only basic c

🔴 0 · Task Completion · Create Studio with page preset (baseline) · anthropic:messages:claude-opus-5

Candidate explicitly refused to implement the required API, stating '@sanity/presets' doesn't exist. Did not call definePage with name:'page', did not import createPresetsRegistry from @sanity/presets, and did not include definePage result in schema.types. Instead provided altern

Create Studio with page preset (gold) (9)

🔴 0 · Task Completion · Create Studio with page preset (gold) · anthropic:messages:claude-opus-4-6

The candidate response does not use createPresetsRegistry from @sanity/presets, does not call definePage with name: 'page', and does not place the result in schema.types. Instead, it invents a presets plugin function that is not in the reference. None of the rubric cr

🔴 0 · Task Completion · Create Studio with page preset (gold) · anthropic:messages:claude-sonnet-4-6

The rubric requires: (1) definePage called with name:'page', (2) createPresetsRegistry imported from @sanity/presets and called to obtain definePage, (3) definePage result in schema.types, (4) valid Studio config export. Candidate imports pageType from @sanity/presets/page and ad

🔴 0 · Task Completion · Create Studio with page preset (gold) · anthropic:messages:claude-opus-4-8

Response does not use createPresetsRegistry or definePage at all. Instead invents a non-existent pageType plugin from @sanity/presets. None of the four rubric criteria are met: no definePage call, no createPresetsRegistry import, no schema.types entry from definePage, and the exp

🔴 0 · Doc Coverage · Create Studio with page preset (gold) · openai:responses:gpt-5.4

Candidate correctly uses defineConfig and plugins array from docs. However, @sanity/presets and pagePreset() are not documented in the reference - these are hallucinated/assumed. The core plugin installation pattern matches docs, but the specific preset implementation details had

🔴 0 · Task Completion · Create Studio with page preset (gold) · anthropic:messages:claude-fable-5

Response does not use createPresetsRegistry or definePage at all. Instead invents a non-existent pageType plugin pattern. None of the four rubric criteria are met: no definePage call, no createPresetsRegistry import, no schema.types entry from definePage, and the exported config

🔴 0 · Doc Coverage · Create Studio with page preset (gold) · anthropic:messages:claude-fable-5

Candidate correctly used defineConfig and plugins from docs. However, @sanity/presets, pageType, and all claims about what the preset provides (page builder, slug field, SEO metadata, rich text, etc.) are entirely fabricated - none of this appears in the reference. The re

🔴 0 · Task Completion · Create Studio with page preset (gold) · openai:responses:gpt-5.6-terra

Response does not use createPresetsRegistry or definePage at all. Uses non-existent pageType import from @sanity/presets. No criteria from rubric are met - missing createPresetsRegistry import, missing definePage call with name:'page', and schema.types is empty instead of contain

🔴 0 · Doc Coverage · Create Studio with page preset (gold) · openai:responses:gpt-5.6-terra

Candidate correctly uses documented defineConfig and plugins array pattern. However, @sanity/presets and pageType() are not documented in the reference at all - these are hallucinated/assumed. The core plugin installation pattern is correct but the specific preset impleme

🔴 0 · Task Completion · Create Studio with page preset (gold) · anthropic:messages:claude-opus-5

Response does not use createPresetsRegistry from @sanity/presets or call definePage with name: 'page'. Instead, it guesses at pageType import and provides a fallback with manual schema types. None of the required rubric criteria are met.

Extend a page document with a campaign category (gold) (2)

🔴 0 · Code Correctness · Extend a page document with a campaign category (gold) · anthropic:messages:claude-opus-4-6

The candidate imports createPresetsRegistry from @sanity/presets and calls createPresetsRegistry().definePage(...). Neither @sanity/presets nor definePage appear in the authoritative reference. The reference only documents defineConfig, definePlugin, kit, `plugins

🔴 0 · Code Correctness · Extend a page document with a campaign category (gold) · anthropic:messages:claude-sonnet-4-6

The response imports createPresetsRegistry from '@sanity/presets' which is not documented in the reference. The definePage function is fabricated - there is no evidence in the reference that such an API exists. The reference only documents defineConfig, definePlugin, `plu

💡 Recommendations

  • 🔴 studio-presets (score: 44) — needs significant doc improvements.

Generated by AI Literacy Framework · Sep 29, 2026, 12:37 AM UTC · view detailed results

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Store caching currently creates several gigabytes of redundant cache entries per CI run and can evict Playwright caches.

Review effort: Balanced
Findings: 1 Medium severity

Open (1)
What changed in this PR

Upgrades the monorepo to pnpm 12.6.0 and updates CI and Vercel tooling for its lockfile format.

Changes:

  • Enables pnpm 12, automatic deduplication, and updated workspace policies.
  • Migrates CI setup to pnpm/setup@v3.
  • Upgrades Vercel and documents compatibility requirements.
File Description
.github/​actions/​setup/​action.yml Migrates CI setup to pnpm/setup.
.github/​workflows/​ailf-eval.yml Uses the new runtime input.
AGENTS.md Documents pnpm 12 behavior.
CONTRIBUTING.md Updates prerequisites.
README.md Updates prerequisites.
e2e/​README.md Documents Vercel compatibility.
package.json Bumps pnpm and Vercel.
pnpm-lock.yaml Regenerates resolutions with pnpm 12.
pnpm-workspace.yaml Enables deduplication and trust exclusions.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread .github/actions/setup/action.yml Outdated
Brings in Renovate's lock file maintenance (#2039). pnpm-lock.yaml
conflicted: it takes main's refreshed lockfile, and pnpm 12.6.0 re-applies
this branch's changes on top (the packageManagerDependencies block and the
vercel@60.1.3 tree with its optional-peer suffixes). No other version moves;
sanity stays on 6.17.0-next.108, and pnpm-workspace.yaml needed no changes.
sanity-io/ui has no such workflow, and autoDedupe now dedupes on every
non-frozen install. On the current lockfile, pnpm 12.6.0's `pnpm dedupe`
is also not idempotent: each run toggles the optional peers
(@vitejs/devtools, oxc-transform-react) of the sanity and
@sanity/tsdown-config entries of plugins/@sanity/debug-preview-url-secret-plugin.
Every dedupe PR the workflow automerged would change pnpm-lock.yaml on
main and trigger the next one. Document the toggle so it doesn't get
committed.
With `cache: true`, pnpm/setup uploads a new ~645 MiB store entry after
every job, even when the lockfile is unchanged, because its cache keys
include the run id. In this PR's CI that meant 12 entries (7.7 GB) per
push, which evicts the Playwright browser caches the e2e jobs rely on.
Each job also spent about 12s restoring and 12s saving that entry.

pnpm/setup still restores and saves its small lockfile-verification
cache, so jobs keep skipping the ~28s supply-chain check.

sanity-io/ui keeps `cache: true` and has the same issue. Revert this
commit to go back to ui's setting.
Copilot AI review requested due to automatic review settings September 29, 2026 01:20

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The PR description incorrectly states that caching remains enabled and the dedupe workflow is retained.

Review effort: Balanced
Findings: 2 Low severity

Open (2)
Resolved since last review (1)

# The store cache would upload a new ~645 MiB entry after every job (its keys
# include the run id) and evict the Playwright browser caches. The lockfile
# verification cache is restored and saved regardless of this setting.
cache: false
Comment thread AGENTS.md Outdated

Because pnpm 12 records that block, `pnpm-lock.yaml` is **two YAML documents**: a leading env document with `packageManagerDependencies`, then the dependency graph. Any tool that reads the lockfile must parse multi-document YAML; a single-document parser silently sees only the env document and concludes the project has no dependencies. This is why the `vercel` CLI must stay at `58.9.5` or newer, with the versions `pnpm install` reports as trust downgrades listed in `trustPolicyExclude`—see [`e2e/README.md`](e2e/README.md) (`Vercel studio preview`).

`autoDedupe: true` in `pnpm-workspace.yaml` collapses compatible duplicate versions on every non-frozen install, so the lockfile arrives deduped without a separate dedupe workflow. Frozen installs never rewrite the lockfile. It requires pnpm >= 12.6.0.
This reverts commit f59a3be ("ci: drop the Dedupe lockfile workflow").
The workflow stays; the next commit adapts it for pnpm 12.
On the current lockfile, pnpm 12.6.0's `pnpm dedupe` is not idempotent:
each run toggles the optional peers (@vitejs/devtools,
oxc-transform-react) of plugins/@sanity/debug-preview-url-secret-plugin's
sanity and @sanity/tsdown-config entries. With a single run, the workflow
would open a toggle-only PR after every lockfile change on main, and each
automerged dedupe PR would trigger the next.

A second run undoes the toggle and keeps real dedupes, which are
idempotent: starting from this lockfile with #2052's duplicates put back,
two runs remove exactly those duplicates. Document the toggle in AGENTS.md
so installs that only flip it don't get committed.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The central CI migration, large lockfile rewrite, and deliberate provenance exemptions warrant final human review.

Review effort: Balanced
Findings: 2 Low severity

Open (2)

@stipsan
stipsan merged commit 9d9c8a5 into main Sep 29, 2026
24 checks passed
@stipsan
stipsan deleted the cody/upgrade-pnpm-12-48e4 branch September 29, 2026 01:32
cursor Bot pushed a commit that referenced this pull request Sep 29, 2026
Brings in the pnpm 12.6.0 upgrade from #2048.

pnpm-workspace.yaml keeps both sides: main's autoDedupe,
strictPeerDependencies and trust-policy changes (including its pruned
entries and the vercel exclusions), plus this branch's vitest catalog
bumps and why-is-node-running@3.2.2 exclusion.

pnpm-lock.yaml is main's lockfile regenerated with pnpm 12.6.0 run
directly, so the packageManagerDependencies document stays as on main.
Versions only move for this branch's vitest tree. The regeneration also
flipped the known @vitejs/devtools/oxc-transform-react toggle in
@sanity/debug-preview-url-secret-plugin's sanity and
@sanity/tsdown-config peer suffixes; that part is reverted to main's
variant, which drops the two snapshots it added. A clean main plus
this branch's pnpm-workspace.yaml regenerates the same lockfile before
that revert, and `pnpm install --frozen-lockfile` accepts the result.
cursor Bot pushed a commit that referenced this pull request Sep 29, 2026
Brings in pnpm 12.6.0 from #2048.

package.json keeps both sides: main's packageManager pnpm@12.6.0 and
vercel ^60.1.3, without this branch's removed lint-staged block.

pnpm-lock.yaml is main's lockfile regenerated with pnpm 12.6.0.
Versions only move for this branch's own changes (oxfmt 0.71.0 with
tinypool 2.2.0; husky, lint-staged and their exclusive dependencies
removed). pnpm also re-derives the peer contexts of snapshots that have
oxfmt as an optional peer, which adds sanity@6.17.0-next.108 and
@sanity/cli@8.13.0 variants. Clean main plus this branch's package.json
and catalog changes yields the identical lockfile. pnpm format changes
nothing.
cursor Bot pushed a commit that referenced this pull request Sep 29, 2026
package.json and pnpm-workspace.yaml keep both sides: main's pnpm 12.6.0
pin, vercel bump, autoDedupe and exclude lists, and this branch's
Changesets devDependencies, Storybook scripts and catalog entries.
pnpm-lock.yaml is main's, regenerated with pnpm 12.6.0 invoked directly
and deduped once more, so debug-preview-url-secret-plugin keeps main's
optional-peer variant and the diff against main holds only this branch's
changes.
cursor Bot pushed a commit that referenced this pull request Sep 29, 2026
Brings in pnpm 12.6.0 from #2048. pnpm-workspace.yaml keeps both sides:
main's autoDedupe, strictPeerDependencies and trustPolicyExclude changes,
plus this branch's catalog bumps and patchedDependencies entry.

pnpm-lock.yaml is main's, regenerated with pnpm 12.6.0 run directly.
Versions only move for this branch's toolchain (tsdown 0.23.0,
@sanity/tsdown-config 0.28.2, @sanity/pkg-utils 13.0.3, rolldown-plugin-dts
0.28.6, @sanity/vanilla-extract-tsdown-plugin 0.4.2 and their
dependencies). autoDedupe also folds main's
@sanity/vanilla-extract-integration 0.1.19 into the 0.1.20 that the tsdown
plugin needs. The rolldown-plugin-dts patch applies under pnpm 12 with the
same hash. The regeneration flipped the optional-peer toggle of
plugins/@sanity/debug-preview-url-secret-plugin; `pnpm install
--fix-lockfile` reverted it, so those entries match main's. The Studio
stays on 6.17.0-next.108.
stipsan added a commit that referenced this pull request Sep 29, 2026
… and ui's Changesets setup (#2046)

<!-- CURSOR_AGENT_PR_BODY_BEGIN -->
Moves `@sanity/themer` from sanity-io/ui (`packages/themer`) to
`plugins/@sanity/themer`, adds a React Storybook in `dev/storybook`
modeled on sanity-io/ui's `apps/storybook`, and puts the whole repo on
sanity-io/ui's Changesets setup.

## What's in this PR

1. **Changesets setup identical to sanity-io/ui (repo-wide).**
`.changeset/config.json` gains `"format": "oxfmt"` and
`"bumpVersionsWithWorkspaceProtocolOnly": true` (only the changelog
`repo` differs from ui's). `@changesets/cli` ^3.0.3 and
`@changesets/changelog-github` ^1.0.1 are root devDependencies, and
there's a `changeset` script. `release.yml` drops the npm@11 pin, since
`@changesets/cli` 3.0.3 handles npm 12's `pnpm info --json` output
(changesets/changesets#2274). The Renovate changesets workflow is ui's.
2. **Catalog aligned with sanity-io/ui for what themer needs:**
`@sanity/ui` ^4.2.7, `@sanity/icons` ^5.2.2, `motion` ^13.4.4,
`@sanity/vanilla-extract-vite-plugin` ^0.2.20. These are the ranges
`@sanity/themer@0.8.0` was published with. As AGENTS.md asks for uniform
shared-dependency updates, one combined patch changeset covers the 43
published plugins whose `dependencies` ranges change with the
`@sanity/ui`, `@sanity/icons` and `motion` bumps;
`@sanity/vanilla-extract-vite-plugin` is a devDependency everywhere, so
it doesn't reach a published range.
3. **The move.** Source copied from sanity-io/ui `main` (`b28d66c`),
which is exactly the published 0.8.0. The version and CHANGELOG carry
over, so the next release is 0.8.1. `package.json` follows the monorepo
template (plugins repository links, `build`/`prepack` scripts) with the
published exports, `sideEffects`, and dependency ranges.
`@sanity/themer-legacy` (moving to sanity-io/themer) is the published
^0.2.1. `xstate`, `@xstate/react`, and `scroll-into-view-if-needed` move
to the catalog, since other plugins now share them;
`sanity-plugin-dashboard-widget-vercel` gets its own patch changeset for
its `xstate` range (^5.32.5 → ^5.33.2). The test studio switches from
`^0.8.0` on npm to `workspace:*`; `themerTool()` was already wired in
its Home workspace.
4. **Storybook in `dev/storybook`**, next to the test studios, set up
like sanity-io/ui's `apps/storybook`: Storybook 10.6 with the a11y,
docs, links, themes and vitest addons, the React Compiler through
`@vitejs/plugin-react`, and the vanilla-extract plugin for workspace
plugins' `.css.ts` source. The preview imports `@sanity/ui/styles.css`,
since the published dist doesn't. The global decorator renders every
story in the theme `buildTheme` from `@sanity/themer` builds from a
preset picked in the toolbar, next to the light/dark toggle. It adds
`pnpm dev:storybook`, `pnpm storybook:build`, `pnpm test:browser`, ui's
`apps/storybook/vercel.json` unchanged (as `dev/storybook/vercel.json`),
a `storybook-test` CI job like ui's, and eslint-plugin-storybook's CSF
rules through oxlint's JS plugins, as in ui. Unlike ui's job,
`storybook-test` also installs Playwright's system deps when it restores
the browser cache, as this repo's e2e workflow does (Bugbot flagged that
the cache doesn't hold them). Like the other `dev/*` apps, it's skipped
by `pretest` and the CI build job.

## Layout

No existing folder is renamed or moved, and there's no new top-level
folder. `@sanity/themer` goes into the existing `plugins/@sanity/`
folder next to the other `@sanity/*` plugins, not ui's
`packages/themer`. The Storybook goes into the existing `dev/` folder as
`dev/storybook`, next to `dev/test-studio` and `dev/e2e-studio`, so the
`dev/*` workspace glob and the `./dev/*` build filters already cover it.
The root only gains workspace-level config.

## How faithful the move is

- **Build output vs. the npm tarball of 0.8.0:** JavaScript is identical
apart from chunk hashes, and `bundle.css` is byte-identical. The `.d.ts`
files have the same API; tsdown 0.22 writes `declare …; export {…}`
where 0.23 wrote `export declare …`.
- **Source changes:** `type` modifiers on type-only imports (plugins'
`strictest` tsconfig has `verbatimModuleSyntax`), one inline
`oxlint-disable-next-line react/exhaustive-effect-dependencies` with its
reason (the rule is on here and off in ui; the `[palette]` dependency is
deliberate), and the README license link. The tsconfig is `strictest`
without `noUncheckedIndexedAccess`, which ui never enabled
(`sanity-plugin-graph-view` and `plugin-kit` do the same).
- **No React Compiler typings stub:** ui's
`typings/babel-plugin-react-compiler.d.ts` isn't carried over. Here the
real `babel-plugin-react-compiler` types resolve (it's a linked optional
peer of `@sanity/tsdown-config`), so `defineConfig()` picks the
single-config overload, and a wrong `reactCompiler` value is still
flagged (checked with a bad `target`, a misspelled `transform`, a number
`target` and an unknown key).
- **Tests:** themer's Vitest config sets `testTimeout: 30_000`. PR CI
here runs every project with coverage, and the whole-theme `buildTheme`
tests took 8s there against the 5s default (ui runs them without
coverage).
- **Versions:** every themer runtime dependency and the whole Storybook
stack resolve to the same versions as in ui's lockfile.

## What Cody needs to do

- **npm trusted publishing:** publishing runs from
`.github/workflows/release.yml`, which this repo already had; it now
uses ui's version/publish flow. The trusted publisher for
`@sanity/themer` must be repository `sanity-io/plugins`, workflow
`release.yml`, no environment.
- **Vercel:** the Storybook carries ui's `apps/storybook/vercel.json`
unchanged, as `dev/storybook/vercel.json` (`pnpm build` into
`storybook-static`). To deploy it like ui's `sanity-ui-storybook`,
create a project with root directory `dev/storybook`; the file supplies
the build command and output directory. No other Vercel config is
touched: `dev/e2e-studio/vercel.json` and the `plugins-studio` and
`plugins-e2e-test-studio` projects stay as they are, and both previews
built fine on this PR.
- **Secrets and env vars:** none new. The touched workflows use names
this repo already has: `ECOSPARK_APP_ID`, `ECOSPARK_APP_PRIVATE_KEY`,
`vars.ECOSPARK_CLIENT_ID`, `TURBO_TOKEN` and `vars.TURBO_TEAM`.
`storybook-test` uses none.
- **Ruleset:** no swap needed. The `main` ruleset requires `lint`,
`build`, `test` and the two Socket Security checks, and those job names
are unchanged. Optionally, to match ui (which requires `build`, `knip`,
`Test node@lts / ubuntu-latest`, `storybook-test` and its two Vercel
checks), add `storybook-test` and `knip`, and the Storybook's Vercel
check once that project exists.
- **CODEOWNERS:** untouched, per the transfer convention. Add an owner
for `plugins/@sanity/themer` if you want one.

## Open questions

1. **Patch, not major.** The transfer skill defaults to a major, but
themer is 0.x, nothing breaks, the next release should continue from the
published version, and themer's own deprecation notice reserves 1.0 for
removing `@sanity/themer/legacy`. So the changeset is a patch (0.8.1).
The color move made the same call.
2. **Stories.** sanity-io/ui had no themer stories. I ported the ui
stories that exercise what themer builds (theme
Debug/Build/Canvas/Color, and the color palette story on
`buildPalette`'s output) and added stories for the tool's theme
thumbnails and navbar color wheel. Those two aren't public API, so their
stories import them from `plugins/@sanity/themer/src` by relative path.
Keep them, or public API only?
3. **`sanity` peer range.** I kept 0.8.0's optional `^3 || ^4 || ^5 ||
^6` instead of the monorepo's `catalog:peer` (`^5 || ^6.0.0-0`), which
would drop Studio v3/v4. As with 0.8.0 from npm, it doesn't match the
`sanity@next` prereleases here, so the test studio shows a peer warning.
4. **`bumpVersionsWithWorkspaceProtocolOnly`.** With ui's setting,
Changesets no longer rewrites `@sanity/sfcc`'s intentionally wide
`sanity-plugin-internationalized-array` peer (`^4.0.0 || ^5.0.0`), the
only internal dependency that doesn't use `workspace:`. That seems
right, but it's a behavior change.

## Remaining alignment with sanity-io/ui (not in this PR)

This PR stays scoped to the themer move. Matching ui's tooling happens
in five separate PRs, one per tool, each owned by another worker. All
five exist:

- **pnpm** 11.21.0 → 12.6.0 with ui's pnpm setup, including the
`pnpm/setup@v3` CI setup action: #2048 (merged; this branch is merged
with it)
- **oxfmt** ^0.63 → ^0.71 with ui's config: #2047. Dropping the
`@sanity/plugin-kit/oxfmt` preset's Prettier-era `.changeset/*.md`
`singleQuote: false` override, which ui did when it enabled Changesets'
`format: "oxfmt"`, belongs there too; #2047 doesn't include it yet.
- **vitest** ^4.1 → ^5.0, with `@vitest/coverage-v8`: #2050. If it
merges first, the rebase of this PR raises the Storybook's
`@vitest/browser-playwright` to `^5.0.2` and re-resolves
`@storybook/addon-vitest` in the lockfile, since otherwise two stale
vitest 4 packages stay behind. #2050's worker checked that the Storybook
tests pass on vitest 5 with those two changes (`@storybook/addon-vitest`
10.6 officially peers on vitest 3/4; ui runs it on 5 with loose peers).
- **oxlint** ^1.83 → ^1.86 and **oxlint-tsgolint** ^7.0.2001 → ^7.0.2003
with ui's config: #2049. It turns `react/exhaustive-effect-dependencies`
off, as ui does, so if it merges first, the rebase of this PR drops its
one suppression for that rule in `ImagePaletteSection.tsx`, which would
otherwise be an unused-directive error.
- **tsdown** ^0.22 → ^0.23 with **@sanity/tsdown-config** ^0.27 → ^0.28,
which requires tsdown 0.23: #2051

All five also touch files this PR changes (`pnpm-workspace.yaml`,
`pnpm-lock.yaml` and `AGENTS.md`; `package.json` and `README.md` for
#2047 and #2048; `oxlint.config.ts` for #2049;
`.github/workflows/ci.yml` and `.gitignore` for #2050), so whichever PR
merges later rebases and regenerates the lockfile.

Smaller differences left to Renovate or later PRs: `oxc-transform-react`
^0.147 → ^0.151 (#1988), `@types/node` ^24.13 → ^24.19, `jsdom` ^29 →
^30, `knip` ^6.33 → ^6.38, ui's macOS test run, and ui's root
`engines.node: ">=22.13"`.

## Verification

- `pnpm format`, `pnpm knip`, `pnpm lint`, `pnpm build` (54 tasks,
including both studios and the Storybook), `pnpm test run` (235 files,
1,509 tests) and `pnpm test:browser` (7 stories) all pass locally,
starting from `pnpm install --frozen-lockfile`.
- A `changeset version` dry run in a scratch worktree, with ui's config
and `GITHUB_TOKEN`, bumps themer 0.8.0 → 0.8.1 and
dashboard-widget-vercel 4.0.28 → 4.0.29 (before the combined changeset;
`changeset status` now plans 46 patch releases: the 43 plugins, themer,
and `sanity-plugin-studio-smartling` and `sanity-plugin-transifex`,
which follow `sanity-translations-tab` as internal dependents), credits
@stipsan through the `author:` line, and writes CHANGELOGs that pass
`oxfmt --check`.
- `zizmor .github --min-severity high` reports no findings.

## Transfer verification

- [x] `package.json` dependencies, peerDependencies and exports verified
against the published 0.8.0
- [x] `LICENSE` not added: ui's credits only Sanity.io; the README links
this repo's LICENSE
- [x] Test studio uses the workspace package (`themerTool()` was already
wired)
- [x] `pnpm format`, `pnpm knip`, `pnpm lint`, `pnpm build`, `pnpm test
run` pass
- [x] Changeset added (patch; see open question 1)
- [ ] Trusted publishing for `@sanity/themer` configured with
`sanity-io/plugins` and `release.yml`

## Maintainer follow-up

- [ ] Remove `packages/themer` from sanity-io/ui (with its entries in
ui's root `build`/`test` scripts and `pkg-pr-new.yml`) once this lands,
on your go-ahead
- [ ] Point ui's docs and AGENTS.md mentions of `@sanity/themer` to
`plugins/@sanity/themer`
- [ ] Transfer open themer issues from sanity-io/ui and label them
`themer`

## Screenshots

The theme cards of every preset, in the dark scheme:

[Storybook: themer tool theme thumbnails for every
preset](https://cursor.com/agents/bc-7516e1ad-c642-5cd2-97f4-35288fbda30a/artifacts?path=%2Fopt%2Fcursor%2Fartifacts%2Fstorybook_themer_theme_thumbnails_presets.png)

The palette `buildPalette` generates for the Verdant preset, light
scheme (its backgrounds replace black and white, its accent the blue
scale, its text color the gray scale):

[Storybook: generated Verdant palette with contrast
badges](https://cursor.com/agents/bc-7516e1ad-c642-5cd2-97f4-35288fbda30a/artifacts?path=%2Fopt%2Fcursor%2Fartifacts%2Fstorybook_themer_palette_verdant_light.png)

The Debug theme story in the Pink Synth preset, both schemes:

[Storybook: Debug theme story in the Pink Synth
preset](https://cursor.com/agents/bc-7516e1ad-c642-5cd2-97f4-35288fbda30a/artifacts?path=%2Fopt%2Fcursor%2Fartifacts%2Fstorybook_themer_debug_pink_synth.png)


<sub>To show artifacts inline, <a
href="https://cursor.com/dashboard/cloud-agents#team-pull-requests">enable</a>
in settings.</sub>
<!-- CURSOR_AGENT_PR_BODY_END -->

<div><a
href="https://cursor.com/agents/bc-7516e1ad-c642-5cd2-97f4-35288fbda30a?cursor_ref=pr_footer&cursor_cta=open_in_web"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-web-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-web-light.png"><img
alt="Open in Web" width="114" height="28"
src="https://cursor.com/assets/images/open-in-web-dark.png"></picture></a>&nbsp;<a
href="https://cursor.com/background-agent?bcId=bc-7516e1ad-c642-5cd2-97f4-35288fbda30a&cursor_ref=pr_footer&cursor_cta=open_in_cursor"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-cursor-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-cursor-light.png"><img
alt="Open in Cursor" width="131" height="28"
src="https://cursor.com/assets/images/open-in-cursor-dark.png"></picture></a>&nbsp;</div>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
cursor Bot pushed a commit that referenced this pull request Sep 29, 2026
Brings in the pnpm 12.6.0 upgrade (#2048).

- pnpm-workspace.yaml: kept both sides (this branch's oxlint catalog
  entries and #2048's pnpm settings and exclusions); git merged it
  without conflicts.
- pnpm-lock.yaml: took main's lockfile and let pnpm 12.6.0 re-apply this
  branch's catalog change. The result differs from main only by the
  oxlint and oxlint-tsgolint catalog specifiers. The re-resolution also
  flipped the known pnpm 12.6.0 peer toggle on
  plugins/@sanity/debug-preview-url-secret-plugin, which AGENTS.md says
  not to commit; a second re-resolution (`pnpm dedupe`, as the Dedupe
  lockfile workflow does) undid it.
stipsan added a commit that referenced this pull request Sep 29, 2026
…ommit hook on lefthook (#2047)

<!-- CURSOR_AGENT_PR_BODY_BEGIN -->
Brings oxfmt in line with sanity-io/ui's `main`: the same version, the
same config, and the same scripts and CI wiring. The pre-commit hook
moves from husky and lint-staged to lefthook, set up like
sanity-io/sanity's. Supersedes #1919.

## Commits

1. **`chore(deps): update oxfmt to ^0.71.0`**: the catalog goes from
^0.63.0 to ui's ^0.71.0, with ui's oxfmt resolution: oxfmt 0.71.0 and
`tinypool` 2.2.0, the same versions and integrity hashes as ui's
lockfile. There's a patch changeset for `@sanity/plugin-kit`, whose
optional `oxfmt` peer uses the catalog.
2. **`chore(format): align the oxfmt config with sanity-io/ui`**: root
`oxfmt.config.ts` now ignores `storybook-static/**` and `CHANGELOG.md`,
like ui's `.oxfmtrc.json`. `.vscode/settings.json` points
`oxc.fmt.configPath` at `oxfmt.config.ts` instead of the `.oxfmtrc.json`
that #1565 deleted. AGENTS.md records the parity with ui and the
exceptions.
3. **`chore: drop the husky and lint-staged pre-commit hook, like
sanity-io/ui`**: superseded by commit 6, which brings the hook back on
lefthook.
4. **`chore(format): drop the oxfmt preset's .changeset/*.md override,
like sanity-io/ui`**: the `@sanity/plugin-kit/oxfmt` preset loses its
Prettier-era `singleQuote: false` override for `.changeset/*.md`, as
ui's `.oxfmtrc.json` did when ui switched Changesets to oxfmt formatting
(sanity-io/ui#2957). It gets a patch changeset for `@sanity/plugin-kit`,
since the preset is published. This part moved here from #2046 because
it's formatter config. AGENTS.md now explains how changesets get quoted.
5. **`chore(format): run pnpm format`**: requotes this PR's own
changeset from commit 1. Everything else already passed under oxfmt
0.71.
6. **`chore: run oxfmt on staged files with lefthook, like
sanity-io/sanity`**: see below.

## Pre-commit hook: husky and lint-staged → lefthook

This mirrors how sanity-io/sanity replaced husky and lint-staged
(sanity-io/sanity#12755, then sanity-io/sanity#12767 and
sanity-io/sanity#12953):

- **`lefthook` ^2.1.14** as a root devDependency, sanity's range. It
resolves to 2.1.14, as in sanity's lockfile.
- **`allowBuilds: lefthook: true`** in `pnpm-workspace.yaml`, as in
sanity. Lefthook's own install script runs `lefthook install` during
`pnpm install`, and it skips when `CI` is set, which covers GitHub
Actions and Vercel. There's no root `postinstall` script: sanity dropped
its `lefthook install --reset-hooks-path` postinstall in #12767 for this
package build.
- **`lefthook.yml`** with sanity's layout: a parallel `pre-commit` with
two jobs that run oxfmt on staged files with `stage_fixed: true`. The
jobs are `format-scripts` (`*.{js,jsx,ts,tsx,mjs,cjs,mts,cts}`) and
`format-other` (`*.{md,yml,yaml,css,json,jsonc}`).
- `mts`, `cts` and `jsonc` are additions to sanity's globs. They cover
this repo's `.mts`, `turbo.jsonc` and `knip.jsonc` files, which the old
lint-staged glob partly covered.
- sanity's `oxlint --fix` line is left to the oxlint alignment (#2049).
- **Direct binary calls:** the jobs call `node_modules/.bin/oxfmt`
instead of sanity's `pnpm oxfmt`. Running pnpm in a hook triggers pnpm
12's implicit install, which flips the lockfile's peer toggle (see
AGENTS.md). sanity switched to `pnpm oxfmt` in #12953 because lefthook
doesn't put `node_modules/.bin` on `PATH`; the path solves that too.
- **Lockfile:** `pnpm-lock.yaml` gains only lefthook 2.1.14 and its
platform packages (100 lines). A plain install would also have bumped
the Studio's `next` overrides from `6.17.0-next.108` to
`6.17.0-next.112`, so I resolved it through a local registry proxy that
served the locked versions as `next`, as AGENTS.md describes. The
lockfile has no `localhost` or `tarball:` entries.
- **Docs:** AGENTS.md documents the hook, `LEFTHOOK=0`, the one-time
reset for clones that ran husky, and `pnpm
--config.verify-deps-before-run=false <script>` for local runs. README's
note that formatting also runs in a pre-commit hook is back.

## Merges with `main`

`main` is merged in, not rebased, so nothing was force-pushed. After
each merge, `pnpm-lock.yaml` is `main`'s lockfile regenerated with pnpm
12.6.0, invoked directly so `packageManagerDependencies` stays, minus
the peer toggle that AGENTS.md describes.

- **#2039 (lock file maintenance):** only the lockfile conflicted.
- **#2048 (pnpm 12.6.0):** `package.json` conflicted next to the removed
`lint-staged` block. The resolution keeps both sides: `main`'s
`packageManager: pnpm@12.6.0` and `vercel` ^60.1.3, without the block. A
follow-up commit restores `main`'s side of the peer toggle, which the
regeneration had flipped.
- **#2046 (themer move):** only the lockfile conflicted. `pnpm format`
then requoted #2046's three changesets in a separate commit:
`dashboard-widget-vercel-xstate`, `shared-deps-sanity-ui-icons-motion`
and `themer-plugins-monorepo`, 45 frontmatter lines, quote-only. The new
`plugins/@sanity/themer` package and `dev/storybook` already passed.
- **#2053 (Version Packages):** it released and deleted those three
changesets, so the modify/delete conflicts resolve to `main`'s
deletions. It didn't change the lockfile, and `pnpm format` changes
nothing after the merge.

`pnpm-lock.yaml` differs from `main` only by this PR's own changes:

- oxfmt and its bindings go from 0.63.0 to 0.71.0, and `tinypool` from
2.1.0 to 2.2.0.
- husky, lint-staged and their four exclusive dependencies are removed.
- lefthook 2.1.14 and its platform packages are added.

No other version moves, and the Studio stays on `6.17.0-next.108`.

## How it compares with ui

- **Options:** identical, overrides included (`printWidth`, `semi`,
`singleQuote`, `bracketSpacing`, `quoteProps`, `sortImports`,
`sortPackageJson`). They come from the `@sanity/plugin-kit/oxfmt`
preset.
- **Ignore patterns:** `dist/**`, `storybook-static/**`,
`pnpm-lock.yaml` and `CHANGELOG.md` in both. Two differences remain:
  - `turbo/**/*.hbs` is plugins-only (the turbo generator templates).
- ui's `**/sanity.types.ts` is left out. In ui it was for
`apps/docs/sanity.types.ts` and has matched nothing since the docs app
went static. Here, `pnpm typegen` formats
`dev/test-studio/sanity.types.ts` on purpose: the test studio runs
typegen with `formatGeneratedCode: false` and then `pnpm format`.
Ignoring the file would make the next TypeGen PR unformat it.
- **Config file:** plugins keeps `oxfmt.config.ts` re-exporting the
published preset instead of ui's `.oxfmtrc.json`. #1565 made that switch
so the preset is exactly what the monorepo formats with, and
plugin-kit's `verify-package` expects plugins to use it. oxfmt, the Oxc
extension and Changesets' formatter detection all read
`oxfmt.config.ts`. See the open question.
- **Scripts and CI:** the `format` script is identical to ui's (`oxfmt .
--no-error-on-unmatched-pattern`). Neither repo checks formatting on
PRs, and both run the same shared `format-if-needed.yml` on `main`, so
no workflow changes are needed. `oxfmt .` also formats
`.github/workflows/*.yml`, which that workflow can't push, and 0.71
changes none of them. ui has no pre-commit hook; this repo keeps one, on
lefthook.
- **VS Code:** ui points `oxc.fmt.configPath` at its config file, but
plugins pointed it at a file that doesn't exist. I drove `oxfmt --lsp`
directly: with the old setting, the editor silently formats with oxfmt's
defaults (double quotes, semicolons, bracket spacing, unsorted imports),
and with `oxfmt.config.ts` it matches `pnpm format`.

## Changesets

- **Changesets formats what it writes with oxfmt.**
`.changeset/config.json` sets `"format": "oxfmt"` (from #2046). I called
`@changesets/write` (what `changeset add` uses) directly. With the
override it writes `"@sanity/plugin-kit": patch`; without it,
`'@sanity/plugin-kit': patch`, the same as `pnpm format`.
- **`changeset version` is unaffected.** It only hands `CHANGELOG.md`
files to the formatter. This PR ignores those, and oxfmt skips them and
exits 0 thanks to `--no-error-on-unmatched-pattern`.
- **`changeset status` parses the single-quoted frontmatter.** It lists
this PR's two changesets for `@sanity/plugin-kit` 10.0.11.

## What to expect after merging

- **Renovate changesets get a follow-up format PR, as in ui.** The
shared Renovate changesets workflow uses
`mscharley/dependency-changesets-action`, which always writes double
quotes and has no formatting option. After a Renovate PR merges, the
Auto format workflow opens (or updates) a `chore(format): 🤖 ✨` PR that
requotes the changeset. ui has had one after most Renovate merges since
sanity-io/ui#2957: sanity-io/ui#2971, sanity-io/ui#2980,
sanity-io/ui#2986, sanity-io/ui#2998, sanity-io/ui#3002,
sanity-io/ui#3013 and sanity-io/ui#3024.
- **Clones that ran husky need a one-time reset.** They still have
`core.hooksPath=.husky/_`. There, `lefthook install` refuses to install
and prints how to fix it, and `pnpm install` continues. Until the reset,
husky's leftover runner does nothing, so commits simply aren't
formatted. `git config --unset core.hooksPath && pnpm install` fixes it.

## Related PRs

- **#1919** (Renovate, oxfmt ^0.71.0): superseded. This PR has the same
bump, with the lockfile regenerated on current `main`, plus the config
alignment. Close it if Renovate doesn't once this merges.
- **#2046** (themer move) and **#2048** (pnpm 12): merged, and merged
into this PR.
- **#2049** (oxlint) conflicts with this PR in `pnpm-workspace.yaml`,
where its `oxlint` catalog lines sit next to `oxfmt`. Keep both sides.
It could also add sanity's `oxlint --fix` line to the `format-scripts`
job, calling `node_modules/.bin/oxlint` directly.
- **#2050** (vitest 5) conflicts only in `pnpm-lock.yaml`.
- **#2051** (tsdown) doesn't conflict.
- Whichever PR lands later merges `main` and regenerates the lockfile
without committing the peer toggle.

## What Cody needs to do

- Add the `🤖 bot` label: the agent's label tool couldn't edit PRs in
this repository.
- Triage Socket's alert. It flags `lefthook@2.1.14`'s install script
(`node postinstall.js`) as a low-priority warning. That script is what
installs the hook: `allowBuilds: lefthook: true` enables it, as in
sanity-io/sanity, and it skips when `CI` is set. Mark it as acceptable
risk if you agree. Socket's required checks pass either way.
- After merging, tell contributors to run `git config --unset
core.hooksPath && pnpm install` once, so lefthook replaces husky's hook.
- Merge the Renovate follow-up format PRs as they come, as in ui.
- No Vercel, secrets or ruleset changes, and no new secrets or env vars.

## Open question

**`.oxfmtrc.json`:** should plugins switch to ui's JSON config file,
too? That would mean no longer re-exporting the plugin-kit preset, and
it would reverse #1565. I kept `oxfmt.config.ts`, since the formatting
result is already the same as ui's.

## Verification

- **The hook**, tested on throwaway commits, with `pnpm-lock.yaml`
untouched each time:
  - Staged TS and JSON files get formatted and re-staged.
- A tracked `.vscode/settings.json` (its folder is gitignored) is
re-staged without the `[FAILED]` that lint-staged printed.
  - A staged `CHANGELOG.md`, which oxfmt ignores, passes as-is.
- In a partially staged file, only the staged hunk is formatted and the
unstaged hunk is kept.
- The commit that adds the hook ran it on its own files in 0.4 seconds.
- **Clones that ran husky:** with `core.hooksPath=.husky/_`, `lefthook
install` exits 1 with instructions and installs nothing, as described
above.
- **Local checks** at the head commit, with pnpm 12.6.0 on Node 24.21
and `CI=true` (lefthook's install script skips):
- `pnpm install --frozen-lockfile`, which leaves `pnpm-lock.yaml`
untouched
  - `oxfmt --check` (2,018 files)
  - `oxlint` and `knip`
  - the full turbo build (54 tasks)
  - `vitest run` (235 files, 1,509 tests)
  - `pnpm test:browser` (4 files, 7 stories)
- **CI** is fully green at the head commit (`123af257`): build, lint,
knip, all four test shards, coverage, `storybook-test`, both e2e
browsers, Bugbot, and both Vercel checks, whose installs run lefthook's
install script. The branch is up to date with `main`.
<!-- CURSOR_AGENT_PR_BODY_END -->

<div><a
href="https://cursor.com/agents/bc-0b8c4f47-d9af-5929-8254-d25847e7e2dd?cursor_ref=pr_footer&cursor_cta=open_in_web"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-web-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-web-light.png"><img
alt="Open in Web" width="114" height="28"
src="https://cursor.com/assets/images/open-in-web-dark.png"></picture></a>&nbsp;<a
href="https://cursor.com/background-agent?bcId=bc-0b8c4f47-d9af-5929-8254-d25847e7e2dd&cursor_ref=pr_footer&cursor_cta=open_in_cursor"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-cursor-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-cursor-light.png"><img
alt="Open in Cursor" width="131" height="28"
src="https://cursor.com/assets/images/open-in-cursor-dark.png"></picture></a>&nbsp;</div>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>

This branch was successfully deployed

1 active deployment
Preview – plugins-studio — a7e1057f Deployed Sep 29, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants