Json values - #453
Open
freekh wants to merge 35 commits into
Open
Conversation
Version Packages
Address Copilot review on #446: mark ValidationWorkerFactory as a type import so it is erased at emit time. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Address Copilot review on #446: the @valbuild/core imports are all types, so import them with `import type` to avoid emitting a runtime dependency. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Address Copilot review on #446: only deserializeSchema is needed at runtime; import the rest with `import type`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ssue Fix validation worker loading issue
Version Packages
|
freekh
changed the base branch from
main
to
replace-quickjs-with-val-modules-for-validation
June 28, 2026 08:32
Contributor
Author
|
Make sure validation hash in c.json makes it easy to know when we must revalidate. |
Validate the content of every .jsonValues() entry server-side by loading each backing *.val.json via its lazy import thunk and checking it against the record's item schema (validateJsonValuesEntries), wired into ValOps.validateSources. The record-level executeValidate only asserts the marker shape, so this performs the deferred deep validation. Add Internal.resolveJsonValues(source): recursively resolves all JsonSource markers by invoking their thunks, returning a fully-inlined source. This backs the eager fetchVal/useVal path (a jsonValues module's local source is markers, not content, so it must be resolved before stega-encoding). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The c.json validation sha is composed as "<schemaHash>-<contentHash>" with the item-schema hash as a readable PREFIX (not hashed together). This lets us decide an entry needs revalidation by comparing the prefix to the current schema hash WITHOUT reading the *.val.json content — a schema change flags affected entries by scanning only the shas already present in the .val.ts. - computeJsonEntrySha(itemSchema, content): the composite sha - getJsonItemSchemaHash / getSchemaHashFromJsonSha: schema-hash prefix - jsonEntrySchemaHashIsStale(storedSha, itemSchema): content-free staleness check - isJsonEntryShaCurrent(storedSha, itemSchema, content): full freshness check Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
fetchValKey (RSC) and useValKey (client) resolve a SINGLE .jsonValues() entry by key — loading only that entry's backing *.val.json via one dynamic import — instead of the eager fetchVal/useVal which load the whole record. This is the runtime-scaling counterpart for 10K+ entry routers/records. - fetchValKey: initFetchValKeyStega in next/src/rsc/initValRsc.ts - useValKey: useValKeyStega in next/src/client/initValClient.ts (promise cache + React.use / Suspense) Production path resolves the local module's import thunk (Internal.getJsonImport). Enabled/Studio draft path is a TODO (needs the single-entry fetch endpoint). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- /json endpoint (ApiRoutes contract + ValServer handler): loads the committed content of one .jsonValues() entry by key (getBaseSources -> marker -> getJsonImport -> thunk). Generic ValRouter dispatch routes it automatically. Unblocks the Studio lazy-load and the enabled/draft runtime path. - examples/next: app/support/[slug] uses s.router(...).jsonValues() with two *.val.json entries; page.tsx renders one via fetchValKey. Registered in val.modules.ts; fetchValKey exported from val/rsc.ts. Example typechecks clean. Note: the Studio still cannot edit jsonValues entries until the UI lazy-load consumes /json (resolvePath guard correctly surfaces "content not loaded"). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When the Studio opens a path that descends into a .jsonValues() entry whose content isn't loaded, fetch it via GET /json and fold it into the source view: - ValSyncEngine: jsonEntryContents cache + requestJsonEntry(mfp, key) (GET /json, then invalidateSource to re-render); applyJsonEntryContents substitutes loaded content for the marker in getPatchedSource (before patches apply, so field edits layer on top). Caches reset on engine reset. - ValFieldProvider: findUnloadedJsonEntryKey detects a path descending into an un-loaded marker; useSourceAtPath + useSchemaAtPathInternal trigger requestJsonEntry and render "loading" until content resolves (then the entry's fields render normally). This resolves the intentional resolvePath "content not loaded" guard in the Studio. Editing shows optimistic updates; persistence to *.val.json needs the commit flow (pending). UI typecheck + ValSyncEngine tests green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Opening a .jsonValues() entry in the Studio surfaced two errors caused by lazy
json markers ({ _type:"json", _sha }) reaching code that expected real content:
- "Expected a c.json(...) entry, got 'object'": once an entry's *.val.json is
loaded, the UI substitutes its content for the marker, but RecordSchema's
executeValidate rejected anything that wasn't a marker. Now it validates
inline/loaded content against the item schema and only DEFERS actual markers
(so loaded entries validate live; unloaded markers are skipped, still validated
server-side once loaded).
- "Schema not found for ...entry._type/._sha": source+schema walkers descended
into an un-loaded marker and treated _type/_sha as object fields. Skip json
markers (opaque until loaded) in traverseSchemas, getRouteReferences, and
createSearchIndex (search still indexes the entry's path, not its content).
Core + UI typecheck; core jsonValues + server validation + ValSyncEngine tests
pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…e could skip validation of json values, however, the sha param would only tell us when we HAD to re-validate, not when we could skip validation
… loading and add isJsonValuesRecordSchema check
…ing) Closes the Phase 3 defects for `.jsonValues()`, none of which were covered by tests. Rename/duplicate now commit. `ValOps.prepare` classified only `op.path`, so the `move` the Studio emits when renaming a record key fell through to "Unsupported op 'move' on jsonValues entry". It now classifies `op.from` too and has `move`/`copy` arms: load the source entry's content, drop the old thunk (move only), insert the new one at the generated path, and null the old file. Cross-record and cross-entry move/copy are rejected rather than silently corrupting -- `rebaseContentOp` sliced `from` by the destination's prefix without checking the two matched. Published edits no longer appear to revert. `jsonEntryContents` was cleared only on init/reset, so the pre-edit content was re-substituted after publish. Invalidating on the /sources/~ refresh is not enough: `publish()` never refetches sources, and a content-only edit leaves the module source (bare markers) byte-identical so `sourcesSha` never changes. Publish now invalidates its own entries. The stale flag is cleared when a request STARTS, not on success, so an invalidation landing mid-flight wins over the in-flight response. A failed /json load no longer spins forever. `jsonEntryErrors` memoizes the failure -- stopping the refetch-on-every-remount -- and the field hooks render an error instead of a permanent loading state. Adds `retryJsonEntry` and `ensureJsonEntry`; the rename flow awaits the latter so the patch carries real content rather than an opaque marker that would 404 on the new key. Nested `.jsonValues()` is now rejected at startup. It was supported by the op classifier but broken in the /json endpoint, the Studio substitution, and validation -- where it silently skipped content validation entirely. `findNestedJsonValuesRecords` feeds a ModulesError from `initSources`, so /sources/~ fails naming the module. The classifier still reports `recordPath` truthfully; the consumers reject. Also fixes a pre-existing bug found on the way: `analyzePatches` pushed one `patchesByModule` entry per non-file op, so `prepare` re-applied the whole patch once per op. Idempotent for `replace` (which is why no test caught it), but it duplicates `add`s and breaks `move`. Locked: a rename relocates to the generated convention path, so renaming a hand-placed file moves it out of its directory. Root-only is decision #7. Tests: +7 ValOpsFS.jsonValues, +6 jsonValuesPatch, +1 validateJsonValues pinning the root-only contract, and a new jsonValues suite in ValSyncEngine.test.ts (previously zero coverage) with /json and /save added to the mock client. Full gate green: lint, format, -r typecheck, 1078 tests, root build, examples/next build. The manual Studio walkthrough (V1-V9 in the tracker) has NOT been run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
Author
We no longer use a hash in the c.json so this is not important. |
Renaming an entry and then reloading the Studio left it unopenable. The
pending `move` relocates the bare `{_type:"json"}` marker to the new key, so
the field requests GET /json for that key -- but /json only resolves keys in
the committed source, which still has the old one. It 404s, and (since the
previous commit) that renders as a hard error instead of a silent spinner.
`resolveBaseJsonEntryKey` walks the pending ops newest-first and undoes
whole-entry renames until it lands on a key the base source actually has.
Fetching under the base key is also what makes the entry render:
`applyJsonEntryContents` substitutes into the base source and the `move` patch
then relocates the content to the new key on its own.
This is the reload half of the rename fix. The `ensureJsonEntry` guard in
ChangeRecordPopover only covers the case where the content happened to be
loaded at the moment of the rename; it does not survive a page load.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`getSources` applied every non-file op with jsonOps directly against the module
source, where a `.jsonValues()` entry is an opaque `{_type:"json"}` marker. A
draft content edit at ["<entryKey>","title"] therefore failed with "Cannot
replace object element which does not exist", and because the module is then
marked poisoned, EVERY later patch for it was skipped with "previous errors
exists". Observed live on the example's support router.
The Studio dodges this by always passing apply_patches=false, but the eager RSC
`fetchVal` sends undefined (which defaults to true) and hits it.
Ops are now routed by `classifyJsonValuesOp`:
- content sub-ops are skipped -- the module source is genuinely unaffected,
since the content lives in the entry's `*.val.json` (draft content is served
by the single-entry /json endpoint);
- whole-entry add/replace push the MARKER (with patch_id) rather than the
content, so the record's key set is right for drafts while record validation
still only sees `isJson`;
- remove passes through; move/copy become add-marker (+ remove of the source
key for a move).
Schemas are serialized lazily and only for modules that have patches, so the
non-jsonValues case is unaffected.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds the server half of the draft read path: `/json` can now replay pending
patches onto an entry, so draft edits are visible to callers that do not own the
patch chain themselves (the runtime in draft mode).
- `applyJsonValuesEntryPatches` in patch/jsonValuesPatch.ts is the pure replay:
route ops with `classifyJsonValuesOp`, rebase content sub-ops with
`rebaseContentOp`, apply with jsonOps. Returns content / deleted / error. It is
the read-side twin of the commit flow in `ValOps.prepare` -- same routing, but
it produces a value instead of files and never touches the `.val.ts`.
- `rebaseContentOp` moved out of ValOps.ts into jsonValuesPatch.ts next to the
rest of the entry-patch helpers, and is now exported and directly tested.
- `ValOps.getJsonEntry(path, key, {applyPatches})` resolves one entry: committed
content from the import thunk (works in fs and http mode, no extra I/O), then
pending patches on top. The ValServer handler is now a thin adapter over it.
- `/json` gains `apply_patches`, defaulting to TRUE to mirror /sources/~. The
Studio passes false explicitly: it owns in-flight client patches the server has
not seen and applies them itself, so server-side application would
double-apply.
Also fixes the test harness: this suite evaluates the module in a `vm` whose
`require` resolved `c.json(() => import("./x.val.json"))` relative to the test
file rather than the temp root, so entry thunks could not load at all.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two halves of the enabled/draft runtime path.
Edit tags. `stegaEncode` seeds its recursion only from the selector branch, so
calling it on a `.jsonValues()` entry's raw content -- plain JSON with no
Path/GetSchema symbols -- left `recOpts` undefined and every string hit the
`!recOpts` bail in the encoder. All four jsonValues call sites were therefore
silently identity transforms: entries had NO click-to-edit at all, even with Val
enabled. `stegaEncode` now takes an optional `root: {path, schema}` seed, and
`getJsonEntryStegaRoot` builds it from the entry's path plus the serialized item
schema. A `title` field ends up tagged
`/app/support/[slug]/page.val.ts?p="/support/faq"."title"` -- the same shape
`findUnloadedJsonEntryKey` walks, so click-to-edit and lazy load line up.
Note this needed a seed parameter rather than a real sub-selector:
`newSelectorProxy` is not exported from core, it would require the private
`RecordSchema["item"]` class instance, and re-entering the selector branch would
make `getModuleIds` report the sub-path as a module id, breaking store lookups
keyed by ModuleFilePath.
Draft content (RSC). `fetchValKey` and the jsonValues branch of `fetchValRoute`
now read through the in-process `/json` endpoint when Val is enabled, so
uncommitted Studio edits are visible; they fall back to the local committed thunk
when the draft read yields nothing. Both also call SET_AUTO_TAG_JSX_ENABLED,
without which a page whose only Val call is fetchValKey/fetchValRoute gets no
data-val-path attributes even with correct stega strings.
The client hooks get the stega seed but still render committed content in draft
mode -- same limitation `useValStega` has today. Wiring them to drafts needs the
overlay emitter to carry patched sources, which is a separate change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…itted-only Ticks the /json draft-awareness and getSources boxes in Phase 2 and the edit-tag + RSC draft boxes in Phase 4. Records what is still open with the reason: client useValKey/useValRoute render committed content in draft mode because overlayEmitter is handed the raw un-patched /sources/~ module, and draft-added routes are unreachable via fetchValRoute because params -> key resolves from the local source. The manual Studio walkthrough (V1-V9) is still unrun. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
freekh
commented
Jul 28, 2026
| ? null | ||
| : syncEngine.getJsonEntryError(moduleFilePath, unloadedJsonKey), | ||
| ); | ||
| const resolvedSchemaAtPathRes = useMemo(() => { |
Contributor
Author
There was a problem hiding this comment.
Instead of as const, prefer annotating the expected return type. Add this to rules.md (or agents.md)
…loaded entries
The Studio's global scans (route refs, keyOf refs, referenced files) and search
silently skip un-loaded `{_type:"json"}` markers. That makes the delete gate and
the rename fixup answer "no references" for a ref living inside an entry the user
happens not to have opened — nondeterministically, since it depends on which
entries were visited this session.
Phase 6 records the fix:
- Scoping rule (direction decides): incoming refs need only the record's key set,
which the markers already carry, so the common case loads NOTHING. Content is
needed only when a jsonValues entry's own item schema points outward at the
thing being edited — a schema-only predicate. `route` is the one
over-approximation (SerializedRouteSchema names no target module).
- Batch `/json` (`keys` + `offset`/`limit`), with `initSources`/`fetchPatches`
hoisted out of the per-entry loop.
- Record list view: it already renders a per-entry preview for every key,
unvirtualized, so a jsonValues record shows N broken previews today. Virtualize
with @tanstack/react-virtual and load only the rendered window; skeletons on all
three preview paths so a marker never reaches a preview component.
- Search: lazy (nothing before the first query), partial results immediately, with
a percentage while the index fills.
- Windowed `.render()` list layouts — gated on a decision, since renders are null
Studio-wide today (`/sources/~` only computes them when apply_patches is true).
- Order of work (8 steps), V10–V18, and two decision items for Fredrik: browser
caching of `/json`, and the renders pincer (select is server-only, patched
sources are client-only).
Also supersedes V1 ("zero /json on open") and adds watch-list entries for
`jsonEntryContents` eviction and for skipping markers being unsafe in scans that
gate mutations.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds support for storing the values of records in json files. This lets Val records scale from 100s to 1000s of entries.
The jsonValues is isomorphic content meaning you can switch back and forth between using or not using jsonValues and let the validation move content from and to json.