Skip to content

✨ Select a component's result with yq, and render a value as YAML (#934) - #949

Closed
taras wants to merge 5 commits into
mainfrom
agent/repl-yq-yaml-934-20261017
Closed

taras wants to merge 5 commits into
mainfrom
agent/repl-yq-yaml-934-20261017

Conversation

@taras

@taras taras commented Oct 18, 2026

Copy link
Copy Markdown
Owner

Closes #934.

Why

A document can read a value producer, but it cannot say which part of the value
it wants. <File path="package.json" as="pkg" /> binds the whole file; selecting
.scripts from it means writing an eval block to do by hand what the producer
already produced. The Sidekick has the sharper version of the same problem: a
reply that asks a person a question gets the whole answer back as feedback,
including the parts it never asked for.

This adds one shared query to component values — yq, the syntax yq.rs
implements — applied to the value a producer actually returned, after that
producer's own validation and before capture, publication and designated
feedback. It adds <Yaml value={...} /> beside <Json> so a selected value can
be rendered explicitly in either form.

What changes

Before:

<File path="package.json" as="pkg" />

binds the file's whole text, and a reply's <Elicit> feedback carries every
field the person filled in.

After:

<File path="package.json" yq="from_json | .scripts" as="scripts" />
<Json value={scripts} />
<Yaml value={scripts} />

binds the selected object, and a queried Sidekick question sends back only what
the reply selected while the history keeps the whole original answer.

  • yq is an engine-owned prop beside slot and as, so a producer never sees
    it and no producer has to opt in.
  • Strings stay strings: JSON or YAML text is decoded explicitly with
    from_json / from_yaml in the expression.
  • A query returns exactly one value. Several results say to collect an array,
    no match is [], and null / false are values. An invalid expression fails
    visibly and binds nothing — never the unfiltered input as a fallback.
  • Without yq, every producer behaves exactly as it did.
  • <Yaml> renders the same JSON types <Json> does, and capturing its text
    does not change the structured binding underneath.

How it works

producer runs once → its own `returns` validation → invocation joins and tears
down → result middleware → `yq` selection → strict JSON → the selected value is
assigned to the binding environment → the execution's computed-bindings resource
carries it onto the ordinary document result → a designated occurrence delivers
it to its own destination

The engine owns a lazily-created, execution-scoped yq.rs instance with no
ambient filesystem, process, environment or clock access; the wasm bytes are
committed and verified rather than fetched.

For the Sidekick, a queried occurrence writes three things: the marked intent
before the work, the whole original outcome, and one acknowledged selected
presentation. Which occurrence a provider call belongs to is answered from the
scope that call is running in — canonical dispatch names the invocation it is
running the body of on that body's own Effection scope — so nested questions and
concurrent <All>/<Spawn> branches each keep their own receiver, however
their suspensions interleave.

Review guide

Start with: packages/core/src/value-query.ts and
packages/core/tests/yq-component.test.ts

Then review:

  1. packages/core/src/component-result.ts and packages/core/api.ts — the
    public result/query seam a consumer sees.
  2. packages/core/src/expand.ts — where selection sits in the invocation's
    order: validation and joined teardown first, assignment after.
  3. packages/core/src/component-result-destinations.ts and
    packages/cli/src/repl/sidekick/{controls,questions,feedback,model}.ts —
    per-occurrence receivers and the selected-presentation record.
  4. packages/core/src/components/Yaml.ts, Json.ts, json-serialization.ts.
  5. packages/core/src/yq-wasm.ts, scripts/generate-yq-wasm.ts,
    scripts/tests/yq-distribution.test.ts — the pinned engine and its
    distribution proof.

Look carefully at:

  • component-result-destinations.ts — a receiver belongs to one occurrence.
    Nothing correlates by component name, completion order or "the last one".
  • A failed query assigns nothing and leaves earlier bindings intact.
  • Teardown order: the producer's invocation joins before anything transforms
    what it returned.

What must stay true

  • yq runs once, over the value one call returned — enforced by the single
    componentResult seam in expand.ts, checked by yq-component.test.ts
    (producer-crossing counts) and component-returns.test.ts.
  • A query never re-runs a producer or rewrites its saved effects — checked by
    the two-producer and producer-once rows in yq-component.test.ts.
  • An invalid expression binds nothing and falls back to no value — checked by
    the yq='!!!' row: <File> prints the refusal, the first binding still
    holds, the second name stays unresolved.
  • Each queried Sidekick question keeps one marked intent, one whole original
    answer and one acknowledged selected presentation for its own occurrence —
    checked by the nested and overlapping rows in
    repl-sidekick-lifecycle.test.ts and by repl-sidekick-controls.test.ts.
  • Feedback for a queried question carries the selection only, never the raw
    answer and never a silent "unknown" — checked by the same rows plus the cold
    incomplete case in repl-sidekick-recovery.test.ts.
  • Builds install nothing and the engine's bytes are the verified pinned ones —
    checked by scripts/tests/yq-distribution.test.ts and deno task verify:clean.

How to verify it

  • deno task test packages/core/tests/yq-component.test.ts proves selection,
    cardinality, explicit decoding, producer-once and the query-refusal account;
    it fails if a query rebinds the original text or runs a producer twice.
  • deno task test packages/core/tests/yaml-component.test.ts proves <Yaml>
    renders the same values <Json> does and serializes once; a second renderer
    read would make the counting-getter row fail.
  • deno task test packages/cli/tests/repl-sidekick-lifecycle.test.ts proves a
    reply's question is this owner's, and that nested and overlapping queried
    questions each keep their own receiver; it fails if one occurrence files its
    order against another's destination.
  • deno task test packages/cli/tests/repl-entries.test.ts proves a selected
    capture reaches the next entry and that an earlier entry's history does not
    move.
  • deno task verify:clean proves a clean checkout installs only in preparation
    and that the builds leave node_modules and deno.lock byte-identical.

Scope

Included

  • Shared yq selection on component values, with its engine, distribution and
    static validation.
  • <Yaml value={...} /> beside <Json>.
  • Per-occurrence result destinations and the Sidekick's selected-presentation
    record, acknowledgment and feedback.

Intentionally unchanged

New dependencies

  • Package: yq.rs@0.1.0, as committed verified wasm bytes plus its vendored
    licence and third-party notices.
  • Used for: the query syntax itself.
  • Why existing dependencies are insufficient: nothing in the tree implements a
    JSON query dialect, and shelling out to a binary is not available to a
    document execution.

Generated or mechanical changes

  • packages/core/src/yq-wasm.ts and yq-wasm.provenance.json come from
    scripts/generate-yq-wasm.ts; the generator's --check mode is a CI gate.
  • packages/core/src/yq-vendor-notices/ is the upstream licence text.

Risks and limitations

  • The engine is the pinned yq.rs port, not jq: the documentation says yq
    rather than advertising jq compatibility.
  • Distribution evidence (real source, compiled and packed npm consumption) and
    the live journey were produced on earlier heads of this branch. This final
    commit changes five files and no dependency, loader, asset, package mapping,
    build flag, runtime adapter, retained format or transport, so those results
    are reused rather than re-timestamped. They are stated as historical.
  • The live journey used the Codex ACP transport, bounded by the vendored Claude
    Code version (2.1.238 against 2.1.251 required); it is not a successful Claude
    transport proof.

Review and ownership evidence

  • Both reviewers independently PASS this exact signed head
    cf0678bf23764ebc2dfb41701026bae620416d8e, tree
    65e1ea81d97048b0e36b9b58af038d82f85ce3be: architecture review and planning
    review, each on the identical SHA.
  • deno task verify:clean was run on this committed candidate before this PR
    was opened: exit 0 in 366 s, a clean checkout of cf0678bf, with tracked
    files, node_modules and deno.lock unchanged.
  • Focused evidence on the final bytes of this head: lifecycle + generated-xmd-root
    • elicit-component + repl-sidekick-controls, 34 passed / 175 steps;
      deno check --frozen and pnpm run lint both exit 0. The integrated parent's
      group runs (F1 22/199, F2 40/120, F3 47/244, F4 50/258) stand for the owners
      this head does not change.

Scope confirmation

  • Every changed file supports the purpose described above.
  • Unrelated cleanup and formatting changes are excluded.
  • Generated or mechanical changes are clearly identified.
  • The description matches the final diff and test results.

taras added 5 commits October 18, 2026 05:42
A shared `yq` attribute selects one JSON value out of what a component
returned, and `<Yaml>` renders a value as a YAML document. Both are one
WebAssembly engine, built once per execution and only when something asks it a
question: a document that writes neither decodes and compiles nothing.

The engine's bytes are committed source. `scripts/generate-yq-wasm.ts` writes
`packages/core/src/yq-wasm.ts` from the pinned `yq.rs@0.1.0` package, verifying
its name, version, length and digest, and emits the provenance and the vendor
notices beside it. A compiled binary and an installed npm package therefore
carry the engine in their own static import graph rather than reaching for a
path, a fetch or a host filesystem.

Selection sits in one fixed place in an invocation's life: the producer's own
contract accepts what it produced, its invocation joins, and only then is the
expression asked. An expression repairs no return its schema rejected, and an
answer that is not one JSON value refuses with the way to ask for several.

A host that evaluates generated source on somebody's behalf can own where one
occurrence's selected result goes. That destination is stated once, as a value
canonical execution copies before any document code runs, and is authenticated
against the exact definition this execution's capture published. It is reachable
through no context, no name and no metadata, and a receiver belongs to one
occurrence and is revoked when that invocation settles.

The REPL Sidekick uses it. A reply may write `yq` on any control it is admitted
to; the intent is marked before the work, the original outcome is recorded as it
always was, and a separate acknowledged `control-presentation` record carries
the selected value. What a next turn is sent is that record — so an
acknowledgement that never landed sends nothing rather than a raw fallback.

Static document validation now takes its engine-owned props from
`ENGINE_OWNED_PROPS` rather than keeping its own list, which is what lets an
entry a run executes also pass the REPL's preflight.
Four reviewed contracts were not held, and one authoring rule was unfinished.

A canonical `<Elicit>` the reply queried sent back the whole answer. Core opened
a destination for that exact occurrence, but the question writer never received
it: the intent went down unmarked and no presentation followed, so a reply that
asked for one field was told both. The destination now hands its occurrence to
the question provider before the asking, which marks the intent, retains the
whole answer as the operation's own evidence, and files the order that carries
one acknowledged selected value. A question nobody answered files nothing and
keeps its marker, so an interrupted selection reads as one.

A genuine uncaptured destination skipped the producer's own return schema. A
destination relaxes required-as — it does not relax what the component agreed
to produce — so a constant expression could turn a rejected return into a
delivered answer. The uncaptured path now validates a declared `returns` after
joined teardown and before the query, middleware and delivery.

An admitted entry that failed still produces its Submit receipt, and querying
that receipt wrote a history nothing could read back: the projector rejected
the presentation its own supported path had just written. The intent's
reservation is what distinguishes the two cases, and both the projector and the
feedback owner now read it. A submission refused before admission reserved
nothing and still keeps refused feedback.

`ComponentResult` and its consumer types are exported from `packages/core/api.ts`
under the unchanged name, with destination authority left on the host boundary.

`yq` is now a reserved declaration name beside `slot` and `as`; static
validation diagnoses a known non-string before any effect; an unresolved
dynamic query reports the site as not statically checkable rather than valid;
and generated preflight refuses a known-invalid query before the first effect.

Four frozen assertions are completed with the fixtures that already existed: a
halted initializer waited to real settlement, an evaluator that throws carrying
partial output, a queried Markdown body whose later element and cleanup precede
the result seam, and a cold-opened history whose marked selection was never
acknowledged.
A queried `<Elicit>` written inside another queried `<Elicit>`'s content sent
the outer question's whole answer back as feedback. The outer occurrence
captured the field it selected, but what the next turn received was both
fields, and the outer intent carried no marker at all — so a later reader
could not tell that a selection had been owed there.

The cause was a single shared cell. Canonical `<Elicit>` is core's own body, so
the Sidekick cannot ask the occurrence lookup which invocation it is in: it
held the occurrence most recently opened instead. Opening the inner
destination overwrote that cell, and the inner question consumed it, leaving
the outer question with nothing to file against.

Core now names the occurrence it is dispatching for exactly as long as that
body runs. Protected dispatch already had both the invocation and the body, so
the route establishes the pairing there and restores the previous one however
the body leaves. The answer therefore nests the way the bodies do: a question
asked inside another's content reaches its own receiver, and the enclosing
question reaches its own again when its body resumes. An invocation that never
reaches its body is never the answer, and an unwinding out of order removes its
own entry rather than somebody else's.

The Sidekick reads that instead of keeping a cell. Nothing is consumed, nothing
is keyed by name, completion order or a value a document could replace, and
the receiver still belongs to the exact invocation canonical capture
authenticated.

Each occurrence keeps its whole original answer as its own evidence and gets
one acknowledged selected presentation. Cancellation, refusal and joined
lifetimes are unchanged.
Main now carries the corrected #939 and the approved #940, and the corrected
#939 deleted the per-name root publication protocol #934 was built on. This
merge keeps both histories and moves #934's selected capture onto the parent's
own model.

`root-binding-publications.ts` stays deleted. The capture no longer publishes
a record per name: it validates the producer's own return, lets the invocation
join, asks the query, and assigns the selected value to the actual binding
environment. The execution's own computed-bindings resource carries current
eligible root data onto the ordinary result from there, and the successor
starts from a detached copy of what that accepted result retained.

What that changes in observable behaviour, and what the evidence now states:

A mutation made before the result is part of that entry's retained data,
because the result carries the entry's current root rather than a snapshot
taken at capture time. A later entry therefore carries its own edit onward to
its successor without re-exporting the name, while the earlier entry, its cold
reading and the copy it handed over all stay as they were.

A rejected query assigns no binding at all. `<File>` prints the refusal, the
run continues, and the earlier selected value is still in the result with no
second value and no fallback to the producer's text beside it.

A refused ordinary result acknowledges no computed map: the run reports the
persistence failure, the handle refuses rather than answering with data no
later run could start from, and nothing compensating is written.

One accepted assertion changed its number rather than its claim. A bound
operand with a counting getter is now read twice — once by the renderer and
once by the execution copying eligible root data — because that second read is
the parent's accepted behaviour for every bound value. The claim is unchanged:
the renderer reads once, and three reads would mean it serialized twice.

The R1 receiver correction is preserved through the generated-XMD conflict:
protected dispatch still names the occurrence it is dispatching for the
duration of that body, beside the parent's fragment and root semantics.
Both reviews of the integrated head found the same remaining defect, and it
is this: the occurrence a protected body is dispatching for was kept in one
array per execution, innermost last, and read as "the last one entered".
That is a nesting answer. Two admitted concurrent branches are not nested —
a branch that has entered its question's body and is waiting inside its own
content is still on that array when a sibling branch asks, and the sibling
reads the waiting one's receiver.

The retained observation: one `<All>` with two `<Spawn>` children, each
asking a queried canonical `<Elicit>`. The first branch's question content
holds an ordinary `<REPL.Bindings />` read open, so the second branch asks
first; releasing the read lets the first branch ask while the second is
still waiting to be answered. Both people's answers arrive normally and both
selected captures reach their bindings — but the first branch filed its
order against the second's receiver, so its own destination had nothing to
present, and its successful selected fact came back as incomplete.

The answer is now a fact about where the asking is, rather than about what
this execution did last. Canonical dispatch says which invocation it is
running the body of on that body's own scope, and the provider asks for it
from the scope it is asking in. Concurrent branches are concurrent scopes,
so each receives its own however their suspensions interleave, and a body
nested in another's content shadows the enclosing one only within itself.
The context is this route's own and unexported, and what it holds is still
resolved through the receivers this route opened: an invocation it opened
nothing for — including one another execution's dispatch is running —
reaches no receiver.

Nothing else moved. Authentication is still the exact definition or the
selected protected name, revocation is still per invocation, the answer is
still core's own `<Elicit>`, and each successful queried question still
writes one marked intent, one whole original answer and one acknowledged
selected presentation. No composition was restricted.

The new lifecycle row drives that overlap through the real owner, with the
real evaluator, the real form provider and the reply's own ordinary read
held by the case. It asserts both captures, both whole answers, both marked
intents, one acknowledged presentation per question order, the intervening
ordinary `{}` read that owes none, and feedback carrying the two selected
values with no incomplete result and no unselected field. The nested row
stays.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found 6 redundant comments. Inline suggestions to remove them below.

// An occurrence whose body filed nothing recorded no intent to
// present, so there is nothing for a presentation to belong to — a
// refused original outcome and an unqueried occurrence both land
// here, and both keep the feedback they already had.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Redundant comment — restates what the code does.

Suggested change
// here, and both keep the feedback they already had.

// its entry's receipt, and the selection is of that answer. The failure is
// still the operation's own and still reaches the reply through the outcome
// it belongs to. A submission refused before admission reserved nothing, so
// it keeps refused feedback as it always has.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Redundant comment — restates what the code does.

Suggested change
// it keeps refused feedback as it always has.

component,
kind: "answered",
// The range outside the payload, so a reply can tell what it selected from
// how much the producer actually observed.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Redundant comment — restates what the code does.

Suggested change
// how much the producer actually observed.

// omitted one would describe a vocabulary narrower than the one the response
// is admitted under — and a hand-kept copy would stop being true the next
// time core admits a pure component. Their returns are the reply's own to
// render: nothing is performed, so there is nothing to answer back.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Redundant comment — restates what the code does.

Suggested change
// render: nothing is performed, so there is nothing to answer back.

// No execution terminal: an expansion driven directly resolves its imports
// through the ordinary public Component Api, exactly as it always has.
// And no destinations: a result has nowhere to go that an execution did
// not state.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Redundant comment — restates what the code does.

Suggested change
// not state.

if (asBinding === undefined) {
// Queried and uncaptured: the selected value is what this element
// renders. A string renders as itself and anything else renders
// nothing, which is the rule an uncaptured result already follows.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Redundant comment — restates what the code does.

Suggested change
// nothing, which is the rule an uncaptured result already follows.

@github-actions

Copy link
Copy Markdown

PR #949: ✨ Select a component's result with yq, and render a value as YAML (#934)

61 files, +8866 / -185

Scope

🔴 PR has 9051 lines changed. Split into focused PRs.

🟡 9051 lines changed. PRs under 400 receive more thorough review.

🟡 61 files changed. Are all changes related?

🟡 Changes span 10 directories.

🟡 PR mixes config and source changes.

Structural

🟡 Type declarations with no consumers: SubmitForReply.
Symbol Declared at Refs in diff Why flagged
SubmitForReply packages/cli/src/repl/sidekick/requests.ts:187 1 referenced ≤1× within the added diff (pre-existing usages not counted)

🟡 2 console statements.

Oxlint structural signals:

  • no-unused-vars ×31: packages/cli/tests/fixtures/repl/issue-883/sidekick-harness.ts, packages/cli/tests/fixtures/repl/issue-883/recovery-phase.ts, packages/cli/tests/fixtures/repl/issue-883/mounted-program.ts (+6)
  • no-unnecessary-type-assertion ×9: packages/cli/tests/fixtures/repl/issue-883/recovery-phase.ts, packages/core/src/expand.ts, packages/cli/tests/fixtures/repl/issue-883/sidekick-harness.ts (+2)
  • no-empty-function ×2: packages/core/src/execute.ts, scripts/build-npm.ts
  • no-unnecessary-type-arguments ×1: packages/cli/src/repl/sidekick/owner.ts

Slop

  • packages/cli/src/repl/sidekick/controls.ts:497 — // here, and both keep the feedback they already had.
  • packages/cli/src/repl/sidekick/feedback.ts:438 — // it keeps refused feedback as it always has.
  • packages/cli/src/repl/sidekick/feedback.ts:471 — // how much the producer actually observed.
  • packages/cli/src/repl/sidekick/profile.ts:297 — // render: nothing is performed, so there is nothing to answer back.
  • packages/core/src/expand.ts:864 — // not state.
  • packages/core/src/expand.ts:3378 — // nothing, which is the rule an uncaptured result already follows.

Oxlint slop signals:

  • no-console ×3: scripts/generate-yq-wasm.ts, scripts/build-npm.ts
  • no-inferrable-types ×2: packages/core/src/expand.ts

Static Analysis

Oxlint: 174 diagnostics across 19 files (25 rules)
Density: 0.020 violations/added-line

no-shadow (38): packages/cli/src/repl/sidekick/model.ts, packages/cli/tests/fixtures/repl/issue-883/recovery-phase.ts, packages/cli/tests/fixtures/repl/issue-883/mounted-program.ts (+8)
no-unused-vars (31): packages/cli/tests/fixtures/repl/issue-883/sidekick-harness.ts, packages/cli/tests/fixtures/repl/issue-883/recovery-phase.ts, packages/cli/tests/fixtures/repl/issue-883/mounted-program.ts (+6)
unbound-method (19): packages/cli/src/repl/program.ts, packages/cli/tests/fixtures/repl/issue-883/recovery-phase.ts, packages/cli/tests/fixtures/repl/issue-883/sidekick-harness.ts (+3)
no-unsafe-type-assertion (17): packages/cli/src/repl/sidekick/questions.ts, packages/cli/tests/fixtures/repl/issue-883/recovery-phase.ts, packages/core/src/expand.ts (+4)
no-floating-promises (17): packages/cli/src/repl/program.ts, packages/cli/tests/fixtures/repl/issue-883/recovery-phase.ts, packages/cli/tests/fixtures/repl/issue-883/mounted-program.ts (+1)
no-unnecessary-type-assertion (9): packages/cli/tests/fixtures/repl/issue-883/recovery-phase.ts, packages/core/src/expand.ts, packages/cli/tests/fixtures/repl/issue-883/sidekick-harness.ts (+2)
no-array-sort (8): packages/cli/src/repl/sidekick/model.ts, packages/cli/tests/fixtures/repl/issue-883/recovery-phase.ts, packages/cli/src/repl/sidekick/feedback.ts (+2)
consistent-function-scoping (4): packages/cli/tests/fixtures/repl/issue-883/mounted-program.ts, packages/core/src/execute.ts
no-base-to-string (4): packages/core/src/expand.ts
consistent-return (4): packages/cli/src/repl/program.ts, packages/core/src/execute.ts, packages/core/src/document-validation.ts
no-console (3): scripts/generate-yq-wasm.ts, scripts/build-npm.ts
no-this-alias (2): packages/core/src/components/component-resolution.ts
no-control-regex (2): packages/cli/tests/fixtures/repl/issue-883/mounted-program.ts
no-empty-function (2): packages/core/src/execute.ts, scripts/build-npm.ts
no-inferrable-types (2): packages/core/src/expand.ts
no-useless-default-assignment (2): packages/cli/src/repl/program.ts
no-duplicate-type-constituents (2): packages/core/src/execute.ts
no-useless-fallback-in-spread (1): packages/core/src/expand.ts
no-underscore-dangle (1): packages/core/src/expand.ts
no-unnecessary-type-arguments (1): packages/cli/src/repl/sidekick/owner.ts
restrict-template-expressions (1): packages/core/src/expand.ts
no-unnecessary-type-conversion (1): packages/cli/src/repl/sidekick/controls.ts
no-misused-spread (1): packages/cli/src/repl/sidekick/controls.ts
no-unnecessary-template-expression (1): packages/cli/src/repl/sidekick/feedback.ts
no-implied-eval (1): packages/core/src/expand.ts

Correctness

No extraneous code patterns detected.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add <Query> for XMD values

1 participant