Skip to content

2.1: manifest covers every ts(export) type; coverage == closure-dtos.txt (on #43, merges #13) - #53

Open
cunninghamcard-bit wants to merge 8 commits into
exp/u1.8-codegen-pocfrom
exp/2.1-manifest
Open

cunninghamcard-bit wants to merge 8 commits into
exp/u1.8-codegen-pocfrom
exp/2.1-manifest

Conversation

@cunninghamcard-bit

@cunninghamcard-bit cunninghamcard-bit commented Oct 12, 2026 •

Copy link
Copy Markdown
Owner

2.1: manifest covers every type (phase 2)

Basis: PHASES.md:104 (2.1), arcships#200 §6.5 (docs/aisdk-architecture-alignment.md:1317-1357 @22580efd, esp. :1350), RFC-0042 §7.2 (:204-218), §7.3 (:222-224), D11 default a (DECISIONS.md:90-93).

Stacked on #43 (exp/u1.8-codegen-poc); #13 (exp/u1.5-closure-inventory) is merged in with git merge (commit 9cbc73f), so this diff also shows #13's files until #13 lands: 3 files (docs/plan/closure.md and scripts/gen_closure_inventory.py from #13, plus docs/plan/closure-dtos.txt that the extended script now writes). Does not touch exp/restart.

What changed

file change
scripts/gen_closure_inventory.py also writes docs/plan/closure-dtos.txt: [exported] = every #[ts(export)] type as <name> <category> <file>; [derive-ts-without-export]; [excluded] = tools/aimux-web/src/wire.rs (web-tool wire). Category per file (CATEGORY_BY_FILE, default core). Built from tracked files at the branch's base, not frozen. --check covers both files.
tools/aimux-codegen/src/extract.rs type model extracted from aimux-core/src with syn: serde renames/rename_all/rename_all_fields, enum representation (external / internal tag / adjacent / untagged), skip, default, skip_serializing_if, flatten, alias, other, with, from/into/try_from proxies, generics + ts(concrete), ts(rename), ts(type/as) codecs, non-generic type aliases reached by DTOs. A field type that is not a primitive/container/JSON/DTO/alias/registered codec fails generation with file:line. Hand-written serde without a ts(type) codec fails too.
tools/aimux-codegen/src/manifest.rs generated/manifest.json: schemaRevision, generator (name+crate version), sdk = all 7 entries read from fixtures/aisdk/VERSIONS.json when generating, inputHash (sha256 over aimux-core/src + VERSIONS.json), types, auxiliaryTypes, aliases. generated/coverage.txt: <name> <category> <file> per type. Coverage ≠ list → error, nothing written.
src/main.rs default run regenerates the PoC files and the manifest; --check verifies both; --src/--versions/--list overrides.
tests/manifest_test.rs acceptance + mutations below.

Dependencies only in tools/aimux-codegen: syn 2, quote, proc-macro2 (span-locations for file:line), sha2 (already used in the workspace).

Why syn rather than serde-reflection

  • serde-reflection traces Deserialize impls at runtime: it needs a sample value for every enum variant/untagged arm, cannot see skip_serializing_if/default/aliases, and handles untagged, internally tagged enums, flatten and hand-written serde (ToolChoice) poorly or not at all. Those are exactly the cases RFC-0042:212-216 cares about.
  • syn reads the Rust source, which is the protocol's source of truth (RFC-0042:200). It reports file:line, and it adds no runtime trait bound or dependency to aimux-core. Downside: serde's rename rules are reimplemented in the extractor. wire_names_match_the_current_ts_exports guards this by checking 864 field and variant names against the ts-rs output that serde actually produces: 129 structs and 229 variant tags, 0 mismatches.

Result

  • List and manifest: 193 exported types (user 21, provider 13, core 83, error 4, recording 25, multimodal 38, tool 9). Shapes: 131 struct, 60 enum (28 internal tag, 20 external, 12 untagged), 2 hand-written (ToolChoice, plus AiMuxError with a ts(as) proxy).
  • 193, not 192: ToolChoice (aimux-core/src/tool.rs:210-213) splits #[ts(\n export, type = ..)] over lines, so the single-line git grep '#[ts(export' count of 192 missed it. 193 + 1 non-exported (AiMuxErrorWire, error.rs) + 7 aimux-web = 201 derive(TS) sites, matching closure.md.
  • AiMuxErrorWire is reached through AiMuxError's from/into proxy. It is in the manifest as auxiliaryTypes (not part of coverage) and appears in the list's [derive-ts-without-export] section.

Mutation table (each must turn red)

# mutation how result
1 a type leaves the manifest remove #[ts(export)] of HttpExchange (scratch copy) coverage: missing from manifest: HttpExchange recording aimux-core/src/recording.rs (test removed_type_is_named)
1b list line deleted (real tree, manual) delete HttpExchange line in closure-dtos.txt --check: coverage: not in closure-dtos.txt: HttpExchange …
2 fake list line add FakeDto core aimux-core/src/fake.rs coverage: missing from manifest: FakeDto … (fake_list_line_is_named)
3 extra manifest type new exported ExtraDto coverage: not in closure-dtos.txt: ExtraDto core aimux-core/src/extra_dto.rs (extra_manifest_type_is_named)
4 VERSIONS.json change ai 7.0.127 → 7.0.999 lib + CLI --check: manifest.json is out of date; regenerated manifest contains 7.0.999 (versions_change_makes_manifest_stale)
5 unregistered field type pub at: std::time::Instant aimux-core/src/extra_dto.rs:3: unregistered type \Instant` …; CLI exits 1 and writes no files (unregistered_field_type_fails_generation`)
5b hand-written serde without codec derive(TS) + export, no serde derive ExtraDto has hand-written serde and no #[ts(type|as)] codec
6 serde rule change (real tree, manual) tool.rs first rename_all camelCase → snake_case --check: manifest.json is out of date
7 hand edit of manifest (manual) "toolCallId" → "tool_call_id" --check: out of date

Local evidence

cargo test -p aimux-codegen: 8 (1.8) + 16 (2.1) passed. cargo clippy -p aimux-codegen --all-targets -- -D warnings clean. cargo fmt --all -- --check OK. aimux-codegen --check up to date. python3 scripts/gen_closure_inventory.py --check up to date. Builds were run one at a time with -p aimux-codegen only.

CRLF (commit 655b800)

inputHash normalizes CRLF→LF for every hashed input (manifest.rs lf()). Done in code rather than via .gitattributes because a code fix works for every checkout, while .gitattributes only helps checkouts that respect it. Test crlf_checkout_gives_identical_manifest converts a scratch copy of aimux-core/src and VERSIONS.json to CRLF and asserts the same inputHash, a byte-identical manifest/coverage, and that --check passes against the committed files. Mutation: making lf() return its input unchanged turns the test red: hash sha256:db22a853… ≠ sha256:bce4b495…. LF hashes are unchanged, so the manifest did not need regenerating.

Reviewer checklist (commit 6489f30)

  1. 193 is computed from source. closure_dto_list_is_reproducible_from_source runs python3 scripts/gen_closure_inventory.py --check, asserts [exported] 193, and asserts that ToolChoice (multi-line #[ts(\n export, at tool.rs:210-213) is listed. Mutation: setting TS_EXPORT back to the single-line #[ts\(export → STALE: docs/plan/closure-dtos.txt, test red.

    • The new test also turned up a bug in the script: it counted #[derive(.., TS)] / #[ts(export)] written as text in the generator's own docs and test fixtures (a fake TypeDef entry; 201→205 derives). tools/aimux-codegen/ is now excluded from the ts-rs scans, which restores 201. closure.md was regenerated; the only change is that extract.rs (now 1134 lines) enters the >1000-line table.
  2. Serde-rule mutations. serde_attribute_mutations_change_the_modelled_field makes one change per rule in a scratch copy. For each it compares the manifests with inputHash removed, asserts that the specific field changed, and asserts that --check reports manifest.json is out of date:

    rule mutation manifest field: before → after
    skip_serializing_if FunctionTool.description loses it (tool.rs:20) omitWhenEmpty: true → absent
    flatten same field gains flatten (aimux-core has no flatten today) flatten: absent → true
    untagged remove #[serde(untagged)] on AudioInput (transcription_model.rs:25) representation.style: untagged → external
    rename_all FunctionTool camelCase → snake_case (tool.rs:13) input_schema.wire: inputSchema → input_schema
    field rename DecisionBooleanCriteria rename = "true" → "yes" (decision_model.rs:135) wire: true → yes

    Control: comment_only_edit_leaves_manifest_identical_except_input_hash adds a comment line to tool.rs. inputHash changes, and everything else in the manifest is byte-identical. Mutation proof: making the extractor ignore flatten turns the test red at flatten: mutated (left Null, right true).

  3. 864-name ts-rs comparison is the committed test wire_names_match_the_current_ts_exports (asserts >800 names checked, 0 problems). It runs in the CI job "Rust core (fmt + clippy + test)" via cargo test --workspace.

  4. No new dependencies in the library crates. cargo tree -p aimux-core -e normal,build and -p aimux-ffi, base 8455db9a vs this head: the same set of crates (210 / 258 lines). The only differences are workspace paths and patch versions from separate local lockfiles (cc 1.7→1.8, serde_json .152→.154, syn 3.0.7→3.0.8; Cargo.lock is not tracked). cargo tree -i syn@2.0.119 -e normal: aimux-codegen only. In aimux-core, syn 2 is still reached only through existing proc-macros (ts-rs-macros, thiserror-impl, strum_macros, tracing-attributes, async-stream-impl). cargo tree -i sha2 -e normal: aimux-codegen and aimux-provider-utils (already there). quote and proc-macro2 are direct dependencies of aimux-codegen only.

serde with codecs (follow-up commit)

  • A field that uses serde with / deserialize_with / serialize_with must have #[ts(type)] or a FIELD_CODECS entry (extract.rs, keyed by owner, field and serde fn). Otherwise generation fails at the field's file:line.
  • Registered:
    • Recording.schema (recording.rs:73) → codec currentRecordingSchema (u32; decode accepts only RECORDING_SCHEMA).
    • BooleanCriteria.true/false (evaluation_model.rs:62-73) → codec presence. The manifest type is {"triState": EvaluationInput}, meaning absent key | explicit null | value, instead of option(option(..)).
  • decision_model.rs:23 (with = "tagged_file_bytes") already has #[ts(type)].
  • Tests:
    • boolean_criteria_manifest_entry_is_tri_state checks the tri-state encoding, the codec name, omitWhenEmpty/default, and Recording.schema's codec.
    • unregistered_serde_with_field_fails_generation gives extra_dto.rs:5: ExtraDto.a uses serde \deserialize_with=custom` but has no `#[ts(type)]` and no FIELD_CODECS registration`.
  • Mutation: removing the Recording.schema registration → aimux-core/src/recording.rs:74: Recording.schema uses serde \deserialize_with=deserialize_current_schema` but has no …(line 74 is the field; its attribute is on :73).--check` and 8 manifest tests turn red.
  • Proxy/skip/cfg_attr handling is deferred to 2.2.

Not in scope / notes

  • No per-language codegen (2.2). No extra-file detection in --check (2.4). aimux-provider V4 types are out of scope (3.3).
  • Category is assigned per file (e.g. content.rs → multimodal; model_catalogue.rs/model_id.rs → provider; everything not listed → core). The table exists in both the Python script and extract.rs, and the coverage lines compare name category file, so if the two tables drift the check fails.
  • CRLF: resolved, see above.
  • inputHash covers all of aimux-core/src, so any edit there makes the manifest stale until regenerated (RFC-0042:222 "input hash").
  • Generic parameters in field types appear as ref with the parameter name (generics lists them on the type).

…osure-dtos.txt

- scripts/gen_closure_inventory.py writes docs/plan/closure-dtos.txt
  (name, category, file; derive(TS)-without-export and aimux-web wire
  in separate sections). Multi-line #[ts(\n export are counted.
- tools/aimux-codegen: extract.rs (syn) builds the type model from
  aimux-core/src serde attributes, incl. discriminated unions;
  manifest.rs writes generated/manifest.json (sdk read from
  fixtures/aisdk/VERSIONS.json, generator, inputHash, schemaRevision)
  and generated/coverage.txt; both under --check.
- Unregistered field types and hand-written serde without a codec fail
  generation with file:line; coverage mismatch fails and writes nothing.
- tests/manifest_test.rs: coverage == list, mutations (removed type,
  fake list line, extra type, VERSIONS.json change, unregistered type),
  wire names cross-checked against bindings/node/src/types.
A CRLF checkout (core.autocrlf=true) now yields the same inputHash and a
byte-identical manifest. Normalized in code rather than .gitattributes so
it holds regardless of checkout. Test: crlf_checkout_gives_identical_manifest.
- closure_dto_list_is_reproducible_from_source: runs
  gen_closure_inventory.py --check; asserts [exported] 193 incl. the
  multi-line #[ts(\n export) ToolChoice.
- serde_attribute_mutations_make_manifest_stale: skip_serializing_if,
  flatten, untagged, rename_all, field rename each make --check stale.
- gen_closure_inventory.py skips tools/aimux-codegen in ts-rs scans (its
  docs/test fixtures quote the attributes as text); closure.md
  regenerated (extract.rs enters the >1000-line table).
…CS entry

- extract.rs: a with/deserialize_with/serialize_with field without
  #[ts(type)] must match FIELD_CODECS (owner, field, serde fn), else
  generation fails at the field's file:line.
- Register Recording.schema (currentRecordingSchema) and
  BooleanCriteria.true/false (presence): the manifest type is
  triState(EvaluationInput) = absent | null | value, not Option<Option>.
- Tests: boolean_criteria_manifest_entry_is_tri_state,
  unregistered_serde_with_field_fails_generation. Manifest and
  closure.md regenerated.
…-only control

serde_attribute_mutations_change_the_modelled_field compares manifests
without inputHash: skip_serializing_if -> omitWhenEmpty, flatten ->
flatten, untagged -> style untagged->external, rename_all -> wire
inputSchema->input_schema, field rename -> wire true->yes.
comment_only_edit_leaves_manifest_identical_except_input_hash is the
control.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant