Skip to content

refactor: replace the DEL/OSTW evidence model with ordinary tests (#112) - #122

Open
e54-bot wants to merge 8 commits into
mainfrom
feat/112-tests-first
Open

refactor: replace the DEL/OSTW evidence model with ordinary tests (#112)#122
e54-bot wants to merge 8 commits into
mainfrom
feat/112-tests-first

Conversation

@e54-bot

@e54-bot e54-bot commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

Summary

Implements #112 under the approved tests-first model (wrightkit/.github#60): DEL/OSTW verification is now expressed as ordinary tests and test data instead of a first-class evidence subsystem.

  • Defect fixes found while wiring the dormant corpus live (separate commits):
    • fix(syntax): interpolated-string post-hole cursor was close.end + 1 under half-open spans — dropped the first char of every post-hole text part and panicked on multi-byte chars.
    • fix(semantic): NodeId was a per-file counter used as keys in project-wide semantic maps; cross-file collisions made EErrorSeverity resolve to EUX_MapState and index OOB in HIR lowering (plus ~26 phantom diagnostics on mobawatch). NodeId now carries FileId; the member write is bounded.
  • Evidence model removed: src/compatibility.rs (579 LOC), the maintainer compatibility/compatibility CLI commands, // status:/// evidence:/// matrix: headers, matrix evidence lists, and the compiler-utility.ostw-reference-evidence matrix entry.
  • Corpus harness simplified (tests/corpus.rs): // expect: only, recursive walk (except projects/), all five unknown fixtures resolved to verified concrete outcomes, third-party fixtures keep // source:/// license: for real attribution while WrightKit-authored fixtures carry none.
  • compatibility/ostw/ converted, not deleted blindly: vendored mobawatch + protect-ban now run end-to-end in tests/real_projects.rs; the reconstruction boundary in tests/reconstruction-fixtures/ is verified by tests/reconstruction.rs — including a manifest-conformance check that every declared bound id resolves through the real binding tables. ~180 files of dormant observation DBs, probe data, and the oracle runner removed.
  • Docs/ADR: compatibility.md + workshop-conformance.md rewritten to the verification surfaces; provenance/limitations/inventory/cli/README/AGENTS de-evidenced; ADR-0005 records the tests-first decision and partially supersedes ADR-0002's evidence framing. Issue Ground DEL/OSTW compatibility tests in real-project regressions and Workshop conformance evidence #26's tracked real-project coverage now exists.

Net: ~19,700 lines removed, ~500 added.

Independent review

A subagent that did not author the change reviewed the full diff: verdict approve-with-nits, no correctness bugs. Actionable findings were fixed in 96dac33 (manifest/code drift wired into real assertions, corpus phase-counting gap, stale docs/claims, .gitattributes/Cargo.toml leftovers); remaining nits are documented non-issues or out of scope.

Test plan

  • cargo fmt --all -- --check
  • cargo clippy --all-targets -- -D warnings
  • cargo test --workspace --all-targets — 16 test targets green, including new real_projects + reconstruction suites
  • cargo run --quiet -p deltin-rs-cli -- support --check — 128 entries valid
  • git diff main --check

Teakowa added 7 commits October 4, 2026 05:53
@Teakowa
fix(syntax): keep full post-hole text in interpolated strings
e204620 
Refs #112.

Token spans are half-open byte ranges, so a hole's close.end is already
the byte after '}'. Resuming the text cursor at close.end + 1 silently
dropped the first character of every post-hole text part and panicked
outright when that character was multi-byte — found while exercising the
vendored mobawatch project.
@Teakowa
fix(semantic): scope NodeId by file so project tables cannot collide
ac732c9 
Refs #112.

NodeIds were per-parse counters starting at 1, but every semantic table
(types, resolution, scopes, symbol maps) is project-wide: decls in
different files with the same sequence position resolved to each other's
symbols. In the vendored mobawatch project, EErrorSeverity's decl
resolved to EUX_MapState's enum and the member-write pass indexed past
its member list — plus 26 phantom diagnostics from mis-resolved symbols.

NodeId now carries its FileId; the member write is also bounded so a
diagnosed decl cannot panic the lowerer.
@Teakowa
refactor(compat): replace the evidence model with the ordinary corpus…
eccad3b 
… harness

Refs #112.

- Drop src/compatibility.rs and the 'compatibility'/'maintainer
  compatibility' CLI commands: the 579-line taxonomy wrapped the same
  parse/project/semantic/HIR pipeline the corpus harness already runs.
- The corpus harness keeps concrete expected outcomes (ok, parse-error,
  semantic-error, hir-error) and now also walks regressions/.
- Resolve every 'expect: unknown' fixture to its verified outcome:
  the two incomplete-syntax fixtures are parse-error, the two
  struct-ref fixtures compile under the decided unknown-type contract
  (documented in their // note: headers), and invalid-ds-toml is
  already covered by a dedicated project test.
- Remove // status:, // evidence:, and // matrix: bookkeeping headers.
- Keep // source: and // license: only on fixtures derived from the
  pinned ItsDeltin/OSTW revision; drop self-referential del-rs provenance
  from WrightKit-authored fixtures.
@Teakowa
refactor(matrix): drop per-entry evidence-path bookkeeping
35fbc24 
Refs #112.

Every feature carried evidence = ["docs/inventory.md"] (or a list of
compatibility-package paths) solely to satisfy mechanical path-exists
validation; no production or test consumer reads the field. Support
claims are verified by the ordinary test suite, so the field, the
path checks, and the ostw-reference-evidence entry it existed to
justify are removed.
@Teakowa
refactor(tests): turn the pinned reference package into live tests
f1623b3 
Refs #112.

compatibility/ostw/ held ~180 files of dormant evidence: observation
databases (corpus.json, reference.json, results.json), a langserver
oracle runner, and upstream-output probes no test consumed.

- The vendored mobawatch and protect-ban projects move to
  tests/real-projects/ and now run end-to-end through
  load -> semantic -> HIR -> Workshop lowering in tests/real_projects.rs.
- The reconstruction boundary moves to tests/reconstruction-fixtures/
  and is verified by tests/reconstruction.rs: positive fixtures must
  reconstruct to OSTW the native parser accepts, reject fixtures must
  fail with structured reconstruct-* codes, and every fixture dir must
  be declared in support-boundary.json.
- The probe observation data, point-in-time result databases, and the
  oracle runner are removed: their conclusions are already encoded in
  the implementation, and no executable test consumed them.
@Teakowa
docs: route compatibility verification through ordinary tests
c16e1bb 
Refs #112.

- compatibility.md becomes the verification contract: the corpus
  harness, real-project runs, and the reconstruction boundary replace
  the evidence package and its report schema.
- workshop-conformance.md drops the evidence-classification report in
  favor of the test surfaces; the real-project coverage issue #26
  tracked now exists under tests/real-projects/.
- provenance.md keeps the pinned upstream identity and licensing rules
  but drops the deleted observation databases and limits fixture
  headers to third-party-derived material.
- limitations.md, cli.md, README files, AGENTS.md, and the
  architecture/inventory docs drop evidence-model terminology for
  declared support state and executable tests.
- ADR-0005 records the tests-first decision and partially supersedes
  ADR-0002's evidence-model framing.
@Teakowa
test: verify the reconstruction boundary against its manifest
96dac33 
Refs #112. Review follow-ups:

- support-boundary.json now declares the two codes the reconstructor
  actually emits (dangling-subroutine, unsupported-localized-string),
  and the reject test asserts emitted codes are declared.
- declared_bound_ids_resolve_in_binding_tables checks every manifest
  boundActionId/boundValueId/enumDomain/enumMember against the real
  binding tables, making the bound_* APIs live consumers and catching
  declared-but-unbound drift.
- The corpus harness now walks tests/corpus recursively (minus
  projects/) and fails expect: ok on project-phase or unclassified
  errors instead of counting only semantic/HIR phases.
- Fixed stale claims: signature.rs binding-exercise comment,
  README reconstruction/verification rows, bound_* docstrings.
- Removed .gitattributes (its only rules targeted deleted paths) and
  repaired the stale Cargo.toml exclude pattern.
- Matrix structs deny unknown fields so retired keys fail loudly.
@e54-bot e54-bot mentioned this pull request Oct 4, 2026
Merged
6 tasks done

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@e54-bot @Teakowa