Skip to content

feat(uast): real-world language coverage + durability hardening - #25

Merged
dmytrogajewski merged 1 commit into
mainfrom
feature/language-coverage-durability
Jul 28, 2026
Merged

dmytrogajewski merged 1 commit into
mainfrom
feature/language-coverage-durability

Conversation

@dmytrogajewski

Copy link
Copy Markdown
Contributor

Summary

Closes the "complexity/cohesion skip some languages" bug class end-to-end: every declared language now detects functions on real open-source code, and the static-analysis path is hardened against hangs, crashes, and signal-deafness. Validated against the frozen Go oracle by the full 505-cell compat gate (0 un-allowlisted divergences) and a 37-repo OSS sweep.

Language coverage

Engine (cf-uast): multiple conditioned mapping rules per tree-sitter node type (declaration order, first passing when: wins, unconditional = catch-all), dispatched by pattern root type; dotted-path conditions (value.type == "arrow_function").

Mappings: JS/TS/TSX arrow + function-expression declarators and class-property methods; Java constructors → Method; Elixir def/defp/defmacro/defmacrop/defmodule; Clojure defn/defn-/defmacro/deftest/defmethod/fn; SQL CREATE FUNCTION; Lua M.f = function + table-field functions; Haskell do/lambda binds incl. $-chained; Fortran program units; un-hijacked .nim and .md extensions from injected sub-grammars.

OSS sweep deltas (files with ≥1 detected function): nim 0→65%, elixir 0→86%, clojure 0→84%, sql 0→40%, fortran 17→99%, typescript 35→58%, javascript 52→77%, tsx 78→96%, haskell 61→87%, java 84→88%, lua 85→89%, powershell timeout→44%. Reusable harness in tests/oss-coverage/.

Durability

  • Exponential unmapped-subtree re-walk fixed (2^depth): posh-git 300s-timeout → 0.19s; compat matrix wall 2909s → 1404s
  • SIGINT/SIGTERM cooperative cancellation (measured 27ms → exit 130, no truncated report; double-signal kills)
  • Stack hardening in cf-complexity (iterative visitors + stacker guards)
  • 30s per-file parse watchdog; 64 MiB per-file read cap; EPIPE-safe stdout (exit 141); skipped-file counter on stderr
  • Go parity: single-FILE analysis roots work in all 8 static walkers; empty-source uast parse emits null

Compat system

  • 46 tree-sitter grammars vendored (all languages wired); grammar-conformance test regenerated
  • Differential fuzz layer repaired (never ran since the Go removal: missing go.mod, stale rust/ paths) — self-check now proves it catches planted bugs
  • Allowlist: evidence-backed go-nondeterminism entry (cohesion summary picked from randomized Go map iteration — proven by flipping live-oracle runs); tracked oracle-defect (--include-vendored@hercules: frozen Go exceeds the oracle's own 600s budget, fixed Rust finishes in 13s); tracked invalid-UTF-8 token-offset gap with distilled evidence

Test plan

  • cargo fmt --all -- --check
  • cargo test green on all touched crates (uast, mappings, mapping, commands, complexity, cohesion, file-history)
  • Full compat gate tests/compat/run.sh full: 359 PASS / 145 EXPECTED_EMPTY / 0 un-allowlisted FAIL/SIM; meta-gate still catches planted bugs
  • 37-repo OSS sweep re-run with the new binary
  • Signal/EPIPE/timeout behaviors measured directly

🤖 Generated with Claude Code

https://claude.ai/code/session_018yqzfJ42Q1PzEbu4LRUmRe

Language coverage — every declared language now detects functions on real
OSS code (validated by a 37-repo sweep, tests/oss-coverage/):

- Engine: multiple conditioned mapping rules per tree-sitter node type
  (declaration order, first passing conditions wins, unconditional rule is
  the catch-all) dispatched by pattern root type, plus dotted-path
  conditions (`value.type == "arrow_function"`) in cf-uast lowering.
- Mappings: JS/TS/TSX arrow-function and function-expression declarators
  and class-property methods; Java constructors as Method (were Struct);
  Elixir def/defp/defmacro/defmacrop/defmodule; Clojure
  defn/defn-/defmacro/deftest/defmethod/fn; SQL CREATE FUNCTION; Lua
  `M.f = function` and table-field functions; Haskell do/lambda binds
  (incl. `$`-chained); Fortran program units; un-hijacked `.nim` from
  nim_format_string and `.md` from markdown_inline.
- Sweep deltas: nim 0->65%, elixir 0->86%, clojure 0->84%, sql 0->40%,
  fortran 17->99%, typescript 35->58%, javascript 52->77%, tsx 78->96%,
  haskell 61->87%, java 84->88%, lua 85->89%, powershell timeout->44%.

Durability:

- Fix exponential unmapped-subtree re-walk in cf-uast lowering (2^depth;
  posh-git 300s-timeout -> 0.19s, full compat matrix 2909s -> 1404s).
- SIGINT/SIGTERM cooperative cancellation of static walks (27ms exit 130,
  no truncated report; second signal force-kills).
- Stack hardening in cf-complexity (iterative visitors + stacker guards).
- 30s per-file tree-sitter parse watchdog; 64MiB per-file read cap.
- EPIPE-safe stdout (exit 141 instead of panic on `| head`).
- Skipped files counted and surfaced on stderr.
- Go parity: single-FILE analysis roots work in all 8 static walkers;
  empty-source `uast parse` emits `null`.

Compat system repairs:

- Differential fuzz layer ran for the first time since the Go removal:
  added standalone go.mod, fixed repo-root/rust-dir/seed/distill paths,
  fixed self-check; it now proves it catches planted bugs.
- Grammar conformance test regenerated for the 46 newly vendored grammars
  (UNWIRED emptied, dead-rule allowlist extended).
- Allowlist: evidence-backed go-nondeterminism entry for the cohesion
  summary message (Go picks it from randomized map iteration); tracked
  oracle-defect entry for --include-vendored@hercules (frozen Go exceeds
  the oracle's own 600s budget; Rust finishes in 13s); tracked
  invalid-UTF-8 token-offset gap with distilled fuzz evidence.
- cf-file-history unit test updated to the Go-parity nil-slice `null`.

Full 505-cell compat gate: 0 un-allowlisted divergences; the only red
cells are the documented oracle defect and the tracked invalid-UTF-8 gap.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018yqzfJ42Q1PzEbu4LRUmRe
@cursor

cursor Bot commented Jul 27, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@dmytrogajewski
dmytrogajewski merged commit d271483 into main Jul 28, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants