Explore a 5 KB lexical language detector inspired by gpu-lexer

[ealmloff]

Summary

Prototype the compact lexical hashing idea from https://gpu-lexer.vercel.app as a separate CPU language detector:

begin/end window → hashed tokens + adjacent pairs → log1p/L2 counts
                 → int4 Dense(512,16) → ReLU → int4 Dense(16,48)

The BTQ1 export is 4,752 raw bytes, including scales, biases and header; gpu-lexer's advertised 27.5 KB is a Brotli-compressed browser bundle. betlang::detect stays on the production model while this experiment measures the accuracy cost.

Includes QAT training/export, NumPy inference, an independent Rust example and a manifest-based comparison with the production detector. Training and comparison results will be added after the current run finishes. See scripts/TINY_STUDENT.md for format and commands.

Verification

• cargo check --all-targets --locked
• cargo clippy --locked --all-targets -- -D warnings
• cargo fmt --check
• cargo test --all-targets --locked
• cargo build --release --example tiny
• BETLANG_TINY_EXAMPLE=... python scripts/test_tiny_student.py — 5 tests, including Python/Rust feature and logit parity on arbitrary bytes and window boundaries.
• python -m flake8 scripts/{tiny_student,train_tiny_student,test_tiny_student}.py --select E9,F63,F7,F82

Accuracy evaluation is still in progress; this is not a proposal to replace the production model yet.

Link to Devin session: https://dioxus.staging.devinenterprise.com/sessions/5842494daec94db89dea35705547b830
Open in Devin Desktop: https://dioxus.staging.devinenterprise.com/desktop/session/5842494daec94db89dea35705547b830?variant=devin-insiders
Requested by: @ealmloff

[staging-devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR that start with 'DevinAI' or '@devin'.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring