Replace wordseq detector with Shannon-bit binary CNN ensemble

BENCHMARKS.md

@@ -3,7 +3,28 @@
 Baselines are informational. They are useful for spotting large regressions, but
 not part of Betlang's semver contract.
 
-## 2026-05-15
+## 2026-08-22 (Shannon-bit binary CNN ensemble)
+
+Environment:
+
+- Host: `aarch64-apple-darwin`, `arm64` (Apple M4 virtual CPU)
+- Rust: `rustc 1.95.0 (59807616e 2026-04-14)`
+
+Native command:
+
+```bash
+cargo bench --bench detect
+```
+
+| Case | Bytes | Median time | Throughput |
+|---|---:|---:|---:|
+| short | 68 | 41.068 ms/inference | 1.6558 KB/s |
+| full window | 4623 | 43.355 ms/inference | 106.63 KB/s |
+
+The five-model binary ensemble trades latency for accuracy against the
+previous single wordseq model below.
+
+## 2026-05-15 (previous wordseq model)
 
 Environment:
 

Cargo.toml

@@ -21,21 +21,17 @@ include = [
     "/RELEASING.md",
     "/assets/confusion-by-size.png",
     "/assets/confusion-overall.png",
-    "/assets/magika/source-student-q4.bin",
+    "/assets/bnn/source-bnn2.bin",
     "/benches/detect.rs",
     "/examples/*.rs",
     "/snippets/demo.rs",
     "/src/**",
     "/tests/**",
 ]
 
-[dependencies]
-fearless_simd = "0.4"
-
 [dev-dependencies]
 criterion = { version = "0.5", default-features = false, features = ["cargo_bench_support"] }
 ignore = "0.4.25"
-rand = "0.8"
 rayon = "1.10"
 
 [lib]

MODEL_CARD.md

@@ -2,12 +2,15 @@
 
 ## Artifact
 
-- File: `assets/magika/source-student-q4.bin`
-- Format: weights-only MSQ1 quantized tensor payload
-- Size: 47,840 bytes
-- SHA-256: `8493d2d3757572c8661141e414b1c0755aa08d4c4e5382dfbbc6b73b02d89083`
-- Architecture: `wordseq-b1024-k3-m2048-tiny-3conv-hidden`
-- Tokenizer: word-unit tokenizer version 3
+- File: `assets/bnn/source-bnn2.bin`
+- Format: packed "BBN2" binary-CNN ensemble payload
+- Size: 1,165,464 bytes
+- SHA-256: `fe36df96d9b6d268118b21b764c3be6dda33596e0fafeacd2bc0b26e3fc143ca`
+- Architecture: ensemble of five binary CNNs over Shannon-coded byte windows
+  (binary weights and activations, XNOR + popcount convolutions, integer
+  thresholds, OR-pooling, segmented popcount heads, i16 quantized classifier)
+- Input: canonical Shannon coding of the raw 2048-byte Magika begin/end
+  window into a fixed 16,384-bit stream plus a codeword-boundary bitplane
 - Output head: 48 model labels exposed one-to-one as public `Language` variants
 
 ## Intended Use
@@ -19,94 +22,87 @@ decisions, malware classification, or legal identification of file provenance.
 
 ## Training Source
 
-The student was originally trained from Google's Magika v3.3 teacher
-predictions over a source-language corpus assembled from extension-suffixed
-files (the `bigorig` split, extracted from a GitHub partial-clone blob index of
-roughly 6,000 popular code repositories).
-
-The shipped artifact is that model fine-tuned from its exported checkpoint on a
-rebuilt public corpus (see `scripts/build_finetune_corpus.py`): per-language
-samples from `bigcode/the-stack-smol-xl` and `bigcode/the-stack`, GitHub repo
-files for labels absent from The Stack, and a small synthetic set targeting the
-Markdown/YAML bare-list ambiguity reported in issue #5, including train-only
-bare `- item` lists.
-
-The fine-tune distills the same Magika v3.3 teacher one-vs-all: per-class
-binary cross-entropy against the teacher's raw head-label marginals, plus a
-hard term against the teacher argmax discounted by the teacher's in-head
-probability mass. Unlike the original softmax distillation, this never
-renormalizes away the probability mass the teacher assigns to labels outside
-the 48-label head (such as `txt`), so inputs the teacher considers ambiguous
-or out-of-scope train toward uniformly low logits and keep low softmax
-confidence at inference.
-
-The shipped run additionally self-distills from a ~2x larger intermediate
-parent (`wordseq-b1536-k3-m2048-med-3conv-hidden`, 0.9492 test teacher parity)
-trained on the same corpus with the same one-vs-all scheme; the parent's
-per-class sigmoid marginals are cached with `scripts/cache_self_distill.py`
-and added as a second BCE term.
-
-The model distills teacher probabilities and filesystem-extension labels. It
-does not contain original source files, but its labels and soft targets are
-derived from the training corpus and Magika teacher.
+The models are trained on a rebuilt public corpus (see
+`scripts/bnn/build_corpus.py`): per-language samples from
+`bigcode/the-stack-smol-xl` and `bigcode/the-stack`, GitHub repo files for
+labels absent from The Stack, and synthetic sets targeting the Markdown/YAML
+bare-list ambiguity and other confusable pairs.
+
+Inputs are the raw Magika begin/end byte windows encoded with a canonical
+Shannon code learned from training byte frequencies (add-one smoothing,
+12-bit maximum code length, codes ordered by `(length, byte)`). No tokenizer
+is used. Each window becomes two bitplanes: the Shannon code bits and the
+codeword-boundary markers.
+
+Training distills Google's Magika v3.3 teacher predictions alongside
+filesystem-extension labels. The binary networks are trained binary from
+scratch (annealing a float network collapses); sparse learnable thresholds,
+OR-pooling, and segmented popcount heads make the fully-binarized networks
+trainable. Ensemble members are fine-tuned on short inputs and synthetic hard
+pairs, and ensemble weights are grid-searched under fixture and edge-case
+constraints (`scripts/bnn/weight_search.py`).
+
+The model does not contain original source files, but its labels and soft
+targets are derived from the training corpus and Magika teacher.
 
 ## Evaluation
 
-Held-out filesystem-label test split of the rebuilt corpus (34,087 files,
-train/valid/test repositories are disjoint, rows where the teacher keeps at
-most 10% of its probability mass on the head labels are excluded):
+Held-out filesystem-label test split of the rebuilt corpus (27,465 files,
+train/valid/test repositories are disjoint), evaluated with exact integer
+export semantics (`scripts/bnn/eval_export2.py`):
 
-- `test_fs_accuracy=0.942353`
-- `macro_recall=0.939690`
-- `test_teacher_parity=0.944055`
+- `test_accuracy=0.941635`
+- `test_macro_recall=0.943226`
 
-For comparison, the pre-fine-tune artifact scores `test_fs_accuracy=0.926160`,
-`macro_recall=0.929904`, and `test_teacher_parity=0.922111` on the same split.
-The rebuilt split is balanced across all 48 labels (including rare classes),
-so these numbers are not comparable to the `bigorig` metrics reported for
-earlier artifacts.
+The previous shipped wordseq artifact (`source-student-q4.bin`) scores
+`test_accuracy=0.934972` and `test_macro_recall=0.934863` on the same split.
 
-On ambiguous bare `- item` lists (valid YAML and valid Markdown), the median
-top-1 probability drops from 0.92 (pre-fine-tune) to 0.51, while YAML or
-Markdown remains the top prediction.
+On ambiguous bare `- item` lists (valid YAML and valid Markdown), the model
+reports a split YAML/Markdown distribution with top-1 probability below 0.9
+rather than picking one with certainty.
 
-Most remaining confusion sits on genuinely ambiguous pairs where the teacher
-also splits its probability on the confused files: `c`/`cpp`,
+Most remaining confusion sits on genuinely ambiguous pairs: `c`/`cpp`,
 `javascript`/`typescript`, `markdown`/`yaml`, `ini`/`toml`, `batch`/`shell`,
-and `php`/`html`. Several other cells are corpus extension-label noise (for
-example `.vb` files containing SQL dumps) where the teacher agrees with the
-model on 80%+ of the confused files.
+and `php`/`html`.
 
-The README confusion matrix groups the same held-out split by file-size bucket.
+## Inference Semantics
+
+All heavy compute is binary CPU work: activations and weights are single bits
+packed into 64-bit words, convolutions are XNOR + popcount
+(`z' = 2 * matches - fan_in`), basis outputs combine with fixed-point integer
+coefficients (`FIX=65536`), activations are integer threshold comparisons,
+pooling is bitwise OR, and the classifier is an i16 integer dot product over
+segmented popcounts with per-class float scale/bias. Rust inference is
+bit-exact against the Python integer reference
+(`cargo test --release python_parity`).
 
 ## Known Weaknesses
 
 - Very short inputs are intentionally rejected when fewer than eight
   non-whitespace bytes are available.
-- Ambiguous snippets can put several languages close together even when a human
-  can infer the language from file naming context. A bare `- item` list with no
-  heading is valid YAML and valid Markdown; the model reports a split
-  YAML/Markdown distribution for such inputs rather than picking one with
-  certainty.
-- The classifier uses content only. It does not inspect file names, extensions,
-  shebangs outside the model window, repository metadata, or build-system
-  context.
+- Short files (< 512 bytes) are harder than full windows; accuracy on a
+  short-truncated validation set is ~0.78 versus ~0.94 on full windows.
+- Ambiguous snippets can put several languages close together even when a
+  human can infer the language from file naming context.
+- The classifier uses content only. It does not inspect file names,
+  extensions, shebangs outside the model window, repository metadata, or
+  build-system context.
 - Non-source formats are out of scope unless represented by a public
   source-language variant.
 
 ## Reproducibility
 
-Training and evaluation scripts live under `scripts/`. The frozen recipe is
-documented in `scripts/TRAINING.md`, including the expected external Magika
-teacher assets, cache layout, training command, evaluation command, and
-expected metrics.
+Training and evaluation scripts live under `scripts/bnn/`. The pipeline is
+documented in `scripts/bnn/TRAINING.md`, including corpus construction,
+Shannon codebook learning, binary training, fine-tuning, ensemble weight
+search, export, and the Rust/Python parity check.
 
 The published crate package intentionally includes only the runtime model
 artifact and user-facing docs. Training scripts and generated analysis files
 remain repository artifacts.
 
 ## Attribution
 
-The embedded student model was trained from outputs of Google's Magika teacher
+The embedded models were trained from outputs of Google's Magika teacher
 model. Magika is published by Google under Apache-2.0. Betlang's source code is
 MIT licensed; keep Magika attribution with redistributed model artifacts.