Repository navigation
New-user README and docs/ guides, release loader fix and 1.0-240M work - #10
Merged
Merged
Conversation
…l audit, NPU export, and SFT/DPO tooling
…ded offsets across val.bin
…22,500/22,888 watchdog gates
…SWA-3 soup base initialization
…PPL, SWA-3 3.4844 / 32.60 PPL)
…ect SWA checkpoints in archiver
…ce model with step_1999 resolution
…PO max-len to 1024 for reasoning traces
…cher -MaskFile selects the loss mask (train.mask.notrunc.bin zeroes the 7,943 docs cut at max_seq_len 2048), -WarmupSteps passes --warmup-steps, and -TrainLog captures train.py output as UTF-8. Defaults keep the old behaviour, so watchdog_gate_final.py is unaffected. The launcher now logs [FAIL] on a non-zero train.py exit instead of always reporting success.
Resumes the run paused at step 750 by calling train.py directly from the I: archive checkpoint, keeping max-steps 2000 and cooldown from 1500 with no second warmup. launch_sft_130m_v3.ps1's resume path would retarget to step + Steps and shift the cooldown. Takes and releases GPU.lock and logs train.py output to out/sft_130m_v3_notrunc.log.
…set DPO margins, tokenizer metadata - check_eval_contamination: prompt_overlap() (exact, containment, short n-gram) and --dpo-pairs; eval_honest_suite replaces the 19 prompts that overlapped DPO pairs and refuses to run on overlap - free_think: _stream_bpe records sampled <think>/</think>; FreeThinkSession.model_closed_think; eval_freethink(_showcase) report it instead of the always-true wrapper flag - train_dpo_130m_v3: margin/acc/loss over all pairs at start, every --full-eval-every and end, written to <out-dir>/pair_eval.json - train.py: inherit tokenizer metadata on resume; BPE runs without a tokenizer now raise; 130M SFT launchers pass --tokenizer
…nce dataset & 240M invariants - scripts/train_dpo_130m_v3.py: fix prompt prefix inversion to canonical [USER, lang_id], aligning Token-Shift (mu-mix) with SFT and adding dynamic Dravidian/English script routing - scripts/audit_dpo_pairs.py: preference pair auditor checking <think> symmetry, length ratios, and abstention gradients - scripts/build_dpo_pairs_v2.py: dataset rebuild pipeline producing 900 balanced pairs (81.2% symmetric <think>, 0.98x length ratio, +398 net positive abstention gradient) - scripts/train_dpo_v2_both.py & eval_dpo_v2_candidates.py: dual candidate training and 5-gate decontaminated evaluation battery - .agents/ARCHITECTURE.md: add Section 9 codifying 240M non-negotiable architectural mandates (4096 doc packing with alpha_t=0/conv/shift boundary reset, SFT abstention prior mass, and inference scratchpad priming) - .agents/documentation.md: record DPO v2 campaign and decontaminated scorecard
… 240M SSD - Document-ID Gated Finite Prefix Sums in Chunked SSD (C=64) - Prevents NaN from (-inf) - (-inf) in intra-chunk prefix sum differences - Vectorizes intra-chunk attention, chunk-to-state write, inter-chunk decay, and state-to-chunk read
…(d_0 - m_0) - Replaces d_0 with d_prev = d_0 - m_0 in Equations (3) and (4) of Section 9.1 - Completely seals t=0 seam boundaries when EOT lands on index C-1 of the previous chunk
…stemic abstention generator & 4096 FFD packer
…-point document reset and V0 highway
…o train.py with 8GB VRAM optimizations
… and live smoke test
…conv/scale/RMSNorm parity
…_sft_stem_open_v2.py
…lback and ignore raw jsonl
…e guard, watchdog scripts, and master blueprints
README now opens with a four-step quick setup (clone, install, download the three Hugging Face files, run chat.py with --device cuda) instead of training instructions, plus short briefs that link to the new guides. New docs/: - SETUP.md: language legend at the top, GPU/CUDA install, both weight download routes, per-language setup for all nine languages, chat.py options and in-chat commands, Python usage, troubleshooting. - ARCHITECTURE.md: readable explainer of the recurrent model. - ROADMAP.md: 0.9 status, 1.0-240M sessions, data licensing, how to help. - DOCUMENTATION.md: moved from the root; adds a "using the code" part ahead of the existing math. - figures/: existing diagrams plus four new on-brand charts, drawn by scripts/plot_docs_figures.py. fix(chat): the safetensors loader ignored slab_v3 from config.json and loaded with strict=False, silently dropping 63 tensors (token shift, write gate, V0 highway). Every user of the Hugging Face release got gibberish. It now passes slab_v3/slab_v4 through and warns when any checkpoint tensor fails to load. requirements.txt: add safetensors, which chat.py needs for the release.
uncoalesced
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
People who wanted to try parentheses-0.9 couldn't find out how from the README, which mostly covered training. This PR rewrites it around a quick setup and adds a
docs/folder. While testing the setup I found why a Bluesky tester got gibberish: the chat script was loading the Hugging Face weights wrong.Loader bug (scripts/chat.py)
When
--checkpointpoints at the Hugging Face folder,load_model_and_tokenizerbuilt its config without readingslab_v3fromconfig.json, then loaded withstrict=False. That silently dropped 63 tensors (token shift, write gate, value highway), so every user of the release got number soup. The loader now passesslab_v3andslab_v4through and prints a warning if any checkpoint tensor fails to load. After the fix, the model on a GPU gives fluent (if often wrong) English and routes Kannada correctly.requirements.txtalso gainssafetensors, which this path needs.Docs
README.md: four-step quick setup near the top (clone, install, download the three HF files, run with--device cuda), the CPU-only PyTorch trap, an honest "what it can and can't do yet", and short briefs linking each guide.docs/SETUP.md: legend with a jump-to-your-language table for all nine languages, GPU setup, both weight download routes, per-language sections, every chat option and in-chat command, Python usage, and troubleshooting.docs/ARCHITECTURE.md: a readable explainer of the recurrent model, with figures.docs/ROADMAP.md: 0.9 status, the 1.0-240M sessions (including the Session 4 regression), the data licensing problem, and how to help.docs/DOCUMENTATION.md: moved from the root, with a new "using the code" part ahead of the unchanged math.docs/figures/: the existing diagrams plus four new charts, drawn byscripts/plot_docs_figures.py.Each guide was drafted, then checked by a separate reviewer against the code, with the documented commands and Python snippets run on a real install.
Also in this branch
This branch carries the existing 1.0-240M work (slab_v4 backbone, Net2WiderNet growth, SFT/DPO builders, session launch scripts and watchdogs, license guard) and a merge of main.
.agents/is no longer tracked: its journal was scrubbed of local paths first and is now in.gitignore.Known issues, not fixed here
nit, because two Kolami marker words are common Telugu words. The docs tell people to pass--lang tel.LICENSE-MODELholds Apache 2.0 text, while the 0.9 weights are CC BY-NC 4.0 on Hugging Face. The README links the CC BY-NC licence directly.nitas Gondi.