Repository navigation
Port the census watcher: the newest HF models against the generated registry - #242
Merged
Merged
Conversation
|
Docs7 for 1bit-monster/engine
Commit |
…egistry
1bit-MONSTER's watcher (Testing/hf_new_models.py) compiled a C++ probe of
rcpp_arch_from_string through census_coverage.build_mapper(). This repository's
registry is generated data (registry/architectures.json), keyed by the HF
architecture class the census counts, so the watcher is a plain lookup: no g++
probe and no strip_arch. That is how PORTING.md's step 5 could land the sweep
(census.yml) and drop the watch half without a compile-time cost.
tools/census_watch.py polls the newest text-generation models (createdAt
descending), reads each one's class from the config inline in the listing and
compares it against the registry; exit 1 is the alert. scripts/census-watch.sh is
the entry point (log file + optional origin/main refresh), the
1bit-census-watch.{service,timer} pair is the 04:30 UTC timer from a
~/census-main worktree, and census-watch.yml is the CI path with the deduped
alert issue. State lives outside the checkout ($CENSUS_WATCH_STATE, default
~/.1bit/census-watch-state.json), so a run never dirties the tree it reads.
Two bugs the first live runs found, both pinned by the test:
- Pagination. HF ignores `p=` on a sorted listing: every page returned the same
first 100 models, so --limit 120 fetched 101 and still called it 120. The walk
follows the Link header's cursor instead, the way tools/census.py sweeps.
- State shape. classify() returned (id, reason) tuples for the unverifiable
models, so remember() wrote tuple keys and json.dump failed on the sorted keys
— the run reported its alert and then died before saving state.
First run against main's registry, the 120 newest text-generation models: 102 with
an architecture, 94 mapped, so six uncovered classes the alert had been blind to
since it stopped running on 2026-09-23 -- Qwen3_5MVLAAbsorbedForCausalLM (2
models, e.g. TelperionAI/Qwen3.8-27B-GLA-g2), ArgonneModel, GPT,
Glm5NextForConditionalGeneration, NcpSmolForCausalLM and NimbusMindForCausalLM.
NcpSmolForCausalLM sits past model 101, so the pagination bug had made it
unreachable.
ctest census_watch drives classify()/report()/remember()/sweep_newest() against
synthetic batches, synthetic pages and the committed registry (offline: no
network), so a regression cannot leave the daily alert permanently green.
bong-water-water-bong
force-pushed
the
census-watch
branch
from
September 30, 2026 03:15
06fb511 to
b5a9d01
Compare
…rite The systemd timer is the alert path for now; the CI path (a workflow with issues: write) goes through an RFC first, per CONTRIBUTING. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bong-water-water-bong
added a commit
that referenced
this pull request
Oct 1, 2026
The daily new-model watcher runs on GitHub's schedule (04:30 UTC) and files one deduped census-watch issue when an uncovered architecture class arrives, so the alert no longer depends on someone looking at the development box's systemd unit. Hardened beyond #242's first version: - the workflow token is read-only by default (permissions: {}); only the alert job holds issues: write, and it runs no repository code, only gh on the watch job's output - checkout does not persist credentials - Hugging Face text (model ids, class names) goes into the issue inside a code fence with backticks and '@' stripped, capped at 40 lines - workflow_dispatch input simulate_alert tests the issue path on demand Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
docs/PORTING.mdstep 5 lists.github/workflows/census-{watch,sweep,autopr}.yml; only the sweep landed (census.yml). The watcher's source —Testing/hf_new_models.pyand its compiled probe ofrcpp_arch_from_stringthroughTesting/census_coverage.py— does not exist in this repository, and the systemd unit that ran it pointed at~/census-main, a worktree of the pre-split repository that no longer has amain. Every daily run since 2026-09-23 died203/EXEC("Unable to locate executable"), so the new-model alert was off for a week with nothing turning red.What
The registry is generated data now, so the watcher is a lookup, not a compile: no g++ probe and no
strip_arch, becauseregistry/architectures.jsonis keyed by the HF architecture class the census counts.tools/census_watch.py— polls the newest text-generation models (createdAt descending), reads each one's class from the config inline in the listing, and compares it againstregistry/architectures.json(the same readtools/census.py'scoverage()makes, so "uncovered" means "not in the number the census reports"). Exit 1 is the alert.scripts/census-watch.sh— entry point: log file, exit contract,CENSUS_WATCH_REFRESH=1to fast-forward a detached checkout (refuses a checkout on a branch or with local changes).scripts/1bit-census-watch.{service,timer}— 04:30 UTC daily, from a~/census-mainworktree.docs/registry.md— "The watch": the alert contract and how to triage each kind.State lives outside the checkout (
$CENSUS_WATCH_STATE, default~/.1bit/census-watch-state.json), so a run can never dirty the tree it reads. Gated repos (no config) are reported as unverifiable and never fail the run; derivatives (a quantized GGUF, a LoRA adapter) have no config by design and are skipped, because the release they derive from carries the config and is what gets checked.No
pull_requesttrigger on the workflow, deliberately. The live watch is red whenever any newest model's class is unmapped — that is the alert — and the census reports a mapped share below 100% by design, so a PR gate would be red on every pull request and mean nothing. The watcher's own contract is checked offline byctest census_watch(run byci.yml); the live alert runs on a schedule.Two bugs the first live runs found, both pinned by the test
p=on a sorted listing — every page returned the same first 100 models, so--limit 120fetched 101 and still reported 120. The walk follows theLinkheader's cursor instead, the waytools/census.pysweeps the full listing. This one mattered:NcpSmolForCausalLM(below) sits past model 101 and was unreachable.Evidence
First clean run against this registry, the 120 newest text-generation models:
So six classes arrived that no backend maps —
Qwen3_5MVLAAbsorbedForCausalLM(2 models, e.g.TelperionAI/Qwen3.8-27B-GLA-g2),ArgonneModel(2),GPT,Glm5NextForConditionalGeneration,NcpSmolForCausalLMandNimbusMindForCausalLM. Triage is a follow-up, not part of this PR: none is inregistry/significant.json, so each needs either a real mapping (only where a pinned backend's code really accepts it) or a recorded reason insignificant.json+docs/arch-gaps.md.Testing
ctest census_watchrunstests/census_watch_test.py: 27 offline checks (no network) drivingclassify(),report(),remember()andsweep_newest()against synthetic batches, synthetic listing pages and the committed registry. The committed registry always passes, so a regression that made the alert stop firing would leave the daily check permanently green — the same reasoning behindregistry_gaps_test.py.The systemd path was run for real on the development box: the unit fails with exit 1 on the six classes above, the log lands in
~/.1bit/logs/census-watch-*.log, and the 04:30 UTC timer is enabled.Note for review
The Actions workflow is not in this PR anymore: it needs
issues: write, so it goes through RFC #246 first (CONTRIBUTING). The systemd timer is the alert path until then.