Skip to content

Port the census watcher: the newest HF models against the generated registry - #242

Merged
bong-water-water-bong merged 4 commits into
mainfrom
census-watch
Sep 30, 2026
Merged

bong-water-water-bong merged 4 commits into
mainfrom
census-watch

Conversation

@bong-water-water-bong

@bong-water-water-bong bong-water-water-bong commented Sep 30, 2026 •

Copy link
Copy Markdown
Collaborator

Why

docs/PORTING.md step 5 lists .github/workflows/census-{watch,sweep,autopr}.yml; only the sweep landed (census.yml). The watcher's source — Testing/hf_new_models.py and its compiled probe of rcpp_arch_from_string through Testing/census_coverage.py — does not exist in this repository, and the systemd unit that ran it pointed at ~/census-main, a worktree of the pre-split repository that no longer has a main. Every daily run since 2026-09-23 died 203/EXEC ("Unable to locate executable"), so the new-model alert was off for a week with nothing turning red.

What

The registry is generated data now, so the watcher is a lookup, not a compile: no g++ probe and no strip_arch, because registry/architectures.json is keyed by the HF architecture class the census counts.

  • tools/census_watch.py — polls the newest text-generation models (createdAt descending), reads each one's class from the config inline in the listing, and compares it against registry/architectures.json (the same read tools/census.py's coverage() makes, so "uncovered" means "not in the number the census reports"). Exit 1 is the alert.
  • scripts/census-watch.sh — entry point: log file, exit contract, CENSUS_WATCH_REFRESH=1 to fast-forward a detached checkout (refuses a checkout on a branch or with local changes).
  • scripts/1bit-census-watch.{service,timer} — 04:30 UTC daily, from a ~/census-main worktree.
  • docs/registry.md — "The watch": the alert contract and how to triage each kind.

State lives outside the checkout ($CENSUS_WATCH_STATE, default ~/.1bit/census-watch-state.json), so a run can never dirty the tree it reads. Gated repos (no config) are reported as unverifiable and never fail the run; derivatives (a quantized GGUF, a LoRA adapter) have no config by design and are skipped, because the release they derive from carries the config and is what gets checked.

No pull_request trigger on the workflow, deliberately. The live watch is red whenever any newest model's class is unmapped — that is the alert — and the census reports a mapped share below 100% by design, so a PR gate would be red on every pull request and mean nothing. The watcher's own contract is checked offline by ctest census_watch (run by ci.yml); the live alert runs on a schedule.

Two bugs the first live runs found, both pinned by the test

  • Pagination. HF ignores p= on a sorted listing — every page returned the same first 100 models, so --limit 120 fetched 101 and still reported 120. The walk follows the Link header's cursor instead, the way tools/census.py sweeps the full listing. This one mattered: NcpSmolForCausalLM (below) sits past model 101 and was unreachable.
  • A false green. A run that checked 0 models printed "OK — no uncovered class". It now says nothing new since the last run (N models already classified).

Evidence

First clean run against this registry, the 120 newest text-generation models:

census-watch: 120 newest text-generation models checked, 102 with an architecture,
94 mapped, 6 uncovered class(es), 3 unverifiable (gated/no-config),
15 derivative (no config by design)

So six classes arrived that no backend maps — Qwen3_5MVLAAbsorbedForCausalLM (2 models, e.g. TelperionAI/Qwen3.8-27B-GLA-g2), ArgonneModel (2), GPT, Glm5NextForConditionalGeneration, NcpSmolForCausalLM and NimbusMindForCausalLM. Triage is a follow-up, not part of this PR: none is in registry/significant.json, so each needs either a real mapping (only where a pinned backend's code really accepts it) or a recorded reason in significant.json + docs/arch-gaps.md.

Testing

ctest census_watch runs tests/census_watch_test.py: 27 offline checks (no network) driving classify(), report(), remember() and sweep_newest() against synthetic batches, synthetic listing pages and the committed registry. The committed registry always passes, so a regression that made the alert stop firing would leave the daily check permanently green — the same reasoning behind registry_gaps_test.py.

The systemd path was run for real on the development box: the unit fails with exit 1 on the six classes above, the log lands in ~/.1bit/logs/census-watch-*.log, and the 04:30 UTC timer is enabled.

Note for review

The Actions workflow is not in this PR anymore: it needs issues: write, so it goes through RFC #246 first (CONTRIBUTING). The systemd timer is the alert path until then.

@context7

context7 Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

Docs7 for 1bit-monster/engine

Result Status Action
Deployment ➖ Not used —
Content review ➖ Did not run. This site has no agent runs available this month. Wait for the monthly reset or check your Docs7 plan. —

Commit 4e5f791

…egistry

1bit-MONSTER's watcher (Testing/hf_new_models.py) compiled a C++ probe of
rcpp_arch_from_string through census_coverage.build_mapper(). This repository's
registry is generated data (registry/architectures.json), keyed by the HF
architecture class the census counts, so the watcher is a plain lookup: no g++
probe and no strip_arch. That is how PORTING.md's step 5 could land the sweep
(census.yml) and drop the watch half without a compile-time cost.

tools/census_watch.py polls the newest text-generation models (createdAt
descending), reads each one's class from the config inline in the listing and
compares it against the registry; exit 1 is the alert. scripts/census-watch.sh is
the entry point (log file + optional origin/main refresh), the
1bit-census-watch.{service,timer} pair is the 04:30 UTC timer from a
~/census-main worktree, and census-watch.yml is the CI path with the deduped
alert issue. State lives outside the checkout ($CENSUS_WATCH_STATE, default
~/.1bit/census-watch-state.json), so a run never dirties the tree it reads.

Two bugs the first live runs found, both pinned by the test:

- Pagination. HF ignores `p=` on a sorted listing: every page returned the same
  first 100 models, so --limit 120 fetched 101 and still called it 120. The walk
  follows the Link header's cursor instead, the way tools/census.py sweeps.
- State shape. classify() returned (id, reason) tuples for the unverifiable
  models, so remember() wrote tuple keys and json.dump failed on the sorted keys
  — the run reported its alert and then died before saving state.

First run against main's registry, the 120 newest text-generation models: 102 with
an architecture, 94 mapped, so six uncovered classes the alert had been blind to
since it stopped running on 2026-09-23 -- Qwen3_5MVLAAbsorbedForCausalLM (2
models, e.g. TelperionAI/Qwen3.8-27B-GLA-g2), ArgonneModel, GPT,
Glm5NextForConditionalGeneration, NcpSmolForCausalLM and NimbusMindForCausalLM.
NcpSmolForCausalLM sits past model 101, so the pagination bug had made it
unreachable.

ctest census_watch drives classify()/report()/remember()/sweep_newest() against
synthetic batches, synthetic pages and the committed registry (offline: no
network), so a regression cannot leave the daily alert permanently green.
bong-water-water-bong and others added 3 commits September 30, 2026 05:06
…rite

The systemd timer is the alert path for now; the CI path (a workflow with
issues: write) goes through an RFC first, per CONTRIBUTING.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong
bong-water-water-bong merged commit aeee65e into main Sep 30, 2026
10 of 11 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the census-watch branch September 30, 2026 08:24
bong-water-water-bong added a commit that referenced this pull request Oct 1, 2026
The daily new-model watcher runs on GitHub's schedule (04:30 UTC) and files one deduped
census-watch issue when an uncovered architecture class arrives, so the alert no longer
depends on someone looking at the development box's systemd unit.

Hardened beyond #242's first version:
- the workflow token is read-only by default (permissions: {}); only the alert job holds
  issues: write, and it runs no repository code, only gh on the watch job's output
- checkout does not persist credentials
- Hugging Face text (model ids, class names) goes into the issue inside a code fence with
  backticks and '@' stripped, capped at 40 lines
- workflow_dispatch input simulate_alert tests the issue path on demand

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant