Repository navigation
Description
What
What is concretely wrong. The four always-on units 3d-printer-vision@k1se, 3d-printer-vision@u1, 3d-printer-watch@k1se and 3d-printer-watch@u1 (shipped by supervise_units.py, Restart=always, WantedBy=default.target) run all day although nothing prints most of the time. Idle, they hold the vision models in memory, keep sockets and a camera loop open, rewrite heartbeat files and write journal lines.
Evidence, measured 2026-11-08 on the owner's workstation with no print running (60 s window, cgroup cpu.stat, VmRSS of every process in the unit's cgroup, inotify on the 3d-cli cache, state and data directories):
unit RSS MiB CPU % journal lines per minute
3d-printer-vision@k1se 778.6 0.006 6
3d-printer-vision@u1 774.4 0.011 6
3d-printer-watch@k1se 69.7 0.132 0
3d-printer-watch@u1 101.4 4.502 0
TOTAL 1724.1 4.650 12
file writes in 60 s: 104 inotify events on 10 files (heartbeat tmp+rename every 5-15 s: vision, watch and bed state json)
The K1 SE was switched off during the measurement (No route to host), and its vision unit still held 778 MiB and wrote an error line every 10 s.
Design. One tiny always-on supervisor per printer (3d-printer-ondemand@NAME, stdlib only, one raw-socket HTTP request every 30 s: Moonraker GET /printer/objects/query?print_stats&webhooks, 331 bytes; creality-stock: one short read of the printer's websocket on port 9999) decides print active (printing or paused) and runs systemctl --user start or stop on the existing 3d-printer-watch@NAME and 3d-printer-vision@NAME units. The workers lose their [Install] section: only the supervisor starts them, and it stops them completely (no ONNX model, no timers, no files) a configurable grace period (default 300 s) after the print ends. An unreachable printer is retried with backoff (30 s growing to 5 min). A supervisor restart never touches the workers; at start it reconciles: a print already running starts them, an idle printer with running workers gets one fresh grace period.
Why polling and not a websocket subscription: Moonraker pushes notify_proc_stat_update every second to every websocket client, so a subscription costs a 1 Hz wake-up and a permanently open connection on the printers' weak boards, against one 331-byte request per 30 s. systemd path or socket activation cannot observe a remote printer.
Related: #715 (lazy scorer inside the long-lived vision process; this ticket removes the process itself while idle), #640 and #641 (a print started from anywhere must always be supervised: kept, the supervisor detects it within one poll), #639 (the units are shipped by 3d printer supervise install).
Why (motivation)
The owner's requirement (2026-11-08): the printer services must unload everything from memory and spend no CPU and no disk while no print runs; only fractions of a percent for polling are acceptable. Today they take 1.7 GiB of RAM and 4.65 percent of a core all day.
User impact
You run agents, builds and reviews on the same workstation as the printers. Today two idle printer-camera processes hold about 1.5 GiB of memory and a busy watcher burns CPU all day, so everything else gets slower and the memory guard of the commit and review gates refuses to run. After this change the printers cost you nothing until a print starts, and a print started from the touchscreen, Fluidd or a slicer is still supervised within 30 seconds, so you lose no protection.
Cost of inaction
You keep losing 1.7 GiB of RAM and a few percent of CPU around the clock, and the memory guard keeps refusing your review and commit gates on this machine.
Acceptance criteria
- With no print running, the four worker units are inactive and only the two supervisors are active; total RSS of the supervisors is below 40 MiB and CPU over a 60 s window is below 0.2 percent; proof: the idle-footprint output pasted in the ticket — forced: no visual surface: the proof is the pasted command output, pytest results and the real-systemd simulated run in printer supervision: workers run only while a print runs; a tiny supervisor polls one request every 30 s and starts and stops them #747 (comment)
- No file under the 3d-cli cache, state and data directories is written during an idle 60 s window (inotify event count 0); proof: the idle-footprint output — forced: no visual surface: the proof is the pasted command output, pytest results and the real-systemd simulated run in printer supervision: workers run only while a print runs; a tiny supervisor polls one request every 30 s and starts and stops them #747 (comment)
- A print starting (simulated Moonraker state) starts watch and vision within one poll, and its end stops both after the grace period, leaving no worker process; proof: pytest on a loopback fake Moonraker plus a real-systemd run against the simulated state, output pasted — forced: no visual surface: the proof is the pasted command output, pytest results and the real-systemd simulated run in printer supervision: workers run only while a print runs; a tiny supervisor polls one request every 30 s and starts and stops them #747 (comment)
- A supervisor restart during a print leaves the workers running and re-adopts them; an unreachable printer backs off and never starts workers; proof: named pytest cases — forced: no visual surface: the proof is the pasted command output, pytest results and the real-systemd simulated run in printer supervision: workers run only while a print runs; a tiny supervisor polls one request every 30 s and starts and stops them #747 (comment)
- The supervise install, status and uninstall commands and the printers doctor know the new unit: workers absent while the printer is idle is OK, absent while it prints is FAIL; proof: cases in tests/test_printer_supervise.py — forced: no visual surface: the proof is the pasted command output, pytest results and the real-systemd simulated run in printer supervision: workers run only while a print runs; a tiny supervisor polls one request every 30 s and starts and stops them #747 (comment)
Screenshots
(none)
Deploy
Deployed: no
Activity
fix(printer): the supervisor stops stray idle workers, and a SIGTERM …
alex-mextner commentedon Nov 8, 2026
Result: on-demand supervision implemented, deployed and measured on the owner's workstation
Branch:
feat/printer-ondemand-747(pushed, not merged). Commits: a behaviour-preserving split of the supervise modules for the complexity gate (Refs #707), the feature, the docs, and a follow-up fix found while deploying.Design as built
3d-printer-ondemand@NAME(lib/printer_ops/ondemand.py, stdlib only,Type=notify,WatchdogSec=600,MemoryMax=96M). Every 30 s it asks the printer ONE question and decides "a print is active" (printingorpaused): MoonrakerGET /printer/objects/query?print_stats&webhooks(331 bytes, raw socket, nohttp.client/ssl/yaml imports); creality-stock: one ws:9999 state read.notify_proc_stat_updateto every websocket client each second (a 1 Hz wake-up plus an open connection on the printers' small boards) versus one 331-byte request per 30 s. systemd path/socket activation cannot see a remote printer. Start latency is at most one poll (the first layer is minutes after the start: heating).3d-printer-watch@NAME/3d-printer-vision@NAMEunits, now without[Install](nothing starts them at boot). The supervisor runssystemctl --user startwhen a print runs andstop(completely: no ONNX model, no sockets, no timers, no heartbeat files) a configurable grace period after it ends (default 300 s;--grace, orTHREED_ONDEMAND_GRACE_Sin a drop-in). A worker that dies mid-print is started again at the next poll; a worker started by hand while idle is stopped after one grace period (stray look every 10 min).StatusText:printer=idle|printing|unreachable phase=idle|active|grace polled=<epoch>), logs only transitions.3d printer supervise install|status|uninstalland the printers doctor judge the new model (a stopped worker is right for an idle or switched-off printer, a FAIL for a printing one, a WARN within one poll lag). New3d printer ondemand NAME [--once]. The supervise assess/render/install/apply and the vision-doctor functions were split into helpers first (complexity gate).supervise.py/supervise_units.py).Numbers, same 60 s window and method before and after (no print running; K1 SE switched off, U1 idle)
RSS is the sum of
VmRSSof every process in each unit's cgroup, CPU is the cgroupcpu.statusage_usecdelta over 60 s, file writes are inotify events (create, modify, close-write, moved-to, delete) on the 3d-cli cache, state, data and$XDG_RUNTIME_DIR/3d-clidirectories./proc/PID/iois not readable for the user's own processes in this environment (permission denied), so inotify and journal-line counts are the disk evidence; the supervisor opens no file for writing at all. The supervisors have 1 thread each; cgroupMemoryPeakis 22 MB including the one-shot resolver child at start. Cumulative CPU of each supervisor over its first 10 minutes, including the start-up resolve, was 0.16 s.The real workers were stopped by the supervisor itself 5 minutes (the grace period) after the new supervisors started:
u1: grace over: workers stoppedat 20:20:17 (journal:3d-printer-vision@u1: 1.1G memory peak,3d-printer-watch@u1: 4min CPU over 1h32 wall clock, 35.1M swap peak).Start/stop transition, SIMULATED printer state (no print was running; the real printers were not touched)
A loopback fake Moonraker whose
print_stats.stateI flipped by hand, the real shipped3d-printer-ondemand@.servicetemplate under real systemd (runtime drop-inTHREED_ONDEMAND_POLL_S=5,THREED_ONDEMAND_GRACE_S=20), workers replaced by two dummy services that allocate 200 MB like a loaded model:Unit files of the simulation were removed afterwards.
Tests (all on the branch)
tests/test_printer_ondemand.py(57): decision machine, probes against the loopback fake Moonraker and fake Creality (state mapping, 503 = Klippy down, chunked reply, garbage, hung server), supervisor with fake systemd and manual clocks (idle = zero log lines, start/stop transitions, grace, backoff 30/60/120, restart reconcile, stray workers, refused start/stop logged once, systemctl timeout), config resolution.tests/e2e/test_printer_ondemand_workflows.py(7): realbin/3d(--help,--onceidle/printing/complete/unreachable) and the REAL supervisor process with a fakesystemctl: READY and StatusText overNOTIFY_SOCKET, RSS below 25 MB, SIGHUP-driven print start, stop after grace, SIGTERM leaves workers, unreachable printer, broken config waits instead of crash-looping.rig lint check(complexity), shellcheck, fast pytest of the changed files' tests (507 passed, 1 skipped for the main commit), mypy over 1269 files: all green.What changed on this machine
3d printer supervise install --all --yeswas run from the branch worktree: the shipped templates replaced the hand-made--dry-runones (.bak-<time>copies kept next to them), the twointentional-off.confgate drop-ins were retired (.retired-<time>), the four worker units were disabled,3d-printer-ondemand@u1and@k1sewere enabled and started. The units run from the branch worktree, so after the merge re-run3d printer supervise install --all --yesfrom the main checkout (the managed templates are updated without--yes).3d printer supervise status,3d printers doctor u1 k1se --check printer-supervisionand3d printer vision doctorare green.Owner follow-ups (not done, need root or other repositories)
installrenamed the symlinks, so the dotfiles still hold the always-on definitions. A dotfiles re-link would bring them back: delete3d-printer-watch@.service,3d-printer-vision@.serviceand the retiredintentional-off.confentries from the dotfiles unit directory (thelive.conf/live-observer.confdrop-ins can stay).backups-user-gates: add3d-printer-ondemand@.service xpto its gate list once the units run from the main checkout, and keepretryfrom starting the on-demand workers (the supervisor stops strays after at most 10 minutes plus the grace period). Already passed to the agent that owns that script.watch.bed.enabled) now only runs during a print and its grace period; raiseTHREED_ONDEMAND_GRACE_Sfor longerbed_clearevents.3d printer bed statusand the start gate still take a fresh reading on demand.feat(printer): timer-driven one-shot supervisor replaces the resident…
feat(printer): timer-driven one-shot supervisor replaces the resident…
alex-mextner commentedon Nov 12, 2026
Shipped: merged via #750.
Merge commit: b9fb559
This does NOT close the ticket — acceptance still needs proof. Run
task read #747for the outstanding criteria, thentask check/task done.alex-mextner commentedon Nov 12, 2026
Merged in #750 (b9fb559). Post-merge acceptance on the real machine (the simulation touched no real printer):
3d printer supervise install --all --yesmigrated both printers:3d-printer-ondemand@.servicewritten,3d-printer-watch@.serviceand3d-printer-vision@.servicereplaced with regular files (.bak-20261112T020949copies kept), bothintentional-off.confdrop-ins retired.ls -l <home>/.config/systemd/user/3d-printer-*.servicenow shows-rw-rw-r--files, no symlinks into<home>/.files(the.bak-*copies and live-observer drop-in symlinks are untouched).3d printer supervise status: k1se and u1 OK,3d-printer-ondemand@<name>.timeractive/waiting (enabled), workers inactive and on demand.idle_measure.py 90): 8 one-shot invocations, 293.9 ms CPU in total = 0.325 % of one core; 0 file events in the state/cache dirs; 0 journal lines; 0 of 86 samples with a residentprinter_ops.ondemandprocess; both timers active, both check services inactive.sim747/run.py, instancesim747): workers active 5.6 s after the print started; deleting the state file mid-print left the same worker PID running; workers stopped 23.3 s after the print ended (grace 20 s + one 5 s tick); no leftover processes.omp_accounts.exclude);--effort lowcompleted it. Deferred findings are in printer ondemand: follow-ups from the #747 review rounds (hand-run state, doctor fix, stray rate, status escalation) #749.