Skip to content

printer supervision: workers run only while a print runs; a tiny supervisor polls one request every 30 s and starts and stops them #747

Closed

Description

@alex-mextner

What

What is concretely wrong. The four always-on units 3d-printer-vision@k1se, 3d-printer-vision@u1, 3d-printer-watch@k1se and 3d-printer-watch@u1 (shipped by supervise_units.py, Restart=always, WantedBy=default.target) run all day although nothing prints most of the time. Idle, they hold the vision models in memory, keep sockets and a camera loop open, rewrite heartbeat files and write journal lines.

Evidence, measured 2026-11-08 on the owner's workstation with no print running (60 s window, cgroup cpu.stat, VmRSS of every process in the unit's cgroup, inotify on the 3d-cli cache, state and data directories):

unit                                    RSS MiB   CPU %    journal lines per minute
3d-printer-vision@k1se                    778.6   0.006    6
3d-printer-vision@u1                      774.4   0.011    6
3d-printer-watch@k1se                      69.7   0.132    0
3d-printer-watch@u1                       101.4   4.502    0
TOTAL                                    1724.1   4.650    12
file writes in 60 s: 104 inotify events on 10 files (heartbeat tmp+rename every 5-15 s: vision, watch and bed state json)

The K1 SE was switched off during the measurement (No route to host), and its vision unit still held 778 MiB and wrote an error line every 10 s.

Design. One tiny always-on supervisor per printer (3d-printer-ondemand@NAME, stdlib only, one raw-socket HTTP request every 30 s: Moonraker GET /printer/objects/query?print_stats&webhooks, 331 bytes; creality-stock: one short read of the printer's websocket on port 9999) decides print active (printing or paused) and runs systemctl --user start or stop on the existing 3d-printer-watch@NAME and 3d-printer-vision@NAME units. The workers lose their [Install] section: only the supervisor starts them, and it stops them completely (no ONNX model, no timers, no files) a configurable grace period (default 300 s) after the print ends. An unreachable printer is retried with backoff (30 s growing to 5 min). A supervisor restart never touches the workers; at start it reconciles: a print already running starts them, an idle printer with running workers gets one fresh grace period.

Why polling and not a websocket subscription: Moonraker pushes notify_proc_stat_update every second to every websocket client, so a subscription costs a 1 Hz wake-up and a permanently open connection on the printers' weak boards, against one 331-byte request per 30 s. systemd path or socket activation cannot observe a remote printer.

Related: #715 (lazy scorer inside the long-lived vision process; this ticket removes the process itself while idle), #640 and #641 (a print started from anywhere must always be supervised: kept, the supervisor detects it within one poll), #639 (the units are shipped by 3d printer supervise install).

Why (motivation)

The owner's requirement (2026-11-08): the printer services must unload everything from memory and spend no CPU and no disk while no print runs; only fractions of a percent for polling are acceptable. Today they take 1.7 GiB of RAM and 4.65 percent of a core all day.

User impact

You run agents, builds and reviews on the same workstation as the printers. Today two idle printer-camera processes hold about 1.5 GiB of memory and a busy watcher burns CPU all day, so everything else gets slower and the memory guard of the commit and review gates refuses to run. After this change the printers cost you nothing until a print starts, and a print started from the touchscreen, Fluidd or a slicer is still supervised within 30 seconds, so you lose no protection.

Cost of inaction

You keep losing 1.7 GiB of RAM and a few percent of CPU around the clock, and the memory guard keeps refusing your review and commit gates on this machine.

Acceptance criteria

Screenshots

(none)

Deploy

Deployed: no

Links

Activity

  1. alex-mextner commented on Nov 8, 2026

    @alex-mextner
    OwnerAuthor

    Result: on-demand supervision implemented, deployed and measured on the owner's workstation

    Branch: feat/printer-ondemand-747 (pushed, not merged). Commits: a behaviour-preserving split of the supervise modules for the complexity gate (Refs #707), the feature, the docs, and a follow-up fix found while deploying.

    Design as built

    • One tiny always-on unit per printer, 3d-printer-ondemand@NAME (lib/printer_ops/ondemand.py, stdlib only, Type=notify, WatchdogSec=600, MemoryMax=96M). Every 30 s it asks the printer ONE question and decides "a print is active" (printing or paused): Moonraker GET /printer/objects/query?print_stats&webhooks (331 bytes, raw socket, no http.client/ssl/yaml imports); creality-stock: one ws:9999 state read.
    • Trigger choice. Websocket subscription rejected: Moonraker pushes notify_proc_stat_update to every websocket client each second (a 1 Hz wake-up plus an open connection on the printers' small boards) versus one 331-byte request per 30 s. systemd path/socket activation cannot see a remote printer. Start latency is at most one poll (the first layer is minutes after the start: heating).
    • Workers are the unchanged 3d-printer-watch@NAME / 3d-printer-vision@NAME units, now without [Install] (nothing starts them at boot). The supervisor runs systemctl --user start when a print runs and stop (completely: no ONNX model, no sockets, no timers, no heartbeat files) a configurable grace period after it ends (default 300 s; --grace, or THREED_ONDEMAND_GRACE_S in a drop-in). A worker that dies mid-print is started again at the next poll; a worker started by hand while idle is stopped after one grace period (stray look every 10 min).
    • Printer unreachable: backoff 30 s, 60 s, then every 2 min (changed from the 5 min in this ticket's first text: a heated start takes minutes), one journal line per change of reason, never starts workers; while workers run it counts like an ended print (grace applies).
    • Supervisor restart: the workers are separate units and are left alone; at start it reconciles (a print already running starts them when down; an idle printer with running workers gets one fresh grace period). A config change (mtime) or SIGHUP re-resolves the printer address in a one-shot child that exits at once, so the supervisor never imports yaml or the fleet registry.
    • No disk while idle: no heartbeat file; state goes to systemd through sd_notify (StatusText: printer=idle|printing|unreachable phase=idle|active|grace polled=<epoch>), logs only transitions.
    • 3d printer supervise install|status|uninstall and the printers doctor judge the new model (a stopped worker is right for an idle or switched-off printer, a FAIL for a printing one, a WARN within one poll lag). New 3d printer ondemand NAME [--once]. The supervise assess/render/install/apply and the vision-doctor functions were split into helpers first (complexity gate).
    • Related: #715 (lazy scorer inside the vision process: complementary, it also shrinks the vision worker while a print runs; it has not landed and touches the same supervise files, so expect a mechanical merge conflict in supervise.py/supervise_units.py).

    Numbers, same 60 s window and method before and after (no print running; K1 SE switched off, U1 idle)

    BEFORE (4 always-on workers)                       RSS MiB   CPU %    journal lines/min
    3d-printer-vision@k1se                              778.6    0.006    6
    3d-printer-vision@u1                                774.4    0.011    6
    3d-printer-watch@k1se                                69.7    0.132    0
    3d-printer-watch@u1                                 101.4    4.502    0
    TOTAL                                              1724.1    4.650    12
    file writes in the 3d-cli cache/state/data dirs: 104 inotify events on 10 files
    
    AFTER (2 supervisors, workers inactive)
    3d-printer-ondemand@k1se                             18.5    0.000    0
    3d-printer-ondemand@u1                               18.6    0.001    0
    TOTAL                                                37.0    0.001    0
    file writes in the 3d-cli cache/state/data dirs: 0 events on 0 files
    workers: 3d-printer-watch@u1, 3d-printer-vision@u1, 3d-printer-watch@k1se, 3d-printer-vision@k1se all inactive
    

    RSS is the sum of VmRSS of every process in each unit's cgroup, CPU is the cgroup cpu.stat usage_usec delta over 60 s, file writes are inotify events (create, modify, close-write, moved-to, delete) on the 3d-cli cache, state, data and $XDG_RUNTIME_DIR/3d-cli directories. /proc/PID/io is not readable for the user's own processes in this environment (permission denied), so inotify and journal-line counts are the disk evidence; the supervisor opens no file for writing at all. The supervisors have 1 thread each; cgroup MemoryPeak is 22 MB including the one-shot resolver child at start. Cumulative CPU of each supervisor over its first 10 minutes, including the start-up resolve, was 0.16 s.

    The real workers were stopped by the supervisor itself 5 minutes (the grace period) after the new supervisors started: u1: grace over: workers stopped at 20:20:17 (journal: 3d-printer-vision@u1: 1.1G memory peak, 3d-printer-watch@u1: 4min CPU over 1h32 wall clock, 35.1M swap peak).

    Start/stop transition, SIMULATED printer state (no print was running; the real printers were not touched)

    A loopback fake Moonraker whose print_stats.state I flipped by hand, the real shipped 3d-printer-ondemand@.service template under real systemd (runtime drop-in THREED_ONDEMAND_POLL_S=5, THREED_ONDEMAND_GRACE_S=20), workers replaced by two dummy services that allocate 200 MB like a loaded model:

    [20:22:43] simulated printer: standby   supervisor: printer=idle phase=idle      watch: inactive  vision: inactive
    print starts (simulated):               workers active after ~4 s (poll 5 s)
    [20:22:50] simulated printer: printing  supervisor: printer=printing phase=active  watch: active 210 MiB  vision: active 210 MiB
    print ends (simulated), grace 20 s:
    [20:23:00] simulated printer: complete  supervisor: printer=idle phase=grace      watch: active  vision: active   (10 s after the end)
    [20:23:18] simulated printer: complete  supervisor: printer=idle phase=idle       watch: inactive  vision: inactive (27 s after the end)
    
    journal: sim: print started: starting 3d-printer-watch@sim.service, 3d-printer-vision@sim.service
             sim: print ended: workers stay 20 s, then stop
             sim: grace over: workers stopped
    supervisor restart mid-print (systemctl restart twice): watch worker pid before=2274570 after=2274570, workers still active;
             journal: sim: workers already running at start: kept for a fresh grace period
    

    Unit files of the simulation were removed afterwards.

    Tests (all on the branch)

    • tests/test_printer_ondemand.py (57): decision machine, probes against the loopback fake Moonraker and fake Creality (state mapping, 503 = Klippy down, chunked reply, garbage, hung server), supervisor with fake systemd and manual clocks (idle = zero log lines, start/stop transitions, grace, backoff 30/60/120, restart reconcile, stray workers, refused start/stop logged once, systemctl timeout), config resolution.
    • tests/e2e/test_printer_ondemand_workflows.py (7): real bin/3d (--help, --once idle/printing/complete/unreachable) and the REAL supervisor process with a fake systemctl: READY and StatusText over NOTIFY_SOCKET, RSS below 25 MB, SIGHUP-driven print start, stop after grace, SIGTERM leaves workers, unreachable printer, broken config waits instead of crash-looping.
    • Updated and added supervise status/install/uninstall/doctor/deploy_check tests (new cases a to l).
    • Pre-commit gate on each commit: ruff, rig lint check (complexity), shellcheck, fast pytest of the changed files' tests (507 passed, 1 skipped for the main commit), mypy over 1269 files: all green.

    What changed on this machine

    3d printer supervise install --all --yes was run from the branch worktree: the shipped templates replaced the hand-made --dry-run ones (.bak-<time> copies kept next to them), the two intentional-off.conf gate drop-ins were retired (.retired-<time>), the four worker units were disabled, 3d-printer-ondemand@u1 and @k1se were enabled and started. The units run from the branch worktree, so after the merge re-run 3d printer supervise install --all --yes from the main checkout (the managed templates are updated without --yes). 3d printer supervise status, 3d printers doctor u1 k1se --check printer-supervision and 3d printer vision doctor are green.

    Owner follow-ups (not done, need root or other repositories)

    • The old hand-made templates were symlinks into the dotfiles repository; install renamed the symlinks, so the dotfiles still hold the always-on definitions. A dotfiles re-link would bring them back: delete 3d-printer-watch@.service, 3d-printer-vision@.service and the retired intentional-off.conf entries from the dotfiles unit directory (the live.conf / live-observer.conf drop-ins can stay).
    • backups-user-gates: add 3d-printer-ondemand@.service xp to its gate list once the units run from the main checkout, and keep retry from starting the on-demand workers (the supervisor stops strays after at most 10 minutes plus the grace period). Already passed to the agent that owns that script.
    • Trade-off to accept or change: the watcher's camera bed monitor (watch.bed.enabled) now only runs during a print and its grace period; raise THREED_ONDEMAND_GRACE_S for longer bed_clear events. 3d printer bed status and the start gate still take a fresh reading on demand.
  2. alex-mextner commented on Nov 12, 2026

    @alex-mextner
    OwnerAuthor

    Shipped: merged via #750.
    Merge commit: b9fb559
    This does NOT close the ticket — acceptance still needs proof. Run task read #747 for the outstanding criteria, then task check/task done.

  3. alex-mextner commented on Nov 12, 2026

    @alex-mextner
    OwnerAuthor

    Merged in #750 (b9fb559). Post-merge acceptance on the real machine (the simulation touched no real printer):

    • 3d printer supervise install --all --yes migrated both printers: 3d-printer-ondemand@.service written, 3d-printer-watch@.service and 3d-printer-vision@.service replaced with regular files (.bak-20261112T020949 copies kept), both intentional-off.conf drop-ins retired. ls -l <home>/.config/systemd/user/3d-printer-*.service now shows -rw-rw-r-- files, no symlinks into <home>/.files (the .bak-* copies and live-observer drop-in symlinks are untouched).
    • 3d printer supervise status: k1se and u1 OK, 3d-printer-ondemand@<name>.timer active/waiting (enabled), workers inactive and on demand.
    • Idle, 90 s window (idle_measure.py 90): 8 one-shot invocations, 293.9 ms CPU in total = 0.325 % of one core; 0 file events in the state/cache dirs; 0 journal lines; 0 of 86 samples with a resident printer_ops.ondemand process; both timers active, both check services inactive.
    • Simulated printer on real systemd (sim747/run.py, instance sim747): workers active 5.6 s after the print started; deleting the state file mid-print left the same worker PID running; workers stopped 23.3 s after the print ended (grace 20 s + one 5 s tick); no leftover processes.
    • Review: final round complete on all three roles with no blocking findings. Earlier rounds were incomplete because the consistency seat timed out on the omp route and its claude fallback is excluded by policy (omp_accounts.exclude); --effort low completed it. Deferred findings are in printer ondemand: follow-ups from the #747 review rounds (hand-run state, doctor fix, stray rate, status escalation) #749.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions