diff --git a/docs/architecture/lib-map.md b/docs/architecture/lib-map.md index d6aef129..f05922c0 100644 --- a/docs/architecture/lib-map.md +++ b/docs/architecture/lib-map.md @@ -333,6 +333,7 @@ Every current root Python module is listed here so future additions are intentio - `bed_norm.py` — the lighting-normalised evidence channel of the bed verdict (#723): brightness mismatch of the frame against the reference (dark/middle/bright quantiles of the bed ROI, or a known light-state difference) and the local-texture difference (luminance over its neighbourhood spread, invariant to gain/offset/gamma) a pixel needs in addition to the colour difference when the light differs; `bed_state.py` wires it (`_under_light_change`). Numpy + OpenCV. - `printer_ops/bed_override.py` — the owner's recorded word that the bed is empty (#723): `bed check --assume-empty WHY` writes `assume-empty-.json` (evidence dir) and `NAME.assumed.json` (state dir: ts, why, ttl, job_marker = Moonraker's newest history job + print_stats.filename); `honoured` is what the preflight `bed` line reads (ttl, no new job, never a printing printer or a missing plate). Stdlib only. - `bed_calibrate.py` — `3d printer bed calibrate NAME` (#723): how much of the plate the camera sees at each Z for a printer without a camera calibration — the guarded Z-only grid (`bed_view.excursion`: lock, displaced-Z marker, restore in a `finally`), one lit frame per Z, the plate followed from its outline in the empty-bed reference / `watch.bed.plate_mask` by chained ECC affine transforms (visible share of the moving polygon per Z; a plate that cannot be followed ends the chain, the Zs are not invented), stored in `calibration.json` that `bed_state.z_match` reads (`watch.bed.views` overrides). Numpy + OpenCV. +- `bed_pose.py` — where the camera is relative to the bed reference (2026-11-13, K1 SE camera rides with the toolhead): the scene round the bed region is correlated (same view), matched by SIFT + RANSAC homography (moved: `bed_state.classify` warps the frame onto the reference) or cannot be aligned (lost: a plate-missing / occupied read becomes unknown with the `bed reference --capture` fix). Numpy + OpenCV. - `bed_refs.py` — which empty-bed reference stands for a frame (#723): the identity (kind, plate type `watch.bed.plate`, light state on/off/unknown, Z) and file names, the preference order (same plate and light first, another light state a marked fallback), the age and staleness (`watch.bed.reference_max_age_days`), the `JudgeContext` a verdict is made in, whether a verdict may become the reference (`refresh_refusal`), and the journal `NAME.refs.log`. Stdlib only. - `bed_refresh.py` — `3d printer bed reference NAME --refresh [--assume-empty WHY]` and `watch.bed.auto_reference` (#723): the bed check's flow (best view, light on, bed back) whose confident clear frame replaces the stale reference, the owner's word as the alternative, the journal; numpy + OpenCV via `bed_state.py`. - `bed_view.py` — the auto view of the printer bed (#303 follow-up): pure picker of the best inspection Z (share of the plate in the camera frame, reference at that Z, clearance floor, axis ceiling), the guarded Z-only move + judge + restore flow `bed_state.py status` runs (`watch.bed.auto_view`). Stdlib only; the pixels stay in `bed_state.py`. diff --git a/docs/commands/printer.md b/docs/commands/printer.md index 7cf5aaad..cb2401d1 100644 --- a/docs/commands/printer.md +++ b/docs/commands/printer.md @@ -955,6 +955,27 @@ until someone takes them off. The camera can see that, so nobody has to tell an `unknown`, "out of the camera's view … it sees the bed again at Z 210 or below", and no reference is taken there: it would read every low part as a clear bed ([#402](https://github.com/alex-mextner/3d-cli/issues/402)). +- **A camera that moved** (`lib/bed_pose.py`). The K1 SE camera rides with the toolhead, so after a job + or a home the same empty plate is seen from another place: the whole picture, background included, is + shifted and zoomed by tens of px. Against a reference taken from elsewhere that read «70 % of the bed surface + changed» — `plate missing`, FAIL, start refused, on a plate that was on (2026-11-13). The frame is now + *located* against its reference before it is judged, by the scene round the bed region (what a plate or a part + cannot change). **Same view** (the edge structure of the scene correlates ≥ 0.85, or in another light the + scene does not shift, or the SIFT homography moves the bed under 4 px — a fixed camera, in any light): + judged as before. **Moved** (SIFT features agree on a plausible homography, after warping the scene lines up + again, edge correlation ≥ 0.65, and ≥ 90 % of the bed region is still in the frame): the frame is warped onto + the reference and judged there with the reference's ROI; `reason`, the metrics `pose_shift_px` / + `pose_inliers` and `pose` in the verdict say so, the evidence frame marks the pixels on the camera frame; a + `--no-plate` reference taken from another view is not used. **Lost** (no consistent match, the scene lines + up only far away — the camera sits lower or closer —, or too little of the bed region is in the frame): + the frame was not judged, so every verdict — a clear bed with the plate on too, never a PASS — becomes `unknown` + (`pose_shift_px` in the metrics) and the preflight `bed` line + a WARN naming the fix: with the plate on and the bed empty take a reference from where the camera is now + (`3d printer bed reference NAME --capture`), or put the toolhead back where the reference was taken; + `bed check --assume-empty "why"` then works as for any unknown. A plate that is really off and a part that is + really on still FAIL when the camera moved and could be aligned. A K1 SE `watch.bed.roi` polygon is a + region of the reference's picture: redraw it when the reference is taken from another view than the one it + was drawn on, or it covers mostly the wall (a whole-frame ROI leaves no scene to locate by: judged as before). - **Status.** `3d printer bed status u1` — `clear` / `occupied` / `unknown` and plate `present` / `missing` / `unknown`, with the reason and an evidence frame (ROI cyan, changed pixels red) in `~/.cache/3d-cli/vision/bed/`. `--max-age S` takes the watcher's last verdict when it was diff --git a/lib/bed_pose.py b/lib/bed_pose.py new file mode 100644 index 00000000..70717c90 --- /dev/null +++ b/lib/bed_pose.py @@ -0,0 +1,222 @@ +"""bed_pose.py — has the camera moved since the reference frame was taken, and by how much (bed_state.py). + +A reference stands for the picture the camera sees. The K1 SE's camera rides with the toolhead, so after a +job (or a home, or a Z move) the same empty plate is seen from another place: the whole picture, background +included, is shifted and zoomed by tens of pixels. Compared pixel by pixel with a reference taken from +elsewhere that is "70 % of the bed surface changed" — a plate-missing FAIL for a plate that is on +(2026-11-13, K1 SE start preflight). The pixels of a part or a missing plate change a REGION of an otherwise +unchanged scene; a camera that moved changes the whole scene. So before judging, the frame is located against +the reference, by the scene OUTSIDE the bed region (what a plate or a part cannot change): + + same the scene round the bed lines up (edge strength correlates, or the frames differ in light only and + phase correlation finds no shift, or the SIFT homography moves the bed under `SAME_PX`): the camera + stands where it stood; judged as is (the fixed U1 camera: always this). + moved the scene is shifted but a homography from matched SIFT features explains it (inliers, a plausible + quad) and after warping the scene round the bed lines up again (`MIN_FIT`) with `MIN_VISIBLE` of + the bed region still in the frame: the frame is warped onto the reference and judged there. + lost the scene round the bed correlates under `LOST_NCC` and no homography is found, or the scene lines up + only far away (the camera sits lower or closer: parallax), or too little of the bed region is in the + frame. Nothing about the plate or parts can be concluded from pixels. + blind too little texture in either frame, no scene round the bed region (a whole-frame ROI), or no match + although the scene round the bed still correlates (a night light, a big part on a fixed camera): to + tell nothing; judged as is. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import TYPE_CHECKING, Literal + +if TYPE_CHECKING: + import numpy as np + +# thresholds tuned on real K1 SE frames (tests/data/bed) and the U1's light-change frames +SAME_NCC = 0.85 # scene correlation at which the camera surely stands where it stood (a 3 px shift: 0.87) +LIGHT_RESPONSE = 0.25 # a light change or parts lower the correlation (U1 night vs day 0.5-0.65): phase correlation + # of the two frames then at least this sure (real pans 0.8-1.0, other views 0.07-0.19) ... +SAME_PX = 4.0 # ... that the scene moved under this many px of the 480-px frame (also for the homography; the judge + # itself forgives 3 px: `shift_px`) +LOST_NCC = 0.3 # a frame is `lost` only when the scene round the bed also differs this much; a scene that still + # correlates (a night light, a part, spaghetti on a fixed camera) is judged as it always was +MIN_SCENE_PX = 1000 # fewer scene pixels outside the bed region: nothing to locate the camera by +RING_PX = 15 # the bed region is grown by this much before it is left out of the scene +MIN_KEYPOINTS = 40 # fewer SIFT features in either frame: nothing to locate it by +MIN_INLIERS = 15 # a homography backed by fewer matched features is chance, not a pose +MIN_SHARE = 0.25 # ... and by less than this share of the ratio-test matches +RATIO = 0.75 # Lowe ratio test +REPROJ_PX = 3.0 # RANSAC reprojection tolerance at PROC_W +AREA_RANGE = (0.25, 4.0) # the frame's area after the move: more than 4x or under a quarter is a mismatch +MAX_STRETCH = 1.6 # singular value ratio of the homography's linear part (shear + aspect) +MIN_FIT = 0.65 # scene correlation after aligning (real frames: aligned 0.8-0.96, not aligned 0.3-0.55) +MIN_VISIBLE = 0.9 # share of the bed region that must stay inside the moved frame (the rest is not judged) + +State = Literal["same", "moved", "lost", "blind"] +SAME: State = "same" +MOVED: State = "moved" +LOST: State = "lost" +BLIND: State = "blind" + + +@dataclass(frozen=True) +class Pose: + state: State + why: str + ncc: float = 1.0 # correlation of the scene round the bed, unaligned + inliers: int = 0 + shift_px: float = 0.0 # how far the bed region moved (PROC_W px) + visible: float = 1.0 # share of the bed region the frame covers (moved) + h: np.ndarray | None = None # current frame -> reference frame (moved) + covered: np.ndarray | None = None # bool: reference pixels the moved frame covers (moved) + + def info(self) -> dict[str, object]: + return {"state": self.state, "ncc": round(self.ncc, 3), "shift_px": round(self.shift_px, 1), + "inliers": self.inliers, "visible": round(self.visible, 3)} + + +class _Blind(Exception): + """Too little texture to match by.""" + + +def _structure(img: np.ndarray) -> np.ndarray: + import cv2 + import numpy as np + + grey = cv2.GaussianBlur(cv2.cvtColor(img, cv2.COLOR_BGR2GRAY), (5, 5), 0).astype(np.float32) + mag = np.hypot(cv2.Sobel(grey, cv2.CV_32F, 1, 0), cv2.Sobel(grey, cv2.CV_32F, 0, 1)) + return cv2.GaussianBlur(mag, (9, 9), 0) + + +def _ncc(a: np.ndarray, b: np.ndarray, mask: np.ndarray) -> float: + """Normalised correlation of two structure maps inside `mask`: 1 the same picture, ~0 another view.""" + import numpy as np + + x, y = a[mask], b[mask] + x, y = x - float(x.mean()), y - float(y.mean()) + norm = float(np.sqrt((x * x).sum() * (y * y).sum())) + return float((x * y).sum() / norm) if norm > 0 else 0.0 + + +def _phase(a: np.ndarray, b: np.ndarray) -> tuple[float, float]: + """(shift in px, how sure) of the translation between two structure maps: light-proof, exact for a pan.""" + import cv2 + import numpy as np + + window = cv2.createHanningWindow((a.shape[1], a.shape[0]), cv2.CV_32F) + (dx, dy), response = cv2.phaseCorrelate(a.astype(np.float32), b.astype(np.float32), window) + return float(np.hypot(dx, dy)), float(response) + + +def _plausible(h: np.ndarray, shape: tuple[int, int]) -> str | None: + """Why the homography `h` cannot be a camera move (the frame's corners must stay a convex quad of about the + same area and aspect, same orientation), or None.""" + import cv2 + import numpy as np + + height, width = shape + corners = np.array([[[0, 0], [width, 0], [width, height], [0, height]]], np.float32) + quad = cv2.perspectiveTransform(corners, h)[0] + edges = np.roll(quad, -1, axis=0) - quad + turns = edges[:, 0] * np.roll(edges[:, 1], -1) - edges[:, 1] * np.roll(edges[:, 0], -1) + if not np.isfinite(quad).all() or float(turns.min()) <= 0: + return "the matched features imply a folded or mirrored view" + area = float(cv2.contourArea(quad)) / (width * height) + if not AREA_RANGE[0] <= area <= AREA_RANGE[1]: + return f"the matched features imply an implausible zoom (area x{area:.2f})" + s = np.linalg.svd(h[:2, :2], compute_uv=False) + if s[1] <= 0 or s[0] / s[1] > MAX_STRETCH: + return "the matched features imply an implausible shear of the view" + return None + + +def _matches(ref: np.ndarray, cur: np.ndarray) -> tuple[np.ndarray, np.ndarray, int]: + """(points in `cur`, their matches in `ref`, features found in `cur`): SIFT + the ratio test. + Raises _Blind when either frame has too few features.""" + import cv2 + import numpy as np + + sift = cv2.SIFT_create(1500) + kr, dr = sift.detectAndCompute(cv2.cvtColor(ref, cv2.COLOR_BGR2GRAY), None) + kc, dc = sift.detectAndCompute(cv2.cvtColor(cur, cv2.COLOR_BGR2GRAY), None) + if dr is None or dc is None or min(len(kr), len(kc)) < MIN_KEYPOINTS: + raise _Blind + pairs = cv2.BFMatcher().knnMatch(dc, dr, k=2) + good = [m for m, n in (p for p in pairs if len(p) == 2) if m.distance < RATIO * n.distance] + src = np.array([kc[m.queryIdx].pt for m in good], np.float32).reshape(-1, 2) + dst = np.array([kr[m.trainIdx].pt for m in good], np.float32).reshape(-1, 2) + return src, dst, len(kc) + + +def _homography(ref: np.ndarray, cur: np.ndarray) -> tuple[np.ndarray | None, int, str]: + """(H mapping `cur` onto `ref`, inliers, why not): matches + RANSAC; H None when there is no fit. + Raises _Blind when either frame has too few features.""" + import cv2 + + src, dst, found = _matches(ref, cur) + if len(src) < MIN_INLIERS: + return None, 0, f"only {len(src)} of {found} features of the frame match the reference's" + h, inl = cv2.findHomography(src, dst, cv2.RANSAC, REPROJ_PX) + n = 0 if inl is None else int(inl.sum()) + if h is None or n < max(MIN_INLIERS, MIN_SHARE * len(src)): + return None, n, f"only {n} of {len(src)} matched features agree on one view" + h = h / h[2, 2] + bad = _plausible(h, ref.shape[:2]) + return (None, n, bad) if bad else (h, n, "") + + +def _lost(why: str, ncc: float, inliers: int, shift: float = 0.0, visible: float = 1.0) -> Pose: + """`lost` when the scene round the bed does not correlate with the reference's; else nobody can tell.""" + if ncc < LOST_NCC: + return Pose(LOST, why, ncc, inliers, shift, visible) + return Pose(BLIND, f"cannot tell whether the camera moved ({why}), but the scene round the bed still correlates " + f"({ncc:.2f})", ncc, inliers, shift, visible) + + +def locate(ref: np.ndarray, cur: np.ndarray, roi: np.ndarray) -> Pose: + """Where is `cur` (BGR, same size as `ref`) relative to `ref`; `roi` is the reference's bed mask.""" + import cv2 + import numpy as np + + if not roi.any(): # classify reports an empty ROI itself + return Pose(BLIND, "the bed region of the reference is empty: nothing to locate the camera by") + scene = ~cv2.dilate(roi.astype(np.uint8), np.ones((RING_PX, RING_PX), np.uint8)).astype(bool) + if int(scene.sum()) < MIN_SCENE_PX: + return Pose(BLIND, "the bed region of the reference fills the frame: no scene round it to locate the camera by") + sr, sc = _structure(ref), _structure(cur) + c = _ncc(sr, sc, scene) + if c >= SAME_NCC: + return Pose(SAME, "the camera stands where the reference was taken", c) + shift, response = _phase(sr, sc) # the frames differ in light, or in what is on the bed, not in view? + if response >= LIGHT_RESPONSE and shift < SAME_PX: + return Pose(SAME, "the camera stands where the reference was taken (the picture differs)", c, 0, shift) + try: + h, inliers, why = _homography(ref, cur) + except _Blind: + return Pose(BLIND, "too little texture to tell whether the camera moved", c) + if h is None: + return _lost(f"the camera does not see the scene of the reference ({why})", c, inliers) + height, width = roi.shape + ys, xs = np.nonzero(roi) + # how far the bed region's corners (and centre) moved: a zoom or a turn about the centre moves its border, not it + box = np.array([[[xs.min(), ys.min()], [xs.max(), ys.min()], [xs.max(), ys.max()], [xs.min(), ys.max()], + [xs.mean(), ys.mean()]]], np.float32) + shift = float(np.linalg.norm(cv2.perspectiveTransform(box, np.linalg.inv(h))[0] - box[0], axis=1).max()) + if shift < SAME_PX: + return Pose(SAME, "the camera stands where the reference was taken", c, inliers, shift) + covered = cv2.warpPerspective(np.ones(cur.shape[:2], np.uint8), h, (width, height), flags=cv2.INTER_NEAREST) > 0 + covered = cv2.erode(covered.astype(np.uint8), np.ones((5, 5), np.uint8)) > 0 + seen = float((covered & roi).sum()) / int(roi.sum()) + if seen < MIN_VISIBLE: # a trusted homography (plausible, many inliers) says so: no `LOST_NCC` doubt applies + return Pose(LOST, f"the camera moved ~{shift:.0f} px and only {seen:.0%} of the bed region of the reference " + "is still in its frame", c, inliers, shift, seen) + warped = cv2.warpPerspective(cur, h, (width, height), flags=cv2.INTER_LINEAR, borderMode=cv2.BORDER_REPLICATE) + # how well the scene lines up after the move, measured where a plate or a part cannot have changed it. A frame + # that aligns on the far scene but not near the bed (parallax: the camera sits lower or closer than it did) + # leaves the bed unaligned, and the diff along every edge of it reads as parts. + around = covered & scene + fit = _ncc(sr, _structure(warped), around if int(around.sum()) >= MIN_SCENE_PX else covered) + if fit < MIN_FIT: + return _lost(f"the camera moved ~{shift:.0f} px and the scene does not line up with the reference after " + f"aligning it (fit {fit:.2f}, needs {MIN_FIT:.2f}: it sees the bed from another height or " + "angle)", c, inliers, shift, seen) + return Pose(MOVED, f"the camera moved ~{shift:.0f} px since the reference ({inliers} features matched, fit " + f"{fit:.2f}): the frame was aligned to it", c, inliers, shift, seen, h, covered) diff --git a/lib/bed_refs.py b/lib/bed_refs.py index 83017fd9..bca848fe 100644 --- a/lib/bed_refs.py +++ b/lib/bed_refs.py @@ -159,6 +159,11 @@ def refresh_refusal(verdict: Mapping[str, Any], view_min: float) -> str | None: the frame, `watch.bed.auto_view_min`), with the printer's light on (or no light to switch).""" if verdict.get("bed") != "clear" or verdict.get("plate") != "present": return f"the bed check says bed {verdict.get('bed')}, plate {verdict.get('plate')}, not clear with the plate on" + pose = verdict.get("pose") + if isinstance(pose, Mapping) and pose.get("state") in ("moved", "lost"): + return (f"the camera is not where the reference was taken (it is {pose.get('state')}, ~{pose.get('shift_px')} px " + "off): stored from here, a watch.bed.roi polygon would keep its place in the frame, not on the plate; " + "take it by hand and check the outline: `3d printer bed reference NAME --capture`") metrics = verdict.get("metrics") or {} seen = metrics.get("bed_in_view") if isinstance(seen, (int, float)) and seen < view_min: diff --git a/lib/bed_state.py b/lib/bed_state.py index 492e0461..9e91b32e 100644 --- a/lib/bed_state.py +++ b/lib/bed_state.py @@ -26,6 +26,9 @@ plate: with a `--no-plate` reference too, whichever reference the bed surface matches better; with the plate reference only, most of the surface changed -> missing (plate off, or the bed covered); surface under 2% of the frame -> unknown. + pose: the K1 SE camera rides with the toolhead, so the frame is first LOCATED against the reference + (bed_pose.py): same view -> judged as is; moved but alignable -> warped onto the reference; + unalignable -> a `plate missing` / `occupied` read from pixels becomes unknown with the fix. ROI: fleet `watch.bed.roi` (a polygon of the bed in the frame at the reference Z) if given, else the camera calibration (vision_geometry: bed surface and the volume above it up to `volume_height_mm`, projected at the reference Z), else the whole frame. @@ -89,6 +92,7 @@ ) from printer_ops.bed_light import LightSession, write_block # noqa: E402 +import bed_pose # noqa: E402 import bed_refs # noqa: E402 from bed_view import view_in_progress # noqa: E402 @@ -695,7 +699,6 @@ def once(norm: bool) -> Verdict: def classify(img: np.ndarray, refs: list[Reference], z: float | None, printer: str, th: Thresholds = TH, match: ZMatch | None = None, ctx: bed_refs.JudgeContext | None = None) -> Verdict: import cv2 - import numpy as np match = match or ZMatch() hidden = out_of_view(match, z, th) @@ -705,6 +708,49 @@ def classify(img: np.ndarray, refs: list[Reference], z: float | None, printer: s plate_ref, noplate_ref, early = _select_refs(img, refs, z, printer, match, ctx) if early is not None: return early + base = plate_ref or noplate_ref + assert base is not None + pose = bed_pose.locate(base.image, img, base.volume) + plate_ref, noplate_ref, pose = _refs_for_view(img, plate_ref, noplate_ref, pose) + base = plate_ref or noplate_ref + assert base is not None + if pose.state == bed_pose.MOVED: # judge in the reference's picture, inside what the moved frame still covers + assert pose.h is not None and pose.covered is not None + img = cv2.warpPerspective(img, pose.h, (img.shape[1], img.shape[0]), flags=cv2.INTER_LINEAR, + borderMode=cv2.BORDER_REPLICATE) + plate_ref, noplate_ref = ((_within(base, pose.covered), None) if base is plate_ref + else (None, _within(base, pose.covered))) + v = _classify_selected(img, plate_ref, noplate_ref, z, printer, th, match, ctx) + return _apply_pose(v, pose, printer, base.roi_source) + + +def _refs_for_view(img: np.ndarray, plate_ref: Reference | None, noplate_ref: Reference | None, + pose: bed_pose.Pose) -> tuple[Reference | None, Reference | None, bed_pose.Pose]: + """The references that stand for the view `img` is seen from, and the pose against the first of them. `pose` is + the one against `plate_ref` (or against `noplate_ref` when that is all there is). A bare-bed reference taken + from another view than the frame's is not used (it would read every plate as present); when the bare-bed one + is the one taken from here and the with-plate one is not, the bare-bed one is all there is.""" + if plate_ref is None or noplate_ref is None or pose.state == bed_pose.BLIND: + return plate_ref, noplate_ref, pose + other = bed_pose.locate(noplate_ref.image, img, noplate_ref.volume) + if pose.state == bed_pose.SAME and other.state in (bed_pose.SAME, bed_pose.BLIND): + return plate_ref, noplate_ref, pose + if pose.state != bed_pose.SAME and other.state == bed_pose.SAME: + return None, noplate_ref, other + return plate_ref, None, pose + + +def _within(ref: Reference, valid: np.ndarray) -> Reference: + """`ref` judged only where the moved frame covers the reference's picture.""" + return replace(ref, volume=ref.volume & valid, surface=ref.surface & valid) + + +def _classify_selected(img: np.ndarray, plate_ref: Reference | None, noplate_ref: Reference | None, + z: float | None, printer: str, th: Thresholds, match: ZMatch, + ctx: bed_refs.JudgeContext | None) -> Verdict: + import cv2 + import numpy as np + base = plate_ref or noplate_ref assert base is not None stats = _light_stats(base, img, th, ctx) @@ -724,6 +770,48 @@ def classify(img: np.ndarray, refs: list[Reference], z: float | None, printer: s return _vet_reference(v, ref, ctx, printer) +def _apply_pose(v: Verdict, pose: bed_pose.Pose, printer: str, roi_source: str) -> Verdict: + """The verdict of a frame located against its reference (bed_pose.py), with the pose in `info` and `pose_ncc` + in the metrics. `moved`: say so, and bring the evidence masks (in the reference's picture) back onto the camera + frame. `lost`: the frame was not judged against a picture it no longer sees, so EVERY verdict, a clear bed with + the plate on included, is `unknown` with the way to a reference that fits (never a PASS).""" + metrics = {**v.metrics, "pose_ncc": pose.ncc} + info = {**v.info, "pose": pose.info()} + reason = v.reason + # a `watch.bed.roi` polygon is a region of the frame: a reference taken from another place keeps it where it was + drawn = ("; watch.bed.roi is drawn on the frame: check that it outlines the plate from the new place (the cyan " + "outline of the evidence frame)") if "watch.bed.roi" in roi_source else "" + if v.plate == MISSING and pose.state != bed_pose.LOST: # the plate is off, or the reference is wrong + reason += (f"; if the plate IS on, retake the reference with the plate on and the bed empty: " + f"`3d printer bed reference {printer} --capture`{drawn}") + if pose.state == bed_pose.MOVED: + assert pose.h is not None + import cv2 + import numpy as np + + back = np.linalg.inv(pose.h) + + def unwarp(mask: Any) -> Any: + if mask is None: + return None + return cv2.warpPerspective(mask.astype(np.uint8), back, (mask.shape[1], mask.shape[0]), + flags=cv2.INTER_NEAREST) > 0 + + return replace(v, reason=f"{reason}; {pose.why}", changed=unwarp(v.changed), roi=unwarp(v.roi), info=info, + metrics={**metrics, "pose_shift_px": pose.shift_px, "pose_inliers": float(pose.inliers)}) + if pose.state != bed_pose.LOST: + return replace(v, reason=reason, metrics=metrics, info=info) + read = ", ".join(f"{k} {w}" for k, w in (("bed", v.bed), ("plate", v.plate)) if w != UNKNOWN) + found = f"the pixels alone read `{read}`: " if read else "the pixels alone: " + return replace(v, bed=UNKNOWN, plate=UNKNOWN, info=info, + metrics={**metrics, "pose_shift_px": pose.shift_px, "pose_inliers": float(pose.inliers)}, + reason=(f"cannot judge the bed: the camera is not where it was when the reference was taken, so the " + f"frame was not judged at all, a clear read is as void as a changed one ({pose.why}; " + f"{found}{v.reason}). Fix: with the plate on and the bed empty take a reference from where " + f"the camera is now: `3d printer bed reference {printer} --capture`{drawn}; or put the " + f"toolhead back where the reference was taken")) + + def occupied_confident(v: Verdict, th: Thresholds, view_min: float) -> bool: """Is this `occupied` one a start gate may FAIL on? A changed region of the normal occupied size (blob `occupied_blob_px` or `occupied_min` of the ROI: not the lenient low-view size) seen with at least diff --git a/tests/data/bed/k1se_idle_far.jpg b/tests/data/bed/k1se_idle_far.jpg new file mode 100644 index 00000000..cb29043b Binary files /dev/null and b/tests/data/bed/k1se_idle_far.jpg differ diff --git a/tests/data/bed/k1se_idle_moved.jpg b/tests/data/bed/k1se_idle_moved.jpg new file mode 100644 index 00000000..2d645784 Binary files /dev/null and b/tests/data/bed/k1se_idle_moved.jpg differ diff --git a/tests/data/bed/k1se_idle_ref.jpg b/tests/data/bed/k1se_idle_ref.jpg new file mode 100644 index 00000000..5c56c44c Binary files /dev/null and b/tests/data/bed/k1se_idle_ref.jpg differ diff --git a/tests/data/bed/k1se_old_ref.jpg b/tests/data/bed/k1se_old_ref.jpg new file mode 100644 index 00000000..51b372a8 Binary files /dev/null and b/tests/data/bed/k1se_old_ref.jpg differ diff --git a/tests/data/bed/k1se_part_patch.jpg b/tests/data/bed/k1se_part_patch.jpg new file mode 100644 index 00000000..a86c84e7 Binary files /dev/null and b/tests/data/bed/k1se_part_patch.jpg differ diff --git a/tests/test_bed_pose.py b/tests/test_bed_pose.py new file mode 100644 index 00000000..d6b7fdae --- /dev/null +++ b/tests/test_bed_pose.py @@ -0,0 +1,309 @@ +"""The K1 SE bed check when the camera is not where the reference was taken (2026-11-13 false "plate missing"). + +The K1 SE camera rides with the toolhead, so the same empty plate is seen from another place after every job: the +whole picture (background included) shifts and zooms by ~100 px. Against a reference from elsewhere that was +"70 % of the bed surface changed" and a plate-missing FAIL that refused the start. The frames in tests/data/bed +are real (scaled to 640 px): `k1se_idle_ref` is the reference taken at the idle Z 124.8, `k1se_idle_moved` the +same empty plate 10 minutes earlier from ~100 px away, `k1se_idle_far` from a view that cannot be aligned, +`k1se_old_ref` the older Z 125.0 reference (another day, another place). A plate taken off is the idle frame with +the bed surface replaced (no real K1 frame of a bare bed exists from the idle view); parts are cut from a real frame. +""" + +from __future__ import annotations + +import dataclasses +from pathlib import Path +from typing import Any + +import pytest +from printer_ops import bed as pbed +from printer_ops import preflight +from test_bed_state import K1_ROI, U1_OCCUPIED, FakeCamera, device, frame, u1_ref + +np = pytest.importorskip("numpy") +cv2 = pytest.importorskip("cv2") +import bed_pose as P # noqa: E402 (needs numpy + OpenCV) +import bed_state as B # noqa: E402 + +DATA = Path(__file__).parent / "data" / "bed" +IDLE_Z = 124.8 + + +@pytest.fixture(autouse=True) +def xdg(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path: + for var in ("XDG_CONFIG_HOME", "XDG_STATE_HOME", "XDG_DATA_HOME", "XDG_CACHE_HOME"): + monkeypatch.setenv(var, str(tmp_path / var.lower())) + return tmp_path + + +def ref_of(name: str, roi: tuple[tuple[float, float], ...] = K1_ROI, z: float = IDLE_Z) -> B.Reference: + img = frame(name) + m = B.polygon_mask(roi, img.shape[:2]) + return B.Reference("plate", z, 0.0, "k1se.watch.bed.roi", img, m, m, DATA / name) + + +def classify(img: Any, ref: B.Reference) -> B.Verdict: + return B.classify(img, [ref], ref.z, "k1se") + + +def part_patch() -> Any: + """A 3D-printed figure lying on the K1's plate (2026-11-11), cut from a real frame.""" + return cv2.resize(cv2.imread(str(DATA / "k1se_part_patch.jpg")), (112, 71), interpolation=cv2.INTER_AREA) + + +def with_part(img: Any, x: int, y: int) -> Any: + out = img.copy() + patch = part_patch() + out[y:y + patch.shape[0], x:x + patch.shape[1]] = patch + return out + + +def moved_by(img: Any, dx: float, dy: float, zoom: float) -> Any: + """`img` as a camera that moved sees it: shifted and zoomed about the frame centre.""" + h, w = img.shape[:2] + m = np.array([[zoom, 0, dx + (1 - zoom) * w / 2], [0, zoom, dy + (1 - zoom) * h / 2]], np.float32) + return cv2.warpAffine(img, m, (w, h), flags=cv2.INTER_LINEAR, borderMode=cv2.BORDER_REPLICATE) + + +# ---- the 2026-11-13 failure ------------------------------------------------------------------------------- + +@pytest.mark.parametrize("name", ["k1se_idle_moved.jpg", "k1se_idle_ref.jpg"]) +def test_empty_plate_against_an_old_reference_from_another_view_is_never_plate_missing(name: str) -> None: + """Both preflight frames showed the plate on and empty; judged against the only reference (Z 125.0, taken + from another place) the pixels read ~70 % changed, which used to be `plate missing` and FAIL.""" + v = classify(frame(name), ref_of("k1se_old_ref.jpg", z=125.0)) + assert v.metrics["surface_vs_plate"] > 0.6 # the pixel evidence the old verdict stood on is still there ... + assert (v.bed, v.plate) == ("unknown", "unknown"), v.reason # ... but it is no verdict + assert "the camera is not where it was when the reference was taken" in v.reason + assert "3d printer bed reference k1se --capture" in v.reason # how to fix it + + +def test_the_preflight_line_for_that_verdict_is_a_warn_with_the_way_out_not_a_fail() -> None: + v = classify(frame("k1se_idle_moved.jpg"), ref_of("k1se_old_ref.jpg", z=125.0)) + check = preflight.check_bed_verdict({"printer": "k1se", "bed": v.bed, "plate": v.plate, "reason": v.reason, + "source": "camera now", "age_s": 0.0}) + assert check.status == preflight.WARN + assert "bed reference k1se --capture" in check.detail and '--assume-empty "why"' in check.detail + + +def test_empty_plate_seen_from_a_camera_that_moved_passes_against_a_reference_from_the_idle_view() -> None: + """20:37 frame, ~100 px from where the 20:38 reference was taken: aligned to it, clear with the plate.""" + v = classify(frame("k1se_idle_moved.jpg"), ref_of("k1se_idle_ref.jpg")) + assert (v.bed, v.plate) == ("clear", "present"), v.reason + assert v.metrics["pose_shift_px"] > 50 and "the frame was aligned to it" in v.reason + + +def test_empty_plate_at_the_idle_view_passes() -> None: + v = classify(frame("k1se_idle_ref.jpg"), ref_of("k1se_idle_ref.jpg")) + assert (v.bed, v.plate) == ("clear", "present"), v.reason + assert v.metrics["pose_ncc"] > 0.99 and "pose_shift_px" not in v.metrics + + +def test_a_view_that_cannot_be_aligned_is_unknown_never_a_finding() -> None: + """The 20:30 frame sees the plate from nearer and lower: the scene aligns, the bed does not.""" + v = classify(frame("k1se_idle_far.jpg"), ref_of("k1se_idle_ref.jpg")) + assert (v.bed, v.plate) == ("unknown", "unknown"), v.reason + assert "does not line up" in v.reason and "bed reference k1se --capture" in v.reason + assert "watch.bed.roi is drawn on the frame" in v.reason # the polygon has to follow the camera, too + + +# ---- what must still FAIL ---------------------------------------------------------------------------------- + +def plate_off(img: Any, ref: B.Reference) -> Any: + """The idle frame with the plate surface replaced by a bright bare-metal-like surface.""" + out = img.copy() + noise = np.random.default_rng(7).integers(-25, 25, size=out.shape) + bare = np.clip(np.array([185, 180, 175]) + noise, 0, 255).astype(np.uint8) + out[ref.surface] = bare[ref.surface] + return out + + +def test_a_plate_that_was_taken_off_is_missing_from_the_idle_view() -> None: + ref = ref_of("k1se_idle_ref.jpg") + v = classify(plate_off(ref.image, ref), ref) + assert v.plate == "missing", v.reason + assert "plate off, or the bed covered" in v.reason + assert "if the plate IS on, retake the reference" in v.reason # the fix, in case the reference is the wrong one + + +@pytest.mark.parametrize(("dx", "dy", "zoom"), [(-40.0, 25.0, 0.93), (35.0, -20.0, 1.06)]) +def test_a_plate_that_was_taken_off_is_still_missing_when_the_camera_moved(dx: float, dy: float, zoom: float) -> None: + ref = ref_of("k1se_idle_ref.jpg") + v = classify(moved_by(plate_off(ref.image, ref), dx, dy, zoom), ref) + assert v.plate == "missing", v.reason + assert v.metrics["pose_shift_px"] > 10 and "aligned" in v.reason + + +def test_a_part_on_the_plate_is_occupied_from_the_idle_view() -> None: + ref = ref_of("k1se_idle_ref.jpg") + v = classify(with_part(ref.image, 230, 170), ref) + assert v.bed == "occupied" and v.plate == "present", v.reason + + +def test_a_part_on_the_plate_is_occupied_when_the_camera_moved_and_the_evidence_marks_it_on_the_frame() -> None: + """The part is pasted on the moved frame at x 100..212, y 175..246: the red evidence is drawn there.""" + ref = ref_of("k1se_idle_ref.jpg") + img = with_part(frame("k1se_idle_moved.jpg"), 100, 175) + v = classify(img, ref) + assert v.bed == "occupied" and v.plate == "present", v.reason + assert v.changed is not None and v.roi is not None and v.changed.shape == img.shape[:2] + ys, xs = np.nonzero(v.changed) + assert 100 <= xs.mean() <= 212 and 175 <= ys.mean() <= 246, (xs.mean(), ys.mean()) + assert v.roi.shape == img.shape[:2] and v.roi.any() + + +def test_a_part_seen_from_a_view_that_cannot_be_aligned_is_not_a_finding_either() -> None: + v = classify(with_part(frame("k1se_idle_far.jpg"), 100, 150), ref_of("k1se_idle_ref.jpg")) + assert v.bed == "unknown" and "bed reference k1se --capture" in v.reason + + +def test_a_small_camera_move_is_aligned_too_not_judged_as_the_same_view() -> None: + """A 15 px shift still correlates (0.4) and used to count as the same view: edges then read as changes.""" + ref = ref_of("k1se_idle_ref.jpg") + pose = P.locate(ref.image, moved_by(ref.image, 15.0, 8.0, 1.0), ref.volume) + assert pose.state == P.MOVED and 12 < pose.shift_px < 25 + v = classify(moved_by(plate_off(ref.image, ref), 15.0, 8.0, 1.0), ref) + assert v.plate == "missing" and v.metrics["pose_shift_px"] > 12, v.reason + + +def test_a_camera_that_moved_so_that_part_of_the_bed_is_out_of_frame_is_unknown_never_clear() -> None: + """A part in the 35 % of the bed region the moved frame no longer covers must not pass as a clear bed.""" + ref = ref_of("k1se_idle_ref.jpg") + img = moved_by(ref.image, 250.0, 0.0, 1.0) + pose = P.locate(ref.image, img, ref.volume) + assert pose.state == P.LOST and pose.visible < P.MIN_VISIBLE, pose.why + v = classify(img, ref) + assert (v.bed, v.plate) != ("clear", "present") and v.bed != "occupied", v.reason + + +def test_a_bare_bed_reference_from_another_view_is_not_used_against_the_aligned_frame() -> None: + """Aligned to the with-plate reference, the frame is not aligned to a --no-plate one taken elsewhere: that one + would read every plate as present (its surface differs everywhere) or tie.""" + plate = ref_of("k1se_idle_ref.jpg") + bare = dataclasses.replace(ref_of("k1se_idle_far.jpg"), kind="noplate") + v = B.classify(plate_off(plate.image, plate), [plate, bare], IDLE_Z, "k1se") + assert v.plate == "missing", v.reason + assert "bare-bed one" not in v.reason + + +def test_a_zoom_about_the_bed_is_a_move_even_though_the_bed_centre_stays_put() -> None: + """The toolhead moving along the camera axis zooms about the bed: its centre stays, its border moves ~6 px.""" + ref = ref_of("k1se_idle_ref.jpg") + ys, xs = np.nonzero(ref.volume) + cx, cy, zoom = float(xs.mean()), float(ys.mean()), 1.1 + m = np.array([[zoom, 0, cx * (1 - zoom)], [0, zoom, cy * (1 - zoom)]], np.float32) + img = cv2.warpAffine(plate_off(ref.image, ref), m, (ref.image.shape[1], ref.image.shape[0]), + borderMode=cv2.BORDER_REPLICATE) + v = classify(img, ref) + assert v.plate == "missing" and v.metrics["pose_shift_px"] > P.SAME_PX, v.reason + + +def test_a_bare_bed_reference_taken_from_here_judges_the_frame_when_the_with_plate_one_was_not() -> None: + """The with-plate reference is the old one from another place, the bare-bed one was taken from where the camera + is now: the plate is off, and that is what the bare-bed reference says.""" + idle = ref_of("k1se_idle_ref.jpg") + old = ref_of("k1se_old_ref.jpg", z=IDLE_Z) + bare = dataclasses.replace(idle, kind="noplate", image=plate_off(idle.image, idle)) + v = B.classify(plate_off(idle.image, idle), [old, bare], IDLE_Z, "k1se") + assert v.plate == "missing", v.reason + assert v.info["pose"]["state"] == "same" + assert "does not see the scene" not in v.reason + + +def test_an_empty_bed_region_is_reported_by_the_classifier_not_crashed_on() -> None: + ref = ref_of("k1se_idle_ref.jpg") + empty = np.zeros(ref.image.shape[:2], bool) + assert P.locate(ref.image, frame("k1se_idle_moved.jpg"), empty).state == P.BLIND + v = classify(frame("k1se_idle_moved.jpg"), dataclasses.replace(ref, volume=empty, surface=empty)) + assert v.bed == "unknown" and "empty or tiny" in v.reason + + +@pytest.mark.parametrize("with_a_part", [False, True]) +def test_a_frame_whose_camera_is_lost_is_never_clear_even_when_its_pixels_look_like_the_reference( + with_a_part: bool, monkeypatch: pytest.MonkeyPatch +) -> None: + """A lost pose means the frame was not judged: `3d printer start` must not go ahead on a `clear` read from + pixels that happen to match. The locator is forced to say `lost` on frames that look clear (the real idle frame, + and the same frame with a part on the plate in a spot the pixels compare against the reference).""" + ref = ref_of("k1se_idle_ref.jpg") + img = with_part(ref.image, 230, 170) if with_a_part else ref.image + monkeypatch.setattr(P, "locate", lambda *_: P.Pose(P.LOST, "the camera does not see the scene of the reference", + 0.1, 7, 0.0)) + pixels = B._classify_selected(img, ref, None, IDLE_Z, "k1se", B.TH, B.ZMatch(), None) + assert pixels.bed == ("occupied" if with_a_part else "clear"), pixels.reason # what the pixels alone say + v = classify(img, ref) + assert (v.bed, v.plate) == ("unknown", "unknown"), v.reason + assert v.info["pose"]["state"] == "lost" and v.metrics["pose_shift_px"] == 0.0 + assert "was not judged at all" in v.reason and "bed reference k1se --capture" in v.reason + check = preflight.check_bed_verdict({"printer": "k1se", "bed": v.bed, "plate": v.plate, "reason": v.reason, + "source": "camera now", "age_s": 0.0}) + assert check.status == preflight.WARN # never PASS, never the FAIL of a finding either + + +def test_a_whole_frame_roi_has_no_scene_to_locate_by_and_is_judged_as_before() -> None: + ref = ref_of("k1se_idle_ref.jpg") + whole = np.ones(ref.image.shape[:2], bool) + assert P.locate(ref.image, frame("k1se_idle_moved.jpg"), whole).state == P.BLIND + + +@pytest.mark.parametrize("name", [*U1_OCCUPIED, "u1_spaghetti_printing.jpg", "u1_z200_light_1703.jpg", + "u1_z200_light_2013.jpg"]) +def test_a_fixed_camera_is_never_a_lost_camera_whatever_happened_on_the_bed(name: str) -> None: + """The U1's camera never moves: parts, spaghetti and a night light must never be called a moved camera and + turned into `unknown` by it (the start gate FAILs on these); at worst the locator cannot tell and the verdict + is the pixels' as ever.""" + ref = u1_ref() + assert P.locate(ref.image, frame(name), ref.volume).state in (P.SAME, P.BLIND) + + +# ---- the locator --------------------------------------------------------------------------------------------- + +def test_locate_same_moved_lost_and_blind() -> None: + ref = ref_of("k1se_idle_ref.jpg") + assert P.locate(ref.image, ref.image, ref.volume).state == P.SAME + moved = P.locate(ref.image, frame("k1se_idle_moved.jpg"), ref.volume) + assert moved.state == P.MOVED and moved.h is not None and moved.covered is not None + assert P.locate(ref.image, frame("k1se_old_ref.jpg"), ref.volume).state == P.LOST + black = np.zeros_like(ref.image) + assert P.locate(ref.image, black, ref.volume).state == P.BLIND # nothing to locate: judged as before + + +def test_the_k1_in_a_dark_room_against_a_daylight_reference_is_the_same_view() -> None: + """Another light is not another view: the daylight reference against the dark-room frame is not aligned.""" + ref = frame("k1_z125_clear_ref.jpg") + roi = B.polygon_mask(K1_ROI, ref.shape[:2]) + assert P.locate(ref, frame("k1_z125_clear_later.jpg"), roi).state == P.SAME + + +# ---- end to end: the check on a stored reference --------------------------------------------------------------- + +def test_check_against_the_only_old_reference_warns_and_a_new_reference_then_passes(xdg: Path) -> None: + dev = device("k1se", watch={"bed": {"enabled": True, "roi": [list(p) for p in K1_ROI]}}, model="Creality K1 SE") + cfg = pbed.BedConfig(enabled=True, roi=K1_ROI) + B.capture_reference(dev, cfg, plate=True, frame=(DATA / "k1se_old_ref.jpg").read_bytes(), frame_z=125.0, now=1.0) + cam = FakeCamera("k1se_idle_moved.jpg") + v = B.one_shot(dev, cfg, max_age=None, camera=cam, printer_fn=lambda d: ("complete", IDLE_Z), sleep=lambda s: None) + assert (v["bed"], v["plate"]) == ("unknown", "unknown") and "bed reference k1se --capture" in v["reason"] + assert preflight.check_bed_verdict(v).status == preflight.WARN + + B.capture_reference(dev, cfg, plate=True, camera=FakeCamera("k1se_idle_ref.jpg"), + printer_fn=lambda d: ("complete", IDLE_Z), now=2.0) + v = B.one_shot(dev, cfg, max_age=None, camera=cam, printer_fn=lambda d: ("complete", IDLE_Z), sleep=lambda s: None) + assert (v["bed"], v["plate"]) == ("clear", "present"), v["reason"] + assert preflight.check_bed_verdict(v).status == preflight.PASS + + +def test_a_clear_judged_from_a_moved_camera_never_becomes_the_reference(xdg: Path) -> None: + """Stored from there, the reference would keep the ROI polygon where it was drawn: another place on the plate.""" + import bed_refs + + dev = device("k1se", watch={"bed": {"enabled": True, "roi": [list(p) for p in K1_ROI]}}, model="Creality K1 SE") + cfg = pbed.BedConfig(enabled=True, roi=K1_ROI) + B.capture_reference(dev, cfg, plate=True, frame=(DATA / "k1se_idle_ref.jpg").read_bytes(), frame_z=IDLE_Z, now=1.0) + v = B.one_shot(dev, cfg, max_age=None, camera=FakeCamera("k1se_idle_moved.jpg"), + printer_fn=lambda d: ("complete", IDLE_Z), sleep=lambda s: None) + assert (v["bed"], v["plate"], v["pose"]["state"]) == ("clear", "present", "moved") + lit = {**v, "light": {"on_for_frame": True}} + assert "not where the reference was taken" in (bed_refs.refresh_refusal(lit, 0.5) or "") + assert bed_refs.refresh_refusal({**lit, "pose": {"state": "same"}}, 0.5) is None + assert "not where the reference was taken" in (bed_refs.refresh_refusal({**lit, "pose": {"state": "lost"}}, 0.5) or "")