Skip to content

Agent integration: pointer/axis IPC + contrib/driftwm-agent + docs - #341

Draft
youssefvdel wants to merge 6 commits into
malbiruk:mainfrom
youssefvdel:feat/agent-desktop
Draft

youssefvdel wants to merge 6 commits into
malbiruk:mainfrom
youssefvdel:feat/agent-desktop

Conversation

@youssefvdel

@youssefvdel youssefvdel commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Adds the missing input path for AI agents and remote viewers, plus a reference adapter.

What

  • msg pointer <x> <y> [--button left|right|middle] — move the seat pointer in the output layout and optionally click. No uinput, so it works in nested sessions where host input devices are invisible to driftwm.
  • msg axis <dx> <dy> — scroll the focused surface by output-layout logical pixels, the same space pointer and cursor use; a notch is 15 px (source wheel, v120 + value), verified in wev.
  • contrib/driftwm-agent — thin adapter (Bun/TS): state, shot --scale, center, focus, snap (AT-SPI), click, type (wtype), msg; plus a WebSocket browser viewer (PNG or --jpeg, wheel/mouse/keyboard, token/view-only options).
  • docs/agents.md — how to drive driftwm with AT-SPI agents; docs/ipc.md pointer/scroll sections; docs/cli.md regenerated.

Verification

  • cargo clippy --all-targets -- -D warnings clean.
  • cargo test --bin driftwm ipc — 113 passed.
  • Live nested session (winit in headless Sway): cursor 640 400 → pointer 400 300 → cursor 400 300; pointer 640 400 --button left lands; axis 0 -15 produces axis_source wheel, value120 -120, value -15 in wev; viewer streams 1600×900 at ~6 fps, JPEG ~39 KB/frame.

Notes

  • Input IPC is user-only (socket 0600). Happy to gate it behind a config flag if preferred.
  • Coordinate space: output layout logical pixels, Y-down (same as cursor), matching the frame a viewer receives.

- msg pointer <x> <y> [--button left|right|middle]: move the seat pointer in the output layout and optionally click there. Works in nested sessions (no uinput) and gives remote viewers/agents a coordinate input path.
- docs/agents.md: how to drive driftwm with AT-SPI agents; docs/ipc.md: pointer command.
- contrib/driftwm-agent: reference adapter (state/shot/focus/center/snap/click/type) and a WebSocket browser viewer with keyboard input.
- cli docs regenerated.
- msg axis <dx> <dy>: scroll the focused surface by wheel notches (verified: wev sees axis_source wheel, value120, value)
- contrib viewer: mouse-wheel forwarding, optional --jpeg (ffmpeg transcode, ~39 KB/frame at 1600x900) and --fps.
- docs/ipc.md scrolling section, cli docs regenerated, hyphen values accepted for pointer/axis args.
Without it the udev backend prefers the system primary GPU. On a headless box that card has no active scanout; an explicit device (e.g. /dev/dri/card0 from vkms) lets driftwm run a real 1920x1080 session with no monitor and no X server. Verified locally: driftwm --backend udev --drm-device /dev/dri/card0 on vkms, output 'Virtual-1' at 1920x1080@60, IPC state + 1920x1080 canvas screenshot.
@Granitosaurus

Copy link
Copy Markdown

msg axis — scroll the focused surface by wheel notches (source wheel, v120 + value), verified in wev

Maybe this should be in pixels to match msg pointer <x> <y>

FocusedWindow gains pid (same socket-credential source as WindowInfo) so agents map focus to AT-SPI without a second state round-trip; stand-ins report None. OutputInfo gains layout_origin (the output top-left in global layout coords, same space msg cursor/pointer use). Reload and cold boot now refresh/seed active_layout display name (was stale English (US) / empty until first keypress).
The agent integration shipped in the compositor itself: inventory, pid, focus+reveal, pointer/axis injection and canvas capture are all 'driftwm msg' subcommands, in-process. The contrib/driftwm-agent CLI was pure indirection over those commands plus a wtype/ydotool shim and a viewer that duplicates RustDesk, so it is removed rather than maintained as a second interface.

agents.md now documents the built-in surface and how a standard AT-SPI client pairs with it: map windows by pid, focus to reveal, capture via the compositor. Notes the two upstream limits that are not ours to fix: agent-desktop lists displays only through Mutter DisplayConfig, and xdg-desktop-portal-wlr writes its Screenshot result to a hardcoded /tmp/out.png (ScreenCast through the same portal works against our screencopy).
`msg pointer <x> <y>` takes output-layout logical pixels and `cursor` reports
the same space, but `msg axis <dx> <dy>` took wheel notches, so a caller had to
know two units to drive one pointer. Reported by Granitosaurus on malbiruk#341.

`axis` now takes the same logical pixels. The frame still carries both
`axis_value120` and `axis`, so clients that scroll discretely keep getting
whole notches: a notch is 15 px, which is the ratio the old notch-based call
already used internally, so one notch in is still one notch out.
@youssefvdel

Copy link
Copy Markdown
Contributor Author

Good catch — changed in 4494e50.

msg axis <dx> <dy> now takes output-layout logical pixels, the same space msg pointer <x> <y> takes and msg cursor reports, so one unit drives the pointer.

The frame still carries both fields, so clients that scroll discretely keep getting whole notches: a notch is 15 px, which is the ratio the old notch-based call already used internally — one notch in is still one notch out.

  • driftwm msg axis 0 15 → one notch down
  • driftwm msg axis 0 45 → three notches down

docs/ipc.md and the generated docs/cli.md are updated, and there is a unit test on the 15 px mapping. Gates: fmt, clippy --all-targets -D warnings, and the full suite (1630 passed) are green.

@youssefvdel

Copy link
Copy Markdown
Contributor Author

While re-verifying the pixel change I could not reproduce scroll delivery to a client, and I want to check my method before trusting the change.

On a live driftwm session (udev backend), with wev focused and the pointer over its surface:

driftwm msg cursor                 -> 250 180 DVI-D-1
driftwm msg axis 0 -1              -> replies 0 -1
wev                                 -> 0 axis events (25 motion events seen earlier)
driftwm msg pointer 250 180        -> cursor moves, wev logs no motion
ydotool mousemove -w -x 0 -y -1    -> 0 axis events

The real wheel case is explained by the scroll binding: src/input/pointer.rs consumes bound scroll and runs the action, and the camera did move (camera 1 -679.9 → 45.9 -702.9), so wheel went to canvas zoom rather than the app.

What I cannot explain is the IPC path: cmd_axis calls pointer.axis(state, frame) directly, and the same session shows no client-facing axis. Same for msg pointer, which moves the cursor without the client seeing motion.

Question: in the run where wev showed axis_source wheel, value120 -120, value -15, was that a nested winit instance under Sway, or driftwm's own session? I tested nested winit under driftwm and saw motion but no axis, so I suspect my probe (wev as the only client on an infinite canvas) rather than the code — but I would rather ask than leave an unverified claim in the description.

@youssefvdel

Copy link
Copy Markdown
Contributor Author

Follow-up with a controlled run, because the first one was inconclusive (my pointer was not actually over the client).

Setup: wev mapped, driven into focus by real input (/dev/uinput pointer), not by msg pointer:

agent-desktop mouse-move --xy 950,530   -> wev logs 6x enter, 19x motion
agent-desktop mouse-click --xy 950,530  -> cursor 949.98 529.98
driftwm msg axis 0 -1                   -> reply 0 -1, wev logs 0 axis events

So focus was real and established before the injection. Real input reaches the client; msg axis does not appear to. msg pointer behaves the same way in my runs: the compositor cursor moves (cursor reports the new position) and the client sees nothing, while a uinput move at the same coordinates does produce enter/motion.

That suggests the IPC path updates seat state without a client-facing frame, rather than my pixel change being wrong. Before I open anything as a bug: is msg pointer/msg axis meant to deliver to the focused client, or only to move the compositor's own cursor for a viewer? The axis doc says a remote viewer forwards wheel events through it, which reads like delivery — but the evidence here says otherwise on this build.

@youssefvdel

Copy link
Copy Markdown
Contributor Author

Reproduced with a real client that actually commits a buffer, so wev is out of the picture.

foot (700x525, focused, 400 lines of scrollback), pointer placed inside it by msg pointer, then a 10-notch scroll, pixel-diffed against a grim screenshot of the same region:

driftwm msg pointer 950 600   -> cursor 950 600, foot focused per `msg state`
driftwm msg axis 0 -150       -> reply 0 -150
grim before/after, crop (700,340)-(1240,750)
-> changed pixels: 0 of 221400
-> camera unchanged: 1 -702.365   (so a scroll binding did not eat it either)

Nothing moved: the terminal content is identical, so the wheel never reached the app, and the camera did not move, so no scroll action consumed it.

Both cmd_pointer and the real input path end up in dispatch_absolute_motion, and cmd_pointer's screen→canvas conversion matches on_pointer_motion_absolute line for line against camera()/zoom(). So I do not think this is a coordinate-space slip in cmd_pointer — it looks like the synthetic frame is being dropped somewhere after pointer.axis().

Happy to instrument and dig further, but this is outside my change: cmd_axis only altered the numbers fed into the frame, and the old notch-based call shows the same behaviour on this box.

@youssefvdel

Copy link
Copy Markdown
Contributor Author

Correction — I was wrong in my two previous comments. Scroll delivery works.

The controlled run, nested (--backend winit) on a build from this branch, with a client that commits a buffer:

foot with 200 lines of scrollback, 700x525, focused
msg pointer 300 250                          -> has_focus = true
msg axis 0 -150                              -> v120 = -1200 (10 notches), has_focus = true
grim before / after, diff                    -> changed bbox (296,165,987,654) = exactly foot's rect

The after-shot shows the terminal had moved from the tail of the output to lines 104-141, and the scrollbar thumb moved. So 150 px / 15 does become 10 notches of v120, and the app does scroll.

I instrumented cmd_axis for that run (has_focus before and after pointer.axis) and then removed the instrumentation — nothing of it is in the tree.

What went wrong in my earlier attempts: on the live udev session wev renders no buffer, and my foot run there did not have pointer focus on the window at the moment of the injection. I do not have a full explanation for that second one, so treat the earlier "the frame is dropped" claim as retracted rather than explained.

The pixel change itself stands: msg axis takes logical pixels, one notch is 15 px, and both axis_value120 and axis are still sent. Gates on the current tree: fmt, clippy --all-targets -D warnings, 1630 tests green.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants