Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 47 additions & 0 deletions docs/agents.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# Driving driftwm with AI agents

driftwm is well suited to computer-use agents: the canvas is queryable and renderable over
its own IPC, and windows carry a stable id plus their client pid. Everything an agent needs
ships in the compositor — there is no adapter process, no shim and no second daemon.

## What the IPC gives an agent

| Need | Command |
|---|---|
| Window inventory (id, app_id, title, canvas position/size, pid, focus) | `driftwm msg --json state` |
| Canvas screenshot at arbitrary scale, including off-screen content | `driftwm msg screenshot -o - --scale N` |
| Spatial navigation | `driftwm msg camera <x> <y>`, `driftwm msg zoom <level>`, `driftwm msg focus <app_id|--id>` |
| Pointer input without uinput | `driftwm msg pointer <x> <y> [--button left|right|middle]` |
| Scroll | `driftwm msg axis <dx> <dy>` |
| Stable window identity | `state.windows[].id` / `[].pid` |
| Bookmarks | `driftwm msg bookmark ...` |

A screenshot is a **canvas re-render**, not a framebuffer copy: an agent does not need the
window on screen. `--scale 2` on a 1600×900 output yields 3200×1800, which is useful for
OCR/vision stages.

`pointer` is deliberately part of the IPC rather than a Wayland protocol: it works in
nested sessions (where uinput events would go to the host compositor) and needs no extra
device permissions. The socket is user-only (0600); treat it as an input-injection surface
and keep it inside the session.

## Pairing with agent-desktop

`agent-desktop` is a standard AT-SPI client: it reads the accessibility tree and drives
elements semantically. It needs no driftwm-specific glue.

- map its windows to driftwm ids by `pid`, which `state` and the `focus` reply both carry;
- `driftwm msg focus <app_id>` activates and reveals a window (foreign-toplevel `activate`
and `xdg-activation` both land on the same path), then its snapshot sees the window;
- `driftwm msg screenshot` is the observation channel. agent-desktop's own Wayland capture
has two known limits: display *listing* is Mutter-only, and the wlr portal's `Screenshot`
implementation writes to a hardcoded `/tmp/out.png`, which an agent that requires
home-relative files will reject. `ScreenCast` through `xdg-desktop-portal-wlr` works
against driftwm's `zwlr_screencopy` surface, which is what remote viewers use.

## Remaining gaps

1. **Video streaming.** Each `msg screenshot` re-renders; a remote viewer should throttle
(5–10 fps) or capture through `zwlr_screencopy`/portal for H.264/WebRTC.
2. **Security.** Input IPC is a deliberate capability; if it ever becomes configurable,
default it on only for the session owner.
30 changes: 30 additions & 0 deletions docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ A trackpad-first infinite canvas Wayland compositor.
With no subcommand, starts the compositor, auto-detecting the backend (udev on a TTY, winit when nested). The `msg` subcommand instead talks to an already-running instance over its IPC socket.

- `--backend <udev|winit>` — Backend to use (default: udev on a TTY, winit if nested)
- `--drm-device <PATH>` — DRM device for the udev backend (e.g. /dev/dri/card0 for vkms). Without it the primary GPU is preferred
- `--config <PATH>` — Use an alternate config file
- `--check-config` — Validate the config and exit
- `--session-file <PATH>` — Durable session file path. Overrides the default; lets a nested winit dev session opt into session restore (it skips it otherwise)
Expand Down Expand Up @@ -65,6 +66,8 @@ driftwm msg --json focus --id 5
| [`bookmark`](#driftwm-msg-bookmark) | List bookmarks, get or set one by `<name>`, or delete with `--delete` |
| [`layout`](#driftwm-msg-layout) | Print the active keyboard layout (full XKB name, e.g. `English (US)`) |
| [`cursor`](#driftwm-msg-cursor) | Print the pointer position in the global output layout (logical pixels, Y-down, the space slurp prints) and the output it is on |
| [`pointer`](#driftwm-msg-pointer) | Move the seat pointer to <x> <y> in the output layout (logical pixels, Y-down, the space `cursor` reports); with --button, click there |
| [`axis`](#driftwm-msg-axis) | Scroll the focused surface by <dx> <dy> in output-layout logical pixels, the space `pointer` and `cursor` use (positive = right/down) |
| [`action`](#driftwm-msg-action) | Run a config action, e.g. `action close-window`, `action switch-layout next` |
| [`screenshot`](#driftwm-msg-screenshot) | Capture a canvas PNG. With no subcommand, captures the active output's current view of the canvas |
| [`debug-counters`](#driftwm-msg-debug-counters) | Print internal collection sizes for leak diagnosis (unstable keys) |
Expand Down Expand Up @@ -337,6 +340,33 @@ Print the pointer position in the global output layout (logical pixels, Y-down,
driftwm msg --json cursor
```

#### `driftwm msg pointer`

```
driftwm msg pointer [OPTIONS] <X> <Y>
```

Move the seat pointer to <x> <y> in the output layout (logical pixels, Y-down, the space `cursor` reports); with --button, click there

- `--button <left|right|middle>`

```bash
driftwm msg pointer 640 400
driftwm msg pointer 640 400 --button left
```

#### `driftwm msg axis`

```
driftwm msg axis <DX> <DY>
```

Scroll the focused surface by <dx> <dy> in output-layout logical pixels, the space `pointer` and `cursor` use (positive = right/down)

```bash
driftwm msg axis 0 -1
```

#### `driftwm msg action`

```
Expand Down
32 changes: 31 additions & 1 deletion docs/ipc.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ JSON, so any language can speak it directly.

Run `driftwm msg <command>` from inside a driftwm session. The commands —
`camera`, `zoom`, `focus`, `move`, `resize`, `opacity`, `pin`, `close`,
`suspend`, `relaunch`, `layout`, `cursor`, `action`, `bookmark`, `screenshot`,
`suspend`, `relaunch`, `layout`, `cursor`, `pointer`, `action`, `bookmark`, `screenshot`,
`state`, `subscribe`, and `debug-counters` — with their arguments, flags, and
JSON reply shapes are documented in the generated [CLI reference](cli.md);
`driftwm msg <command> --help` prints the same for one command. The conventions
Expand Down Expand Up @@ -130,6 +130,36 @@ reaches off-screen content; its four targets, flags, and caveats are in the
[CLI reference](cli.md#driftwm-msg-screenshot) and their wire forms in
[Requests](#requests).

### Scrolling

`axis <dx> <dy>` scrolls the focused surface by a distance in output-layout logical
pixels — the same space `pointer <x> <y>` takes and `cursor` reports (positive =
right/down). It is what a remote viewer forwards from mouse-wheel events; the reply
echoes the deltas. A notch is 15 px, and the frame carries both `axis_value120` (for
clients that scroll discretely) and `axis` (for clients that scroll smoothly), so one
call serves both kinds of client.

```bash
driftwm msg axis 0 -15 # one notch up
driftwm msg axis 0 45 # three notches down
```

### Pointer injection

`pointer <x> <y> [--button left|right|middle]` moves the seat pointer to a point in the
global output layout (logical pixels, Y-down — the same space `cursor` reports) and, with
`--button`, clicks there.

This is the input path for remote viewers and coordinate-based agents: it does not depend
on uinput, so it also works in nested sessions, where host input devices are not visible
to driftwm. The reply is the resulting pointer position:

```bash
driftwm msg pointer 640 400
driftwm msg --json pointer 640 400 --button left
# {"Ok":{"Pointer":{"x":640.0,"y":400.0,"output":"winit"}}}
```

### Subscribing to changes

`subscribe` turns the connection into a live feed instead of polling: one event
Expand Down
13 changes: 9 additions & 4 deletions src/backend/udev.rs
Original file line number Diff line number Diff line change
Expand Up @@ -409,6 +409,7 @@ pub(crate) fn render_if_needed(data: &mut DriftWm) {
pub fn init_udev(
event_loop: &mut EventLoop<'static, DriftWm>,
data: &mut DriftWm,
drm_device: Option<&std::path::Path>,
) -> Result<UdevDevice, Box<dyn std::error::Error>> {
// 1. Create libseat session
let (mut session, session_notifier) = LibSeatSession::new()
Expand All @@ -429,10 +430,14 @@ pub fn init_udev(
tracing::info!("System primary GPU: {}", p.display());
}

// Build ordered candidate list: primary GPU first, then all others.
// On hybrid graphics (iGPU + dGPU), the "primary" GPU may not have
// the display outputs, so we fall back to other devices.
let gpu_paths: Vec<PathBuf> = {
// Build ordered candidate list: an explicit --drm-device wins and is used
// alone (headless vkms must not fall back to the monitor GPU); otherwise
// primary GPU first, then all others (on hybrid graphics the "primary" GPU
// may not have the display outputs).
let gpu_paths: Vec<PathBuf> = if let Some(path) = drm_device {
tracing::info!("Using explicit DRM device: {}", path.display());
vec![path.to_path_buf()]
} else {
let mut paths = Vec::new();
if let Some(ref p) = primary_gpu_path {
paths.push(p.clone());
Expand Down
24 changes: 24 additions & 0 deletions src/ipc/client.rs
Original file line number Diff line number Diff line change
Expand Up @@ -203,6 +203,22 @@ pub enum Msg {
///
/// `--json` reply: `{"Ok":{"Cursor":{"x":2600.5,"y":12.0,"output":"DP-1"}}}`.
Cursor,
/// Move the seat pointer to <x> <y> in the output layout (logical pixels,
/// Y-down, the space `cursor` reports); with --button, click there.
Pointer {
x: f64,
y: f64,
#[arg(long, value_name = "left|right|middle")]
button: Option<String>,
},
/// Scroll the focused surface by <dx> <dy> in output-layout logical
/// pixels, the space `pointer` and `cursor` use (positive = right/down).
Axis {
#[arg(allow_hyphen_values = true)]
dx: f64,
#[arg(allow_hyphen_values = true)]
dy: f64,
},
/// Run a config action, e.g. `action close-window`, `action switch-layout next`.
///
/// Takes the same string you would write in a config keybinding, parsed with
Expand Down Expand Up @@ -405,6 +421,12 @@ fn to_request(msg: &Msg) -> Result<Request, String> {
Msg::Zoom { level } => Request::Zoom(*level),
Msg::Layout { short } => Request::Layout { short: *short },
Msg::Cursor => Request::Cursor,
Msg::Pointer { x, y, button } => Request::Pointer {
x: *x,
y: *y,
button: button.clone(),
},
Msg::Axis { dx, dy } => Request::Axis { dx: *dx, dy: *dy },
Msg::State => Request::State,
Msg::DebugCounters => Request::DebugCounters,
Msg::Subscribe => Request::Subscribe,
Expand Down Expand Up @@ -623,6 +645,8 @@ fn print_response(response: Response) {
Response::Zoom(zoom) => println!("{zoom}"),
Response::Layout(layout) => println!("{layout}"),
Response::Cursor { x, y, output } => println!("{x} {y} {output}"),
Response::Pointer { x, y, output } => println!("{x} {y} {output}"),
Response::Axis { dx, dy } => println!("{dx} {dy}"),
Response::Focused(Some(w)) => {
println!("#{} {}", w.id, w.app_id.as_deref().unwrap_or("(no app_id)"))
}
Expand Down
109 changes: 106 additions & 3 deletions src/ipc/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ use std::path::PathBuf;
use std::rc::Rc;
use std::time::{Duration, Instant};

use smithay::backend::input::{Axis, AxisSource, ButtonState};
use smithay::input::pointer::{AxisFrame, ButtonEvent};
use smithay::reexports::calloop::generic::Generic;
use smithay::reexports::calloop::{Interest, LoopHandle, Mode, PostAction};
use smithay::reexports::wayland_server::Resource;
Expand Down Expand Up @@ -268,6 +270,8 @@ pub(crate) fn dispatch(request: Request, state: &mut DriftWm) -> Reply {
Request::Zoom(arg) => cmd_zoom(arg, state),
Request::Layout { short } => cmd_layout(short, state),
Request::Cursor => cmd_cursor(state),
Request::Pointer { x, y, button } => cmd_pointer(x, y, button, state),
Request::Axis { dx, dy } => cmd_axis(dx, dy, state),
Request::State => Ok(cmd_state(state)),
Request::DebugCounters => Ok(Response::DebugCounters(state.debug_counters())),
// Handled in serve_connection, which has the raw stream Subscribe needs.
Expand Down Expand Up @@ -398,6 +402,83 @@ fn cmd_cursor(state: &mut DriftWm) -> Reply {
})
}

fn cmd_pointer(x: f64, y: f64, button: Option<String>, state: &mut DriftWm) -> Reply {
let output = state.active_output().ok_or("no output")?;
let (camera, zoom, origin) = {
let os = crate::state::output_state(&output);
(os.camera, os.zoom, os.layout_position)
};
let screen = Point::<f64, Logical>::from((x - origin.x as f64, y - origin.y as f64));
let canvas =
driftwm::canvas::screen_to_canvas(driftwm::canvas::ScreenPos(screen), camera, zoom).0;
let serial = SERIAL_COUNTER.next_serial();
let time = std::time::SystemTime::now()
.duration_since(std::time::SystemTime::UNIX_EPOCH)
.map(|d| d.subsec_millis())
.unwrap_or(0);
state.dispatch_absolute_motion(&output, screen, canvas, serial, time);
if let Some(name) = button {
let code = match name.as_str() {
"left" => 0x110,
"right" => 0x111,
"middle" => 0x112,
_ => return Err(format!("unknown button {name}")),
};
let pointer = state.seat.get_pointer().ok_or("no pointer")?;
for button_state in [ButtonState::Pressed, ButtonState::Released] {
let serial = SERIAL_COUNTER.next_serial();
pointer.button(
state,
&ButtonEvent {
button: code,
state: button_state,
serial,
time,
},
);
}
pointer.frame(state);
}
let pointer = state.seat.get_pointer().ok_or("no pointer")?;
let location = pointer.current_location();
let screen =
driftwm::canvas::canvas_to_screen(driftwm::canvas::CanvasPos(location), camera, zoom).0;
Ok(Response::Pointer {
x: origin.x as f64 + screen.x,
y: origin.y as f64 + screen.y,
output: output.name(),
})
}

fn cmd_axis(dx: f64, dy: f64, state: &mut DriftWm) -> Reply {
let pointer = state.seat.get_pointer().ok_or("no pointer")?;
let time = std::time::SystemTime::now()
.duration_since(std::time::SystemTime::UNIX_EPOCH)
.map(|d| d.subsec_millis())
.unwrap_or(0);
let vertical = notches(dy);
let horizontal = notches(dx);
let frame = AxisFrame::new(time)
.source(AxisSource::Wheel)
.v120(Axis::Vertical, vertical)
.v120(Axis::Horizontal, horizontal)
.value(Axis::Vertical, dy)
.value(Axis::Horizontal, dx);
pointer.axis(state, frame);
pointer.frame(state);
Ok(Response::Axis { dx, dy })
}

/// Pixels a wheel notch is worth, so a caller that thinks in pixels still gets
/// discrete wheel scrolling in clients that read `v120` instead of `value`.
const AXIS_PIXELS_PER_NOTCH: f64 = 15.0;

/// `v120` for a pixel distance: 120 units per notch, so one notch in lands as
/// one notch out and a fractional pixel distance stays proportional.
fn notches(pixels: f64) -> i32 {
(pixels / AXIS_PIXELS_PER_NOTCH * 120.0) as i32
}

fn cmd_layout(short: bool, state: &mut DriftWm) -> Reply {
if !short {
return Ok(Response::Layout(state.active_layout.clone()));
Expand Down Expand Up @@ -452,9 +533,14 @@ pub(crate) fn state_info(state: &mut DriftWm) -> StateInfo {
.space
.outputs()
.map(|output| {
let (cam, z, active_bookmark) = {
let (cam, z, active_bookmark, layout_origin) = {
let os = crate::state::output_state(output);
(os.camera, os.zoom, os.active_bookmark.clone())
(
os.camera,
os.zoom,
os.active_bookmark.clone(),
[os.layout_position.x, os.layout_position.y],
)
};
let logical = crate::state::output_logical_size(output);
OutputInfo {
Expand All @@ -464,6 +550,7 @@ pub(crate) fn state_info(state: &mut DriftWm) -> StateInfo {
size: [logical.w, logical.h],
active: active.as_ref() == Some(output),
active_bookmark,
layout_origin,
}
})
.collect();
Expand Down Expand Up @@ -549,7 +636,7 @@ fn suspended_by_selector(
}

/// IPC handle + app_id for a suspended window (its stage `ElementId`), for the
/// `Focused` reply.
/// `Focused` reply. A stand-in has no client, so `pid` is always `None`.
fn suspended_focused_info(state: &DriftWm, id: SuspendedId) -> Option<protocol::FocusedWindow> {
let element = state
.stage
Expand All @@ -558,6 +645,7 @@ fn suspended_focused_info(state: &DriftWm, id: SuspendedId) -> Option<protocol::
Some(protocol::FocusedWindow {
id: state.stage.id_of(element)?.0,
app_id: element.app_id_or_class(),
pid: None,
})
}

Expand Down Expand Up @@ -613,13 +701,20 @@ fn focused_window_info(
state: &DriftWm,
window: &smithay::desktop::Window,
) -> protocol::FocusedWindow {
use smithay::reexports::wayland_server::Resource;
let pid = window.wl_surface().and_then(|s| s.client()).and_then(|c| {
c.get_credentials(&state.display_handle)
.ok()
.and_then(|cred| u32::try_from(cred.pid).ok())
});
protocol::FocusedWindow {
id: state
.stage
.id_of(window)
.expect("window from the stage has an id")
.0,
app_id: window.app_id_or_class(),
pid,
}
}

Expand Down Expand Up @@ -1465,4 +1560,12 @@ mod tests {
assert_eq!(layout_code("us,ru", 2), None);
assert_eq!(layout_code("us,", 1), None);
}

#[test]
fn a_wheel_notch_is_fifteen_pixels_of_scroll() {
assert_eq!(super::notches(15.0), 120);
assert_eq!(super::notches(-15.0), -120);
assert_eq!(super::notches(45.0), 360);
assert_eq!(super::notches(0.0), 0);
}
}
Loading