Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .planning/REQUIREMENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -244,7 +244,9 @@ Which phases cover which requirements. Updated during roadmap creation.
## GUI Chrome Engine Web Browser Canvas & 60fps Video Player Canvas (Phase 59)

- [x] **WEB-01**: Embedded Chromium engine web browser canvas via CDP Screencast frame-sink (and optional CEF off-screen renderer) rendering 60fps RGBA frames into composite pane canvases under private profile isolation (`0700`).
- Note: Phase 59 shipped a placeholder (Chrome was launched with no debugging channel and every call went to the synthetic engine). The real CDP engine (DevTools pipe, screencast frames decoded off the UI thread, per-launch `0700` profile, process-group cleanup) landed with `feat/browser-real-cdp`. No CEF renderer exists. Frame rate is whatever the screencast delivers, not a fixed 60fps.
- [x] **WEB-02**: Full interactive browser navigation (URL bar, mouse clicks/scrolls, keyboard input forwarded via CDP, history back/forward/reload, zoom, TUI sub-cell fallback).
- Note: real as of `feat/browser-real-cdp`: keys, paste, mouse, wheel, resize and zoom reach the page; history comes from `Page.getNavigationHistory`. The TUI view is a half-block preview (no kitty/sixel output yet); tabs, find-in-page, link hints and the agent surface (`browser.text|screenshot|click|type|eval`) are not built.
- [x] **VID-01**: Embedded 60fps RGBA hardware-accelerated video player canvas powered by `ffmpeg` pipe / libavcodec with monotonic audio sync.
- [x] **VID-02**: On-screen transport controls (play/pause, seek, volume, loop, speed), Borland/OIA status bar integration, TUI sub-cell fallback (Kitty/Sixel/HalfBlock), and surface parity (`UNI-R01`).

Expand Down
17 changes: 17 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Changed
- **Browser canvas is now a real browser (`WEB-01..02`).** `:browser open <url>` drives headless Chrome over the DevTools pipe (`--remote-debugging-pipe`, no TCP debugging port) instead of painting a placeholder. Frames come from the screencast and are decoded off the UI thread; the window-manager tick redraws only when a frame or the page state changed. URL, title, loading state and back/forward follow the page, including navigations the page starts itself. See [user guide chapter 14](docs/user-guide/14-browser-canvas.md).
- **Keyboard and mouse reach the page.** With the viewport focused (click the page, `Ctrl+]`, or `:browser focus on`) keys, bracketed paste, clicks, drags and the wheel are forwarded. `Ctrl+]` always returns the keyboard to bilterm. `Ctrl+L` edits the URL only while the page has focus; it is no longer taken from the shell otherwise. Mouse positions are derived from the frame and pane size, so clicks land correctly in the terminal and in `--gui`.
- **Resize and zoom work.** The page follows the pane size, and `:browser zoom` reflows the page (25% to 500%).
- **Honest status.** The status line and `browser.state` report `cdp` only after the DevTools handshake, `cdp (starting)` before it, `cdp (crashed)` with "press r to reload" after a crash, and `synthetic (Chrome not found)` or `synthetic (launch failed: …)` when the placeholder engine is used. The lock icon is shown only for `https://`. `browser.open` returns `ok: false` with an `error` when the URL is refused.
- Nav-bar buttons are hit-tested by display column, so every click lands on the button drawn there.

### Security
- Browser URLs are validated before they reach Chrome: only `http://`, `https://` and `about:blank`; `file://` needs `BILTERM_BROWSER_ALLOW_FILE=1`; `javascript:`, `data:`, `chrome:` and anything starting with `-` are refused. The URL is never passed on Chrome's command line.
- Each browser gets its own profile directory (pid, pane and random suffix) inside the private runtime dir (`0700`); a failure to create it aborts the launch. Closing the canvas kills Chrome's whole process group and removes the profile. Downloads are denied and the sandbox stays on.

### Dependencies
- Added `serde_json` (DevTools messages) and `zune-jpeg` (screencast frame decoding, pure Rust).

### Requirements
- A Chromium-family browser on Linux or macOS for the real engine (`BILTERM_CHROME_PATH` overrides discovery). Without one the placeholder engine is used and the status line says so.

### Fixed
- **Remote tmux link (`:ssh-tmux`) rendering**: the remote tmux client is now as large as the linked pane instead of the whole Bilterm window (full-screen programs and wrapped lines drew in the wrong place); a hydrate restores alternate screen, cursor position and visibility, application cursor keys and mouse modes (re-attaching while `vim` ran left the cursor in the wrong place and stale rows on the primary screen); a UTF-8 character split across two `%output` lines is no longer turned into replacement characters; `%extended-output` is understood.
- **Remote tmux link flooding**: Bilterm asks tmux for `pause-after` flow control and re-captures a paused pane, so a flooding remote pane no longer starves input (Ctrl-C typed in Bilterm now arrives).
Expand Down
36 changes: 36 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,8 @@ serde = { version = "1.0.219", features = ["derive"] }
toml = "0.8.20"
toml_edit = "0.22"
nucleo-matcher = "0.3.1"
serde_json = "1.0.154"
zune-jpeg = { version = "0.5.15", default-features = false, features = ["std", "x86", "neon"] }

[dev-dependencies]
insta = "1.42.2"
Expand Down
115 changes: 115 additions & 0 deletions docs/user-guide/14-browser-canvas.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
# Chapter 14: The Browser Canvas

`:browser open <url>` shows a real web page inside a pane. Bilterm starts a headless Chrome for
that pane, streams its frames into the pane and forwards your keys and mouse to the page.

## Requirements

- Linux or macOS, and a Chromium-family browser: Chrome, Chromium, Brave or Edge. Bilterm looks
for `chromium`, `chromium-browser`, `google-chrome-stable`, `google-chrome`, `brave-browser`
and `microsoft-edge-stable` on `PATH` (plus the standard application paths on macOS). Set
`BILTERM_CHROME_PATH=/path/to/chrome` to pick one.
- No browser found, or the launch fails: the pane shows a painted placeholder (the *synthetic*
engine) and the status line says why. On Windows the synthetic engine is always used.

## Opening and closing

| Action | Command | Key | Mouse |
|---|---|---|---|
| Open a URL | `:browser open <url>` or `:browser <url>` | | |
| Toggle the canvas | `:browser` | `Ctrl+Alt+B`, `<Leader> .` | |
| Close | `:browser close` | | `[✕]` on the nav bar |
| Back / forward | `:browser back`, `:browser forward` | | `[◄]` `[►]` |
| Reload / stop | `:browser reload`, `:browser stop` | | `[↻]` (shows `[■]` while loading) |
| Edit the URL | | `Ctrl+L` while the page has focus | click the URL |
| Zoom | `:browser zoom 25`, `:browser zoom -25` | | |
| Give the keyboard to the page | `:browser focus on` | `Ctrl+]` | click the page, or `[FOCUS:…]` |
| Take the keyboard back | `:browser focus off` | `Ctrl+]` | click the prompt, or `[FOCUS:…]` |

`:bcanvas` is an alias of `:browser`. The same operations are available over RPC and MCP as
`browser.open`, `browser.control` and `browser.state` (`bilterm_browser`).

## What the status line tells you

The status line and `browser.state` name the engine that is really running:

| Label | Meaning |
|---|---|
| `cdp (starting)` | Chrome was launched; the DevTools handshake has not finished. |
| `cdp` | Attached to the page. This is real Chrome. |
| `cdp (crashed)` | The page or Chrome died. The pane shows `… — press r to reload`. |
| `synthetic` | Placeholder engine, by request (`demo://`, `test://`, `about:` URLs or the environment switches below). |
| `synthetic (Chrome not found)` | No browser binary was found. |
| `synthetic (launch failed: …)` | Chrome was found but did not come up; the reason follows. A launch that gives no handshake within 5 seconds is reported this way. |

The nav bar shows a lock only for `https://` pages, `◌` while the page is loading, the page
title after the URL, and `⚠` with the reason when a URL was refused or a navigation failed
(for example `net::ERR_NAME_NOT_RESOLVED`).

## Keyboard

Keys go to the floating prompt until the page has focus. Click the page, press `Ctrl+]` or run
`:browser focus on` to give it the keyboard. While it has focus:

- Every key goes to the page, including `Tab`, `Esc`, arrows, `Enter`, function keys and
`Ctrl`/`Alt` chords. Pasted text is inserted at the page's caret.
- `Ctrl+]` returns the keyboard to bilterm. This always works. (Terminals without the kitty
keyboard protocol send it as `Ctrl+5`; both are accepted.)
- `Ctrl+L` opens the URL editor: type, `Enter` to go, `Esc` to cancel, `Ctrl+U` to clear. A bare
host such as `example.com` gets `https://`.
- The leader key (`Ctrl+B` / `Ctrl+X`), your own key bindings, `Ctrl+Q` and `Ctrl+Alt+A` still
belong to bilterm. So `Ctrl+X` (cut) and `Ctrl+B` are not sent to the page.
- After a crash, `r` relaunches the browser on the last URL (`:browser reload` does the same).

Without focus the page takes no keys. In particular `Ctrl+L` reaches your shell as usual.

## Mouse

Clicks, drags, movement and the wheel on the page are forwarded to it. A click lands where you
point in the terminal and in `--gui`: the position is taken from the size of the frame and of
the pane, not from an assumed cell size. Double clicks are recognised. Clicking the floating
prompt gives the keyboard back to it.

Links that ask for a new window open in the same pane (one page per pane).

## Size and zoom

The page is laid out for the pane: 10 by 20 CSS pixels per cell, between 320x200 and 3840x2160,
and it follows the pane when you resize or split it. `:browser zoom <delta>` changes the zoom in
percent (25% to 500%): the page reflows larger or smaller, as with `Ctrl +` in a desktop browser.

In the terminal the page is drawn with half-blocks, two pixels per cell row, so text on most
pages is not readable there. It is a preview; `bilterm --gui` draws the frame at full size.

## What is allowed

- URLs: `http://`, `https://` and `about:blank`. `javascript:`, `data:`, `chrome:`,
`view-source:` and anything starting with `-` are refused before they reach Chrome.
- `file://` is off. Start bilterm with `BILTERM_BROWSER_ALLOW_FILE=1` to allow it.
- Downloads are denied.
- Chrome's sandbox stays on. Bilterm never passes `--no-sandbox`; if your system cannot run the
sandbox (some containers, running as root), Chrome fails to start and you get
`synthetic (launch failed: …)`.
- Each launch gets its own empty profile in `$XDG_RUNTIME_DIR/bilterm/browser/` (mode `0700`).
Nothing is shared with your normal browser profile: no cookies, logins or extensions. The
profile is deleted when the canvas closes, and leftovers from a killed bilterm are removed the
next time a browser is opened.
- Chrome talks to bilterm over a private pipe. No debugging port is opened.

## Environment switches

| Variable | Effect |
|---|---|
| `BILTERM_CHROME_PATH` | Browser binary to launch. |
| `BILTERM_BROWSER_ALLOW_FILE=1` | Allow `file://` URLs. |
| `BILTERM_BROWSER_SYNTHETIC=1` | Never launch a browser; always use the placeholder engine. |
| `BILTERM_BROWSER_CDP=1` | For tests only: let a test binary launch a real browser. Tests use the placeholder engine otherwise. `BILTERM_BROWSER_CDP=1 cargo test --test browser_cdp_live` runs the live test. |

## Known limits

- One page per pane: no tabs, no find-in-page, no link hints.
- Zooming in lowers the frame resolution (Chrome captures frames at CSS-pixel size).
- No kitty or sixel image output yet, so the terminal view stays a half-block preview.
- Agents cannot yet read page text, take screenshots or click by selector over RPC/MCP.
- The clipboard is the page's own: `Ctrl+C` in the page does not reach the system clipboard.
- JavaScript `alert` dialogs are accepted automatically; `confirm` and `prompt` are dismissed.
2 changes: 2 additions & 0 deletions docs/user-guide/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,8 @@ Welcome to the official User Guide for **Bilterm (`v3.2.0`)** — the Rust + Rat
- The Find window (`Ctrl+Alt+F`, `:find agent|session|history`, View menu): fuzzy ranked search of agents, live and saved sessions, and command history; every row shows its source (`[live]`, `[saved:<name>]`, `[log:<name>]`); secrets are masked and counted; `:finder` verbs; `find.*` RPC and `bilterm_find` MCP; SSH and tmux coverage and known gaps G1 to G10.
13. **[Chapter 13: The Traffic Monitor](13-traffic-monitor.md)**
- The TRAFFIC MONITOR window (`Ctrl+Alt+T`, `:traffic`, `:netmon`): bytes per connection, session, window, pane, origin, locality and direction, sparklines, pause, drill-down, `traffic.*` RPC and `bilterm_traffic` MCP.
14. **[Chapter 14: The Browser Canvas](14-browser-canvas.md)**
- A real web page in a pane (`:browser open <url>`, `Ctrl+Alt+B`): headless Chrome over a private DevTools pipe, keyboard focus and the `Ctrl+]` escape hatch, mouse, resize and zoom, what the status line says about the running engine, allowed URLs, profile isolation, environment switches and known limits.

---

Expand Down
10 changes: 9 additions & 1 deletion src/api/rpc_media.rs
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,16 @@ impl AutomationControlPlane {
};
crate::colon_command::execute_media::execute_browser_command(&mut self.wm, op);

// A URL refused by the policy opens nothing (or leaves a notice on an open canvas).
let refused = match self.wm.pane(target_id).and_then(|p| p.browser_canvas()) {
Some(b) => b.notice().map(str::to_string),
None => Some("browser canvas did not open (URL refused)".to_string()),
};
let mut res = BTreeMap::new();
res.insert("ok".to_string(), JsonValue::Bool(true));
res.insert("ok".to_string(), JsonValue::Bool(refused.is_none()));
if let Some(why) = refused {
res.insert("error".to_string(), JsonValue::String(why));
}
res.insert("url".to_string(), JsonValue::String(url.to_string()));
res.insert("pane_id".to_string(), JsonValue::Number(target_id as i64));
Ok(JsonValue::Object(res))
Expand Down
Loading
Loading