HA autofill, adaptive frames, reconnect, force refresh [server] + LVGL widget, rotation fix, callbacks

[micoli]

First of all, thank you for creating and maintaining RemoteWebView project. IMO, this is a game changer in writing standalone remote UIs.

This PRs cover improvements developed during daily use with a Waveshare ESP32-S3 Touch LCD 4" (480×480 MIPI RGB, GT911 touch) displaying a Home Assistant dashboard.
The changes span both repositories; I'm opening this issue here and will reference the companion client PR below.

Branches:

• server branch: https://github.com/micoli/RemoteWebViewServer/tree/autofill-autoreconnect
• companion client branch: https://github.com/micoli/RemoteWebViewClient/tree/improve-rotation-reconnect

---

Server-side changes (autofill-autoreconnect branch)

1 — Home Assistant credential autofill

Problem. After a server restart or session expiry, the Home Assistant login page appears on the panel. With a touch-only display in a fixed location, typing credentials is impractical.

Fix. A new haAutofill.ts plugin detects when the browser navigates to an auth/authorize URL and injects a CDP Runtime.evaluate script that:

• Traverses the shadow DOM (HA uses Web Components) to locate input[name="username"] and input[name="password"]
• Fills the fields and dispatches input/change events so the Lit-based form registers the values
• Clicks the submit button automatically

Credentials are read from environment variables; the feature is opt-in:

HA_CREDENTIAL_AUTOFILL=true
HA_USERNAME=myuser
HA_PASSWORD=mypassword

The hassio add-on config.yaml exposes these as user-configurable options.

---

2 — Adaptive frame sending: transmit only when something changed

Problem. The server forwarded every Chrome screencast frame regardless of whether the page had actually changed. On a static dashboard this wasted bandwidth and drove unnecessary JPEG decode cycles on the ESP32.

Fix. adaptiveMinFrameInterval is tracked per device and updated from the FrameStats packets already sent back by the client. The broadcaster now also drops stale queued frames when a newer one arrives, so the client always gets the freshest tile set rather than working through a backlog.

---

3 — DeviceList and KillDevice protocol verbs

Problem. The web debug client had no way to know which ESP32 devices were connected, nor to force-disconnect a device with stale state.

Fix. Two new message types:

ID	Type	Direction	Description
7	DeviceList	server → browser	JSON list of device IDs, URLs, last-active timestamps
8	KillDevice	browser → server	Force-disconnect a device by ID

getDeviceSummaries() and broadcastDeviceList() are exported from deviceManager.ts. The broadcaster now tracks browser clients (web debug UI) separately from device clients (ESP32), ensuring DeviceList packets are never forwarded to embedded hardware.

---

4 — Reliable auto-reconnection

Problem. When an ESP32 reconnected after a network interruption, the screen stayed blank until Chrome emitted its next natural screencast frame — which could be many seconds on a static page.

Fix. On reconnect, ensureDeviceAsync() now:

1. Issues Page.captureScreenshot immediately and sends it as a full frame
2. Re-sends the current URL to the client's URL sensor
3. Restarts Page.startScreencast for ongoing live updates

Stale device sockets are evicted on reconnect while browser observer connections survive. Broken CDP sessions are detected and recreated gracefully.

---

5 — Force page refresh option

Problem. open_url() with the current URL was a no-op on the server — no navigation occurred, so a hung page could not be recovered remotely.

Fix. A new kFlagOpenURLForce flag in the OpenURL protocol packet instructs the server to reload the page even when the URL is unchanged. Exposed in the ESPHome component as open_url(url, force=true).

---

Notes

• The test environment in commit dcc1089 is marked for removal before merge — it was used for local integration testing only.

---

Client-side changes (improve-rotation-reconnect branch — micoli/RemoteWebViewClient)

1 — Fix rotation: physical vs logical display dimensions

Problem. draw_pixels_at() writes to the physical frame buffer, bypassing ESPHome's software rotation transform. The component was passing display_->get_width() / get_height() — the logical dimensions — to the server. At 90° or 270°, width and height are swapped, so the server encoded tiles in the wrong coordinate space, producing a sheared or clipped image.

Fix. setup() reads display_->get_rotation(), swaps dimensions for 90°/270°, and sends the physical dimensions to the server. validate_rotation() is also inlined in __init__.py because esphome.components.display.validate_rotation is not part of the stable public API and caused ImportError on some ESPHome versions.

---

2 — Dual-mode: standalone display or LVGL canvas widget

Problem. When LVGL is active it owns the display driver permanently. The previous direct draw_pixels_at() calls conflicted with LVGL's rendering loop, producing tearing or corruption.

Fix. The component now works in two modes selected at YAML parse time:

• Standalone mode (existing behaviour): display_id provided, tiles written directly to the display.
• LVGL widget mode: a remote_webview: widget inside an lvgl: page. ESPHome creates lv_canvas_t; set_obj() allocates a PSRAM RGB565 buffer, copies tile data into it, and calls lv_obj_invalidate() to let LVGL handle the repaint.

display_id and touchscreen_id are now Optional in the schema.

remote_webview:         # top-level component
  id: rwv
  server: host:port
  url: http://...

lvgl:
  pages:
    - id: webview_page
      widgets:
        - remote_webview:
            remote_webview_id: rwv

This will allow, for example, to display the remote_webview under a toolbar with reload and link buttons to other pages:

    - id: webview_page
      bg_color: 0x000000
      bg_opa: COVER
      pad_all: 0
      widgets:
        - obj:
            align: TOP_LEFT
            width: 480
            height: 56
            pad_all: 0
            border_width: 0
            radius: 0
            bg_color: 0x0d1117
            bg_opa: COVER
            widgets:
              - button:
                  id: btn_reload
                  align: TOP_LEFT
                  x: 0
                  y: 0
                  width: 100
                  height: 56
                  pad_all: 0
                  radius: 0
                  bg_color: 0x000000
                  bg_opa: "80%"
                  widgets:
                    - label:
                        text: ""
                        text_font: montserrat_14
                        align: CENTER
                  on_click:
                    then:
                      - lambda: |-
                          id(rwv).open_url(id(current_url_sensor_id).state, true);
              - button:
                  id: btn_home
                  align: TOP_LEFT
                  x: 100
                  y: 0
                  width: 100
                  height: 56
                  pad_all: 0
                  radius: 0
                  bg_color: 0x000000
                  bg_opa: "80%"
                  widgets:
                    - label:
                        text: ""
                        text_font: montserrat_14
                        align: CENTER
                  on_click:
                    then:
                      - lambda: |-
                          id(rwv).open_url("${url1}", true);
              - button:
                  id: btn_home2
                  align: TOP_LEFT
                  x: 200
                  y: 0
                  width: 100
                  height: 56
                  pad_all: 0
                  radius: 0
                  bg_color: 0x000000
                  bg_opa: "80%"
                  widgets:
                    - label:
                        text: "HA"
                        text_font: montserrat_14
                        align: CENTER
                  on_click:
                    then:
                      - lambda: |-
                          id(rwv).open_url("${url2}", true);
        - remote_webview:
            remote_webview_id: rwv
            align: TOP_LEFT
            y: 56
            width: 480
            height: 424

---

3 — on_connect / on_disconnect automation triggers

Problem. There was no ESPHome automation hook for WebSocket connection state — useful for a splash screen, status LED, or uptime logging.

Fix. Two new triggers, symmetric with on_frame_update:

remote_webview:
  on_connect:
    - logger.log: "server connected"
  on_disconnect:
    - logger.log: "server disconnected"

connect_pending_ and disconnect_pending_ are std::atomic<bool> flags set in the WebSocket event callback (ISR context) and consumed in loop(), keeping automations safely on the ESPHome main task.

---

4 — Web debug client improvements: FrameStats and new server verbs

The debug web client (web_client/) is updated to:

• Parse FrameStats packets and display the current adaptive MFI in the status bar
• Handle the new DeviceList / KillDevice verbs from the server (list connected devices, force-disconnect)
• The UI has been redesigned around three new areas:

  1. Statusbar replaces the old single-line metrics div. It displays:

     • A coloured connection-state dot (green = connected, yellow = connecting, red = error, grey = idle)
     • Active device ID
     • Zoom selector (inline, inside the statusbar)
     • Frame counter
     • MFI (adaptive min frame interval in ms) — updated live from FrameStats packets, giving direct visibility into the bandwidth/latency trade-off the server is adapting to
     • Last error message (truncated with ellipsis)

  2. Device list panel (sidebar) populated from DeviceList packets:

     • Each connected device shown as a card with its ID, current URL, and last-active timestamp
     • Active device highlighted with an accent border
     • Per-device "Kill" button sends KillDevice to force-disconnect from the server
     • Clicking a card switches the active stream to that device

  3. Message log (beside the canvas) with per-type filter toggle buttons: shows all incoming binary packets in real time, filterable by MsgType (Frame, Touch, FrameStats, CurrentURL, DeviceList, …).

  4. Input history: server address and URL fields persist the last 10 entries in localStorage and expose them as <datalist> autocomplete options.

  5. Persistent browser ID: a stable random identifier stored in localStorage (rwv-browser-id) is reused across page reloads, so the server logs show a consistent identity for the debug observer.

---

5 — Upgrade to ESPHome 2026.5.3

Target updated to the current stable release.