Use the phone as a trackpad for an external display

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "localdesktop"
-version = "1.0.4"
+version = "1.1.0"
 edition = "2021"
 build = "build.rs"
 default-run = "build_apk"

README.md

@@ -12,6 +12,12 @@ Wayland compositor in the Android NDK plus proot) and replaces what runs on it.
 ideally with a keyboard and mouse. It brings a desktop to Android; it does not turn the desktop
 into a phone UI.
 
+On a phone, connect a monitor (USB-C to HDMI, a dock, or a wireless display): the desktop moves
+to it and the phone becomes a trackpad, with buttons to turn the screen and to type with the
+Android keyboard.
+
+<img src="docs/img/external-display-monitor.jpg" width="560" alt="elementary OS on a 4K monitor"> <img src="docs/img/external-display-trackpad.png" width="120" alt="The phone as its trackpad">
+
 ## How it works
 
 1. The APK carries an elementary OS 8 root filesystem as a ~190 MB xz archive
@@ -24,7 +30,13 @@ into a phone UI.
    Classic session inside it: Wingpanel, the Dock, notifications and the apps.
 
 Rendering is done on the CPU (Mesa llvmpipe): Android's GPU drivers cannot be used from a glibc
-userland.
+userland. llvmpipe compiles shaders for the CPU it reads from `/proc/cpuinfo`; on phones whose
+firmware turns off SVE (Snapdragon 8 Gen 1 to 3, for example) the app hides the CPU model from the
+guest, or Gala would crash on the first shader that uses it.
+
+The desktop's scale follows Android's display density, lowered when the desktop would otherwise be
+smaller than 540×900 logical pixels (but never below half the size of Android's own UI): a 520 dpi
+foldable gets 2x on both its screens instead of 3x.
 
 ### Updates
 
@@ -61,7 +73,7 @@ Crash reporting is off. Set `SENTRY_DSN` when building to send crashes to your o
 Android 12 and later kill an app's child processes once it has more than 32 of them (the "phantom
 process killer"); everything in the desktop then disappears at once. The session is trimmed to
 about 23 processes at idle, which leaves room for a couple of apps (Terminal and Web together use
-6). To open more at the same time, lift the limit once:
+7). To open more at the same time, lift the limit once:
 
 - Android 14 and later: Developer options → **Disable child process restrictions**.
 - Otherwise, from a computer with adb:
@@ -81,8 +93,8 @@ about 23 processes at idle, which leaves room for a couple of apps (Terminal and
   managed by Android. Screen locking is too.
 - Translations are stripped to keep the APK small, so the desktop is in English.
 - Shut down and restart in the session menu (there is no logind). Log out starts a fresh session.
-- Known rough edges: the panel and dock sometimes appear only after the first tap, and sound is
-  not working reliably yet.
+- Known rough edges: sound is not working reliably yet, and on narrow screens (a foldable's cover
+  screen) some windows open wider than the screen.
 
 ## Building
 

docs/architecture.md

@@ -57,7 +57,9 @@ The stages, in order:
    `/etc/localdesktop`, so they survive OS updates) and seed the user's home directory.
 4. `setup_machine_id` — give D-Bus a machine ID.
 5. `setup_network_and_time` — write Android's DNS servers and timezone into the guest.
-6. `fix_xkb_symlink` — repoint an absolute xkb symlink to a relative one (it's resolved from NDK, not the chroot root).
+6. `setup_cpuinfo` — give the guest a `/proc/cpuinfo` without the CPU model when the kernel has SVE
+   turned off, so llvmpipe's shaders do not use it (`core::cpuinfo`).
+7. `fix_xkb_symlink` — repoint an absolute xkb symlink to a relative one (it's resolved from NDK, not the chroot root).
 
 The first stage is the heaviest: the archive is ~1 GB unpacked. Whether it needs to run is
 decided by a stamp file holding the archive's SHA-256 from its manifest.

patches/winit/src/platform/android.rs

@@ -186,3 +186,9 @@ pub mod activity {
     #[doc(hidden)]
     pub struct AndroidApp;
 }
+
+/// The physical key winit reports for an Android `KeyEvent` key code.
+#[cfg(android_platform)]
+pub fn physical_key(keycode: u32) -> crate::keyboard::PhysicalKey {
+    crate::platform_impl::keycodes::to_physical_key(keycode.into())
+}

patches/winit/src/platform_impl/android/mod.rs

@@ -27,7 +27,7 @@ use crate::window::{
     WindowButtons, WindowLevel,
 };
 
-mod keycodes;
+pub(crate) mod keycodes;
 
 pub(crate) use crate::cursor::{
     NoCustomCursor as PlatformCustomCursor, NoCustomCursor as PlatformCustomCursorSource,

patches/xbuild/xbuild/src/gradle/mod.rs

@@ -252,7 +252,19 @@ pub fn build(env: &BuildEnv, libraries: Vec<(Target, PathBuf)>, out: &Path) -> R
         scaler.optimize();
         let anydpi = res.join("mipmap-anydpi-v26");
         std::fs::create_dir_all(&anydpi)?;
-        std::fs::write(anydpi.join("ic_launcher.xml"), IC_LAUNCHER)?;
+        // A monochrome layer is the icon's silhouette, which for an opaque icon is a square:
+        // themed icons then show the icon in its own colors instead.
+        let variants: &[&str] = if scaler.is_opaque() {
+            let xml = String::from_utf8_lossy(IC_LAUNCHER).replace(
+                "  <monochrome android:drawable=\"@mipmap/ic_launcher_monochrome\" />\n",
+                "",
+            );
+            std::fs::write(anydpi.join("ic_launcher.xml"), xml)?;
+            &["foreground"]
+        } else {
+            std::fs::write(anydpi.join("ic_launcher.xml"), IC_LAUNCHER)?;
+            &["foreground", "monochrome"]
+        };
         let dpis = [
             ("m", 48),
             ("h", 72),
@@ -270,10 +282,11 @@ pub fn build(env: &BuildEnv, libraries: Vec<(Target, PathBuf)>, out: &Path) -> R
                 let mut icon = std::fs::File::create(dir.join("ic_launcher.png"))?;
                 scaler.write(&mut icon, opts)?;
             }
-            for variant in ["foreground", "monochrome"] {
+            // Adaptive icon layers are 108 dp where the legacy icon is 48 dp.
+            for variant in variants {
                 let mut icon =
                     std::fs::File::create(dir.join(format!("ic_launcher_{}.png", variant)))?;
-                scaler.write(&mut icon, opts)?;
+                scaler.write_adaptive_layer(&mut icon, size * 108 / 48)?;
             }
         }
         manifest.application.icon = Some("@mipmap/ic_launcher".into());

patches/xbuild/xcommon/src/lib.rs

@@ -75,6 +75,27 @@ impl Scaler {
         Ok(())
     }
 
+    /// Whether the icon has no transparent pixels (after `optimize`).
+    pub fn is_opaque(&self) -> bool {
+        !self.img.color().has_alpha()
+    }
+
+    /// An adaptive icon layer, `size` px for its 108 dp, with the whole icon in the 72 dp that
+    /// launchers show. The icon's edge pixels continue to the layer's edges, so a full-bleed
+    /// icon shows no other color around it and a transparent one stays transparent.
+    pub fn write_adaptive_layer<W: Write + Seek>(&self, w: &mut W, size: u32) -> Result<()> {
+        let inner = size * 2 / 3;
+        let offset = (size - inner) / 2;
+        let resized = self
+            .img
+            .resize_exact(inner, inner, FilterType::Lanczos3)
+            .to_rgba8();
+        let edge = |v: u32| v.saturating_sub(offset).min(inner - 1);
+        let layer = RgbaImage::from_fn(size, size, |x, y| *resized.get_pixel(edge(x), edge(y)));
+        layer.write_to(w, ImageOutputFormat::Png)?;
+        Ok(())
+    }
+
     pub fn to_vec(&self, opts: ScalerOpts) -> Vec<u8> {
         let mut buf = vec![];
         let mut cursor = Cursor::new(&mut buf);

rootfs/overlay/etc/elementary-mobile/xdg/autostart/pantheon-gnome-keyring-secrets.desktop

@@ -0,0 +1,4 @@
+[Desktop Entry]
+Type=Application
+Name=pantheon-gnome-keyring-secrets
+Hidden=true

rootfs/overlay/usr/local/bin/elementary-mobile-session

@@ -84,20 +84,39 @@ export LIBGL_ALWAYS_SOFTWARE=1 GSK_RENDERER=cairo
 export WEBKIT_DISABLE_SANDBOX_THIS_IS_DANGEROUS=1 ELECTRON_DISABLE_SANDBOX=1
 export WEBKIT_DISABLE_DMABUF_RENDERER=1 WEBKIT_DISABLE_COMPOSITING_MODE=1
 # Android kills apps with more than 32 child processes, so the session avoids helper processes:
-# a forked bus instead of dbus-run-session, and Gala started directly rather than supervised.
+# a forked bus instead of dbus-run-session, and one shell that keeps Gala running.
 bus_pid_file="$XDG_RUNTIME_DIR/session-bus.pid"
 
+# What Gala starts and leaves behind when it dies. A new Gala starts them again, and Wingpanel and
+# the Dock refuse to run twice, so it would keep restarting them every second.
+gala_helpers='(/usr/libexec/)?(io\.elementary\.(wingpanel|dock|notifications)|gala-daemon-gtk3|mutter-x11-frames)'
+
+# Gala runs outside gnome-session: it reports readiness to systemd only, so as a session component
+# it would hit gnome-session's registration timeout. Its screen locker also crashes without
+# XDG_SESSION_ID when logind is missing. Without a window manager the screen goes black and stops
+# responding, so it is started again if it dies.
+run_gala() {
+    trap 'kill "$wm" 2>/dev/null; wait "$wm"; exit' TERM
+    while :; do
+        started=$(date +%s)
+        XDG_SESSION_ID=elementary-mobile gala &
+        wm=$!
+        wait "$wm"
+        log "Gala exited with status $?, restarting it"
+        pkill -x -f "$gala_helpers"
+        # Do not spin if it cannot start at all.
+        [ $(($(date +%s) - started)) -ge 10 ] || sleep 10
+    done
+}
+
 # Start a fresh session after "Log Out", like a display manager with automatic login.
 while kill -0 "$xwayland" 2>/dev/null; do
     DBUS_SESSION_BUS_ADDRESS=$(dbus-daemon --session --fork --print-address=1 --print-pid=3 3>"$bus_pid_file")
     export DBUS_SESSION_BUS_ADDRESS
 
     gsettings set org.gnome.desktop.interface scaling-factor "$scale"
 
-    # Gala runs outside gnome-session: it reports readiness to systemd only, so as a session
-    # component it would hit gnome-session's registration timeout. Its screen locker also crashes
-    # without XDG_SESSION_ID when logind is missing.
-    XDG_SESSION_ID=elementary-mobile gala &
+    run_gala &
     gala=$!
 
     gnome-session --session=elementary-mobile

rootfs/overlay/usr/local/share/dbus-1/services/org.gnome.zeitgeist.Engine.service

@@ -0,0 +1,5 @@
+# Shadows zeitgeist's own service file (the session bus prefers /usr/local/share), which starts the
+# daemon from a shell that waits for it: one more process towards Android's limit of 32.
+[D-BUS Service]
+Name=org.gnome.zeitgeist.Engine
+Exec=/usr/bin/zeitgeist-daemon

src/android/accessibility.rs

@@ -17,6 +17,10 @@ pub enum AppUserEvent {
     AccessibilityInputReady,
     /// The first-run install finished; switch from the progress page to the desktop.
     SetupFinished,
+    /// An external monitor's surface appeared, changed or went away.
+    ExternalDisplayChanged,
+    /// A mouse (or touch) on the external monitor moved, clicked or scrolled.
+    ExternalPointer,
 }
 
 #[derive(Clone, Copy, Debug)]
@@ -147,21 +151,23 @@ fn enqueue_key_event(action: jint, key_code: jint, scan_code: jint, event_time_m
         _ => return false,
     };
 
-    if scan_code <= 0 {
+    if scan_code <= 0 || !bridge().lock().unwrap().runtime_active {
         return false;
     }
 
+    inject_key(scan_code as u32, state, event_time_ms.max(0) as u64)
+}
+
+/// Type a key into the desktop.
+pub fn inject_key(scancode: u32, state: ElementState, event_time_ms: u64) -> bool {
     let mut bridge = bridge()
         .lock()
         .expect("Failed to lock accessibility bridge");
-    if !bridge.runtime_active {
-        return false;
-    }
 
     bridge.pending_events.push_back(AccessibilityKeyEvent {
-        scancode: scan_code as u32,
+        scancode,
         state,
-        event_time_ms: event_time_ms.max(0) as u64,
+        event_time_ms,
     });
 
     let proxy = bridge.proxy.clone();