Skip to content

fix(network): serialize developer SSH reconciliation - #642

Merged
boburning merged 2 commits into
mainfrom
fix/network-ssh-reconcile-race
Aug 31, 2026
Merged

boburning merged 2 commits into
mainfrom
fix/network-ssh-reconcile-race

Conversation

@boburning

@boburning boburning commented Aug 31, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • makes bloom-network-services the sole SSH/SFTP lifecycle owner during boot reconciliation
  • removes the compatibility path's duplicate background bloom-dev-ssh start
  • serializes primary developer-SSH start and stop operations with a bounded, stale-owner-recoverable lock
  • adds a concurrent-reconcile regression test with exclusive listener ownership
  • lets the public rollback operation explicitly reject a booted unconfirmed candidate and enter signed recovery without repeated failed boots
  • preserves the retained known-good payload and prevents recursive rollback rejection
  • refreshes the canonical legacy manifest and dependency inventory for the exact shipping bytes

This is a corrective NET-10/NET-11 and integrated F1 checkpoint slice. It is independent and targets current upstream main; it is not stacked on another open implementation PR.

Fixes #641.

Root cause

bloom-network invoked the compatibility services path and then canonical bloom-network-services reconciliation. The compatibility path started bloom-dev-ssh in the background while the canonical path started it synchronously. Concurrent helpers could race stop_dropbear and prepare_runtime_key. On a read-only firmware home using Bloom's bind-mount fallback, the losing helper could unmount /home after the surviving Dropbear listener was already live, leaving TCP/22 reachable but removing its authorized_keys path.

The recovery CLI also required a candidate to consume the full failed-boot budget before rollback could be published. The bounded automatic path remains, but an authenticated operator can now reject phase testing immediately. The state transition is fixed, records recovery_reason operator_requested, retains the signed known-good authority, and refuses rollback recursion.

Validation

  • git diff --check
  • POSIX shell syntax validation for all changed runtime scripts
  • canonical legacy manifest: 326 components, 1,967 files, 260,662,316 bytes
  • provenance policy: 7 components
  • transfer policy: 6 roots, 5 read-write, 1 read-only
  • developer SSH tests: 13/13
  • focused network, transaction, Wi-Fi, listener, manifest, and dependency tests: 134/134
  • focused SSH/update-state/rollback/CLI tests on the final two-commit stack: 60/60
  • exhaustive shell integration suite on the SSH correction: 819/819 across 8 shards; slowest shard 81 seconds
  • pull-request shell CI on the SSH correction: green in 2m22s

The latest commit will rerun PR shell CI. A fresh signed full ARM hardware artifact is being built from the final head after the obsolete in-flight build was cancelled.

Device evidence

Observed on Miyoo Mini Plus model 354 during the first controlled boot of signed candidate 0.1.0-dev-55e0664b built from main commit 55e0664:

  • Wi-Fi associated and TCP/22 presented the pinned Dropbear host key.
  • Only public-key authentication was allowed.
  • The retained Ed25519 automation key was offered but rejected before signature.
  • Bloom Remote Keys reported the same durable key fingerprint ending b9002791.
  • The candidate's bloom-dev-ssh and bloom-dropbearmulti were byte-identical to the previously working 2933f70 release.
  • No second reboot, Wi-Fi/profile mutation, key mutation, host-key bypass, password authentication, or alternate identity was attempted.
  • The candidate is unconfirmed; retained known-good release 0.1.0-dev-2933f706 and the pre-update snapshot remain intact.

Recovery and remaining device gate

The failed card will be inspected offline before mutation. Recovery will use the exact fixed scripts only to regain strict pinned-key access, explicitly reject the unconfirmed candidate, publish the retained signed known-good rollback once, and then install the fresh signed PR artifact. No candidate will be confirmed with modified or unsigned runtime bytes.

Before merge/confirmation, the final signed Mini Plus candidate must demonstrate strict pinned-key SSH continuity, listener audit, installed version and structured health, stable Wi-Fi, exact update state, and no repeated reboot/rollback loop.

Limitations

Remote evidence does not certify physical controls, LCD appearance, audio, gameplay, save semantics, Mini V1-V4, or Mini Flip. No Full update will be confirmed while required checks or the device gate are failing.

@boburning

Copy link
Copy Markdown
Owner Author

Device gate update (2026-08-31): the failed v0.1.0-dev-55e0664b candidate remains reachable at 192.168.1.180:22, and the live ED25519 host key still exactly matches the retained Mini Plus pin (SHA256:X6iLeRVLqybI5kOPe/R4RZ/bkww0iJ15JIltxvbmIEE). Strict key-only authentication still rejects the sole automation key even though its canonical fingerprint matches the durable Bloom Remote Keys entry. This confirms the runtime authorized-key projection failure remains active; no password, host-key bypass, alternate identity, Wi-Fi mutation, or repeat reboot was attempted.

The final PR head is 0b3b5d96609723724c3d5449e4f65628aa8d3246; shell CI is green and the signed hardware artifact from run 33378466169 validates locally. An exact-path offline recovery procedure has been dry-run against a disposable D:-based fixture in both audit and repair modes. It preserves hashes/evidence before replacing only the four corrective runtime scripts. The PR remains intentionally unmerged pending recovery, strict-SSH continuity, rollback to exact signed known-good bytes, and a successful signed final-candidate device checkpoint.

@boburning

Copy link
Copy Markdown
Owner Author

Offline recovery checkpoint (2026-08-31): after the user physically powered down the Plus, the FAT32 card mounted as BLOOMOS with the dirty flag set. A read-only chkdsk found no file/folder corruption; chkdsk /f cleared the dirty flag, and the volume then reported Healthy/OK before writes.

Preserved pre-repair state:

  • installed v0.1.0-dev-55e0664b
  • update phase testing, boot_attempts=1, snapshot 20260831T083944Z-pre-update-3206
  • retained signed known-good 0.1.0-dev-2933f706
  • durable authorized-key fingerprint still SHA256:PzLe/DSOXhmvcSHaJQiEf1EDCqGZydATBSmUK8AmwF0
  • all four original active files copied to D: with SHA-256 before replacement

Recovery-only offline repair replaced exactly four files from final PR head 0b3b5d96609723724c3d5449e4f65628aa8d3246 and verified the on-card hashes:

  • bloom-dev-ssh: 4d5e658268de7d69975d8360547ae607ad7eb745cd07729c39466003fea9d597
  • bloom-update-state: cd31ac2aa8b7b4c18dca1a06488b27111e8875a59a5edd61a5bc97dd514eaa84
  • bloom-update-rollback: 9527357de5f47b1d43889b27187a077124fe8bf3a861aae77cc711ff752444dc
  • update_networking.sh: abf5485064b12e647d034b6224d29b6489a05e45e6bf3f24a22b34710162a052

The update state was unchanged after repair, the FAT volume remained clean/healthy, and Explorer safely ejected the card. This is explicitly an unconfirmed recovery-only runtime patch, not signed-release certification. The next gate is one boot, strict key-only SSH continuity, supported rollback to exact signed known-good bytes, and then the signed final candidate.

Separate physical power evidence: a 3-second power hold showed no Saving/Shutting Down screen; Bloom Shell remained visible until the device powered off roughly 30 seconds later. That behavior is not counted as a shutdown-UI pass and will be tracked separately from this SSH race fix.

@boburning

Copy link
Copy Markdown
Owner Author

Rollback activation checkpoint: PR #642 recovery runtime restored strict key-only SSH on the repaired boot. Read-only gates passed on Mini Plus model 354: aggregate health healthy, update phase testing at boot attempt 2, zero active play sessions, network transaction idle, SSH desired/running/key-only, and listener audit clean with only reviewed key-only SSH.

The supported bloomctl update rollback path rejected the unconfirmed 55e0664b candidate and published retained signed known-good 2933f706 with state rollback_pending, boot attempts 0, snapshot 20260831T122313Z-pre-rollback-2857. The published signed installer onion.pak exactly matched its retained candidate source before recovery substitution.

To keep SSH available on the rollback boot, only the pending installer copy of onion.pak was atomically replaced with the previously validated recovery-only pack: 86,327,297 bytes, SHA-256 f9da08fcdcf8872ee2ca446dcafbb8fe8171fe7b1678ac73a6d365bf30c692df. The remote temporary and final bytes/hash both matched and the temporary file was absent before activation. This pack is explicitly unsigned recovery-only and will not be confirmed or retained as known-good.

Exactly one bloomctl power reboot was issued. The device is now in the expected offline installer interval; no further reboot/input/network mutation is being attempted.

@boburning

Copy link
Copy Markdown
Owner Author

Signed Mini Plus device gate passed for final PR head 0b3b5d96609723724c3d5449e4f65628aa8d3246 using the signed development artifact from hardware run 33378466169.

Exact signed artifact:

  • version 0.1.0-dev-0b3b5d96
  • ZIP 185,667,660 bytes
  • ZIP SHA-256 316b63bb8bc3bdc093aef360fad24c7af54f122862498fa6b358a312a71c4506
  • nested onion.pak 73,786,477 bytes
  • nested SHA-256 b7f4a5183be8e968da5abd20f5cbf5a99849fbbdfed8aaa51f810e5861d9ab1c

Device result on Miyoo Mini Plus model 354:

  • strict pinned-host, key-only SSH returned on the first signed-candidate boot after about 7.5 minutes;
  • installed version exactly v0.1.0-dev-0b3b5d96;
  • update passed testing boot attempt 1 and was confirmed to known_good;
  • aggregate health healthy; system/storage healthy; play-activity database quick-check OK; active sessions 0; snapshots verified; network transaction idle;
  • SSH desired/running with running_key_only; SFTP and Samba desired/stopped off;
  • listener audit clean with only reviewed key-only SSH; no VNC or other prohibited listener/process;
  • SSH boot log showed one Dropbear parent start, successful authentication by the expected key, correct /home/root/.ssh/authorized_keys permissions, no stale lifecycle lock, and the /home tmpfs still mounted;
  • exact installed fix hashes match the final PR source:
    • bloom-dev-ssh 4d5e658268de7d69975d8360547ae607ad7eb745cd07729c39466003fea9d597
    • bloom-update-state cd31ac2aa8b7b4c18dca1a06488b27111e8875a59a5edd61a5bc97dd514eaa84
    • bloom-update-rollback 9527357de5f47b1d43889b27187a077124fe8bf3a861aae77cc711ff752444dc
    • update_networking.sh abf5485064b12e647d034b6224d29b6489a05e45e6bf3f24a22b34710162a052;
  • signed rollback source 2933f706 remains retained;
  • task transfer copies, pending MainUI, RetroArch installer, and empty installer directory were removed after exact validation.

Recovery-only intermediate bytes were restored to exact signed 2933f706 before that rollback was confirmed and are no longer installed or retained by the updater. Evidence is remote/SSH/process/filesystem evidence on one Mini Plus; it does not certify Mini V1-V4 or Flip, physical LCD quality, tactile controls, audio, gameplay/save semantics, or the separately observed missing Saving/Shutting Down screen.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Developer SSH listener loses projected key during concurrent boot reconciliation

1 participant