Skip to content

refactor: migrate fleet configuration to native units - #348

Open
hanthor wants to merge 69 commits into
masterfrom
feat/native-laptop-units
Open

hanthor wants to merge 69 commits into
masterfrom
feat/native-laptop-units

Conversation

@hanthor

@hanthor hanthor commented Oct 27, 2026

Copy link
Copy Markdown
Owner

Changes

  • Replace fleet Ansible applies with ordered native Bash units, mise and chezmoi; preserve independent failure handling and no-secrets operation.
  • Update onboarding, services, documentation and CI; protect retired VPS identities from all unit changes.
  • Add native inspection and behavioral coverage.

Verification

  • Local: Justfile parses; native shell syntax checked; isolated chezmoi inspection produced no HOME changes; retired-host and invalid-tag guards exercised.
  • CI and laptop validation pending. Native check mode intentionally reports unsupported inspections as failures rather than pretending completeness; do not merge until those gaps and fleet behavior are reviewed.

hanthor and others added 20 commits October 25, 2026 19:49
The omp role installed omp but never pointed it at the fleet's auth broker
(`omp auth-broker`, running on punjab), so a rebuilt machine had an empty
~/.omp/agent/config.yml and no provider credentials until someone ran
/login by hand.

The role now sets auth.broker.url from group_vars and, when Bitwarden is
unlocked, writes auth.broker.token from the item omp-auth-broker-token. A
locked vault leaves an existing token alone. The config file is set to 0600
and the role reports `omp auth-broker status`.

just seed-omp-broker (run on punjab) creates the Bitwarden item from
`omp auth-broker token`, which prints the token without rotating it.

The broker's own systemd unit on punjab is still hand-made and unmanaged.
Read the broker token over Tailscale SSH when seeding Bitwarden, and preserve the vault when the server is unreachable.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: James Reilly <5840441+hanthor@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: James Reilly <5840441+hanthor@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: James Reilly <5840441+hanthor@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: James Reilly <5840441+hanthor@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: James Reilly <5840441+hanthor@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: James Reilly <5840441+hanthor@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: James Reilly <5840441+hanthor@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: James Reilly <5840441+hanthor@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: James Reilly <5840441+hanthor@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: James Reilly <5840441+hanthor@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: James Reilly <5840441+hanthor@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: James Reilly <5840441+hanthor@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: James Reilly <5840441+hanthor@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: James Reilly <5840441+hanthor@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: James Reilly <5840441+hanthor@users.noreply.github.com>
@hanthor

hanthor commented Oct 27, 2026

Copy link
Copy Markdown
Owner Author

Read-only laptop smoke: archived commit ad4f8c6, ran native --tags git inspection with secrets disabled on james@himachal-1, and removed the temporary checkout. Exit 0; chezmoi reported two scheduled scripts and no managed-file changes. No configuration apply or service restart was performed.

@hanthor

hanthor commented Oct 27, 2026

Copy link
Copy Markdown
Owner Author

CI also found an existing server-image blocker: ghcr.io/projectbluefin/utah:latest no longer exists. Separate PR #345 changes the base image, but its own installer acceptance still fails on a masked serial-getty@ttyS0.service; that issue is independent of the native fleet migration and is not being bypassed here.

hanthor and others added 7 commits October 27, 2026 18:52
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: James Reilly <5840441+hanthor@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: James Reilly <5840441+hanthor@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: James Reilly <5840441+hanthor@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: James Reilly <5840441+hanthor@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: James Reilly <5840441+hanthor@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: James Reilly <5840441+hanthor@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: James Reilly <5840441+hanthor@users.noreply.github.com>
hanthor and others added 5 commits October 28, 2026 13:56
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: James Reilly <5840441+hanthor@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: James Reilly <5840441+hanthor@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: James Reilly <5840441+hanthor@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: James Reilly <5840441+hanthor@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: James Reilly <5840441+hanthor@users.noreply.github.com>
@hanthor
hanthor marked this pull request as ready for review October 28, 2026 19:13
@hanthor

hanthor commented Oct 28, 2026

Copy link
Copy Markdown
Owner Author

Replacement Punjab is active at 100.103.187.46 under tailnet name punjab. Final home sync and representative hashes passed; Kiro Crew removed. Corrected DHCPv6 on the IPv4-only masquerade interface; full VM start and subsequent guest reboot returned authenticated SSH, broker, Atuin daemon and osquery reporting/polling. Real native just apply-nosecrets passed. Relevant full Python modules: 206 passed; full AWS mock-provider module: 7 passed; VM manifest server dry-run passed. Earlier broader local bootstrap tests had three failures caused by installed Bitwarden CLI shadowing stubs; not represented as a clean full-suite pass. Atuin cloud login remains a pre-existing local/vault key mismatch; history preserved. runs-on stack is DELETE_COMPLETE with both S3 buckets/versions retained. EC2 final shutdown/EIP release runs from the replacement and records a root-only receipt; retained root volume/snapshots/S3 still incur storage charges.

hanthor and others added 2 commits October 28, 2026 19:22
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: James Reilly <5840441+hanthor@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: James Reilly <5840441+hanthor@users.noreply.github.com>
@hanthor

hanthor commented Oct 28, 2026

Copy link
Copy Markdown
Owner Author

Final cutover evidence: replacement is now the session host (100.103.187.46, Ubuntu generic kernel). Its punjab-aws-cutover.service journal records successful completion at 2026-10-28 19:27:15 UTC after observing EC2 stopped, releasing its EIP and retaining recovery storage. AWS logout was intentional. Root-only completion receipt preserves explicit journal provenance; a later check-only authentication failure is retained separately. Current broker, Atuin daemon, update timer, osquery daemon and fleet poll checks pass. Kiro Crew remains masked/inactive with launcher absent. Matrix, auth, School, Reef and Hub each return HTTPS 200. All current-host staging/active-EC2 claims were updated. Retained EBS/snapshots/S3 still incur storage charges.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: James Reilly <5840441+hanthor@users.noreply.github.com>
@hanthor

hanthor commented Oct 28, 2026

Copy link
Copy Markdown
Owner Author

Atuin recovery completed with owner-approved clean cloud replacement. Preserved all 23,133 readable cloud, 512 Punjab and 10 Kerala history UUIDs plus 105 additional raw-shell entries; 23,761 visible entries include one real daemon capture/cloud round-trip verification command. SQLite integrity and every active encrypted record verify. Independent fresh-client download matches the full history UUID set. Punjab, Karnataka and Kerala sync using the existing verified Bitwarden key; prior Punjab/Kerala keys are hidden recovery fields. Original server ciphertext, including 245 records under 11 unavailable keys, is retained locally and in decrypt-tested/checksum-verified age archives on Spindle; Himachal remained offline/untouched. Installed official atuin-sync on Kerala. Fixed the unit to trust atuin status exit status rather than the obsolete [Sync] heading. Full affected Atuin/package-check modules: 20 passed; shellcheck passed. Broader platform module run had one unrelated existing Termux boot-script migration failure; Kerala apk-tools3.0.8 also rejects the package unit existing --no-upgrade option, so its direct package replacement is not claimed as a full APK-unit validation. No secrets/history content committed.

hanthor and others added 2 commits October 28, 2026 21:18
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: James Reilly <5840441+hanthor@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: James Reilly <5840441+hanthor@users.noreply.github.com>
@hanthor

hanthor commented Oct 29, 2026

Copy link
Copy Markdown
Owner Author

2026-10-29 live AWS shutdown re-verification: account181185361136 has no running workloads in the enumerated services across all17 enabled regions. Regional inventory:332 successful reads;8 endpoint errors resolved as unsupported service/region pairs via AWS public SSM availability registry. Another68 AutoScaling/SpotRequest/SpotFleet/EC2Fleet checks all returned empty. Direct final check: Punjab EC2i-085690e02ca98c95a stopped, no us-east-1 EIPs. All3 S3 buckets,250GiB gp3 root volume and5 snapshots retained; both DLM policies disabled. Independent live Kiro subscription API confirms Power and ProPlus both CANCELLED, expiring2026-10-31T23:59:59.999Z; remaining October Power billing is not an active renewal. No AWS mutations were necessary because shutdown/cancellation state was already achieved. Migrated Punjab passed strict-host-key SSH and a fresh omp request explicitly using anthropic/claude-opus-5-5. Private verification receipt retained under ~/backups/aws-shutdown-verification-20261029T013010Z/verification.json. Retained recovery storage is still billable; no zero-bill claim. No tests run for this documentation-only change; evidence is live AWS APIs and actual omp execution.

Preserve historical cluster contexts and vault notes, scope API firewall access to the tailnet, and document verified client rollout.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: James Reilly <5840441+hanthor@users.noreply.github.com>
@hanthor

hanthor commented Oct 29, 2026

Copy link
Copy Markdown
Owner Author

Karnataka K3s documented and live client access rolled out to Punjab, Karnataka, Kerala and Spindle. Each default context is karnataka-k3s and passed authenticated TLS /readyz + Ready-node listing; existing context/cluster/user entries compared and preserved, configs0600. Dedicated Bitwarden note kubeconfig-karnataka-k3s server-refreshed; historical notes untouched. Added all-active-platform unit eligibility, package prerequisites, atomic context merge, safe historical seeder default, and explicit cached-only mode for production hosts. Reproduced API failure before; scoped runtime/permanent firewalld rule restored6443 overtailnet without reload/restart. Full firewall helper still returnsnonzero at pre-existing absent runtime pod-egress policy; not force-activated. 78 kube/check/seeder/firewall tests passed, shellcheck error-level passed, docs built with CI-pinned mdBook0.4.44 and rendered page visually checked. Mumbai/Himachal/Dilli/Kanpur unreachable, Termux8022 refused; repository rollout prepared but no live configuration claimed there. BiharSSH22 refused/unverified and retired VPSes untouched. AWS final deletion still waits for complete off-AWS fleet-backup preservation.

Record owner-approved deletion of historical backups without a new archive, preserve live Karnataka/Spindle services, and remove stale infrastructure recreation and CI entrypoints.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: James Reilly <5840441+hanthor@users.noreply.github.com>
@hanthor

hanthor commented Oct 29, 2026

Copy link
Copy Markdown
Owner Author

Final AWS retirement completed2026-10-29 after explicit owner authorization that old Hive/Spindle backups are unnecessary. Canceled archival transfer and removed only its newlycreated partial payloads. Deleted3S3buckets, all1430versions+413delete markers(185.26GB), oldPunjabEC2,250GiBroot,5snapshots,migrationVPC/network,PunjabSG,2DLMpolicies,5budgets,3IAMusers/credentials,3customroles/profiles andcustompolicy. Final102regional APIchecks across17regions empty; buckets/users/customroles/customerpolicies empty; each5budgetnames returnsNotFoundException. KarnatakaReady,PunjabVMRunning,andfresh explicitAnthropicomp request passed afterdeletion. Removed executableAWSrootmodule/helper/recipes/providerupdate/CIwiring; Gitdcbbfbc retains historicalsource. Removed ONLY obsolete tofu requiredbranchcheck; retained lint/python/tests/justfile/broker bindings/reviewgate. justlint,Justfilerecipesmoke,jq andCI-pinnedmdBookbuild passed; K3s78test result remains aspreviouslyreported. Existinglocalmigrationdata,K3sconfigs,Atuinarchives andliveSpindle/Punjabuntouched. No new completearchive claimed. AWSaccountdefaults/service-linkedroles remain; KiroalreadycanceledexpiresOct31,andfinal/delayedchargespossible. Privatecompletionreceipt: ~/backups/aws-final-retirement-20261029T015927Z/completion.json.

- flatpak: new flatpark_flatpaks list, installed system-wide from the
  flatpark remote; system_flatpaks stays Flathub-only.
- beeper-login unit: Beeper Desktop only supports emailed one-time codes, so
  it starts sign-in via the local Desktop API, reads the code over IMAP from the
  Bitwarden item 'beeper', then unlocks E2EE with the Recovery Code. Skipped
  with --no-secrets, a locked vault, non-desktops or skip_beeper.
retry.usageReservePct 10 -> 0 and omp-route's Claude candidate reserve
10 -> 0 (OMP_ROUTE_CLAUDE_RESERVE_PCT), so sessions stay on Claude until
its window is actually spent. Other omp-route candidates keep 10%.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: James Reilly <jreilly1821@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant