Adds blue-green deployment scripts - #733
Open
klautcomputing wants to merge 1 commit into
Open
klautcomputing wants to merge 1 commit into
klautcomputing wants to merge 1 commit into
Conversation
klautcomputing
commented
May 4, 2026
| fi | ||
|
|
||
| echo "→ Building release binary..." | ||
| LEPTOS_HASH_FILES=true nix develop -c cargo leptos build -rP |
Collaborator
Author
There was a problem hiding this comment.
remove this and the other one
klautcomputing
commented
May 4, 2026
| cd /home/drone/hive | ||
| git pull | ||
| cp /home/leex/hive/.env . # if your .env source-of-truth lives elsewhere | ||
| LEPTOS_HASH_FILES=true nix develop -c cargo leptos build -rP |
Collaborator
Author
There was a problem hiding this comment.
the other one
klautcomputing
force-pushed
the
blue-green
branch
2 times, most recently
from
September 8, 2026 12:39
ea0f2be to
8bbc44c
Compare
Two systemd slots (hive@blue on 3000, hive@green on 3001) behind nginx. deploy.sh stages a release into bin/<colour>/, boots the idle slot, checks it, then swaps the nginx upstream. rollback.sh flips back, refusing when migrations have been applied since the target's commit. Both take an exclusive flock first, so two runs cannot interleave. A slot owns a directory rather than a bare binary because leptos resolves the asset hash manifest as current_exe().parent()/hash.txt. Two binaries sharing a parent would read one manifest between them and each deploy would overwrite the live slot's copy — with LEPTOS_HASH_FILES=true that is a panic on render, not a degraded page. The deploy fails if the build produced no hash.txt rather than shipping a slot that 500s. Staging also carries the outgoing release's hashed assets into the incoming slot, driven by that slot's own pkg.manifest so it is one generation and not the union of every release that ever ran. A page served just before the flip fetches its bundle just after it, from a slot that never built that file. Both slots run every background job for the length of the overlap, so the ones where a second run does real work are guarded: tournament_start, game_cleanup and challenge_cleanup take a transaction-scoped advisory lock; hash_backfill takes a session-scoped one, retried rather than skipped, since the holder is normally the outgoing slot and is about to stop mid-pass. That lock is on a pooled connection, so it is released explicitly on every exit path — dropping the guard hands a held lock to the next borrower. email_drain's claim_batch now leases its batch under FOR UPDATE SKIP LOCKED instead of reading it with a plain SELECT and only marking rows after delivery, which sent every queued mail twice. The lease is the caller's, because only the caller knows its worst case: deliver() is an HTTP call with no timeout of its own, so it gets one, and the batch shrank to fit inside the lease. timeout_sweeper, tournament_cleanup, push_device_sweep and email_cleanup are deliberately left unlocked. They are already idempotent, and the first two notify WebSocket clients — the lock winner may be the slot nginx is not routing to, so locking them would silence the users who are actually connected. tournament_start keeps that flaw on purpose: double-starting a tournament is worse than a missed notification. The overlap window is the drain plus however long the outgoing slot takes to die, and the second term dominated: actix waits shutdown_timeout for connections to close, a WebSocket never closes on its own, and the default is 30s. main.rs sets it to 5 and the drain is 3s, so roughly 8s rather than 33. Closing it properly needs the hubs to share broadcasts over LISTEN/NOTIFY, which is the next piece of work. Error handling fails closed throughout, because the failure that matters here is the one that lets a broken deploy continue: - a git diff that errors tells us nothing about whether the schema changed, so it aborts instead of reading as "no migrations"; likewise grep exiting 2 for an unreadable migration rather than 1 for no match - swap_upstream restores the previous upstream file when *either* nginx -t or the reload fails. That file is the record of which slot is live and the next deploy trusts it over systemd, so leaving it naming a slot nginx never picked up makes that deploy stop the slot still serving traffic - every probe has a request timeout and the liveness loop a wall-clock deadline; a slot that accepts and then says nothing used to hang curl forever with the retry budget never advancing - a failure before the flip stops the slot it brought up, so a failed deploy does not leave two instances running for good. After the flip it stops nothing: by then that slot is serving traffic Server-side, the pieces that only worked with one instance: - the upstream include lives at /etc/nginx/hive-upstream.conf, not conf.d/, which Debian also parses at http scope; the bootstrap's config backup moved out of sites-enabled/ for the same reason — Debian globs it as *, not *.conf - sudoers matches the argument lists the scripts actually send, and covers every sudo they make; a rule that misses one hangs on a password prompt - slots bind loopback, and the deploy refuses to flip to one that doesn't; the box has no firewall and the old 0.0.0.0 bind answered on the public IP - production secrets move to /etc/hive/prod.env; writing them over the repo's tracked .env left the working tree dirty and broke the next pull - nginx listens on 127.0.0.1:3999 for on-box clients, so hive-hydra follows the active slot instead of pinning port 3000 - hive.leex.dev serves the app rather than /var/www/html - releases are staged in bin/<colour>.new and moved into place, so an interrupted deploy that left the idle slot running does not hit ETXTBSY - pg_dump runs alongside the build and keeps the newest BACKUP_KEEP dumps
klautcomputing
force-pushed
the
blue-green
branch
from
September 8, 2026 16:34
8bbc44c to
f9e3c4b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.