Multi-proxy support (Redis + shared MySQL) and cache/SQL race fixes

.claude/known_errors.md

@@ -15,3 +15,21 @@
 - **sqlite-jdbc:** never relocate `org.sqlite` when shading (JNI symbol names). Java 25 has no 32-bit x86 port
   (JEP 503), so those natives can be excluded from the jar.
 - **Local MySQL on macOS:** the Unix socket path must be ≤ 103 chars; use a short `--socket` path and connect over TCP.
+- **Cache vs SQL drift under concurrency:** "decide → update cache → queue SQL write" must be one atomic step, or two
+  threads (accept vs remove, two `/f best` toggles) queue writes in a different order than they changed the cache.
+  Fix: one lock around every state change; SQL writes queued inside it. Found by `ConsistencyTest` (random stress).
+- **Don't apply an event after its write commits (thenRun):** with crossing accepts, a delayed second `Befriended` fired
+  after a later removal and re-linked the pair. Apply immediately under the lock instead.
+- **Cross-proxy ordering:** publish events *through the SQL queue* (after the writes queued before them), otherwise a
+  remote proxy can delete a friendship before its insert lands. Crossing accepts on two proxies need a Redis
+  `SET NX PX` claim on the pair, or the loser's lagging insert resurrects a removed friendship.
+- **Stale profile loads:** a friendship change for a player whose profile is still loading is missed by the load (the
+  write is queued behind it) — track loading players and re-read if something changed.
+- **Detect races deterministically:** a `LaggingDb` executor (queue SQL tasks, run random FIFO batches) reproduces
+  write-timing bugs every run; random multi-threaded stress only catches them ~10% of the time.
+- **Test clocks vs wall time:** RedisNetwork turns absolute expiry into a TTL with real time; tests using a fake clock
+  must set it to `Instant.now()` or keys expire instantly.
+- **Name shadowing:** a local variable named `redis` shadows the `redis.clients...` package in fully-qualified names.
+- **Jedis 8:** `JedisPooled` → `RedisClient.builder()`; `ssl(boolean)` is deprecated → `sslOptions(SslOptions.defaults())`.
+- **GitHub Actions YAML:** an unquoted `run:` containing `: ` (e.g. `...:root: -D...`) is parsed as a mapping and the
+  whole workflow is rejected (run with 0 jobs, named after the file path). Quote such commands.

.github/workflows/build.yml

@@ -0,0 +1,42 @@
+name: build
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+permissions:
+  contents: read
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    services:
+      mysql:
+        image: mysql:8.4
+        env:
+          MYSQL_ALLOW_EMPTY_PASSWORD: "yes"
+          MYSQL_DATABASE: friends_test
+        ports: ["3306:3306"]
+        options: >-
+          --health-cmd="mysqladmin ping -h 127.0.0.1"
+          --health-interval=5s --health-timeout=5s --health-retries=30
+      redis:
+        image: redis:8
+        ports: ["6379:6379"]
+        options: >-
+          --health-cmd="redis-cli ping"
+          --health-interval=5s --health-timeout=5s --health-retries=30
+    steps:
+      - uses: actions/checkout@v7
+      - uses: actions/setup-java@v6
+        with:
+          distribution: temurin
+          java-version: "25"
+      - uses: gradle/actions/setup-gradle@v6
+      # Runs every suite: SQLite, MySQL and real Redis (throwaway tables / key namespaces).
+      - run: "./gradlew build -Dfriends.mysql=127.0.0.1:3306/friends_test:root: -Dfriends.redis=127.0.0.1:6379"
+      - uses: actions/upload-artifact@v7
+        with:
+          name: friends-velocity
+          path: velocity/build/libs/friends-velocity-*.jar

README.md

@@ -11,8 +11,8 @@ Modules:
 ## Build
 
 ```bash
-./gradlew :velocity:shadowJar   # -> velocity/build/libs/friends-velocity-0.2.0.jar
-./gradlew :common:test         # SQLite; add -Dfriends.mysql=host:port/db:user:password to also run on MySQL
+./gradlew :velocity:shadowJar   # -> velocity/build/libs/friends-velocity-0.3.0.jar
+./gradlew :common:test         # see "Tests" below for the MySQL and Redis suites
 ```
 
 Drop the jar in Velocity's `plugins/` folder. Velocity 4.2 itself needs Java 25. sqlite-jdbc loads a native
@@ -40,14 +40,34 @@ Permission: `friends.use` — allowed unless explicitly set to false (Velocity h
 ## Config (`plugins/friends/config.yml`)
 
 `storage.type` `sqlite` (single proxy) or `mysql` (MySQL/MariaDB), `request-expiry-minutes` (default 5),
-`max-friends` (default 5000).
+`max-friends` (default 5000), and the `redis` section for multi-proxy networks.
+
+## Multiple proxies
+
+Set `storage.type: mysql` on every proxy (same database), then `redis.enabled: true` with a unique, stable
+`redis.proxy-id` per proxy. Friends then work across proxies: requests, accept/deny, join/leave messages,
+online status, current server, `/f list` and tab-completion all see the whole network.
+
+- Redis holds only short-lived shared state: who is online where (`<ns>:online`), pending requests (one key
+  each, expiring with the request), a heartbeat per proxy and one pub/sub channel. MySQL stays the source of truth.
+- A proxy that stops heartbeating (crash, freeze) is reaped by the others after ~30-40 s and its players are shown
+  offline. A restarted proxy clears what its previous run left behind.
+- After a Redis reconnect every proxy re-reads the shared state and re-announces its own players.
 
 ## How it works
 
 - Each online player's friends and settings are loaded with one query at login and kept in memory, so
   lists, notifications and tab-completion never hit the database.
-- Writes update the cache immediately and run on a single ordered database thread, never on proxy threads.
-- Friend requests live in memory (they expire after minutes) and are lost on proxy restart.
-- Single proxy only: multiple Velocity instances would need a shared cache/pub-sub (e.g. Redis).
+- Every change is an event: applied under one lock (cache + queued SQL write together, so they can't drift), then
+  published through the SQL queue, so other proxies only see a change once it is stored.
+- Writes run on a single ordered database thread per proxy, never on proxy threads.
+- Friend requests live in memory (plus Redis when enabled) and expire after minutes; they are not kept in SQL.
 - Tables are `friends_players` and `friends_friendships`. Data from the old Bungee/Paper tables
   (`players`, `friends`, `requests`, `settings`) is not migrated.
+
+## Tests
+
+`./gradlew :common:test` runs everything that needs no services (SQLite, in-memory multi-proxy, randomized
+cache-vs-database checks with a deliberately slow database, a concurrency stress test). Add
+`-Dfriends.mysql=host:port/db:user:password` and/or `-Dfriends.redis=host:port` to also run the MySQL and
+real-Redis suites (they use throwaway tables/key namespaces).

build.gradle.kts

@@ -4,7 +4,7 @@ plugins {
 
 allprojects {
     group = "com.friends"
-    version = "0.2.0"
+    version = "0.3.0"
 
     repositories {
         mavenCentral()

common/build.gradle.kts

@@ -9,18 +9,21 @@ dependencies {
     compileOnly("net.kyori:adventure-text-minimessage")
     compileOnly("net.kyori:adventure-text-serializer-legacy")
     compileOnly("org.slf4j:slf4j-api:2.0.17")
+    compileOnly("com.google.code.gson:gson:2.14.0")
 
     // Shaded into the plugin jar.
     implementation("com.zaxxer:HikariCP:7.1.0")
     implementation("org.xerial:sqlite-jdbc:3.53.4.0")
     implementation("com.mysql:mysql-connector-j:26.7.0")
+    implementation("redis.clients:jedis:8.0.1")
 
     testImplementation(platform("net.kyori:adventure-bom:5.2.0"))
     testImplementation("net.kyori:adventure-api")
     testImplementation("net.kyori:adventure-text-minimessage")
     testImplementation("net.kyori:adventure-text-serializer-legacy")
     testImplementation("net.kyori:adventure-text-serializer-plain")
     testImplementation("org.slf4j:slf4j-api:2.0.17")
+    testImplementation("com.google.code.gson:gson:2.14.0")
     testImplementation(platform("org.junit:junit-bom:6.1.3"))
     testImplementation("org.junit.jupiter:junit-jupiter")
     testRuntimeOnly("org.junit.platform:junit-platform-launcher")
@@ -30,4 +33,6 @@ tasks.test {
     jvmArgs("--enable-native-access=ALL-UNNAMED") // sqlite-jdbc loads a native library
     // Opt-in MySQL run: -Dfriends.mysql=host:port/db:user:password
     System.getProperty("friends.mysql")?.let { systemProperty("friends.mysql", it) }
+    // Opt-in Redis run: -Dfriends.redis=host:port (uses a throwaway key namespace)
+    System.getProperty("friends.redis")?.let { systemProperty("friends.redis", it) }
 }

common/src/main/java/com/friends/common/Event.java

@@ -0,0 +1,34 @@
+package com.friends.common;
+
+import java.time.Instant;
+import java.util.List;
+import java.util.UUID;
+
+import com.friends.common.Storage.PlayerRow;
+
+/**
+ * A change other proxies must know about. The proxy where it happens applies it, then publishes it; every other
+ * proxy applies the same event, so single- and multi-proxy setups share one code path ({@code Friends.apply}).
+ */
+public sealed interface Event {
+
+    /** Joined the network. */
+    record Joined(UUID id, Presence presence) implements Event {}
+
+    /** Changed server or status (no notification). */
+    record Updated(UUID id, Presence presence) implements Event {}
+
+    /** Left the network from {@code proxy} (ignored if they are already on another proxy). */
+    record Left(UUID id, String proxy, Instant lastSeen) implements Event {}
+
+    record RequestSent(Request request) implements Event {}
+
+    record RequestDenied(UUID from, UUID to) implements Event {}
+
+    record Befriended(PlayerRow a, PlayerRow b, Instant since) implements Event {}
+
+    record Unfriended(UUID player, List<UUID> friends) implements Event {}
+
+    /** A proxy (re)started or stopped heartbeating: forget everyone who was on it. */
+    record ProxyDown(String proxy) implements Event {}
+}

common/src/main/java/com/friends/common/FriendCommand.java

@@ -17,11 +17,9 @@ public final class FriendCommand {
     private static final int MAX_SUGGESTIONS = 100;
 
     private final Friends friends;
-    private final Platform platform;
 
-    public FriendCommand(Friends friends, Platform platform) {
+    public FriendCommand(Friends friends) {
         this.friends = friends;
-        this.platform = platform;
     }
 
     public CompletableFuture<Void> execute(Online s, String[] args) {
@@ -83,13 +81,17 @@ private static CompletableFuture<Void> usage(Online s, String usage) {
     public List<String> suggest(Online s, String[] args) {
         if (args.length <= 1) {
             String prefix = args.length == 0 ? "" : args[0];
-            return filter(Stream.concat(SUBCOMMANDS.stream(), platform.onlineNames().stream().filter(n -> !n.equalsIgnoreCase(s.name()))), prefix);
+            return filter(Stream.concat(SUBCOMMANDS.stream(), friends.onlineNames().stream().filter(n -> !n.equalsIgnoreCase(s.name()))), prefix);
         }
         String prefix = args[args.length - 1];
         String sub = args[0].toLowerCase(Locale.ROOT);
         if (args.length > 2) return List.of();
         return switch (sub) {
-            case "add" -> filter(platform.onlineNames().stream().filter(n -> !n.equalsIgnoreCase(s.name()) && !isFriend(s, n)), prefix);
+            case "add" -> {
+                var friendNames = friends.friendNames(s.id()).stream().map(n -> n.toLowerCase(Locale.ROOT)).collect(java.util.stream.Collectors.toSet());
+                yield filter(friends.onlineNames().stream()
+                        .filter(n -> !n.equalsIgnoreCase(s.name()) && !friendNames.contains(n.toLowerCase(Locale.ROOT))), prefix);
+            }
             case "accept", "deny", "decline" -> filter(friends.requesterNames(s.id()).stream(), prefix);
             case "remove", "delete", "best", "nickname", "nick" -> filter(friends.friendNames(s.id()).stream(), prefix);
             case "list" -> filter(Stream.of("best"), prefix);
@@ -99,12 +101,9 @@ public List<String> suggest(Online s, String[] args) {
         };
     }
 
-    private boolean isFriend(Online s, String name) {
-        return platform.player(name).map(o -> friends.isFriend(s.id(), o.id())).orElse(false);
-    }
-
     private static List<String> filter(Stream<String> options, String prefix) {
         String p = prefix.toLowerCase(Locale.ROOT);
-        return options.filter(o -> o.toLowerCase(Locale.ROOT).startsWith(p)).limit(MAX_SUGGESTIONS).toList();
+        return options.filter(o -> o.toLowerCase(Locale.ROOT).startsWith(p)).distinct()
+                .sorted(String.CASE_INSENSITIVE_ORDER).limit(MAX_SUGGESTIONS).toList();
     }
 }