Repository navigation
offline sync: resend queued writes under an idempotency key - #81
Conversation
Operator remember/correct take an optional key. The Worker stores it in operator_receipts in the claim's own batch and answers a repeat with the first claim, so kypp sync now resends a write after any failure and it still lands once. Rows sent before keys keep the manual --drop/--resend.
vu1n
left a comment
There was a problem hiding this comment.
VPS review of 5cd2255: 1 finding(s), 1 blocking.
The change adds persistent idempotency keys for hosted writes and improves offline retries. Keyed correction replays omit the claim’s current status, so the replay response handling needs correction before merging.
| // A replayed claim may since have been purged or moved out of the operator's view; report what it is now. | ||
| const status = (await store.anyClaim(id))?.status ?? "gone"; | ||
| if (!human) return { id, status, ...(key ? { key, replayed } : {}) }; | ||
| return { id, superseded, ...(key ? { key, replayed } : {}) }; |
There was a problem hiding this comment.
P2 Include current status in keyed correction responses
Send a keyed correct, purge its returned claim, then resend the same request: the response contains replayed: true but no status, so the caller cannot tell that the correction is now rejected. The advertised replay contract covers both remember and correct. Include status in the keyed correction response while preserving the unkeyed response shape, and add a keyed-correction replay test covering purge or supersession.
There was a problem hiding this comment.
Fixed in f66cb97: a keyed correct now returns status (the claim's current one) next to key and replayed, and the unkeyed answer keeps its old shape. The new test sends a keyed correct, purges it, resends it, and gets replayed: true, status: "rejected" with the claim still purged.
Generated by Claude Code
vu1n
left a comment
There was a problem hiding this comment.
VPS review of f66cb97: 1 finding(s), 1 blocking.
The change adds persistent idempotency keys and automatic retries for queued writes, backed by Worker receipts. It is unsafe to merge as written because retries against an older Worker can duplicate committed writes.
| db.execute("UPDATE outbox SET sent_at = ?, key = ? WHERE seq = ?", (_now(), key, seq)) | ||
| try: | ||
| remote.call(op, **args) | ||
| remote.call(op, key=key, **args) |
There was a problem hiding this comment.
P2 Confirm Worker idempotency support before enabling automatic retries
When an updated CLI syncs against a Worker deployed before this change, that Worker silently ignores key. If it commits a write but its reply is lost, the next sync automatically sends the row again and creates a duplicate; a claim purged between attempts can also be recreated. Persisting a client-generated key does not establish that the server honored it. Check server capabilities before the first send and retain the manual ambiguous-send guard for servers without receipt support. Add coverage for a lost reply from an older Worker; the current Python fake always honors keys.
There was a problem hiding this comment.
Fixed in 033569f. The Worker has a new capabilities op (idempotent_writes: true), and push asks it once before sending. A Worker from before this change answers unknown op. In that case the write goes without a key, and a send that got no 200 waits for the operator's --drop or --resend as before, even after the Worker is upgraded. A keyed row is also never resent to a Worker that stops honouring keys. New tests cover a lost reply from an older Worker (it is applied once and never resent on its own) and that downgrade case.
Generated by Claude Code
vu1n
left a comment
There was a problem hiding this comment.
VPS review of 033569f: 1 finding(s), 1 blocking.
The change adds idempotency keys and automatic retries for queued hosted writes. It is unsafe to merge as written because a failed retry can erase prior-send evidence and allow a duplicate write after a Worker downgrade.
| remote.call(op, key=key, **args) | ||
| except remote.Unreachable: | ||
| with db: # no connection was made, so nothing was applied | ||
| db.execute("UPDATE outbox SET sent_at = NULL WHERE seq = ?", (seq,)) |
There was a problem hiding this comment.
P2 Preserve prior-send evidence when a retry cannot connect
If a keyed write commits but loses its reply, a later sync can pass the capabilities check and then fail to connect when resending the write. This handler clears sent_at, even though the earlier attempt may have committed. If the Worker subsequently stops supporting keys, the guard at line 232 permits another send because sent_at is null; the older Worker ignores the key and inserts a duplicate, potentially restoring a purged claim. Restore the row's previous sent_at on Unreachable instead of always clearing it, and test this retry/downgrade sequence.
There was a problem hiding this comment.
Fixed in 390137e. When an attempt hits Unreachable, the row now gets back the sent_at it had before that attempt, which is NULL only if it was never sent. A keyed write that committed earlier therefore stays on record. The new test covers the case: a lost reply, then a resend that can't connect, then a downgraded Worker. The row is refused with "no longer honours idempotency keys" and the claim is applied once.
Generated by Claude Code
vu1n
left a comment
There was a problem hiding this comment.
VPS review of 390137e: 1 finding(s), 1 blocking.
The change adds idempotency keys and automatic retries for queued writes. It is unsafe to merge because a downgrade-and-upgrade sequence can cause a committed write to be duplicated.
| "and no 200 came back")) | ||
| if keyed is None: | ||
| keyed = _keyed() | ||
| if key and sent_at and not keyed: |
There was a problem hiding this comment.
P2 Clear an unsent key before sending to a Worker without key support
If capabilities succeeds but the first write cannot connect, the row retains its key with sent_at = NULL. If the Worker is then downgraded, this guard allows the row through and preserves its key even though the older Worker ignores it. A committed write whose reply is lost will subsequently be resent after an upgrade, creating a duplicate because the first write has no receipt. When keyed is false and the row has never been sent, clear its key on disk before sending so an ambiguous result requires manual resolution. Add a regression test covering this transition.
There was a problem hiding this comment.
Fixed in c83ad28, and I fixed it at the root rather than just this one sequence. A row now keeps its key only while every send of it went to a Worker that honours keys. Before each send, the key is fixed on disk (keyed) or cleared (not keyed). A sent row is resent on its own only when it still has a key and the Worker still honours keys. The new test runs the sequence you described: a keyed attempt that can't connect, then a downgrade with a lost reply, then an upgrade. The row is refused with "without an idempotency key" and the claim is applied once.
Generated by Claude Code
vu1n
left a comment
There was a problem hiding this comment.
VPS review of c83ad28: 1 finding(s), 1 blocking.
The change adds idempotency keys and Worker receipts so queued writes can retry safely after a lost response. A newly introduced race in cache schema migration can abort concurrent CLI commands and should be fixed before merging.
1 finding(s) dropped by the check
tests/test_offline.py:193Test automatic retries against the actual Worker implementation: The missing combined integration test establishes no reachable defect: CLI tests verify reuse of the persisted key, and actual Worker tests verify receipt replay after purge without recreating the claim.
| os.chmod(path, 0o600) | ||
| db.executescript(_SCHEMA) | ||
| # A cache made before writes carried keys gains the column; push gives each row its key. | ||
| if "key" not in {r[1] for r in db.execute("PRAGMA table_info(outbox)")}: |
There was a problem hiding this comment.
P2 Serialize the cache schema check and migration
When two CLI processes first open an older cache concurrently, both can observe that key is absent before either ALTER runs. The second ALTER then raises sqlite3.OperationalError: duplicate column name: key, aborting the command. The sync lock cannot prevent this because _db() runs before acquiring it, and other cache users do not acquire it. Serialize the schema check and ALTER in a transaction that obtains the write lock before checking, and cover concurrent migration.
There was a problem hiding this comment.
Fixed in d943894. The schema check and the ALTER now run inside BEGIN IMMEDIATE, so a second CLI waits for the write lock and then sees the column already there. The new test opens an old cache from 6 threads at once, with the gap between reading the schema and changing it widened. Without the transaction it reproduces duplicate column name: key, and with it the test passes.
Generated by Claude Code
vu1n
left a comment
There was a problem hiding this comment.
VPS review of d943894: 0 finding(s), 0 blocking.
The change adds persisted idempotency keys for queued writes and Worker receipts that prevent retries from creating duplicate claims. The finding identifies a test coverage gap without demonstrating a reachable defect, so it does not block merging.
1 finding(s) dropped by the check
tests/test_offline.py:55Test queued retries against the actual Worker handler: Although the Python retry test uses a fake Worker, push persists and reuses the key, remote.call transmits it, and operatorClaim atomically stores the receipt and claim; actual handler tests also verify replay deduplication, so the missing combined test demonstrates no concrete failure.
Follow-up to #78. A queued
rememberthat got no 200 used to blockkypp syncuntil the operator ran--drop Nor--resend N, because the Worker had no way to tell a resend from a new write. Now each write carries a key and is resent automatically, and it still lands only once.Contracts touched
POST /v1/operatorremember/correct: new optionalkey(16 to 128 characters from[A-Za-z0-9_-]).operator_receipts, migration 0012) is written in the same D1 batch as the claim.replayed: trueand the claim's current status. A purged claim comes back asrejectedand is never written again.kypp sync:--drop Nis still how to discard a row the Worker keeps refusing.keycolumn added automatically. A row an older client already sent has no key, so it still waits for a manual--dropor--resend.npm run deploy, so it needs no manual step.Verification
npm testinworker: 359 pass, 0 fail.tsc --noEmitis clean.uv run python -m unittest discover -s tests: 343 tests, all OK.python3 scripts/featuremap.py check: 0 broken pointers.operatorClaimmakes the race test fail.keyfrom the request, skipping the guard on old-format rows, or skipping the column migration each maketest_offlinefail.Generated by Claude Code