Skip to content

offline sync: resend queued writes under an idempotency key - #81

Merged
vu1n merged 6 commits into
mainfrom
claude/project-thread-7nrdgz
Nov 6, 2026
Merged

vu1n merged 6 commits into
mainfrom
claude/project-thread-7nrdgz

Conversation

@vu1n

@vu1n vu1n commented Nov 6, 2026

Copy link
Copy Markdown
Owner

Follow-up to #78. A queued remember that got no 200 used to block kypp sync until the operator ran --drop N or --resend N, because the Worker had no way to tell a resend from a new write. Now each write carries a key and is resent automatically, and it still lands only once.

Contracts touched

  • Worker, POST /v1/operator remember/correct: new optional key (16 to 128 characters from [A-Za-z0-9_-]).
    • The receipt row (operator_receipts, migration 0012) is written in the same D1 batch as the claim.
    • Sending the same key again returns the first claim, with replayed: true and the claim's current status. A purged claim comes back as rejected and is never written again.
    • Two sends of one key that race: the receipt's primary key rolls back the losing batch, and the loser returns the winner's claim.
    • The same key with a different write is a 400.
    • A request without a key gets the same answer shape as before.
  • CLI, kypp sync:
    • Before an outbox row is first sent, it gets a key saved to disk, and every retry reuses that key.
    • Any failure other than unreachable keeps the row and the rows after it queued, and the next sync resends it.
    • --drop N is still how to discard a row the Worker keeps refusing.
    • Older caches get the new key column added automatically. A row an older client already sent has no key, so it still waits for a manual --drop or --resend.
  • The migration ships with npm run deploy, so it needs no manual step.

Verification

  • npm test in worker: 359 pass, 0 fail. tsc --noEmit is clean.
  • uv run python -m unittest discover -s tests: 343 tests, all OK.
  • python3 scripts/featuremap.py check: 0 broken pointers.
  • I broke the code on purpose to check the tests catch it:
    • Removing the race fallback in operatorClaim makes the race test fail.
    • Dropping the key from the request, skipping the guard on old-format rows, or skipping the column migration each make test_offline fail.

Generated by Claude Code

Operator remember/correct take an optional key. The Worker stores it in
operator_receipts in the claim's own batch and answers a repeat with the
first claim, so kypp sync now resends a write after any failure and it
still lands once. Rows sent before keys keep the manual --drop/--resend.

@vu1n vu1n left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VPS review of 5cd2255: 1 finding(s), 1 blocking.

The change adds persistent idempotency keys for hosted writes and improves offline retries. Keyed correction replays omit the claim’s current status, so the replay response handling needs correction before merging.

Comment thread worker/src/operator.ts Outdated
// A replayed claim may since have been purged or moved out of the operator's view; report what it is now.
const status = (await store.anyClaim(id))?.status ?? "gone";
if (!human) return { id, status, ...(key ? { key, replayed } : {}) };
return { id, superseded, ...(key ? { key, replayed } : {}) };

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Include current status in keyed correction responses

Send a keyed correct, purge its returned claim, then resend the same request: the response contains replayed: true but no status, so the caller cannot tell that the correction is now rejected. The advertised replay contract covers both remember and correct. Include status in the keyed correction response while preserving the unkeyed response shape, and add a keyed-correction replay test covering purge or supersession.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in f66cb97: a keyed correct now returns status (the claim's current one) next to key and replayed, and the unkeyed answer keeps its old shape. The new test sends a keyed correct, purges it, resends it, and gets replayed: true, status: "rejected" with the claim still purged.


Generated by Claude Code

@vu1n vu1n left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VPS review of f66cb97: 1 finding(s), 1 blocking.

The change adds persistent idempotency keys and automatic retries for queued writes, backed by Worker receipts. It is unsafe to merge as written because retries against an older Worker can duplicate committed writes.

Comment thread kypp/offline.py
db.execute("UPDATE outbox SET sent_at = ?, key = ? WHERE seq = ?", (_now(), key, seq))
try:
remote.call(op, **args)
remote.call(op, key=key, **args)

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Confirm Worker idempotency support before enabling automatic retries

When an updated CLI syncs against a Worker deployed before this change, that Worker silently ignores key. If it commits a write but its reply is lost, the next sync automatically sends the row again and creates a duplicate; a claim purged between attempts can also be recreated. Persisting a client-generated key does not establish that the server honored it. Check server capabilities before the first send and retain the manual ambiguous-send guard for servers without receipt support. Add coverage for a lost reply from an older Worker; the current Python fake always honors keys.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 033569f. The Worker has a new capabilities op (idempotent_writes: true), and push asks it once before sending. A Worker from before this change answers unknown op. In that case the write goes without a key, and a send that got no 200 waits for the operator's --drop or --resend as before, even after the Worker is upgraded. A keyed row is also never resent to a Worker that stops honouring keys. New tests cover a lost reply from an older Worker (it is applied once and never resent on its own) and that downgrade case.


Generated by Claude Code

@vu1n vu1n left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VPS review of 033569f: 1 finding(s), 1 blocking.

The change adds idempotency keys and automatic retries for queued hosted writes. It is unsafe to merge as written because a failed retry can erase prior-send evidence and allow a duplicate write after a Worker downgrade.

Comment thread kypp/offline.py Outdated
remote.call(op, key=key, **args)
except remote.Unreachable:
with db: # no connection was made, so nothing was applied
db.execute("UPDATE outbox SET sent_at = NULL WHERE seq = ?", (seq,))

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Preserve prior-send evidence when a retry cannot connect

If a keyed write commits but loses its reply, a later sync can pass the capabilities check and then fail to connect when resending the write. This handler clears sent_at, even though the earlier attempt may have committed. If the Worker subsequently stops supporting keys, the guard at line 232 permits another send because sent_at is null; the older Worker ignores the key and inserts a duplicate, potentially restoring a purged claim. Restore the row's previous sent_at on Unreachable instead of always clearing it, and test this retry/downgrade sequence.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 390137e. When an attempt hits Unreachable, the row now gets back the sent_at it had before that attempt, which is NULL only if it was never sent. A keyed write that committed earlier therefore stays on record. The new test covers the case: a lost reply, then a resend that can't connect, then a downgraded Worker. The row is refused with "no longer honours idempotency keys" and the claim is applied once.


Generated by Claude Code

@vu1n vu1n left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VPS review of 390137e: 1 finding(s), 1 blocking.

The change adds idempotency keys and automatic retries for queued writes. It is unsafe to merge because a downgrade-and-upgrade sequence can cause a committed write to be duplicated.

Comment thread kypp/offline.py Outdated
"and no 200 came back"))
if keyed is None:
keyed = _keyed()
if key and sent_at and not keyed:

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Clear an unsent key before sending to a Worker without key support

If capabilities succeeds but the first write cannot connect, the row retains its key with sent_at = NULL. If the Worker is then downgraded, this guard allows the row through and preserves its key even though the older Worker ignores it. A committed write whose reply is lost will subsequently be resent after an upgrade, creating a duplicate because the first write has no receipt. When keyed is false and the row has never been sent, clear its key on disk before sending so an ambiguous result requires manual resolution. Add a regression test covering this transition.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in c83ad28, and I fixed it at the root rather than just this one sequence. A row now keeps its key only while every send of it went to a Worker that honours keys. Before each send, the key is fixed on disk (keyed) or cleared (not keyed). A sent row is resent on its own only when it still has a key and the Worker still honours keys. The new test runs the sequence you described: a keyed attempt that can't connect, then a downgrade with a lost reply, then an upgrade. The row is refused with "without an idempotency key" and the claim is applied once.


Generated by Claude Code

@vu1n vu1n left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VPS review of c83ad28: 1 finding(s), 1 blocking.

The change adds idempotency keys and Worker receipts so queued writes can retry safely after a lost response. A newly introduced race in cache schema migration can abort concurrent CLI commands and should be fixed before merging.

1 finding(s) dropped by the check
  • tests/test_offline.py:193 Test automatic retries against the actual Worker implementation: The missing combined integration test establishes no reachable defect: CLI tests verify reuse of the persisted key, and actual Worker tests verify receipt replay after purge without recreating the claim.

Comment thread kypp/offline.py Outdated
os.chmod(path, 0o600)
db.executescript(_SCHEMA)
# A cache made before writes carried keys gains the column; push gives each row its key.
if "key" not in {r[1] for r in db.execute("PRAGMA table_info(outbox)")}:

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Serialize the cache schema check and migration

When two CLI processes first open an older cache concurrently, both can observe that key is absent before either ALTER runs. The second ALTER then raises sqlite3.OperationalError: duplicate column name: key, aborting the command. The sync lock cannot prevent this because _db() runs before acquiring it, and other cache users do not acquire it. Serialize the schema check and ALTER in a transaction that obtains the write lock before checking, and cover concurrent migration.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in d943894. The schema check and the ALTER now run inside BEGIN IMMEDIATE, so a second CLI waits for the write lock and then sees the column already there. The new test opens an old cache from 6 threads at once, with the gap between reading the schema and changing it widened. Without the transaction it reproduces duplicate column name: key, and with it the test passes.


Generated by Claude Code

@vu1n vu1n left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VPS review of d943894: 0 finding(s), 0 blocking.

The change adds persisted idempotency keys for queued writes and Worker receipts that prevent retries from creating duplicate claims. The finding identifies a test coverage gap without demonstrating a reachable defect, so it does not block merging.

1 finding(s) dropped by the check
  • tests/test_offline.py:55 Test queued retries against the actual Worker handler: Although the Python retry test uses a fake Worker, push persists and reuses the key, remote.call transmits it, and operatorClaim atomically stores the receipt and claim; actual handler tests also verify replay deduplication, so the missing combined test demonstrates no concrete failure.

@vu1n
vu1n merged commit 2a86bd5 into main Nov 6, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant