Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions docs/features/hosted-worker.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,11 @@ TypeSafe API's shape, for GitHub Actions OIDC tokens (`worker/src/oidc.ts`) and
`defrag_log`. `markVerified` mirrors `MemoryStore.mark_verified`: change one, change both.
- The operator `snapshot` op is `kypp sync`'s read-only pull: recall's visibility with every
candidate (`visible(..., "all")`), paged by id. A field the offline cache needs must be in `CLAIM_COLUMNS`.
- Operator `remember`/`correct` take an optional idempotency `key`. Its `operator_receipts` row goes
in the claim's own batch, so a repeat (or a racing send whose batch lost on the primary key) returns
the first claim with `replayed: true` and its current `status`; the same key with a different write is refused. Receipts are
kept for good: a write queued offline may be resent at any later sync. The `capabilities` op
(`idempotent_writes: true`) is how the CLI tells this Worker from one that ignores `key`.
- Hosted operator `correct` is an explicit replacement in the existing `memory_claims` store,
not a rank contest. Insert, FTS/pointers, exact-group supersession and audit share one D1 batch;
the returned superseded count comes from that transaction. Equal clocks, query order and lower
Expand Down
19 changes: 13 additions & 6 deletions docs/features/offline.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,19 @@ operator's view of the project (`POST /v1/operator` `snapshot`, 500 claims a pag
- Only `remote.Unreachable` (no connection was made) falls back to the cache or queue. urllib wraps a
failure while sending in `URLError` too, so `remote.py` records whether `connect()` finished
(`_conn.made`); after it, a failure is ambiguous and never falls back.
- A queued write is marked sent before its request and deleted only after a 200. Only `Unreachable`
unmarks it. Anything else (an error answer, a timeout, a dropped connection) may follow a committed
write: the Worker answers 400 for any failure, even after a commit, and operator `remember` has no
idempotency key. A marked write blocks the push until the operator checks and runs `kypp sync
--drop N` or `--resend N`; it is never re-sent on its own (a purge must not be undone by a replay).
The pull still runs, so a stuck write doesn't freeze the copy.
- Before a push, `kypp sync` asks the Worker for `capabilities`. Against one that honours keys, each
queued write gets an idempotency key, saved on disk before its first send and sent again on every
retry. The Worker stores the key in `operator_receipts` in the claim's own batch and answers a repeat
with the first claim, so a write is resent automatically after any failure (an error answer, a
timeout, a lost reply) and still lands once; a purged claim is never written back by a late resend.
A write leaves the queue only after a 200 and blocks the writes behind it until then; a write the
Worker keeps refusing stays until `kypp sync --drop N`. The pull still runs, so a stuck write doesn't
freeze the copy.
- A Worker deployed before keys answers `unknown op` to `capabilities` and ignores `key`, so the write
goes without one, and a row's key is cleared before any such send. A row sent without a key (to such a Worker, or by an older CLI), or a keyed row the
Worker no longer honours, may or may not have landed: it blocks the push until the operator checks and
runs `--drop N` or `--resend N`. A send that never connected restores the row's earlier mark (none if
it was never sent), so an earlier, maybe committed send stays on record.
- The cache is a copy, never an authority: nothing in `offline.py` accepts, promotes or supersedes. The
Worker applies its own rules when a queued write arrives.
- `correct`, `reject` and `purge` are refused offline (`shell._online`): they replace or retire memory
Expand Down
70 changes: 53 additions & 17 deletions kypp/offline.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
import json
import os
import re
import secrets
import sqlite3
import sys
import time
Expand All @@ -33,7 +34,7 @@
CREATE TABLE IF NOT EXISTS forgotten (url TEXT NOT NULL, id TEXT NOT NULL, purged INTEGER NOT NULL, at REAL NOT NULL,
PRIMARY KEY (url, id));
CREATE TABLE IF NOT EXISTS outbox (seq INTEGER PRIMARY KEY AUTOINCREMENT, url TEXT NOT NULL, project TEXT NOT NULL,
op TEXT NOT NULL, args TEXT NOT NULL, queued_at TEXT NOT NULL, sent_at TEXT);
op TEXT NOT NULL, args TEXT NOT NULL, queued_at TEXT NOT NULL, sent_at TEXT, key TEXT);
"""


Expand All @@ -56,6 +57,16 @@ def _db() -> sqlite3.Connection:
os.umask(old)
os.chmod(path, 0o600)
db.executescript(_SCHEMA)
# A cache made before writes carried keys gains the column; push gives each row its key. The check
# and the ALTER share one write transaction, so two CLIs opening an old cache can't both add it.
db.execute("BEGIN IMMEDIATE")
try:
if "key" not in {r[1] for r in db.execute("PRAGMA table_info(outbox)")}:
db.execute("ALTER TABLE outbox ADD COLUMN key TEXT")
db.execute("COMMIT")
except BaseException:
db.execute("ROLLBACK")
raise
return db


Expand Down Expand Up @@ -192,32 +203,56 @@ def forget(claim_id: str, purged: bool) -> None:
db.execute("DELETE FROM claims WHERE url = ? AND id = ?", (_url(), claim_id))


def _keyed() -> bool:
"""Whether the Worker honours idempotency keys. One deployed before them ignores `key`."""
try:
return remote.call("capabilities").get("idempotent_writes") is True
except remote.Unreachable:
raise
except SystemExit as e:
if "unknown op" in str(e.code):
return False
raise


def push(db: sqlite3.Connection) -> int:
"""Send queued writes in order. Each is marked sent before its request and leaves the queue only
after a 200. Only a request that never got a connection is unmarked again. Any other failure (an
error answer, a timeout, a dropped connection) may follow a committed write: the Worker answers 400
for any failure, even after a commit, and has no idempotency key. So a marked write is never sent
again on its own; the operator checks the hosted store and runs `--drop N` or `--resend N`."""
sent = 0
for seq, op, raw, sent_at in db.execute(
"SELECT seq, op, args, sent_at FROM outbox WHERE url = ? ORDER BY seq", (_url(),)).fetchall():
"""Send queued writes in order. Against a Worker that honours keys, each is sent under its own
idempotency key, so it is applied at most once however often it is sent: a write leaves the queue
only after a 200, and any other answer (or none) leaves it, and the writes after it, for the next
`kypp sync` to send again. A row sent without a key (to an older Worker, or by an older CLI) may or
may not have landed, so it is never resent on its own: the operator decides it."""
sent, keyed = 0, None
for seq, op, raw, sent_at, key in db.execute(
"SELECT seq, op, args, sent_at, key FROM outbox WHERE url = ? ORDER BY seq", (_url(),)).fetchall():
if op not in _QUEUEABLE:
raise SystemExit(f"kypp: queued write #{seq} is `{op}`, which is never queued; "
f"drop it with `kypp sync --drop {seq}`")
stored = json.loads(raw)
args = {k: stored.get(k) for k in _QUEUEABLE[op]}
if sent_at:
raise SystemExit(_unsure(seq, args, f"it was sent at {sent_at} and no 200 came back"))
# A row keeps a key only while every send of it went to a Worker that honours keys: it is fixed on
# disk before such a send and cleared before any other, so only a keyed row is resent on its own.
if keyed is None:
keyed = _keyed()
if sent_at and not (key and keyed):
why = "and the Worker no longer honours idempotency keys" if key else "without an idempotency key"
raise SystemExit(_unsure(seq, args, f"it was sent at {sent_at} {why} and no 200 came back"))
key = (key or secrets.token_hex(16)) if keyed else None
with db:
db.execute("UPDATE outbox SET sent_at = ? WHERE seq = ?", (_now(), seq))
db.execute("UPDATE outbox SET sent_at = ?, key = ? WHERE seq = ?", (_now(), key, seq))
try:
remote.call(op, **args)
remote.call(op, key=key, **args)

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Confirm Worker idempotency support before enabling automatic retries

When an updated CLI syncs against a Worker deployed before this change, that Worker silently ignores key. If it commits a write but its reply is lost, the next sync automatically sends the row again and creates a duplicate; a claim purged between attempts can also be recreated. Persisting a client-generated key does not establish that the server honored it. Check server capabilities before the first send and retain the manual ambiguous-send guard for servers without receipt support. Add coverage for a lost reply from an older Worker; the current Python fake always honors keys.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 033569f. The Worker has a new capabilities op (idempotent_writes: true), and push asks it once before sending. A Worker from before this change answers unknown op. In that case the write goes without a key, and a send that got no 200 waits for the operator's --drop or --resend as before, even after the Worker is upgraded. A keyed row is also never resent to a Worker that stops honouring keys. New tests cover a lost reply from an older Worker (it is applied once and never resent on its own) and that downgrade case.


Generated by Claude Code

except remote.Unreachable:
with db: # no connection was made, so nothing was applied
db.execute("UPDATE outbox SET sent_at = NULL WHERE seq = ?", (seq,))
with db: # this attempt never connected; an earlier send may still have landed, so keep its mark
db.execute("UPDATE outbox SET sent_at = ? WHERE seq = ?", (sent_at, seq))
raise
except SystemExit as e:
raise SystemExit(f"{e.code}\n" + _unsure(seq, args, "the Worker didn't answer 200")) from None
if not key:
raise SystemExit(f"{e.code}\n" + _unsure(seq, args, "the Worker (which predates idempotency keys) "
"didn't answer 200")) from None
raise SystemExit(f"{e.code}\nkypp: queued write #{seq} ({args.get('subject')!r}) and the writes after "
f"it stay queued; the next `kypp sync` sends it again under the same key, so it lands "
f"at most once. If the Worker refused it, fix the cause or run "
f"`kypp sync --drop {seq}`.") from None
with db:
db.execute("DELETE FROM outbox WHERE seq = ?", (seq,))
sent += 1
Expand Down Expand Up @@ -252,7 +287,8 @@ def sync_main():
ap.add_argument("--drop", type=int, action="append", default=[], metavar="N",
help="discard queued write #N without sending it (it reached the store, or it was refused)")
ap.add_argument("--resend", type=int, action="append", default=[], metavar="N",
help="send queued write #N again: you checked that the hosted store doesn't have it")
help="send queued write #N again (one sent without an idempotency key): you checked that the "
"hosted store doesn't have it")
args = ap.parse_args()
if not remote.remote_url():
raise SystemExit("kypp sync: set KYPP_URL; the local store needs no sync")
Expand Down
Loading
Loading