-
Notifications
You must be signed in to change notification settings - Fork 1
refactor(activity): qualify Valkey activity identifiers #914
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
seonghobae
wants to merge
70
commits into
main
Choose a base branch
from
fix/activity-stream-semantic-identifiers
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
70 commits
Select commit
Hold shift + click to select a range
c852bec
test(naming): require semantic activity stream parameters
seonghobae 88273a1
test(naming): cover activity stream reader parameters
seonghobae 8d89242
refactor(activity): qualify Valkey stream identifiers
seonghobae 3ac2c00
test(activity): expose reseed window idempotency gap
seonghobae 8c41448
fix(activity): preserve reseed idempotency across retained stream
seonghobae bb3cae0
test(activity): preserve distinct same-summary events
seonghobae ef15ddb
fix(activity): deduplicate exact retained event identity
seonghobae 28db59c
docs(activity): document Valkey boundary invariants
seonghobae 819a4df
test(activity): isolate reseed identity dimensions
seonghobae b689091
test(activity): reproduce concurrent reseed duplication
seonghobae 5b9b1e9
test(activity): model watched reseed transaction
seonghobae 7ebf1fe
fix(activity): make seed dedupe atomic under concurrency
seonghobae 40b823f
test(activity): reproduce unbounded WATCH retries
seonghobae a35925b
fix(activity): bound WATCH conflict retries
seonghobae 6a2d60d
test(activity): keep reseed contention errors identifier-safe
seonghobae 2905bd5
fix(activity): keep contention errors identifier-safe
seonghobae 81c3a49
test(activity): reject identifier-bearing contention causes
seonghobae c744fe9
fix(activity): suppress identifier-bearing conflict causes
seonghobae 8a883bf
test(activity): reject false failure telemetry on WATCH retry
seonghobae 91a1e2f
fix(activity): keep recovered WATCH retries out of failure telemetry
seonghobae 625833c
test(activity): require terminal WATCH exhaustion failure span
seonghobae f7fc32a
fix(activity): mark only terminal WATCH exhaustion as failure
seonghobae 8e8d6d2
test(activity): reject numeric actor identity aliases
seonghobae c22e6c2
fix(activity): preserve exact actor identity types
seonghobae 2707911
test(activity): reject numeric post identity aliases
seonghobae 2e010e2
fix(activity): preserve exact post identity types
seonghobae e9dcc6b
test(activity): cover async identity admission
seonghobae 45db51d
test(activity): bound buyer-facing stream reads
seonghobae 433ec17
fix(activity): enforce retained-window read budget
seonghobae a2aaf24
test(activity): expose UUID stream-key alias
seonghobae 223d210
fix(activity): canonicalize source-post UUID stream keys
seonghobae d5a4ba5
test(activity): preserve legacy UUID stream aliases
seonghobae a0eec57
test(activity): cover canonical read of uppercase legacy stream
seonghobae f4044cf
fix(activity): retain historical uppercase UUID stream reads
seonghobae cf58ffb
test(activity): reject cross-stream sequence chronology
seonghobae bdf76ac
fix(activity): define cross-stream tie ordering
seonghobae 874bb6f
test(activity): expose cross-stream event identity collision
seonghobae a3d0a01
fix(activity): namespace legacy read event identities
seonghobae f9645d0
test(activity): align legacy alias event identity
seonghobae 14f0e50
test(activity): use repository-native async execution
seonghobae 60bd1f4
Merge protected main queue fix into activity identity lane
seonghobae 117e9d0
chore(activity): converge identity lane with protected main
seonghobae c3ffdc0
fix(activity): preserve all retained UUID stream aliases
a1d8412
fix(activity): bound compatibility stream merge
d2254c4
chore(main): non-force converge protected #930
seonghobae 345aaa6
docs(adr): keep activity identity decision proposed
seonghobae c257971
chore(main): non-force converge protected #931
seonghobae 64340c0
fix(activity): align read double with Redis stream contract
75d8460
test(activity): bound canonical stream read round trips
seonghobae 3b122bf
test(activity): exercise canonical UUID read path
seonghobae e174738
fix(activity): collapse canonical stream reads
seonghobae ce53313
docs(activity): record canonical read fast path
seonghobae 7dcc5c1
test(activity): count alias lookup in read round-trip budget
seonghobae 02417fc
fix(activity): pipeline canonical alias probe with bounded read
seonghobae 2b59494
test(activity): model one-exchange pipeline fast path
seonghobae 1b0c6a2
docs(adr): bind canonical activity read to one pipeline exchange
seonghobae 4cfa402
test(activity): bound startup alias-index writes
seonghobae c4a25ce
fix(activity): batch startup alias-index writes
seonghobae 25685ef
docs(adr): bound startup alias-index write batches
seonghobae b757fc7
test(activity): prove alias-index batch ceiling
seonghobae c9ca17d
test(activity): make batch-boundary aliases unambiguously legacy
seonghobae 7d44315
test(activity): reject per-event compatibility read waits
seonghobae 2ee36cf
perf(activity): batch retained-alias history reads
seonghobae a352efa
docs(adr): bound retained-alias activity prefetch
seonghobae 15746e2
test(activity): reject redundant alias admission read
seonghobae 07fa552
fix(activity): reuse alias admission read
seonghobae 3a468e1
docs(adr): reuse complete activity alias admission
seonghobae a9c080c
test(activity): keep reused alias admission fanout bounded
seonghobae 8ef3a42
fix(activity): preserve reused alias fanout ceiling
seonghobae 61ed3a3
docs(adr): preserve reused alias fanout ceiling
seonghobae File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,178 @@ | ||
| # ADR 0363: Canonical activity-stream identity | ||
|
|
||
| - Status: Proposed | ||
| - Date: 2026-09-04 | ||
|
|
||
| ## Context | ||
|
|
||
| `source_post.post_id` is a PostgreSQL `uuid`, while historical HTTP requests | ||
| could spell the same UUID with upper-case digits, braces, omitted hyphens, or | ||
| other PostgreSQL-accepted hyphen placement. Earlier activity writes embedded | ||
| the request spelling in a Valkey stream key. Canonicalizing only new writes can | ||
| therefore strand authorized history, and enumerating selected spellings cannot | ||
| cover PostgreSQL's input grammar. | ||
|
|
||
| The activity feed is a retained operational projection. It is not a source of | ||
| record truth, and its compatibility work must not broaden a request into a | ||
| keyspace scan. | ||
|
|
||
| ## Decision | ||
|
|
||
| New activity writes use PostgreSQL's canonical lower-case, hyphenated UUID | ||
| form. Before the application begins serving reads, startup scans the existing | ||
| activity-key namespace once and records every actually present, UUID-equivalent | ||
| legacy key in a durable canonical-post alias set. A request reads the canonical | ||
| stream and only the aliases in that set. Non-UUID synthetic and legacy keys | ||
| retain exact spelling. | ||
|
|
||
| Cross-stream ordering uses the persisted millisecond component and declared | ||
| canonical-first precedence for equal milliseconds. Stream-local sequence | ||
| numbers never become a global chronology. Alias event identifiers remain | ||
| namespaced so two streams cannot emit the same public identity. | ||
|
|
||
| WATCH retries, retained-window read limits, and alias-index startup are fixed | ||
| product contracts. Failure to build the alias index makes the application | ||
| unready; the product does not silently hide historical activity. | ||
|
|
||
| Alias discovery may find many independent legacy keys. Startup therefore keeps | ||
| at most 128 pending alias-set writes in memory and flushes each bounded group | ||
| through a non-transactional redis-py pipeline. `SADD` result integers are summed | ||
| exactly, preserving the existing count of newly indexed aliases. Minimal test | ||
| adapters without pipeline support retain the sequential compatibility fallback; | ||
| the production client created by `create_valkey_client` uses the batched path. | ||
| The batching changes transport cost only: discovery, durability, and readiness | ||
| semantics remain unchanged. | ||
|
|
||
| The supported deployment is the repository's single `lineageweave` Compose | ||
| backend service. Deployment stops the preceding backend before starting the | ||
| replacement, so no pre-canonical writer may overlap the alias scan. A rolling | ||
| multi-version replica deployment is unavailable until it has a separate | ||
| writer-fencing contract. | ||
|
|
||
| Reads enumerate at most the retained-window number of streams. For a UUID read, | ||
| the common path queues the first bounded alias-index `SSCAN` page and canonical | ||
| `XREVRANGE count=N` in one non-transactional redis-py pipeline. When that scan | ||
| returns cursor zero, its alias members and paired canonical page are both reused | ||
| by the same request. An empty member set completes the canonical-only response | ||
| in that one exchange. A complete non-empty member set enters compatibility | ||
| merge without scanning the alias index again or re-fetching the canonical page; | ||
| only retained alias pages are fetched in the next pipeline exchange. Reusing a | ||
| complete page does not bypass the existing fan-out ceiling: canonical stream plus | ||
| retained aliases may contain at most 1,000 distinct stream keys. A nonzero cursor | ||
| falls back to the bounded compatibility iterator and never treats an incomplete | ||
| alias scan as complete history. | ||
|
|
||
| When aliases exist, production redis-py clients prefetch a bounded slice from | ||
| every admitted stream before merging locally. The per-stream page size is | ||
| `min(event_count, 1000 // stream_count)`, with a minimum of one. When the first | ||
| UUID pipeline already fetched the canonical page, that page is trimmed to this | ||
| per-stream budget before retained aliases are fetched, and the alias-only second | ||
| pipeline fills the remaining stream pages. Thus resident compatibility pages | ||
| still stay within the same 1,000-entry product ceiling. If one stream supplies | ||
| more than its prefetched share, only that exhausted stream is refilled with | ||
| another bounded page. Minimal adapters without pipeline support retain the older | ||
| one-entry incremental merger. Both paths preserve complete retained history, | ||
| canonical-first equal-millisecond ordering, and the final 1..1000 buyer-facing | ||
| output budget. | ||
|
|
||
| ## Consequences | ||
|
|
||
| - Canonical reads retain every historical UUID spelling that actually exists. | ||
| - Startup alias writes use at most 128 pending `(index key, legacy stream key)` | ||
| tuples per production pipeline exchange instead of one awaited network write | ||
| per alias. | ||
| - The alias-free UUID buyer path admits compatibility metadata and fetches the | ||
| bounded canonical window in one redis-py pipeline network exchange. | ||
| - A complete retained-alias admission page is reused rather than discarded: the | ||
| one-alias compatibility fixture now needs two network exchanges, not three. | ||
| - Reused admission pages fail closed above 1,000 distinct canonical-plus-alias | ||
| stream keys before any retained-alias fan-out is issued. | ||
| - The retained-alias production path batches bounded history pages instead of | ||
| issuing one network command per stream and one additional command per returned | ||
| event. | ||
| - Compatibility prefetch retains at most 1,000 stream entries at a time before | ||
| output construction; skewed history refills only the stream that exhausted | ||
| its bounded page. | ||
| - Request-time records and calls remain bounded by the retained stream and output | ||
| limits; excessive alias cardinality is explicitly unavailable. | ||
| - Startup performs a cursor scan and must finish before readiness. | ||
| - The alias index is additional derived Valkey state and can be rebuilt from | ||
| retained activity keys. | ||
|
|
||
| ## Alternatives considered | ||
|
|
||
| - Enumerate common UUID spellings: rejected because PostgreSQL accepts more | ||
| forms than a finite hand-picked list would honestly cover. | ||
| - Await one `SADD` for every discovered alias: rejected because rollout/readiness | ||
| latency then adds one serial Valkey write round trip per historical alias even | ||
| though those writes are independent. | ||
| - Buffer every discovered alias and issue one unbounded pipeline at the end: | ||
| rejected because startup memory and command-buffer size would scale with the | ||
| entire retained keyspace rather than a fixed batch. | ||
| - Perform an alias-index lookup and only afterward issue the canonical | ||
| `XREVRANGE`: rejected because the normal UUID path then has at least two | ||
| sequential Valkey network waits even when the alias set is empty. | ||
| - Discard a complete non-empty first alias page and rescan it before compatibility | ||
| merge: rejected because it adds a deterministic network exchange and repeats | ||
| admission work without improving history correctness. | ||
| - Reuse a complete alias page without reapplying the total-stream ceiling: | ||
| rejected because 1,000 aliases plus the canonical stream would silently expand | ||
| request fan-out beyond the bounded contract formerly enforced by the iterator. | ||
| - Fetch one compatibility entry per stream and then await one `XREVRANGE` per | ||
| returned event: rejected because an authorized 50-event legacy-history panel | ||
| still pays roughly one sequential Valkey wait per result. | ||
| - Fetch `event_count` entries from every compatibility stream in one pipeline: | ||
| rejected because worst-case buffering becomes `event_count * stream_count`, | ||
| up to one million retained records under the existing limits. | ||
| - Scan the activity keyspace on each request: rejected because latency and work | ||
| would scale with unrelated posts. | ||
| - Drop legacy aliases: rejected because canonicalization would hide retained | ||
| authorized history. | ||
|
|
||
| ## Implementation evidence | ||
|
|
||
| - Review `5121693984` identified that the earlier read-budget test counted only | ||
| `XREVRANGE` and omitted the preceding alias-index `SSCAN` network wait. | ||
| - RED `7dcc5c16bc61393fc6a533e216baee8c43363894` counts both direct Valkey waits | ||
| and therefore rejects the earlier two-exchange canonical UUID path. | ||
| - Production repair `02417fcf92969ffe7924f2722b192f40ff953e0d` | ||
| pipelines bounded alias admission with the canonical page while preserving the | ||
| existing compatibility fallback. | ||
| - Test convergence `2b594942c12c37fe68459c02fbf47a2d11a727e4` models the pipeline as one | ||
| network exchange and retains the exact canonical stream/count assertion. | ||
| - Review `5121832019` identified the independent startup N+1 where alias | ||
| discovery awaited one `SADD` network write for every retained legacy key. | ||
| - RED `4cfa402acadb43c3f60e4dba7bad8917230b9460` rejects direct per-alias `SADD` | ||
| on a two-alias fixture while preserving exact durable members and added count. | ||
| - Production repair `c4a25ce1f03ee225874a56c3f102ffa8b48a0621` introduces a bounded 128-write | ||
| non-transactional pipeline batch with a compatibility fallback for minimal | ||
| adapters that do not expose redis-py pipeline support. | ||
| - Review `5121915874` identified the retained-alias buyer-path N+1 left after the | ||
| canonical fast-path repairs: initial probes were concurrent, but each emitted | ||
| event still triggered another awaited `XREVRANGE count=1`. | ||
| - RED `7d44315c81bd942db58af75ea9af9e9885c46dc5` requires a four-event, | ||
| two-stream compatibility read to preserve exact merge order without paying one | ||
| network exchange per event. | ||
| - Production repair `2ee36cf7844526a5f27803100f2b469e31573a98` batches bounded per-stream | ||
| history pages through one non-transactional pipeline and refills only an | ||
| exhausted stream, with total initial buffering capped at 1,000 entries. | ||
| - Review `5122107164` found that the retained-alias path still discarded the | ||
| complete first SSCAN page and canonical XREVRANGE, then repeated alias | ||
| admission and canonical retrieval before merging. | ||
| - RED `15746e2be27effca16c71f445f8d77f5d55fe6b8` requires the one-alias fixture | ||
| to preserve exact four-event ordering in exactly two network exchanges, with | ||
| pipeline batches `[2, 1]`. | ||
| - Production repair `07fa552f9fecbe1cda79a9fccc7eeacd82c4fba5` carries a complete first | ||
| alias page plus the canonical page into compatibility merge and fetches only | ||
| retained alias pages in the second pipeline exchange. | ||
| - Follow-up review `5122123572` found that this reuse path initially skipped the | ||
| iterator's existing total-stream fan-out guard. | ||
| - RED `a9c080c1f2fa030bd2b0cec0e2fb6b43e4328bbc` supplies a complete first page | ||
| with 1,000 aliases and requires fail-closed behavior after the first two-command | ||
| exchange, before a retained-alias pipeline is issued. | ||
| - Production repair `8ef3a42608739e5a32b5841e4a14494df5326a3f` reapplies the 1,000 distinct | ||
| canonical-plus-alias stream ceiling to reused admission pages. | ||
| - redis-py asyncio pipeline documentation specifies that pipeline commands are | ||
| buffered and executed together when awaited through `execute()`; transactions | ||
| remain optional and are not required for these independent read or set-write | ||
| commands. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.