From e92d25e03e76b699673e82802aaf020e89c02a0f Mon Sep 17 00:00:00 2001 From: BobbyAxerol Date: Mon, 19 Oct 2026 13:53:52 +0000 Subject: [PATCH] docs(v2.2.5): W12 decision record - TS writer proof package, verdict W12-A --- DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md | 14 ++ .../releases/v2.2.5/drafts/W12_DECISION.md | 205 ++++++++++++++++++ .../v2.2.5/drafts/W12_DECISION_EVIDENCE.json | 198 +++++++++++++++++ 3 files changed, 417 insertions(+) create mode 100644 upgrade/evidence/releases/v2.2.5/drafts/W12_DECISION.md create mode 100644 upgrade/evidence/releases/v2.2.5/drafts/W12_DECISION_EVIDENCE.json diff --git a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md index e4dc45db..1d85af62 100644 --- a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md +++ b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md @@ -76799,3 +76799,17 @@ locally; release image NOT rebuilt (the coordinator rebuilds rust together with io.weight. No new campaign now (Codex review E). - Unchanged: the v2.2.4 certificate's operational condition (measure and run with unrelated heavy stacks paused); IDR-128 stays OPEN. + +### DL v2.2.5 window: W12 decision (TS writer proof package) - 2026-10-19 (Claude) + +- Decision record `upgrade/evidence/releases/v2.2.5/drafts/W12_DECISION.md` (+ `W12_DECISION_EVIDENCE.json`). + **Verdict W12-A** (writer 2deb90b0, identity mount only; packet section 5 and rollback unchanged). +- Proof items: (1) writer binary main vs deployed: function-level only `LatencyAggregates::observe` (reviewed + latency.rs change) + alignment padding; Python closure diff fully classified; GHCR image NOT verified (registry + unauthorized in this session) -> not complete. (2) T1 runtime routes PASS: Python store = BAR/BOOK_DELTA/ + BOOK_SNAPSHOT (101 commits/s, ~14 ms each, errors 0), TRADE/QUOTE = native Rust checkpoints. (3) T1 failure-step + and cancellation tests added on TS `test/writer-t1-proof` 0c5148e1 (18 passed x3 locally); PR not opened (TS repo + not visible to the available token), CI not run. (4) component-level isolated A/B ABBA on the production trace: + 50 ms = -19 % commits/write IO, +59 ms ACK p50, +12.5 ms BOOK_DELTA apply p50; full-writer A/B not done. +- Host lock `claude-w12-writer-proof` 13:19:58-13:50:05Z, released on completion; containers `--rm`; no production + mutation. Status: proposal; owner/TS go needed only for a later W12-B. diff --git a/upgrade/evidence/releases/v2.2.5/drafts/W12_DECISION.md b/upgrade/evidence/releases/v2.2.5/drafts/W12_DECISION.md new file mode 100644 index 00000000..6e16222c --- /dev/null +++ b/upgrade/evidence/releases/v2.2.5/drafts/W12_DECISION.md @@ -0,0 +1,205 @@ +# DL v2.2.5 window: W12 decision record (TS writer slot) + +Status: proposal for the window owner, 2026-10-19 (Claude). **Verdict: W12-A**: keep writer image 2deb90b0 and switch +only its TLS identity mount, as in `MAINTENANCE_WINDOW_PACKET.md` section 5. W12-B (roll the writer to the TS +v2.0.0 release image) needs every proof item below to PASS. Item 2 passes. Items 1 and 3 are incomplete and item 4 +has only a component-level measurement, so W12-B is not taken in this window. + +Rule (coordinator + Codex review `p184-v225-bounded-window-review-20261019.md` section C): W12-B only when the release +image is final and has its closure proof, its T1 runtime-path proof, the T1 correctness tests and an isolated +interval 0 vs 50 ms A/B; otherwise W12-A. Machine-readable evidence: `W12_DECISION_EVIDENCE.json` (same folder). + +| Item | Result | Evidence | +|---|---|---| +| 1 Binary + image closure | **NOT COMPLETE** (binary: function-level PASS; GHCR image: not verified) | below, 1a-1c | +| 2 T1 runtime route proof | **PASS** | below, 2 | +| 3 T1 tests (Codex C list) | **PARTIAL**: tests pass locally; PR not opened, CI not run | below, 3 | +| 4 Performance A/B 0 vs 50 ms | **PARTIAL**: component-level isolated A/B done; full-writer A/B not done | below, 4 | + +## Inputs + +- Production writer: `market_data_service_p184w`, image `sha256:2deb90b0cc52…` (label revision TS 30f50a91), binary + sha256 `aefe6a47…b433`. Never touched; all production reads were `docker inspect`, `docker exec` reads, + `docker cp` of trace files and host log reads. +- Release candidate: `ghcr.io/bobbyaxerol/executorbroker:sha-3b4466521df2ba78e2a68a6ce1f6da819346fe1b`, digest + `sha256:9a251870cae21e0588f51d9db4f6fe47e293983e593496efd05fefffe9fc0ff9` (TS v2.0.0, main 3b446652). +- Host measurement lock `claude-w12-writer-proof` held 13:19:58-13:50:05Z (released on completion, not at the + deadline). It covered the binary rebuilds and the isolated A/B. Containers: `tsa-eq-*` (builder) and + `ts-p184-t1ab-*` / `ts-p184-w12-*`, all `--rm` with CPU caps. + +## 1. Binary and image closure + +1a. **Writer binary (Rust).** Recipe: the Dockerfile builder `ghcr.io/pyo3/maturin@sha256:e557fd4c…` (maturin 1.8.2, +rustc 1.84.1) and the Dockerfile's third cargo command, run on `git archive` of the revision. +- 30f50a91 -> 2deb90b0's binary: bit-identical rebuild (earlier proof). +- 30f50a91 -> 54c4dbaa (rustfmt only): only panic `Location` line/column words (180) and the build-id differ + (earlier proof). +- 54c4dbaa -> main 3b446652: the only writer source change is `latency.rs` `LatencyAggregates::observe` (the slice + cell is updated in place instead of through `Option::map`, then moved at its last use) plus a doc line in + `session.rs`. `Cargo.lock` is unchanged since 54c4dbaa, and the writer has no dependency on `trading-core`. +- Local rebuild of main: sha256 `448ad37e…7ec6`. Its size equals the deployed binary's (6 547 864 bytes). +- Function-level comparison. Both revisions were built unstripped and disassembled with objdump; addresses and + rip-relative displacements were normalised and ThinLTO `.llvm.N` suffixes removed. Of 4216 functions, 4214 are + identical and none exists on only one side. The two that differ: + - `market_data_writer::latency::LatencyAggregates::observe`: the reviewed change, 1813 -> 1817 instructions with a + different stack-slot assignment; + - `ring_core_0_17_14__p256_point_mul_base_vartime`: trailing alignment `nop` padding only. +- Stripped pair (deployed `aefe6a47` vs main `448ad37e`): the differing sections are `.text`, `.rodata`, + `.eh_frame(_hdr)`, `.gcc_except_table`, `.rela.dyn`, `.got`, `.data.rel.ro`, `.data`, `.init_array` and the + build-id. + - Classification: a code-address shift after the grown `observe()` (jump tables, FDEs, relocations, GOT and vtable + entries) plus the panic-location lines. This follows from the function-level result. + - It was **not verified word by word**, because the unstripped build changes symbol hashes and so is not + byte-equal to the stripped one. + +1b. **Python closure (git and deployed image).** +- The deployed image's 43-file writer closure (`scripts/p184_writer_release_build.py` `WRITER_FILE_ROOTS`) equals git + 30f50a91. So does the 56-file import closure of `services/market_data/main.py`, computed by an AST walk that + includes function-level imports. +- 30f50a91 -> 3b446652 import closure (57 files): one file added, eight changed. Every change was read. + +| File | Change | +|---|---| +| `adapters/market_data/cursor_group_commit.py` (added) | T1 store | +| `adapters/market_data/data_layer_v2.py` | T1 wiring | +| `shared/config.py` | adds `DATA_LAYER_V2_CURSOR_MIN_COMMIT_INTERVAL_MS` = 50 and `MARKET_DATA_V1_COMPAT_ENABLED` | +| `services/market_data/main.py` | V1 feed-lease renewal moves behind the V1 switch | +| `services/market_data/data_layer_bridge.py` | V1 pub/sub, V1 REST recovery and VN quotes move behind the V1 switch | +| `services/market_data/native_plan.py` | lint only (`zip(..., strict=False)` is the default behaviour) | +| `services/market_data/native_worker.py` | lint only | +| `services/market_data/trailing_observation.py` | lint only | +| `alpha_sdk/order_batch.py` | lint only (import order) | + +- In the 43-file set but outside the import closure: `native_handoff.py` and `scripts/p184_n1_cursor_handoff.py`. + Both are lint only. +- Runtime dependencies: `uv.lock`, `pyproject.toml` and `vendor/` are unchanged. The deployed image's interpreter is + CPython 3.10.22 with qdl-sdk 2.4.0. +- Behaviour delta of the V1 switch in production: the live writer logs `market active scope refresh failed + reason=ConnectError` every 30 s, because V1 lease renewal targets the stopped V1 Data Layer. Under v2.0.0 that + renewal is off. The scope update itself is unchanged. + +1c. **GHCR image: NOT VERIFIED.** +- `docker manifest inspect` of the digest returns `unauthorized`, and this session holds no credential for the TS + package. So nothing here compares the image's actual `/app` files, its `/usr/local/bin/market-data-writer` or its + site-packages against the local rebuild. +- `python:3.10-slim` in the Dockerfile is not digest-pinned, so the release image's base layer and interpreter may + differ from 2deb90b0's. +- To complete: pull by digest with a read credential, run `image_probe` (sha256 of `/app`, site-packages and + `/usr/local`) on both images, then check that the image binary equals `448ad37e…` and that the `/app` diff equals + 1b. + +## 2. T1 runtime route proof (production, read-only) - PASS + +- Cursor root `/app/state/data-layer-v2/cursors/fixed-workers-v1/`. Two key snapshots were taken 14 s apart + (12:53:56Z and 12:54:10Z). + - Python workers `count-4-worker-0..3`: BAR/1m, BOOK_DELTA and BOOK_SNAPSHOT keys advance (10 / 11 / 11 routes). + Their TRADE/QUOTE keys do not move; they are leftovers from before the native cutover. + - `native-4` (the Rust `CursorCommitter`, `native_plan.py:203`): TRADE and QUOTE advance (11 QUOTE, 10 TRADE). + - Process list: 4 Python workers plus one `market-data-writer` (native). Env `MARKET_DATA_NATIVE_FEEDS=TRADE,QUOTE`. +- Live store counters (`qdl_cursor_persistence`, 12:53:25-13:08:26Z, 901 s): + +| Worker | commits/s | ACKs per commit | store I/O per commit | pending peak | +|---|---|---|---|---| +| worker-0 | 27.9 | 1.15 | 14.4 ms | 7 | +| worker-1 | 24.5 | 1.11 | 14.2 ms | 7 | +| worker-2 | 20.8 | 1.03 | 13.6 ms | 4 | +| worker-3 | 28.2 | 1.12 | 14.4 ms | 7 | + + All four workers: errors 0. Total: 101 commits/s, 113 ACKs/s. +- ACK path: these feeds ACK inline (`data_layer_v2.py`, non-overlap branch `await acknowledge_checkpoint` before the + next batch). Overlapped ACK applies only to TRADE/QUOTE, which are native here. +- Consequence: under W12-B the 50 ms default would govern the BOOK and BAR routes. Each ACK holds that route's read + loop. +- The T1 setting lives in the shared builder, so every service built from v2.0.0 that streams through + `build_versioned_data_layer_client` gets it. This is outside W12 and noted for TS. + +## 3. T1 tests (Codex C) - PARTIAL + +- On main already (commit 0d1273f9): same-route overlapping ACKs; replace/new generation with older queued work; + file-fsync ENOSPC (single request); aclose while spacing; queue saturation with an unaffected route; replay after + restart never regresses; SIGKILL readback. +- Added on TS branch `test/writer-t1-proof` (from dev 883fc193, commit + `0c5148e1ec86960dcc8c02de14ef7c237c3e243a`, worktree `/home/bobby/.worktrees/ts-writer-t1-proof`): + - failure at each durability step of a coalesced batch with same-route overlap: file fsync ENOSPC, rename EIO, + directory fsync EIO. No request of the failed commit is ACKed, no temp file is left, and the last durable + cursor is kept. In the directory-fsync case the file may already show the applied but never-ACKed offsets, + which is never ahead of apply. Recovery by re-ACK; + - a caller cancelled while its ACK waits for the spacing is released only after its write is durable, and it does + not stall the rest of the batch; + - the TEST-only harness `scripts/p184_t1_cursor_ab.py`; + - plan journal entry E-A17. +- Local preflight: test image `local/trading-system-test:p184-r59` (python 3.10.21) with the vendored qdl-sdk 2.4.0 + wheel, `--network none`. Result: 18 passed, 3 times; the 4 new cases passed. +- No product code change was needed, so there is no v2.0.1. +- Open: + - the branch is pushed, but the PR into `dev` is not opened: the available API token cannot see the TS repository + (404). It can be opened at `https://github.com/BobbyAxerol/ExecutorBroker/pull/new/test/writer-t1-proof`; + - CI has not run; + - the SIGKILL test still does not model power loss at each fsync step. Fault injection now covers each step's + error path, not data loss in the page cache. + +## 4. Interval 0 vs 50 ms A/B (isolated) - PARTIAL + +**Apparatus.** `scripts/p184_t1_cursor_ab.py`, component level. It runs the real SDK 2.4.0 `FileCursorStore` +(interval 0, the builder's choice) against `GroupCommitFileCursorStore` (50 ms); the TS dev code equals main 3b446652 +for these files. Each route runs the writer's inline-ACK loop: a batch of at most 64 events, at most 20 ms wait, the +recorded apply, then an awaited ACK. One store and cursor file per worker, seeded to the production document size. +- Workload: the production writer's own delivery trace (4 workers, 22 BOOK routes, DL gateway-send arrival times, + recorded Redis apply durations), the same 300 s window for every arm (33 985 events), and a 20 s stall at 200 s + for catch-up. +- Container: `--network none`, 2 CPUs, order A1 B1 B2 A2, under the host lock. +- Calibration: arm A reproduces production. It gives 90 commits/s (production 101, which also includes BAR), + 1.17 ACKs per commit (production 1.03-1.15), 13.8-14.0 ms store I/O per commit (production 13.6-14.4) and + BOOK_DELTA arrival -> apply p50 of 24.9 ms (production gateway-send -> Redis reply p50 25.3-27.0 ms). + +| Metric (4 workers) | A1 (0) | A2 (0) | B1 (50) | B2 (50) | +|---|---|---|---|---| +| commits/s | 90.4 | 90.2 | 73.4 | 73.4 | +| ACKs/s | 105.8 | 105.8 | 104.2 | 104.2 | +| ACKs per commit | 1.17 | 1.17 | 1.42 | 1.42 | +| write syscalls (300 s) | 27 112 | 27 061 | 22 015 | 22 012 | +| container write IOs (cgroup) | 29 676 | 29 194 | 24 189 | 23 925 | +| container CPU s | 61.6 | 61.7 | 56.1 | 55.7 | +| host iowait share | 11.2 % | 11.3 % | 10.7 % | 10.8 % | +| pending p99 / peak | 3 / 6 | 3 / 6 | 5 / 6 | 5 / 6 | +| apply -> durable ACK p50 (worst worker) | 15.6 ms | 15.7 ms | 74.6 ms | 74.6 ms | +| apply -> durable ACK p99 (worst worker) | 32.1 ms | 32.8 ms | 99.0 ms | 98.5 ms | +| apply -> durable ACK max | 46.8 ms | 44.8 ms | 107.2 ms | 114.6 ms | +| BOOK_DELTA arrival -> apply done p50 (worst worker) | 24.9 ms | 24.9 ms | 37.6 ms | 37.3 ms | +| catch-up after 20 s stall (worst route) | 0.13 s | 0.14 s | 0.37 s | 0.32 s | + +**Reading.** +- The repeated arms agree (A1/A2 and B1/B2 within 1-2 %). +- At this workload T1 at 50 ms removes about **19 % of cursor commits and write I/O** (90 -> 73 commits/s, 29.4k -> + 24.1k write IOs) and about 9 % of the component's CPU. Host iowait barely moves. +- In exchange, the durable ACK becomes about **+59 ms at p50 and +67 ms at p99**, the inline BOOK_DELTA apply + latency about **+12.5 ms at p50** on the busiest workers, and catch-up after a stall rises from about 0.13 s to + 0.3-0.4 s. +- The reduction is small because most commits carry a single route's ACK (1.17 ACKs per commit): the inline-ACK + loops do not produce enough concurrent ACKs to coalesce. +- As Codex noted, 50 ms is not an upper bound on ACK latency (max 107-115 ms). + +**Not measured.** +- Steady-state inline p95/p99: in this run the injected stall dominates the tails. +- A full writer instance. Stream transport, projection and Redis apply are replaced by the recorded schedule. The + isolated N1/G12 apparatus has no BOOK publisher, and BOOK feeds are the only Python-store routes. +- Power-loss behaviour. + +**Implication for W12-B.** Even with complete proof, the measured trade-off (about -19 % write I/O for about ++12 ms BOOK apply p50 and about +60 ms cursor ACK) argues for +`DATA_LAYER_V2_CURSOR_MIN_COMMIT_INTERVAL_MS=0` on the writer (the builder then installs the plain SDK store, the +current production behaviour) or a T1 redesign before using 50 ms. This is a TS decision, recorded here because it +changes the W12-B env. + +## Decision and what W12-B would still need + +- **W12-A** for this window: the packet's W12 row and rollback are unchanged. +- W12-B later needs: + - GHCR image probe and `/app` diff (1c); + - the TS PR opened and its CI green (3); + - a decision on the interval (4), and, if 50 ms is kept, a full-writer or longer A/B with steady-state + inline-latency tails; + - the W12-B rollback (2deb90b0 + old identity) and its own evidence, as the packet requires. +- Side finding (DL tooling, not fixed here): `scripts/host_measurement_lock.py acquire` treats the agent's own + sibling shell as a foreign job when the `--purpose` text contains a `FOREIGN_PROCESSES` keyword such as "pytest". + The `pgrep -af` match covers the wrapper shell's command line, so such an acquire waits on itself. diff --git a/upgrade/evidence/releases/v2.2.5/drafts/W12_DECISION_EVIDENCE.json b/upgrade/evidence/releases/v2.2.5/drafts/W12_DECISION_EVIDENCE.json new file mode 100644 index 00000000..0b7e776d --- /dev/null +++ b/upgrade/evidence/releases/v2.2.5/drafts/W12_DECISION_EVIDENCE.json @@ -0,0 +1,198 @@ +{ + "schema": "dl.v225.w12-writer-proof-evidence.v1", + "created_utc": "2026-10-19", + "test_only_measurements": true, + "production_writer": { + "container": "market_data_service_p184w", + "image": "sha256:2deb90b0cc52eae7610fa44692f90df077e9db100ca259e38fd652a2dc1c5dc5", + "label_revision": "30f50a91dbd67ce2a482480be598cf509b25333e", + "binary_sha256": "aefe6a47dd2ec57d1ea425ccadc47beae736de172118a48e4dd2b7e0e6f7b433" + }, + "release_candidate": { + "image": "ghcr.io/bobbyaxerol/executorbroker:sha-3b4466521df2ba78e2a68a6ce1f6da819346fe1b", + "digest": "sha256:9a251870cae21e0588f51d9db4f6fe47e293983e593496efd05fefffe9fc0ff9", + "pulled_in_this_proof": false, + "reason": "registry returned unauthorized; no GHCR credential for the TS package in this session" + }, + "item1_binary": { + "local_rebuild_main_3b446652_sha256": "448ad37eb325c16128f500bec949328b4551e7e68bdb1d0ee1d485cc8bb57ec6", + "builder": "ghcr.io/pyo3/maturin@sha256:e557fd4c6a0bd13fc926f0a4a886f4143a7ea8a77593f9fd396d637472ab086f", + "size_bytes_equal": 6547864, + "fmt_commit_54c4dbaa_vs_deployed": "prior proof: only panic Location line/column words (180) and build-id differ", + "unstripped_function_diff_54c4dbaa_vs_3b446652": { + "functions": 4216, + "equal": 4214, + "differ": [ + "market_data_writer::latency::LatencyAggregates::observe (the reviewed latency.rs change; 1813 -> 1817 instructions)", + "ring_core_0_17_14__p256_point_mul_base_vartime (trailing alignment nop padding only)" + ], + "only_one_side": 0 + }, + "stripped_deployed_vs_main_sections_differing": [ + ".note.gnu.build-id", + ".rela.dyn", + ".text", + ".rodata", + ".eh_frame_hdr", + ".eh_frame", + ".gcc_except_table", + ".init_array", + ".data.rel.ro", + ".got", + ".data" + ], + "stripped_classification": "code-address shift after the grown observe() (jump tables, FDEs, relocations, GOT, vtables) plus panic-location lines; not verified word by word" + }, + "item1_closure": { + "deployed_image_43_file_writer_closure_equals_git_30f50a91": true, + "deployed_image_import_closure_56_files_equals_git_30f50a91": true, + "git_import_closure_30f50a91_to_3b446652": { + "added": [ + "adapters/market_data/cursor_group_commit.py" + ], + "changed": { + "adapters/market_data/data_layer_v2.py": "T1 wiring", + "shared/config.py": "T1 setting + MARKET_DATA_V1_COMPAT_ENABLED", + "services/market_data/main.py": "V1 lease renewal behind the V1 switch", + "services/market_data/data_layer_bridge.py": "V1 pub/sub, REST recovery, VN quotes behind the V1 switch", + "services/market_data/native_plan.py": "lint only (import blank line, zip strict=False = default)", + "services/market_data/native_worker.py": "lint only (blank line)", + "services/market_data/trailing_observation.py": "lint only (one statement per line)", + "alpha_sdk/order_batch.py": "lint only (import order)" + }, + "not_in_import_closure_but_in_43_files": { + "services/market_data/native_handoff.py": "lint only", + "scripts/p184_n1_cursor_handoff.py": "lint only (operator script, not run by the writer)" + } + }, + "ghcr_image_side": "NOT_VERIFIED" + }, + "item2_live_routes": { + "window_utc": "2026-10-19T12:53:25Z/13:08:26Z", + "python_store_routes": [ + "BAR/1m", + "BOOK_DELTA", + "BOOK_SNAPSHOT" + ], + "native_rust_checkpoint_routes": [ + "TRADE", + "QUOTE" + ], + "per_worker": { + "worker-0": { + "commits_per_s": 27.9, + "acks_per_commit": 1.15, + "io_ms_per_commit": 14.4, + "pending_peak": 7 + }, + "worker-1": { + "commits_per_s": 24.5, + "acks_per_commit": 1.11, + "io_ms_per_commit": 14.2, + "pending_peak": 7 + }, + "worker-2": { + "commits_per_s": 20.8, + "acks_per_commit": 1.03, + "io_ms_per_commit": 13.6, + "pending_peak": 4 + }, + "worker-3": { + "commits_per_s": 28.2, + "acks_per_commit": 1.12, + "io_ms_per_commit": 14.4, + "pending_peak": 7 + } + }, + "errors": 0, + "production_book_delta_gateway_send_to_redis_reply_ms_p50": [ + 25.3, + 27.0 + ] + }, + "item4_ab": { + "apparatus": "component-level: real SDK 2.4.0 FileCursorStore vs TS GroupCommitFileCursorStore (dev = main code), inline-ACK route loop, production delivery-trace schedule (4 workers, 22 BOOK routes, 300 s, 20 s pause at 200 s), --network none, 2 CPUs, order A1 B1 B2 A2, host lock held 13:19:58-13:50:05Z", + "arms": { + "A1": { + "interval_ms": 0, + "commits_per_s": 90.37, + "acks_per_s": 105.75, + "acks_per_commit": 1.17, + "pending_p99_max": 3, + "pending_peak_max": 6, + "ack_latency_p50_ms_max": 15.624, + "ack_latency_p99_ms_max": 32.057, + "ack_latency_max_ms": 46.787, + "book_delta_inline_done_p50_ms_max": 24.85, + "catchup_max_s": 0.128, + "write_syscalls": 27112, + "cgroup_wios": 29676, + "cgroup_wbytes": 677158912, + "cgroup_cpu_usage_s": 61.601, + "host_iowait_share": 0.112, + "events_replayed": 33985 + }, + "B1": { + "interval_ms": 50, + "commits_per_s": 73.36, + "acks_per_s": 104.23, + "acks_per_commit": 1.421, + "pending_p99_max": 5, + "pending_peak_max": 6, + "ack_latency_p50_ms_max": 74.614, + "ack_latency_p99_ms_max": 99.015, + "ack_latency_max_ms": 107.184, + "book_delta_inline_done_p50_ms_max": 37.557, + "catchup_max_s": 0.371, + "write_syscalls": 22015, + "cgroup_wios": 24189, + "cgroup_wbytes": 550297600, + "cgroup_cpu_usage_s": 56.062, + "host_iowait_share": 0.1069, + "events_replayed": 33985 + }, + "B2": { + "interval_ms": 50, + "commits_per_s": 73.35, + "acks_per_s": 104.19, + "acks_per_commit": 1.421, + "pending_p99_max": 5, + "pending_peak_max": 6, + "ack_latency_p50_ms_max": 74.64, + "ack_latency_p99_ms_max": 98.518, + "ack_latency_max_ms": 114.591, + "book_delta_inline_done_p50_ms_max": 37.316, + "catchup_max_s": 0.321, + "write_syscalls": 22012, + "cgroup_wios": 23925, + "cgroup_wbytes": 549154816, + "cgroup_cpu_usage_s": 55.657, + "host_iowait_share": 0.10798, + "events_replayed": 33985 + }, + "A2": { + "interval_ms": 0, + "commits_per_s": 90.2, + "acks_per_s": 105.76, + "acks_per_commit": 1.172, + "pending_p99_max": 3, + "pending_peak_max": 6, + "ack_latency_p50_ms_max": 15.737, + "ack_latency_p99_ms_max": 32.752, + "ack_latency_max_ms": 44.793, + "book_delta_inline_done_p50_ms_max": 24.863, + "catchup_max_s": 0.137, + "write_syscalls": 27061, + "cgroup_wios": 29194, + "cgroup_wbytes": 674140160, + "cgroup_cpu_usage_s": 61.678, + "host_iowait_share": 0.11343, + "events_replayed": 33985 + } + }, + "not_measured": [ + "inline latency p95/p99 in steady state (the injected pause dominates the tails)", + "full writer instance (Stream transport, projection, Redis apply replaced by the recorded schedule)" + ] + } +} \ No newline at end of file