Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 71 additions & 0 deletions CERTIFICATION_LEDGER.md
Original file line number Diff line number Diff line change
Expand Up @@ -2363,3 +2363,74 @@ Pinned at:
`certificate.json` status PASS, `amendments` = G4-A1, G4-A2, A3 (refreshed), A4 v2 (pre-registered, not needed), A5
(`post_hoc: true`); `gates_without_a5` G11 FAIL. It supersedes the 2026-10-18 certificate (`da2fd484...`).
`release.yml` is unchanged. No tag, no merge.

## DL v2.2.5 - 2026-10-20 production window + certificate (PASS under post-hoc P1; owner confirmation pending)

Compact release on top of v2.2.4: IDR-127 configuration, versioned KN compose + healthchecks, mesh mTLS generation
`20261019T084200Z`, Query image with the IDR-129 fix. Rust digests unchanged; TS writer unchanged (W12-A). Evidence:
`upgrade/evidence/releases/v2.2.5/` (CERTIFICATE.md, certificate.json, endpoint-report.json, RELEASE_NOTES.md,
MAINTENANCE_WINDOW_PACKET.md section 10, window receipts `window-20261020T0128Z/` with `SHA256SUMS`).

### 2026-10-20 - maintenance window (production, coordinator-run, owner GO): R1 R2 R4 PASS, W14 deferred

Pinned at:
- Query `qdl-v2-python:2.2.5-9b6747f` `sha256:753604f8abaa866e2de30c7ccaf36168986bc6991550ee326d53e51b83d1550a`
(query_kn_1/2); every other DL role on its v2.2.4 digest (rust `27c79b53` / `60f83a51` / `7c848aad`, Kafka
`9516fb76`, Redis `dfa18828`, bar edge rc14 `08f356a8` held);
- compose `deploy/compose/qdl-v2-stable-kn.yml` `50c1f014...` + `qdl-v2-stable-kn.v2.2.5.env` `c6770d5d...`;
- TLS generation `20261019T084200Z` (CA `610c0c6f...`, notAfter 2027-10-19); writer `2deb90b0` with the new identity,
STRICT.

- Outage 545 s (writer stop 01:31:04Z -> STRICT READY 64/64 01:40:09Z, incl. ~2 min coordinator wait-loop bug);
execution-ready 53 = baseline; K6 no-op (topics equal); Query max 188.3 MiB during both writer starts.
- R1 PASS (`r1.json` `eb47296d...`): 17 roles healthy, RestartCount 0, trust anchor moved (old identity refused by
Query rc 60 / Stream num=20), Kafka 3 brokers, URP 0, core 3 members.
- R2: window run FAIL only on kafka1-3 `cpus` (compose float32: 2249999872 vs 2.25); tool fixed (1e-6 vCPU tolerance,
`tests.test_kn_stable_compose` 12 OK, negative control fails on the old code); re-run read-only 03:00:55Z **PASS**,
863 checks, 0 problems (`r2-parity-rerun.json` `15415306...`).
- R4 60-min passive soak PASS (`r4-soak.jsonl` `82a4e9aa...`): READY 59/59, 0 auth/protocol, 0 restarts, Query
plateau 260-277 MiB; QUOTE p95 <=25 ms n=7261, TRADE p95 <=25 ms n=1800 (writer native_probe); served age
NOT_MEASURED. Portal running, Codex read-only: a regression sample, not a certification of the IDR-128 envelope.
- W14 monitor DEFERRED (GHCR read:packages missing).
- Re-run only if a pinned digest, the compose/env, the TLS generation or the writer image changes.

### 2026-10-20 - v2.2.5 certificate: status PASS under P1 (POST_HOC_PACKAGING_DECISION); not tagged

- Inherited from v2.2.4 (`certificate.json` `42412548...`, counted run 040311Z): G1-G4, G7-G10, G12-G14; G11 as in
v2.2.4 (PASS under A5, FAIL without). G6 PASS by delta proof (no eligibility code in the IDR-129 diff; execution-ready
53 = baseline).
- G5: the predeclared delta item "concurrent hot-read latency" (CERTIFICATE_SCOPE.md D3 (b)) was **not measured**.
P1 splits G5 by code path (hot path unchanged + maintenance hook ran 0 times in production; warmup path PASS by the
IDR-129 A/B) and records hot reads during a maintenance run (136-160 ms loop block, <= 1 per 120 s, only >= 448 MiB)
as NOT_MEASURED. `gates_without_p1` G5 NOT_MEASURED. **Owner confirmation of P1 is required before the tag**; the
alternative is the isolated hot-read A/B.
- SDK 2.4.1 wheel `9f79e032...` (same as v2.2.4, rebuilt byte-identical). `release.yml` unchanged. No tag.

### 2026-10-20 - Query A/B (G5 D3 (b) hot reads + DR warmup byte compare): CERTIFIED (isolated); v2.2.5 certificate PASS without P1

Pinned at:
- old Query `sha256:3e9e4d3a...` (v2.2.4), new Query `sha256:753604f8...` (v2.2.5); harness
`scripts/v225_query_hot_read_ab.py` with PREREG v2 committed at `0bba6385` before the run (session tool `543ddf18`,
BC re-run `b5b36e72`); one replica per arm at 1 vCPU / 2 GiB, private read-only copy of the market cache.

- Owner 2026-10-20: "Đo luôn rồi tag" (measure, then tag), then (~04:30Z) run without waiting for the foreign
correctness-only lock, interleave arms, judge paired deltas, record foreign activity. Session
`~/.local/state/qdl-v2/releases/v2.2.5/query-ab-20261020c` 05:04-05:50Z: verdict **PASS**
(`query-ab-20261020/receipt.json` `d3899b22...`).
- Hot reads (TS EXECUTION QUOTE/TRADE snapshots, open loop 10/s) during 6 x 64-warmup bursts per arm, A1 B1 A2 B2:
p95 186.0 / 122.9 / 173.1 / 152.4 ms, p99 437.8 / 391.7 / 399.4 / 406.6 ms; paired mean d_p95 -41.9 ms, d_p99
-19.5 ms (bounds +18.0 / +41.9).
- New image maintenance runs 131 / 147 ms inside the scored windows; overlapping reads 7.9 / 44.7 ms.
- Non-200 only 503 DATA_NOT_READY (HNX TRADE without a latest entry), 5.1 % vs 5.0 %.
- Concurrency: other agents' test containers running, no lock file present, load1 8.2-13.3.
- DR warmup byte compare: 65 TS r11 requirements x 2 passes, 130 valid triples; status, content-type and
Content-Length equal; bodies byte-identical after masking the per-request paths that also differ between two
requests to the same old replica. `byte-compare.json` `41ebcc3c575e3fb1cb37ee069cd5eff156bb0ed7ef0dd4b0b5becc51e22ae7e4`
PASS.
- Not counted: attempts 1 (04:50Z, nested client-CA bind dropped) and 2 (05:02Z, volume subpath ignored) measured
nothing.
- Certificate re-issued: G5 PASS by complete delta proof; P1 superseded (never used); `gates_without_a5` keeps G11
FAIL as in v2.2.4. Re-run only if either Query image, the warmup/snapshot code path or the TS r11 manifest changes.
- P1b (coordinator, same session 06:10-06:13Z): alpha DR02/DR03 read shapes (alpha-binance r16 + alpha-okx r15 BAR 1m:
instrument pages, warmup 5/700/10,000, 10,001 typed refusal, universe batch 200, mixed batch 10), 24 shapes x 2
passes: PASS, byte-identical outside per-request paths (`byte-compare-alpha.json` `c884e495...`).
100 changes: 100 additions & 0 deletions DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -77035,3 +77035,103 @@ locally; release image NOT rebuilt (the coordinator rebuilds rust together with
tuning campaign) in `CERTIFICATE_SCOPE.md` section 3b and the new `drafts/RELEASE_NOTES.md`.
- Status: the window is ready for go once this PR is merged (CI green); production-ready only after the window's
R1-R4. Next: coordinator schedules the window per packet section 9.

### DL v2.2.5: window result, R2 parity tolerance fix, certificate and release packaging - 2026-10-20 (Claude)

- Coordinator 2026-10-20: package v2.2.5 after the production maintenance window (coordinator-run, owner GO,
2026-10-20 01:28-02:54:45Z, forward path, no rollback). Branch `docs/v225-release-package` from origin/dev aa89d83
(worktree `/home/bobby/.worktrees/data-layer-dev`). Production read-only in this slice (docker inspect/logs/stats,
one compose-parity run); no host lock taken (none needed: docs, a unit test, a read-only parity check).
- Window facts (receipts copied, non-secret only, to `upgrade/evidence/releases/v2.2.5/window-20261020T0128Z/`, 19
files, `SHA256SUMS`; each file grepped for keys/passwords/tokens: none; `rollback/` and `writer/*.compose.json`
not copied): outage 545 s (writer stop 01:31:04Z -> STRICT READY 64/64 01:40:09Z, incl. ~2 min coordinator
wait-loop bug at W10); Query READY with the new TS identity 01:33:03Z, old identity refused; K6 topics pre == post;
Query max 188.3 MiB during both writer starts; R1 PASS; R4 60-min passive soak PASS; W14 DEFERRED (GHCR
read:packages missing); portal running, Codex read-only.
- R2 fix: `scripts/kn_versioned_compose.py` `parity` compared `cpus` exactly; compose passes `cpus: 2.25` through
float32 to NanoCpus 2249999872, so kafka1-3 failed with 1.3e-7 vCPU. New `CPUS_TOLERANCE = 1e-6` + `cpus_equal`,
`same(..., equal=...)` for that field only. Tests: `tests/test_kn_stable_compose.py` new class
`KnComposeParityCpusTest` (parity() against a faked docker inspect: float32 cap PASS; 2.24/2.26/2.0 vs 2.25 FAIL
with exactly one problem; tolerance bounds). `python3 -B -m unittest tests.test_kn_stable_compose`: 12 OK. Negative
control: the float32 test FAILS against `git show HEAD:scripts/kn_versioned_compose.py`. Re-run read-only against
live 2026-10-20T03:00:55Z with the v2.2.5 env + new generation + kn-secrets: **PASS, 863 checks, 0 problems, 6
intended deltas** (`r2-parity-rerun.json` `15415306...`).
- SDK: `scripts/build_qdl_sdk_release.py` rebuilt into the scratchpad: wheel `9f79e032...`, `.cdx.json`, `.release.json`
byte-identical to v2.2.4 (`git diff b5ab3c2 -- qdl_sdk generated contracts scripts/build_qdl_sdk_release.py`
empty), so `release.yml`'s wheel check holds with the v2.2.4 sha.
- Certificate (`upgrade/evidence/releases/v2.2.5/CERTIFICATE.md`, `certificate.json`, `endpoint-report.json`):
status **PASS under P1 (POST_HOC_PACKAGING_DECISION, owner confirmation required before the tag)**. Finding (E2):
the predeclared G5 delta item D3 (b) "concurrent hot-read latency" was never measured (the IDR-129 harness drives
warmups only; arm H `H.out` `643dcc60...`). P1 splits G5: hot path INHERITED (request path unchanged by the
IDR-129 diff; the maintenance hook ran 0 times in production since the window: `docker logs --since
2026-10-20T01:33:00Z` grep `kind=memory_maintenance` 0/0, RSS 228/234 MiB), warmup path PASS by the A/B (RSS
361-445 vs 472-743 MiB, warmup p50 6.6-8.3 vs 6.4-8.7 s, 429 991-1118 vs 992-1135 per burst), hot reads during a
maintenance run NOT_MEASURED (136-160 ms loop block per run, <= 1 per 120 s, only >= 448 MiB). Without P1:
`gates_without_p1` G5 NOT_MEASURED -> not PASS. Other gates: G1-G4, G7-G10, G12-G14 INHERITED from v2.2.4
(certificate `42412548...`); G11 inherited with A5 (FAIL without); G6 PASS by delta proof (no eligibility code in
the diff, execution-ready 53 = baseline).
- Limits recorded: P1; IDR-128 supported load (envelope measured with heavy stacks paused; window ran with the portal
running); inherited G11/A5, G14, IDR-108; served age / delivery lag / end-to-end NOT_MEASURED passively; W14
deferred (no watchdog); single host; old-identity containers (packet 5c) pending owner cleanup approval; R2 window
FAIL was a tool defect; R4 memory "first 10 min" wording not met literally (plateau 260-277 MiB from ~02:02Z).
- Files: drafts moved to `upgrade/evidence/releases/v2.2.5/` (CERTIFICATE_SCOPE.md, MAINTENANCE_WINDOW_PACKET.md
+ section 10 window result, RELEASE_NOTES.md final, W12_DECISION.md/.json); docstring paths in
`scripts/kn_window_writer.py` and `scripts/kn_mesh_rotation_rehearsal.py`; CERTIFICATION_LEDGER.md v2.2.5 entry.
- Status: implemented and tested locally (tool fix); v2.2.5 runtime production-deployed, certificate PASS under P1,
NOT tagged. Open: owner confirmation of P1 (or the hot-read A/B under the host lock), W14 (TS read:packages token),
old-identity cleanup approval, PR CI. Next: PR into dev (coordinator merges), then the dev -> main release PR;
the coordinator creates the tag.

### DL v2.2.5: G5 D3 (b) measured by an isolated Query A/B; DR warmup byte compare; P1 superseded - 2026-10-20 (Claude)

- Owner via the coordinator: option (b) "Đo luôn rồi tag". At ~04:30Z the owner also said: do not wait for the
Codex lock, because it is correctness only. Run concurrently, interleave the arms, judge on paired old-vs-new
deltas, record foreign activity, and touch nothing of theirs.
- Waiting: from 03:49Z I queued an acquire behind the Codex lock. Codex extended that lock twice, to 05:14:50Z. The
acquire was stopped at 04:49Z on the owner instruction, after which no lock was taken or touched by me. The lock
file was absent during the counted session.
- Tool: `scripts/v225_query_hot_read_ab.py` (new; reuses the IDR-129 harness probe).
- PREREG v1 `7d28d9cb` was never run. PREREG v2 `0bba6385` is the concurrent, interleaved, paired version and was
committed before any run.
- Rules:
- HR1: paired mean d_p95 <= max(10 % of A, 10 ms); paired mean d_p99 <= max(10 % of A, 25 ms).
- HR2: >= 1 maintenance run overlapped by reads; if not, the verdict is INCONCLUSIVE.
- HR3: overlapping reads <= run + p99 + 50 ms.
- HR4: transport and 5xx errors on B no higher than A beyond the prereg allowance.
- HR5: non-200 share on B <= A + 1 pp.
- BC: A/B/A' triples.
- Fixes during the session, rules unchanged:
- `543ddf18`: keep nested binds and volume subpaths of the production Query spec. Attempts 1 and 2 measured
nothing.
- `b5b36e72`: compare-only re-run mode and path-exact masking in the analysis. The BC driver could not write its
raw record, so the BC phase was re-run in the same session after the arms.
- Result (session `query-ab-20261020c`, 05:04-05:50Z): **PASS**.
- Paired mean d_p95 -41.9 ms, d_p99 -19.5 ms.
- Maintenance runs 131 / 147 ms; 2 overlapping reads.
- 503 DATA_NOT_READY 5.1 % vs 5.0 %.
- BC: 65/65 requirements x 2 passes, byte-identical outside the per-request paths (request_id, stream_cursor,
per-row cursor, freshness_ms, provider_session_liveness_ms).
- Receipts are in `upgrade/evidence/releases/v2.2.5/query-ab-20261020/` (`SHA256SUMS`). `byte-compare.json`
`41ebcc3c...` is the DR delta-proof input for TS.
- Certificate re-issued: G5 PASS by complete delta proof; P1 removed and recorded as superseded. Updated:
- `CERTIFICATE.md`
- `certificate.json`
- `endpoint-report.json`
- `RELEASE_NOTES.md`
- `CERTIFICATE_SCOPE.md` header
- `CERTIFICATION_LEDGER.md`
- Cleanup: every `qdl-hr-*` container (`--rm`) and network `qdl-hr-net` removed; 0 left (`docker ps -a` / `network
ls` by label). No image was built. Production was read-only: docker inspect, the read-only cache copy and
docker logs.
- Status: v2.2.5 certificate PASS, not tagged. Next: PR #48 CI and merge (coordinator), then the dev -> main
release PR.
- P1b (coordinator, owner-approved DR delta proof): the BC rule was extended to the TS DR02/DR03 alpha read shapes
(`--alpha`, harness `4e87da2b`). The shapes:
- alpha-binance r16 with its new-generation leaf and the registered r135 JWT;
- alpha-okx r15 with its r135 leaf and JWT;
- per consumer: instrument pages, warmup 5 x 5 symbols, 700, 10,000, a 10,001 refusal, a universe batch of 200 and
a mixed batch of 10.

Same session, 06:10-06:13Z, 24 shapes x 2 passes: **PASS**. Statuses: 44 x 200 and 4 x 400 typed refusals, equal on
both images. Receipt `byte-compare-alpha.json` `c884e495...`, copied to the release folder. 0 `qdl-hr-*` containers
and networks left.
2 changes: 1 addition & 1 deletion scripts/kn_mesh_rotation_rehearsal.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"""Isolated rehearsal of the v2.2.5 maintenance window: atomic mesh mTLS rotation + versioned-compose roll.

Purpose
Run the window procedure of ``upgrade/evidence/releases/v2.2.5/drafts/MAINTENANCE_WINDOW_PACKET.md`` on a
Run the window procedure of ``upgrade/evidence/releases/v2.2.5/MAINTENANCE_WINDOW_PACKET.md`` on a
disposable copy of the KN topology before production sees it, and measure it:

A bring the mini-stack up from the committed compose (``deploy/compose/qdl-v2-stable-kn.yml``) on the OLD
Expand Down
18 changes: 15 additions & 3 deletions scripts/kn_versioned_compose.py
Original file line number Diff line number Diff line change
Expand Up @@ -385,6 +385,17 @@ def image_id(ref: str) -> str:
return json.loads(sh(["docker", "image", "inspect", ref]))[0]["Id"]


# docker compose hands ``cpus`` to the engine through a float32, so 2.25 arrives as NanoCpus 2249999872
# (1.3e-7 vCPU short; DL v2.2.5 window R2, 2026-10-20). Any real cap change is >= 0.01 vCPU.
CPUS_TOLERANCE = 1e-6


def cpus_equal(live_nano_cpus: int, compose_cpus) -> bool:
"""True when the live NanoCpus and the compose ``cpus`` value are the same cap within CPUS_TOLERANCE."""

return abs(live_nano_cpus / 1e9 - float(compose_cpus or 0)) <= CPUS_TOLERANCE


def parity(compose: Path, env_files: list[Path]) -> dict:
cfg = compose_config(compose, env_files)
problems, deltas, checked = [], [], 0
Expand All @@ -393,10 +404,10 @@ def parity(compose: Path, env_files: list[Path]) -> dict:
live_env_clear = dict(e.split("=", 1) for e in json.loads(sh(["docker", "inspect", f"{PROJECT}-{role}-1"]))[0]["Config"]["Env"])
svc = cfg["services"][role]

def same(field, a, b):
def same(field, a, b, equal=None):
nonlocal checked
checked += 1
if a != b:
if not (equal(a, b) if equal else a == b):
problems.append(f"{role}.{field}: live={_short(a)} compose={_short(b)}")

same("image", live["image_id"], image_id(svc["image"]))
Expand All @@ -413,7 +424,8 @@ def same(field, a, b):
svc.get("command"))
same("user", live["user"] if live["user"] != live["image_user"] else None, svc.get("user"))
h = live["host"]
same("cpus", h["NanoCpus"] / 1e9, float(svc.get("cpus", 0)))
same("cpus", h["NanoCpus"] / 1e9, float(svc.get("cpus", 0)),
equal=lambda _a, b: cpus_equal(h["NanoCpus"], b))
same("mem_limit", h["Memory"], int(svc.get("mem_limit", 0)))
same("memswap_limit", h["MemorySwap"], int(svc.get("memswap_limit", 0)))
same("read_only", bool(h["ReadonlyRootfs"]), bool(svc.get("read_only", False)))
Expand Down
2 changes: 1 addition & 1 deletion scripts/kn_window_writer.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"""W12-A of the v2.2.5 window: TS writer composes for the TLS identity switch, readiness check, passive soak.

Purpose
The v2.2.5 window (``upgrade/evidence/releases/v2.2.5/drafts/MAINTENANCE_WINDOW_PACKET.md`` W12) keeps the
The v2.2.5 window (``upgrade/evidence/releases/v2.2.5/MAINTENANCE_WINDOW_PACKET.md`` W12) keeps the
production TS writer image (2deb90b0) and changes exactly one thing for good: the host source of its mesh TLS
identity mount ``/run/secrets/qdl-v2`` moves to the new generation. Because the window outage exceeds the
writer's 120 s execution-surface TTL, the writer first starts once with ``MARKET_DATA_NATIVE_REQUIRE_RESUME=0``
Expand Down
Loading
Loading