Repository navigation
Release v2.2.3 (Data Layer, P18.4 phase D) - #32
Merged
Merged
Conversation
…policy, quota log, SPOF runbook - IDR-040 fix 1: a Binance REST refresh proves a quiet book only when the stream already covers it (lastUpdateId <= last_sequence); REST_AHEAD_OF_STREAM otherwise - no publication, no re-stamp. - IDR-002: consumer route binding V3 seals the manifest request policy per product (additive; manifest digest and JWT revision unchanged). - IDR-029: quota exhaustion logged once per consumer and minute. - docs/runbooks/kn-single-points-of-failure.md: dependency map and limits.
…1-D3 TS Phase A closed on candidate r59b (93/93, strict verifier PASS); D2 tolerant SDK response readers in this release, D1 heartbeat deferred, D3 HA separate. Lists remaining Codex work and close conditions per item.
…zation) to the DL closure TS measured 0 overlapping / 18-21% crossed book-vs-quote reads with book age p50 ~730 ms; the core materializes BOOK_SNAPSHOT on a 1000 ms timer only. Fix belongs in the DL (top-change trigger with a floor), measured for resources before release.
…t path Single owner-approved runtime change: binance_bar_edge to the v2.2.2 Python image (fba8863), ending the Binance 3d overlap retry storm (0.56 vCPU, ~850 req/s per broker). DL observed 4.37/4.30 mean vCPU in two 300 s windows. Product-path coherence 120/120 per venue on current cores with TS IDR-076.
… on final digests
… (PREPARED_NOT_PUBLISHED) Scorecard on source bbfb3be, DL python e0252f4c / rust 64468c72, writer 34202753: PASS G5; FAIL G2 (TRADE BINANCE p95 539.1 ms in trace window 1); DEFERRED G13; NOT_MEASURED G1 G3 G4 G6 G7 G8 G9 G10 G11 G12 G14. SDK 2.2.0 wheel built twice from git archive bbfb3be, byte-identical, sha256 d8226f50. Adds scripts/p184_trace_feed_transit.py (G1/G2/G7 from writer delivery traces) with tests. No runtime change, no push, no tag.
… and attested, per-container provenance), --query-code image; final-image re-run packet
… by measurement (OKX +10 s, Binance +5..+24 s at 16:00Z)
…SIGTERM/SIGHUP/SIGINT run evidence and teardown
…er dumps beside their evidence; monitor records its exit reason
…lock load windows; trace analysis counts duplicate applies and loads slim G6: every view the Query endpoint returns to the stage-100 driver is classified BEFORE client validation against the frozen per-feed policy (TRADE 3000, QUOTE/MARK/BOOK_DELTA 2000, BOOK_SNAPSHOT 60000, BAR1m 180000 ms; other BAR intervals their sealed requirement): ELIGIBLE, ELIGIBLE_QUIET (OBSERVE quiet continuity with a live session), NOT_ELIGIBLE or UNSAFE (eligible although the policy rejects it). Exact counts/max per binding, bounded reservoir for p50/p95, merged across workers into receipt g6_view_quality. No change to the request mix, the validation or the gates. Receipt adds load_window_unix_s and BURST/RECONNECT wall-clock windows so host probes score the same window. Trace analysis: duplicate_applied_event_id (G10, canonical event_id APPLIED twice) and a field-projected loader so a 40-min campaign (~250 parts) fits in memory; the MARK script loads only MARK records.
…is sampler, G9/G10 writer counters, G11 restart probe, G14 table, runner and G1-G14 scorer One campaign run of the frozen protocol (stage-100 final run with BURST, SLOW_READER and RECONNECT windows, >= 30 min) with every probe the v2.2.3 scorecard listed as missing: - G3: MGET poll of the TS last_bar keys, close -> first visibility per slice; expected closes never seen are counted with their lower bound. - G8: PING RTT, INFO commandstats deltas per 10-min window, new SLOWLOG entries, MONITOR client count, on the Redis the writer projects into (TRADING_MARKET_REDIS_URL = redis_marketdata DB1, read from the writer). - G9/G10: writer heartbeat (demanded/session/execution, purpose not_ready rows by reason; quiet TRADE excluded) and typed TRADE log signals (CURSOR_EXPIRED, SlowConsumerError, continuity/SNAPSHOT_REPLACED). No dedicated TRADE integrity counter exists in the writer or SDK. - G11: waits for a coordinator-triggered writer restart; outage start, fault cleared, first valid apply and caught-up per slice; rewind and cursor-not-ahead checks (cursor offsets read without tokens). - G14: per-role mean (cgroup) / p95 / max CPU and max RSS vs limit; the campaign profile adds binance_bar_edge and stable_redis. - p184_gate_campaign.py run|score: read-only preflight (refuses without a long writer trace), PID-scoped children with deadlines and SIGTERM summaries, G4 windows incl. a UTC :00/:30 boundary window, trace analysis in a --rm --network none container, scorecard with PASS/FAIL/NOT_MEASURED where a measurement outside the protocol never passes. All Redis access is an allowlisted read-only RESP client (no MONITOR, no writes).
…d failed attempts
…ution, funding evidence
…TS EXECUTION probe The stage-100 workload reads as ALPHA; the collector classified those reads against the TS D3 EXECUTION table and reported 676 false UNSAFE views. Each read now carries its policy basis, only D3_EXECUTION rows decide the gate, and a read-only probe inside the writer supplies them for the whole load.
…budget The live budget was charged the full decoded weight per subscriber although dispatch shares one Arc<LiveRecord>; a short stall exhausted the 32 MiB budget and 84 streams with empty queues were closed at once (2026-10-10 17:22:48Z). The default per-subscription cap becomes half of the replica total.
….2.1) A fixed start put every client's streams on the first target: in the rc8 campaign stream_kn_1 served all 244 streams and stream_kn_2 none, so one replica's live budget carried the whole load.
…g the lease expiry The INFLIGHT_CAPACITY retry hint is the earliest lease's maximum expiry (30 s). The bar edge slept it, so an hour-boundary cycle stalled ~30 s and most bindings failed their deadline at once (G4, 2026-10-10 18:00Z).
…d grade Funding was anchored at its oldest settlement row and every alpha funding read scored UNSAFE; non-MARK products use the newest observation, and only an EXECUTION MARK read uses provider confirmation, as the Query service does.
All 12 roles run python a0c3649d / rust 96a2c912 (IDR-039 closed). Final campaign 4: PASS G1 G3 G5 G6 G7 G9 G10 G11 G14; FAIL G2 G8 (writer TRADE burst throughput, owner moved it to TS phase E); NOT_MEASURED G4 G12; G13 41/41 DL cells, TS groups deferred. Kafka cleaner throttle, SDK 2.2.1, local CI and every earlier campaign recorded. Status PREPARED_NOT_PUBLISHED.
…3 after recovery The isolated source was BOOK-only, so G12 could only be NOT_MEASURED. Adds TEST_ONLY TRADE, QUOTE, BAR 1m and MARK/INDEX lanes built from the production acquisition mapping, scores every part in the post-catch-up window, and keeps container exit/OOM events as evidence.
The DR08 Query built its cache reader from production topic defaults, so every isolated BAR warmup missed its checkpoint and the TS clones never started BAR. Also keeps the writer clones' stderr in the evidence.
…and retryability KnMarketCacheQueryBackend.history_many stores QueryBackendError per item; the batch route mapped it to a non-retryable INTERNAL_ERROR, hiding retryable DATA_NOT_READY results (seen on isolated BAR warmup, 2026-10-11).
…e a fault NOT_MEASURED Each 1 s sample's Kafka admin calls run in a supervised child with a hard timeout; a SIGSEGV/hang becomes a typed sample_error row naming the call instead of stopping the run (exit 139 after a broker stop, 2026-10-11). A monitor gap over 5 s around catch-up scores the fault NOT_MEASURED.
…K 2.3.0) Additive wire fields SubscribeRequest.max_records_per_response and SubscribeResponse.records; the gateway packs already-queued TRADE/BOOK records without waiting or crossing a control, and old clients see the unchanged single-record shape. Client gRPC cost per record drops ~13x at 64 per response. Phase E step 3.
… contracts With sdk_from_source only qdl_sdk was mounted over a 2.2.2 image, so SDK 2.3.0 read SubscribeResponse.records from an old protobuf and every alpha stream failed (AttributeError, campaign 2026-10-11 10:37Z).
…IDR-088) A quiet key's state-partition watermark can trail the head an earlier ring read recorded; once the ring evicted the key, Query refused the very same record as PRIMARY_BEHIND and the TS writer's slice dropped (stage 100, 2026-10-11 11:37Z, OKX BNB TRADE quiet > 120 s).
…child (IDR-087) AdminClient.list_consumer_group_offsets segfaulted inside librdkafka 2.15.0 during a broker stop (G12 monitor evidence); the bar edge called it in-process. A crash or hang now closes the history gate (fail-safe) instead of killing the bar edge, and the next call respawns the child.
All 12 roles run python ccbea12e / rust 6afdc155. Stage-100 campaign (2026-10-11T12:35Z): PASS G1 G2 G3 G5 G6 G7 G8 G9 G10 G11 G14; G12 PASS (isolated, every part); G4 NOT_MEASURED (rows without a frozen baseline); G13 41/41 DL cells, TS groups deferred. SDK 2.3.0, local CI and every earlier campaign recorded. Status PREPARED_NOT_PUBLISHED.
…81fd67) certificate.json (qdl.scoped-release-certificate.v2, status PASS under the stated limits) and endpoint-report.json from the stage-100 step-2 campaign; the release workflow's certificate and SDK checks pass locally (wheel b9c74157 reproducible across five Python environments).
P18.4 phase D: Data Layer v2.2.3 release candidate (rc13, 481fd67)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release PR for Data Layer v2.2.3: dev at 5c51a42 (merge of #31, head c9f53d1, all required CI green).
Evidence and scoped certificate (status PASS): upgrade/evidence/releases/v2.2.3/. Tag v2.2.3 is created on main after this merges; the release workflow verifies main ancestry, the certificate and the SDK wheel hash before it publishes.