Repository navigation
[PoC] speed up transaction-status reads: ledger hint on getTransaction, WebSocket status push #905
Description
TL;DR
Prototype two ways to shrink the time a client spends learning its transaction's status, and measure what each gets us:
- Ledger hint on
getTransaction. The client knows the latest ledger at submit time. Pass it, and RPC searches only ledgers above it instead of the full retention range. - WebSocket status push. The client does not poll at all. RPC matches each ingested transaction against subscribed listeners and pushes the status over the same socket the submission used.
Why
Phase 3 declares a 2 s end-to-end budget:
| Slice | Target |
|---|---|
| Block time × 3 closes (600 ms) | 1800 ms |
| Tx submission | 60 ms (50 ms network RTT + 10 ms in RPC/Core) |
| Ingest | 80 ms |
| Read to client | 60 ms (50 ms network RTT + 10 ms in RPC) |
Ingest currently misses the 100 ms target on every profile and both hardware classes (min box: SAC 858 / Soroswap 192 / OZ 149 ms; c6id.8xlarge: 749 / 141 / 112 ms). Shrinking the read slice's real cost, and taking the read load off the polling path entirely, gives ingest room on the same hardware. This would help shrink the 50 ms network RTT down to 25ms for tx submission (only consider send) and for read (only consider push time).
Candidate A — ledger hint on getTransaction
Today. The getTransaction request carries only hash and format — no ledger parameter exists. On the full-history read path, TxReader.GetTransaction (cmd/stellar-rpc/internal/fullhistory/storage/stores/txhash/read_assembly.go:49) scans the hot hash index, then every cold per-chunk index in turn. Cold indexes are MPHF-fingerprinted, so a hit is only a candidate that the ledger store must verify. The scan is linear in the number of chunk indexes and does not use ledger bounds, even though ColdReader already exposes MinLedger()/MaxLedger() (txhash/cold_reader.go:70,74).
The change. Accept an optional ledger hint. A transaction submitted after ledger N can only land in ledgers > N, so the reader skips every index whose range ends at or below N — for a just-submitted transaction, that is the hot store and nothing else. Two API shapes to evaluate in the PoC:
- an explicit "recent only" flag, or
- pass the ledger and let RPC decide where to look (falls back to a full scan on a miss, so a wrong hint degrades to today's behavior instead of a wrong answer).
The hot store exposes only Get (txhash/hot_store.go:100) and needs range accessors for the pruning decision.
Candidate B — WebSocket status push
Today. RPC speaks JSON-RPC over HTTP only. The repo has no WebSocket endpoint, no subscription mechanism, and no WS dependency. Clients learn their status by polling getTransaction.
The change. A WebSocket endpoint where a client submits a transaction and subscribes to its hash in one connection. On ingest, RPC checks each transaction against the subscribed hashes and pushes the result. Polling disappears: the read cost per transaction becomes one hash-set lookup inside ingest plus one socket write, and the client learns the status as early as RPC can know it.
What the PoC measures
Run against a Phase 3-style paced load (600 ms closes, the three target profiles):
getTransactionservice time with and without the hint, on the full-history reader, as the cold index count grows.- Push latency: ingest commit → client receives the status over the socket.
- The prize: ingest p99 on the same box while status reads run, per mechanism — does the freed read cost show up in ingest?
- For push: cost of the subscription match inside the ingest loop at Phase 3 transaction rates.
Done when
Both mechanisms are prototyped and measured under Phase 3 load, and the results support a recommendation: which (if either) to productionize, the API shape, and the measured effect on the read slice and on ingest p99.
Related
- Investigate and fix the issues surfaced by the O3 ingestion load test #800 — fixes to the existing ingest path (latency tail, memory, RocksDB config). This PoC attacks the same Phase 3 gap from the read side; the two do not overlap in code.
- Wire the v2 stores into the API handlers and support both v1 and v2 #772 — full-history read-server wiring; the ledger hint lands on that path.
- Full-history query benchmark commands — reads across all data types #856 / Read-endpoint SLAs under ingestion load #870 — query benchmarks; they measure the read slice this PoC shrinks.
- Benchmark transaction-submission latency #869 — tx-submission benchmark; its in-RPC p99 (~5 ms) is the submission half of the same socket in the push design.
- Run the Phase 3 benchmark campaign — ingest and reads at 600 ms blocks #874 — Phase 3 campaign; source of the ingest numbers above.
Metadata
Metadata
Assignees
Type
Projects
- Status
Activity
tamirms commentedon Aug 28, 2026
Findings from the read-path work since this was filed. Full write-up — measured numbers, the measurement standard behind them, and a side-by-side of the options — is in Paths to 10 ms; the code and benchmark tables are in #961.
Where the read slice's time actually goes
A found
getTransactionis O(ledger), not O(index). The hash index resolves in microseconds (an in-process miss costs ~80 µs). The milliseconds are in materialization: the reader fetches the candidate ledger's whole LedgerCloseMeta (~18 MB raw on the sac-6000 stress profile), zstd-decodes all of it (~4 ms), and walks/hashes it via the SDK to locate and pair the transaction (~8 ms). Before #961 it also allocated ~20 MB of garbage per request — which is why 200 rps of lookups OOM-killed the daemon on the 61 GB box.#961 (in review) removes the allocation, not the CPU:
(sac-6000, all-hit spread-hash corpus — misses are not exercised over HTTP; hold-window percentiles; protocol and declared limitations in the doc and the PR.) Target load went from un-servable to stable, and ingest holds its 80 ms budget under read load: whole-replay pace-lag p99 is 50.1–50.9 ms with the 5-minute 1,000 rps hold vs 35.9–42.7 in reps where load ran only ~1 of the 25 minutes, and the worst ledger committed 76.2 ms behind schedule — the hold shows up in the ingest tail but stays inside budget. The 10 ms in-RPC read target is not yet met at target load: the residual is raw per-request CPU, ≈12 ms ⇒ ~83 req/s/core ⇒ ~12 read cores + ~4 ingest at 1,000 rps — the 16-core box at redline (8 physical cores collapses: client-observed p50 7.6 s, 31 % errors).
What this means for the two candidates here
Options for the remaining gap (details, gates, and risks in the doc)
🤖 Generated with Claude Code
tamirms commentedon Sep 23, 2026
Option B spike results:
getTransactionfrom a per-ledger span tableThe problem. The tx-hash index only says which ledger holds a transaction. To answer a
getTransaction, RPC decompresses that whole ledger and walks every transaction in it to find the one asked for. That cost scales with ledger size. On today's pubnet ledgers (about 2 MB uncompressed, a few hundred transactions) it is a few milliseconds. On the Phase 3 stress profile, 6,000 transactions per ledger and about 15 MB uncompressed (2 MB compressed), it is ~12 ms of CPU per lookup: 4 ms to decompress, 8 ms to walk. At 1,000 rps that is most of a 16-core box, which is why p99 sat at 23 ms and why a freeze or index rebuild running alongside tipped the node over.What we built. Each ledger carries a small table saying where every transaction's bytes are, 18 bytes per transaction, so a lookup jumps straight to its transaction instead of walking the ledger. Ledgers over 4 MiB are stored as 4 MiB compressed frames, so a lookup decompresses only the frame it needs instead of the whole ledger; ledgers under 4 MiB, which is every pubnet ledger today, are stored exactly as before. The same layout goes into the cold packs. Every answer is verified by hash, and a bad table is an error, never a silent wrong answer.
Results (sac-6000, 1,000 rps of reads while ingesting at 600 ms):
getTransactionp99The two targets set for the spike, p99 under 10 ms at 1,000 rps during ingest and no ingest slowdown, are both met with room.
Cost. Cold storage +5% on dense chunks, none on pubnet-shaped ones. Memory is lower.
Rollout. The cold tier only changes a chunk when it holds ledgers over 4 MiB, and only for chunks frozen after the upgrade. So this should land before pubnet ledgers approach 4 MiB: then no chunk ever needs rebuilding. If it lands after, chunks frozen in between with large ledgers would have to be re-backfilled to get the cold gain, and a rebuilt chunk would no longer match the one already published, which we want to avoid.
Next. Productize as two PRs on #902: the hot tier first (tables and serving), then the cold tier (frames and tables in packs). They depend on stellar/go-stellar-sdk#6010 and stellar/go-stellar-sdk#6011 and on stellar/streamhash#21, an index mapping fix found during the spike that is worth half of the cold gain.
Branch:
tamirms/gettx-option-b-spike-2. Full numbers and methodology: Option B Final Round.🤖 Generated with Claude Code
https://claude.ai/code/session_01Y7opQUW9UZPKzE6B3tz9b5
tamirms commentedon Sep 23, 2026
Closing as complete because the spike for speeding up getTransaction to meet phase 3 targets has been validated