Query-path Sub-Epic: Adopt zero-copy XDR views and GetLedgerRaw for performance #732

Note: This epic was co-opted into being an epic purely for the query-path components of this project. Tasks relating to ingestion, despite being listed below for completeness, are considered out of scope.

Summary

The new XDR view types (xdr.LedgerCloseMetaView, xdr.LedgerEntryView, xdr.TransactionEnvelopeView, etc.) and the LedgerBackend.GetLedgerRaw method coming from go-stellar-sdk (PR #5941) enable zero-copy partial decoding of XDR data. stellar-rpc has multiple hot paths that currently:

1. Decode entire XDR structures via SQL Scanner auto-decode, then read only a few fields
2. Decode XDR structures and immediately re-marshal sub-fields back to bytes for response/storage
3. Walk full transaction batches when only metadata or a single transaction is needed

This issue tracks the work to migrate those paths to view-based access.

Constraint

The public JSON-RPC API contract is preserved exactly. All changes are internal Go interfaces, types, and implementations.

Prerequisite

This work depends on go-stellar-sdk PR #5941 merging. That PR generates view types for every XDR struct (the wrapper types are []byte-based with field accessors). Convenience helpers on those views (e.g., LedgerCloseMetaView.LedgerCloseTime(), LedgerEntryView.LedgerKey()) will be added per-task as needed.

For ingest-pipeline work (Task 7), an additional SDK addition is needed: a view-based replacement for ingest.NewLedgerTransactionReaderFromLedgerCloseMeta that operates on raw bytes / a LedgerCloseMetaView and yields per-transaction views without full decode.

Measured baseline

In the BSB throughput benchmark from the SDK PR (1000 ledgers from GCS, NumWorkers=100):

• GetLedger (full decode): 167 lps
• GetLedgerRaw (decode skipped): 198 lps
• ~19% throughput improvement when LCM decode is eliminated

This is the unit cost of the LCM decode (~1ms per typical pubnet ledger). Per-call savings on stellar-rpc paths range from ~1ms (single-LCM read) to ~5ms (getTransactionByHash with re-encode round trip).

Tasks

Task 1: Migrate DB ledger read path to bytes / view

Files: cmd/stellar-rpc/internal/db/ledger.go, cmd/stellar-rpc/internal/methods/get_latest_ledger.go, cmd/stellar-rpc/internal/methods/get_health.go, cmd/stellar-rpc/internal/methods/get_fee_stats.go, callers of StreamAllLedgers/StreamLedgerRange

Current: Every db.Select(... []xdr.LedgerCloseMeta ...) triggers xdr.LedgerCloseMeta.Scan in the SQL driver layer (registered in xdr/db.go:13 of the SDK), which calls UnmarshalBinary on the entire LCM bytes. Affected sites:

• getLedgerFromDB (line 328) — backs GetLedger for get_latest_ledger and the JOIN in getTransactionByHash
• getLedgerRangeWithCache (line 221), getLedgerRangeWithoutCache (line 252) — backs get_health and get_fee_stats; reads only LedgerSequence() + LedgerCloseTime()
• StreamAllLedgers (line 154), StreamLedgerRange (line 184) — full LCM scan per row

get_latest_ledger.go additionally re-encodes after the decode:

• Line 34: header.MarshalBinary() to base64 the header
• Line 49: latestLedger.MarshalBinary() to base64 the entire LCM (whose bytes were just discarded by the Scanner)

Proposed change:

• Internal helper getLedgerBytesFromDB reads [][]byte directly (no Scanner auto-decode).
• LedgerReader.GetLedger/LedgerReaderTx.GetLedger return (xdr.LedgerCloseMetaView, bool, error) instead of (xdr.LedgerCloseMeta, bool, error).
• getLedgerRangeWithCache/...WithoutCache use views to read only sequence + close-time.
• StreamAllLedgers/StreamLedgerRange callbacks receive a view; existing callbacks update accordingly.
• get_latest_ledger.go reads view fields directly; uses the raw bytes for LedgerMetadata base64 and slices header bytes via view for LedgerHeader (no re-marshal).

SDK pre-req: LedgerCloseMetaView.LedgerCloseTime(), ProtocolVersion(), raw header bytes accessor.

Motivation: Every get_latest_ledger request, every get_health poll, every get_fee_stats call, every range query, every backfill row pays a full LCM decode for a few-field read. This is the broadest-impact task. get_latest_ledger additionally pays two re-encodes on top.

Effort: medium · Risk: medium · Confidence: high

Task 2: Rewrite getTransactionByHash with view-walk

File: cmd/stellar-rpc/internal/db/transaction.go

Current: getTransactionByHash (line 200) reads:

var rows []struct {
    TxIndex int                 `db:"application_order"`
    Lcm     xdr.LedgerCloseMeta `db:"meta"`  // Scanner auto-decodes the entire LCM
}

Then NewLedgerTransactionReaderFromLedgerCloseMeta walks the just-decoded structure to find the tx at application_order. ParseTransaction (line 238) calls MarshalBinary on Result (line 258), Meta (line 261), and Envelope (line 264) to repackage as bytes for the response. The transactions table only stores (hash, ledger_sequence, application_order) lookups, so every GetTransaction joins back to ledger_close_meta.

Proposed change:

• Read raw bytes from the JOIN (Lcm []byte).
• Wrap as LedgerCloseMetaView, view-walk to application_order, slice tx envelope/result/meta as raw bytes via view accessors.
• ParseTransaction accepts the view + index; constructs Transaction struct with raw byte slices directly — no decode-then-re-encode.

SDK pre-req: LedgerCloseMetaView accessors that yield per-transaction byte slices or transaction-level views (envelope/result/meta).

Motivation: Largest per-call savings of any task (~3-5ms: full LCM decode + walk + 3× MarshalBinary eliminated). getTransaction is typically the highest-frequency endpoint — every wallet polling for tx status hits it.

Effort: medium · Risk: medium-high · Confidence: high

Task 3: rpcdatastore + metaToChunk use GetLedgerRaw

Files: cmd/stellar-rpc/internal/rpcdatastore/ledger_reader.go, cmd/stellar-rpc/internal/methods/get_ledgers.go, cmd/stellar-rpc/internal/methods/mocks.go

Current: rpcdatastore.LedgerReader.GetLedgers (line 65) iterates a sequence range and calls bufferedBackend.GetLedger(ctx, sequence) (line 81) — full LCM decode per ledger. Returns []xdr.LedgerCloseMeta. metaToChunk (get_ledgers.go:294) then loops and calls lcm.MarshalBinary() (line 297) to re-encode each as raw bytes for the response.

Proposed change:

• rpcdatastore.LedgerReader.GetLedgers returns [][]byte; calls bufferedBackend.GetLedgerRaw(ctx, seq) instead of GetLedger.
• metaToChunk becomes a near-passthrough; uses view to slice the header bytes without re-marshaling.
• Update MockDatastoreReader.GetLedgers signature.

SDK pre-req: LedgerCloseMetaView raw header bytes accessor (shared with Task 1).

Motivation: This is the most direct application of the new GetLedgerRaw API and matches the BSB benchmark shape that measured 19% improvement. Every getLedgers request that hits the datastore (history queries beyond local DB retention) pays the decode + re-encode cycle today. Lowest-risk, highest-confidence task.

Effort: low · Risk: low · Confidence: high (proven 19%)

Task 4: View-based event filtering and response

Files: cmd/stellar-rpc/internal/db/event.go, cmd/stellar-rpc/internal/methods/get_events.go

Current:

• db/event.go:428 event scan loop: event.UnmarshalBinary(row.eventData) per row. The scanner callback may filter (only certain contract IDs/types/topics) and discard most events; the decode is paid regardless.
• get_events.go:296: xdr.MarshalBase64(segment) for each topic in non-JSON response output.
• get_events.go:304: xdr.MarshalBase64(v0.Data) for event data in non-JSON response output.

Proposed change:

• Wrap row.eventData as DiagnosticEventView. Run filter checks against view accessors (contract ID, type, topic patterns).
• Only call event.UnmarshalBinary when filter passes AND the scanner callback genuinely needs the decoded form.
• For response output, slice topic and data bytes from the source via view rather than re-encoding.
• Possibly change ScanFunction signature to accept view (or both).

SDK pre-req: DiagnosticEventView accessors for contract ID, type, topics, data.

Motivation: Filter selectivity. A typical contract-scoped getEvents query filters to one contract ID; if 99% of events are rejected, ~99% of decode work is avoided. For unfiltered scans, no gain.

Effort: medium · Risk: medium · Confidence: workload-dependent

Task 5: Migrate ledger entry path to views end-to-end

Files: cmd/stellar-rpc/internal/ledgerentries/main.go, cmd/stellar-rpc/internal/preflight/preflight.go, cmd/stellar-rpc/internal/methods/get_ledger_entries.go, cmd/stellar-rpc/internal/methods/simulate_transaction.go, mocks

Current: Per SnapshotSourceGet invocation (called many times per simulation):

1. preflight.go:54: xdr.SafeUnmarshal(ledgerKeyXDR, &ledgerKey) — DECODE Migrate Soroban RPC from soroban-tools Repository #1 (key from C FFI)
2. ledgerentries/main.go:78: xdr.SafeUnmarshalBase64(entry.Entry, &xdrEntry) — DECODE soroban-rpc: Migrate Soroban Tools to Soroban RPC #2 (entry from captive core)
3. preflight.go:70: entries[0].Entry.MarshalBinary() — RE-ENCODE for C FFI return

Plus simulate_transaction.go:64: xdr.SafeUnmarshal(entryXDR, &entry) of full xdr.LedgerEntry just to call entry.LedgerKey() (line 68). The comment at line 61 acknowledges the decode is just for the key.

Plus get_ledger_entries.go:142,184,189,195: re-marshal keys, entry data, ext for response output.

Proposed change:

• Change LedgerKeyAndEntry type from Entry xdr.LedgerEntry to either Entry []byte (raw bytes) or Entry xdr.LedgerEntryView (wrapper).
• Skip the xdr.SafeUnmarshalBase64 decode in coreLedgerEntryGetter.GetLedgerEntries — base64-decode to bytes, store bytes.
• preflight.go:70: return raw entry bytes directly to C FFI — no MarshalBinary.
• simulate_transaction.go:LedgerEntryChangeFromXDRDiff: use LedgerEntryView.LedgerKey() for partial decode of just the key.
• get_ledger_entries.go: use view accessors for Data, Ext, LastModifiedLedgerSeq.
• Optional, wider scope: if interfaces.FastCoreClient.GetLedgerEntries accepts raw key bytes, also skip preflight decode Migrate Soroban RPC from soroban-tools Repository #1.

SDK pre-req: LedgerEntryView.LedgerKey(), Data, Ext, LastModifiedLedgerSeq accessors.

Motivation: Soroban contracts read many ledger entries per simulation (often 50-200). Each SnapshotSourceGet pays one decode + one re-encode (plus another decode upstream). simulate_transaction's LedgerEntryChange similarly pays a full decode per change just to extract the key. Hot path for any RPC instance serving Soroban traffic.

Effort: medium-high · Risk: medium · Confidence: high

Task 6: Tactical view-based partial decodes

Files: cmd/stellar-rpc/internal/daemon/metrics.go, possibly others if profiling reveals them

Current: metrics.go:86: xdr.SafeUnmarshalBase64(envelopeBase64, &envelope) decodes the entire transaction envelope just to read len(envelope.Operations()) (line 107).

Proposed change: Use TransactionEnvelopeView.OperationCount() to skip the full decode.

SDK pre-req: TransactionEnvelopeView.OperationCount().

Motivation: Per-SubmitTransaction cost. Trivial fix once the view helper exists. Easy starter task.

Effort: trivial · Risk: low · Confidence: small but easy win

Task 7: Migrate ingest pipeline to raw bytes / views

Files: cmd/stellar-rpc/internal/ingest/service.go, cmd/stellar-rpc/internal/db/transaction.go (InsertTransactions, ParseTransaction), cmd/stellar-rpc/internal/db/event.go (InsertEvents, insertEvents), cmd/stellar-rpc/internal/feewindow/feewindow.go (IngestFees), cmd/stellar-rpc/internal/db/ledger.go (InsertLedger), cmd/stellar-rpc/internal/db/migration.go (MigrationApplier.Apply interface), cmd/stellar-rpc/internal/daemon/daemon.go:457 (migration replay loop)

Current:

• service.go:192: s.ledgerBackend.GetLedger(ctx, sequence) returns full-decoded xdr.LedgerCloseMeta.
• All downstream consumers (InsertTransactions, IngestFees, ingestLedgerCloseMeta, InsertEvents) take xdr.LedgerCloseMeta.
• InsertTransactions (line 88) opens ingest.NewLedgerTransactionReaderFromLedgerCloseMeta — full decode walk.
• ParseTransaction (line 238) re-marshals Result, Meta, Envelope (lines 258-264) to bytes for storage.
• db/transaction.go lines 280, 299, 311 each event.MarshalBinary() per event type during ingest.
• db/event.go:260 event.Event.MarshalBinary() for eventBlob; line 274 segment.MarshalBinary() for each topic.
• feewindow.go:164 walks all transactions for fees — decodes envelope + result + meta even when only fee-related fields are read.
• MigrationApplier.Apply takes xdr.LedgerCloseMeta. On first deploy with retention-window migrations, daemon.go:457 walks every ledger in the retention range (months → years) and runs migrations — decode pays per ledger per migration.

Proposed change:

• service.go calls s.ledgerBackend.GetLedgerRaw(ctx, sequence); passes raw bytes downstream.
• InsertTransactions, IngestFees, InsertEvents, ingestLedgerCloseMeta accept (seq uint32, raw []byte) (or a view).
• Use view-based transaction iteration (SDK pre-req) instead of NewLedgerTransactionReaderFromLedgerCloseMeta.
• Slice topic/event bytes from the source LCM rather than re-encoding (eliminates the per-topic MarshalBinary at db/event.go:260,274 and db/transaction.go:280,299,311).
• MigrationApplier.Apply interface changes to accept bytes/view; migrations (transactionTableMigration, feeWindowMigration) use views internally.
• InsertLedger accepts raw bytes (already mostly the case).

SDK pre-req: largest of any task — view-based iteration over a LedgerCloseMetaView's transactions, equivalent to ingest.NewLedgerTransactionReaderFromLedgerCloseMeta but operating on a view and yielding per-tx views/byte slices. Plus accessors for tx.Result.FeeCharged, soroban meta paths, etc.

Motivation:

• Steady-state: every ingested ledger pays decode + multiple per-event/topic re-marshals. Modest per-ledger savings stacked across the entire history.
• One-time first-deploy migration cost (per daemon.go:457 loop): walking the entire retention window with all migrations applied is potentially hours of CPU on a fresh deployment. View-based migrations could reduce that significantly.

Effort: high · Risk: medium-high · Confidence: depends on SDK feature availability

Task 8: View-based simulate → preflight C-FFI input chain

Files: cmd/stellar-rpc/internal/methods/simulate_transaction.go, cmd/stellar-rpc/internal/preflight/preflight.go, cmd/stellar-rpc/internal/preflight/pool.go, preflight.GetterParameters/Parameters types

Current:

• simulate_transaction.go:295: xdr.SafeUnmarshalBase64WithOptions(request.Transaction, &txEnvelope, decodeOptions) — full envelope decode per simulate request.
• Envelope is then field-navigated: Operations() count check (line 303), Memo() type check (line 309), source account derivation (line 320-325), OpBody.Type (line 328), sorobanData.Resources.Footprint extraction (line 338).
• Sub-fields OpBody, Footprint, SourceAccount are passed to preflight via GetterParameters → Parameters (decoded forms).
• preflight.go:189: params.OpBody.MarshalBinary() — re-encode for C FFI
• preflight.go:195: params.Footprint.MarshalBinary() — re-encode
• preflight.go:223: params.OpBody.MustInvokeHostFunctionOp().MarshalBinary() — re-encode
• preflight.go:229: params.SourceAccount.MarshalBinary() — re-encode

Proposed change:

• simulate_transaction.go: don't decode the full envelope. Wrap base64-decoded bytes as TransactionEnvelopeView.
• Use view accessors for Operations() count check, Memo() type check, source account derivation.
• Extract OpBody, Footprint, SourceAccount as raw byte slices via view accessors.
• Change preflight.GetterParameters and Parameters to carry raw bytes (e.g., OpBodyRaw []byte, FootprintRaw []byte, SourceAccountRaw []byte).
• preflight.go:189,195,223,229: pass raw bytes directly to C FFI — no MarshalBinary.

SDK pre-req: TransactionEnvelopeView accessors for operations, memo, source-account chain, op-body bytes, soroban data extraction.

Motivation: Per-simulateTransaction request: full envelope decode + 2-4 sub-field re-encodes saved. Simulate is the heaviest single Soroban-RPC request type.

Effort: medium-high · Risk: medium · Confidence: high

Recommended execution order

Out of scope

The following decode/encode patterns were identified but cannot be addressed via views:

• xdr2json C-FFI re-parse (xdr2json/conversion.go:53, used widely on JSON-format response paths). JSON serialization inherently requires walking every field; no view alternative. Optimization options (caching JSON, skipping JSON entirely) are separate from this work.
• send_transaction.go:103,126 JSON-only conditional decode. Already optimal: only fires on JSON format, and the decoded form is consumed by xdr2json which needs decoded structs.
• get_events.go:88 topic filter encode (segmentFilter.ScVal.MarshalBinary()). Necessary for SQL WHERE clause construction; user-supplied filter encoded once per query.
• go-stellar-sdk/protocols/stellarcore.DiagnosticEventsToSlice decode-then-re-encode round trip (called from send_transaction.go:152). Lives in the SDK, not stellar-rpc. Would require an SDK addition like DiagnosticEventsToRawSlice to fix.

Profiling note

The expected gains above are extrapolated from the BSB benchmark (~1ms per LCM decode unit cost). Real impact for stellar-rpc depends on the production traffic mix. Before committing to the larger tasks (1, 5, 7), running pprof on a staging instance for an hour to identify which paths actually dominate CPU is recommended.