release-v20260505

[amishas157]

• Batch-stream history export commands (Batch-stream history export commands #413)
• Use GCS datastore backend for get_ledger_range_from_times (Use GCS datastore backend for get_ledger_range_from_times #412)

[Copilot]

Pull request overview

This release aggregates two feature lines: (1) refactors multiple history export CLI commands to stream ledger data in batches and write/upload per-batch files, and (2) switches get_ledger_range_from_times to binary-search ledger close times via the GCS datastore backend instead of Stellar history archives.

Changes:

• Introduces a shared batch-export driver (runLedgerBatchExport) plus new input.StreamLedgerBatches/LedgerBatch to support per-batch streaming exports.
• Adds per-ledger extraction helpers (transactions/operations/trades/assets) to support the streaming model.
• Updates ledger-range lookup to use GCS datastore schema + random-access reads; adds unit tests and retains the old history-archive implementation as a separate helper.

Reviewed changes

Copilot reviewed 45 out of 56 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
testdata/transactions/ledger_no_txs.golden	Updates golden output for empty-result transaction export under new batching/sorting behavior.
testdata/trades/ledger_no_trades.golden	Updates golden output for empty-result trade export under new batching/sorting behavior.
testdata/trades/10_ledgers_trades.golden	Golden changes reflecting new output ordering/numbering for batched trade export.
testdata/token_transfers/one_ledger_token_transfers.golden	Golden updates reflecting ordering/numbering changes after batching refactor.
testdata/operations/ledger_no_ops.golden	Updates golden output for empty-result operations export.
testdata/effects/one_ledger_effects.golden	Golden changes reflecting new ordering/numbering for batched effects export.
testdata/effects/ledger_no_effects.golden	Updates golden output for empty-result effects export.
testdata/assets/one_ledger_assets.golden	Golden changes reflecting batching + dedup behavior and ordering/numbering updates.
testdata/assets/ledger_no_assets.golden	Updates golden output for empty-result assets export.
testdata/assets/10_ledgers_assets.golden	Golden changes reflecting batching + dedup behavior and ordering/numbering updates.
internal/utils/main.go	Defaults --strict-export to true; adds AddLedgerBatchFlags/MustLedgerBatchFlags for batch-stream commands.
internal/utils/logger_test.go	Adds tests for EtlLogger.LogError strict/non-strict behavior and strict-export flag default.
internal/input/transactions.go	Adds TransactionsFromLedger helper to extract transaction inputs from a single LCM.
internal/input/trades.go	Adds TradesFromLedger helper to extract trade-producing ops from a single LCM.
internal/input/operations.go	Adds OperationsFromLedger helper to extract operation inputs from a single LCM.
internal/input/ledgers.go	Adds HistoryArchiveLedgerFromLCM adapter for transforms expecting historyarchive.Ledger.
internal/input/ledger_range_test.go	Adds comprehensive unit tests for datastore-backed ledger-range finding and boundary conditions.
internal/input/ledger_range_history_archive.go	Preserves old history-archive ledger-range implementation as GetLedgerRangeHistoryArchive.
internal/input/ledger_range.go	Reimplements GetLedgerRange using GCS datastore random access + iterative lower-bound binary search.
internal/input/ledger_batches_test.go	Adds test covering fatal behavior when StreamLedgerBatches fails to read a ledger.
internal/input/ledger_batches.go	Adds LedgerBatch + StreamLedgerBatches to fetch inclusive ledger ranges in batch-size increments.
internal/input/assets.go	Adds PaymentOperationsFromLedger helper for extracting asset-introducing ops from a single LCM.
cmd/process_ledger_fatal_test.go	Adds end-to-end test asserting transform failures are fatal when StrictExport is true.
cmd/ledger_batch_export.go	Adds shared batch-export command driver used by multiple export commands.
cmd/get_ledger_range_from_times.go	Wires --datastore-path and uses datastore-backed input.GetLedgerRange.
cmd/export_transactions_test.go	Updates CLI tests to folder output + multi-batch coverage for transactions export.
cmd/export_transactions.go	Refactors transactions export to streaming batches via runLedgerBatchExport.
cmd/export_trades_test.go	Updates CLI tests to folder output + multi-batch coverage for trades export.
cmd/export_trades.go	Refactors trades export to streaming batches via runLedgerBatchExport.
cmd/export_token_transfers_test.go	Updates CLI tests to folder output + multi-batch coverage for token transfer export.
cmd/export_token_transfers.go	Refactors token transfer export to streaming batches via runLedgerBatchExport.
cmd/export_operations_test.go	Updates CLI tests to folder output + multi-batch coverage for operations export.
cmd/export_operations.go	Refactors operations export to streaming batches via runLedgerBatchExport.
cmd/export_ledgers_test.go	Updates CLI tests to folder output + multi-batch coverage for ledgers export.
cmd/export_ledgers.go	Refactors ledgers export to streaming batches via runLedgerBatchExport + HistoryArchiveLedgerFromLCM.
cmd/export_ledger_transaction_test.go	Updates CLI tests to folder output + multi-batch coverage for ledger-transaction export.
cmd/export_ledger_transaction.go	Refactors ledger-transaction export to streaming batches via runLedgerBatchExport.
cmd/export_effects_test.go	Updates CLI tests to folder output + multi-batch coverage for effects export.
cmd/export_effects.go	Refactors effects export to streaming batches via runLedgerBatchExport.
cmd/export_contract_events_test.go	Updates CLI tests to folder output + multi-batch coverage for contract events export.
cmd/export_contract_events.go	Refactors contract events export to streaming batches via runLedgerBatchExport.
cmd/export_assets_test.go	Updates CLI tests to folder output + multi-batch coverage for assets export.
cmd/export_assets.go	Refactors assets export to streaming batches via runLedgerBatchExport, retaining cross-batch dedup.
README.md	Updates documented default for strict-export to true.
.pre-commit-config.yaml	Extends hook excludes for specific golden files representing empty-result outputs.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	github.com/​stellar/​go-stellar-sdk@​v0.0.0-20260325174035-031e5bfdc4bd ⏵ v0.5.0	+1
	cloud.google.com/​go/​storage@​v1.42.0 ⏵ v1.62.1	-5
	github.com/​lib/​pq@​v1.10.9 ⏵ v1.12.3
	github.com/​spf13/​cobra@​v1.7.0 ⏵ v1.10.2	-1
	github.com/​spf13/​pflag@​v1.0.5 ⏵ v1.0.10
	github.com/​sirupsen/​logrus@​v1.9.3 ⏵ v1.9.4	+1
	github.com/​spf13/​viper@​v1.17.0 ⏵ v1.21.0	+2

View full report