rpcv2: version the on-disk format before the first release freezes it

[tamirms]

The v2 store writes immutable cold artifacts, so whatever the first release ships can never be retrofitted: a file written without self-description stays that way forever. And today nothing stops a binary from misreading — or destroying — data written by a different version: the resolver treats any catalog state it doesn't recognize as missing work, so rolling back to an older binary after a format change would overwrite the newer binary's artifacts.

This PR adds the minimum that must exist before the format freezes:

Version handshake. catalog.Open checks every catalog key and value against the exact vocabulary this binary writes (catalog/census.go) and refuses to start — before writing anything — when it meets an entry it doesn't know, with a message distinguishing "a newer binary wrote this" from corruption. Upgrades stay seamless; rollback across a format change refuses instead of destroying data. Format-touching releases are roll-forward-only, by policy.

Self-describing files. .bin gains a magic + version prelude (it was the one cold file with none); txhash .idx metadata and ledger-pack app data gain leading version bytes; index.pack gains a build stamp (term-schema version + indexed-field bitmask), so "index lacks this term family" is distinguishable from "no matches". All three pack writers enable the packfile content hash, computed over pre-compression bytes so it survives zstd encoder changes.

Pins against accidental format changes. Every RocksDB column family pins its block-based-table format_version (a grocksdb upgrade could otherwise silently change on-disk bytes), and byte-level golden fixtures pin the term-key derivation, which is de facto on-disk format.

The policy, in design docs: what forces a format bump (a new indexed term family bumps the hot format too; dependency bumps that change RocksDB defaults count; write-path switches must flip freeze and walk together) and the per-tier upgrade shapes. The rest of the versioning design — state@id grammar, per-format read sets, capability gating — deliberately waits for the first release that actually changes a format.

Verification: the full internal/rpcv2 tree is green and golangci-lint --new-from-rev=feature/full-history is clean at every commit. New tests pin the acceptance/refusal matrices (including first-start crash residues and secret redaction), each decoder's newer-version refusal, stamp round-trips, and per-pack hash verification.

Interactions: rebased onto #910; the build-stamp gate runs after #910's record-checksum gate in OpenColdReader. #902's zero-decompression freeze will need a decompress extract or an ingest-time hash handoff for the ledgers content hash when it rebases (recorded in the doc section).

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Establishes release-one storage-format safety through startup validation, self-describing artifacts, integrity metadata, and documented upgrade policy.

Changes:

• Adds catalog census validation and RocksDB format safeguards.
• Versions cold-storage metadata and enables packfile content hashes.
• Adds format goldens, compatibility tests, and upgrade documentation.

Reviewed changes

Copilot reviewed 34 out of 34 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
design-docs/gettransaction-full-history-design.md	Documents versioned txhash formats.
design-docs/full-history-streaming-workflow.md	Defines format upgrade policy.
stores/txhash/cold_merge.go	Validates merged .bin inputs.
stores/txhash/cold_index.go	Parses the new .bin header.
stores/txhash/cold_index_test.go	Updates malformed-header fixtures.
stores/txhash/cold_format.go	Versions index metadata.
stores/txhash/cold_format_test.go	Tests metadata version handling.
stores/txhash/cold_bin.go	Adds .bin magic and version.
stores/txhash/cold_bin_test.go	Pins the new binary layout.
stores/ledger/cold_writer.go	Adds app-data version and hash.
stores/ledger/cold_writer_test.go	Tests ledger hash generation.
stores/ledger/cold_reader.go	Validates app-data versions.
stores/ledger/cold_reader_test.go	Tests version rejection.
stores/event/index.go	Defines term schema identity.
stores/event/format_golden_test.go	Pins term-key bytes.
stores/event/cold_writer.go	Enables event content hashing.
stores/event/cold_reader.go	Validates index build stamps.
stores/event/cold_index.go	Writes stamps and hashes.
stores/event/cold_index_test.go	Tests stamps and hashes.
stores/event/cold_format.go	Adds shared version gates and stamp codec.
stores/event/cold_format_test.go	Updates decoder tests.
stores/blobversion.go	Adds shared blob-version validation.
rpcv2test/rpcv2test.go	Reads the new .bin layout.
rocksdb/rocksdb.go	Pins table format and translates errors.
rocksdb/rocksdb_test.go	Tests zero-tuning format pinning.
geometry/keys.go	Adds lifecycle-state registries.
catalog/secret.go	Centralizes secret width.
catalog/secret_test.go	Tests census-based secret rejection.
catalog/kv_test.go	Uses census-valid fixtures.
catalog/census.go	Implements startup catalog census.
catalog/census_test.go	Tests census acceptance and refusal.
catalog/catalog.go	Runs census before secret minting.
bench/command.go	Warns against live storage use.
backfill/perf_test.go	Pins streaming .bin format.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 34 out of 34 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 33 out of 33 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

cmd/stellar-rpc/internal/rpcv2/stores/event/cold_reader.go:203

• The new schema/mask rejection path has no negative reader test: existing tests decode a valid stamp and reject a wrong pack format, but none opens an index.pack whose stamp has a different schema or field mask. Add lookup-path tests for both mismatches (including an eventless chunk) so this release-critical capability gate cannot be bypassed by a later refactor.

		if schema != TermSchemaVersion || mask != IndexedFieldMask {
			return fmt.Errorf(
				"events: %s was built under term schema %d with field mask %#x; this binary expects "+
					"schema %d with mask %#x (rebuilt index required, or a binary matching the artifact)",
				indexPackPath, schema, mask, TermSchemaVersion, IndexedFieldMask)

cmd/stellar-rpc/internal/rpcv2/stores/txhash/cold_bin_test.go:110

• This refusal test covers foreign magic and version only, leaving the newly added reserved-byte rejection untested in both scanBinHeader and the self-defending merge reader. Add a fixture with one of bytes 5–7 set and assert that both consumers reject it; otherwise the exact prelude vocabulary can regress without a failing test.

// TestColdBin_ScanRejectsForeignHeader pins the header scan's refusals: a
// foreign magic and a newer version byte each fail loudly instead of being
// misread as entry data.
func TestColdBin_ScanRejectsForeignHeader(t *testing.T) {

[Copilot]

Pull request overview

Copilot reviewed 34 out of 34 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

cmd/stellar-rpc/internal/rpcv2/stores/event/cold_writer.go:100

• No event-store test currently asserts that this hash is present or verifies it with eventsPackDecoder; the existing event tests only read records and trailer fields. The analogous ledger and index changes both call ContentHash and Verify. Add the same integration assertion for events.pack so removal or decoder/hash mismatches cannot silently change the release format.

		// Items reach AppendItem in canonical (uncompressed) payload form, so
		// the content hash is independent of the zstd encoder version.
		ContentHash:  true,

cmd/stellar-rpc/internal/rpcv2/stores/event/format_golden_test.go:28

• This golden calls ComputeTermKey with already-encoded bytes, so it pins the hash and field prefix but not the contract-ID/topic value encoding performed by TermsForBytes. A change from raw XDR topic bytes to another representation would leave this test green while changing persisted term identities. Add a fixed marshaled ContractEvent golden through TermsForBytes that covers the contract ID and each topic position.

func TestComputeTermKey_Golden(t *testing.T) {
	val := []byte("stellar-rpc-term-golden")

[Copilot]

Pull request overview

Copilot reviewed 35 out of 35 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 35 out of 35 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

cmd/stellar-rpc/internal/rpcv2/rocksdb/rocksdb_test.go:767

• This assertion only proves that a BBTO was allocated; it still passes if SetFormatVersion is removed or changed, so the new on-disk-format pin is not actually protected against regression. Add a persistence-level assertion (for example, inspect the generated RocksDB OPTIONS entry after opening the store and require format_version=6) so the test pins both application of the option and its release value.

[karthikiyer56]

Summary of the on-disk changes in this PR, the paths whose cost should be measured, and the effect on data directories written before it.

1. On-disk changes by artifact

Artifact	Tier	Before	After	Additional write-path work
txhash .bin	cold	8-byte entry count, 16-byte secret, entries	8-byte SBIN prelude, then the same layout	none; 8 bytes per file
txhash .idx	cold	24-byte user metadata	25-byte user metadata (leading version byte)	none
ledger pack	cold	4-byte app data	5-byte app data (leading version byte); content hash enabled	SHA-256 over the raw ledger bytes
events.pack	cold	unchanged	content hash enabled	SHA-256 over the raw event bytes
index.pack	cold	empty app data	11-byte build stamp; content hash enabled	SHA-256 over the bitmap bytes; the writer moves from a synchronous path to the worker pipeline
catalog RocksDB	catalog	library default table format	table format_version pinned to 6 (the current default)	none
hot chunk RocksDB	hot	library default table format	pinned to 6 (the current default)	none

2. Paths to measure

Path	Cost introduced	Measurement
Ingestion (backfill walk and freeze)	SHA-256 on ledger, event, and index.pack payloads; worker pipeline for index.pack	Backfill throughput over a fixed ledger range, content hash enabled versus disabled. A regression above 10 percent warrants discussion of xxh3 or a backfill-only switch.
Query (getLedgers, getTransaction, getEvents)	none per query; one 11-byte stamp decode per cold index.pack open	none required
Startup	one full catalog scan (census), approximately 20k keys for full pubnet	a single startup-time sample is sufficient

Expected ingestion impact is in the low single-digit percent range, based on SHA-256 throughput of roughly 2.5 GB/s on commodity hardware.

3. Data directories written before this PR

The catalog and the hot RocksDB open without error. The following cold artifacts do not:

Artifact	Failure
ledger pack	unsupported version, or app-data size mismatch
txhash .bin	"not a cold txhash .bin"
txhash .idx	metadata version or size mismatch
index.pack	"build stamp: empty blob" on the first filtered lookup

The catalog continues to mark these chunks frozen, so the resolver does not rebuild them, and the error text attributes the files to a newer binary although they are older. This is expected before the first release. Every development and perf-eval data directory must be deleted in full and re-backfilled after this PR merges; removing only the cold files is insufficient.

[Copilot]

Pull request overview

Copilot reviewed 35 out of 35 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 35 out of 35 changed files in this pull request and generated no new comments.