test(oracle): add manual live soak suite

[ByteYue]

Summary

Add one opt-in, non-gating Wave 7 live soak suite for the complete Gravity Oracle path:

• four equal-power validators;
• three independent live Binance Futures testnet closed index-price kline feeds: NVDAUSDT (feedId 1001), BTCUSDT (1002), and ETHUSDT (1003), all sourceType=3;
• one recently closed binary Polymarket market with a finalized Polygon CTF settlement (sourceType=6);
• one governance proposal that registers all four task instances and resolver callbacks;
• epoch-boundary activation, observer startup, validator JWK votes, quorum, NativeOracle execution, and resolver state verification.

The suite is excluded from the runner default test set and must be named explicitly.

Architecture exercised

1. pre_deploy selects one common closed-minute anchor for all three Binance pairs and discovers one finalized Polygon settlement.
2. It writes an ignored per-run URI mapping; RPC credentials are never committed.
3. Four Gravity validators start from the same genesis.
4. Governance registers three price tasks and one Polymarket task during epoch E.
5. The test proves all 16 observers are absent in E, appear in E+1, and at least three validators certify each issuer.
6. Agreed bytes execute through NativeOracle into one feedId-namespaced PriceFeedResolver and the PolymarketSettlementResolver.
7. Every heartbeat compares a common confirmed block across all four RPC replicas.
8. Contract reads use the exact EIP-1898 canonical block hash plus a progress/resolver/progress seqlock, so an RPC view transition is retried instead of being reported as Oracle divergence.

Generated mappings and metadata are removed on teardown, including copies deployed into node config directories.

Acceptance checks

• independent monotonic delivery nonce, source position, and resolver round for every Binance feed;
• exact onchain close for each pair's requested one-minute index-price bucket;
• four-replica state convergence at one canonical block hash;
• chain and per-feed stall budgets;
• at least three relayers checkpoint every committed feed nonce;
• immutable Polymarket terminal settlement at nonce 1;
• each feed's callback count equals its final delivery nonce;
• exactly one Polymarket callback;
• optional validator restart with block and all three relayer checkpoints recovered;
• observed price-change counts in heartbeat and summary evidence.

Validation

Static

• python3 compileall: PASS
• pytest collect-only: PASS (one manual suite)
• git diff --check: PASS
• tracked path/credential scan: PASS
• deployed relayer config cleanup check: PASS

Three-feed high-frequency live regression

• configured duration: 300s
• actual duration: 305.971s
• heartbeat samples: 56
• every pair: nonce 2 -> 8, 6 advances (minimum 4)
• observed price changes: NVDA 0, BTC 6, ETH 6
• callbacks: NVDA 8, BTC 8, ETH 8, Polymarket 1
• relayer quorum: 4/4 for every pair on every sample
• node4 restart recovery: 11.136s
• result: PASS

Required three-feed 30-minute burn-in

• configured duration: 1,800s
• actual duration: 1,803.466s
• heartbeat samples: 119
• every pair: nonce 2 -> 33, 31 advances (minimum 24)
• maximum observed update gap: 61.007s (budget 360s)
• observed price changes: NVDA 0, BTC 31, ETH 31
• callbacks: NVDA 33, BTC 33, ETH 33, Polymarket 1
• relayer quorum: 4/4 for every pair on every sample
• result: PASS

24-hour soak

A previous single-feed run stayed healthy for 8,349.273s and was intentionally stopped, not failed, to add the requested BTC and ETH coverage. The first formal three-feed attempt stayed healthy for 43,185.741s: every feed reached nonce 722 with 4/4 replica and relayer agreement. It then exposed an Aptos epoch-reconfiguration race because the scheduled node4 restart coincided exactly with the two-hour epoch boundary. Commit 5c56e1f adds a chain-derived five-minute guard around epoch transitions; a fresh formal run is required.

ORACLE_SOAK_DURATION_SECONDS=86400 \
ORACLE_SOAK_POLL_SECONDS=15 \
ORACLE_SOAK_STALL_TIMEOUT_SECONDS=360 \
  ./gravity_e2e/run_test.sh oracle_live_soak --force-init --log-cli-level=INFO

It requires every feed to complete at least 80% of expected minute advances. Node4 becomes eligible to restart at 12 hours and is deferred only while the chain is within five minutes of an epoch boundary. This PR remains draft until the repeated run completes and its summary is posted.

Dependencies

• based on SDK main after test(oracle): add deterministic Polymarket quorum E2E #806;
• genesis contracts pinned to the merged Polymarket resolver commit from contracts enhance: Support specify customized command line args by copying the start.sh to remote server #115;
• node binaries use the gravity-reth and gravity-aptos revisions pinned by current SDK main.

Operations and safety

• Binance values are testnet index data and must be labeled as such in demos.
• Binance testnet indexPriceKlines is public; no Binance API key is required.
• Polygon RPC is supplied only through an environment variable.
• Runtime evidence stays in the ignored suite artifacts directory.
• The runner performs global local gravity_node cleanup; do not run it beside another local Gravity cluster.

Tracks Galxe/gravity-audit#1038.

[ByteYue]

Wave 7 live validation update:

• The first formal 24-hour attempt failed after 6,111 seconds, before validator restart.
• Oracle data remained healthy through the last heartbeat: Binance nonce reached 104, all four relayers had checkpointed it, and the Polymarket settlement remained exactly once at nonce 1.
• The chain stopped at the epoch 103 -> 104 boundary. Two validators timed out for 30 seconds while resetting the prior epoch BufferManager and never started epoch 104, leaving only 2/4 voting power. This was consensus reconfiguration churn, not a Binance, Polygon, JWK quorum, or callback failure.
• The suite originally used a 60-second epoch throughout, which would impose 1,440 reconfigurations during a 24-hour Oracle soak. The same governance proposal now activates the dynamic Oracle tasks and changes the next epoch interval to two hours. The suite asserts the pending and applied EpochConfig states before starting the timer.
• Failure diagnostics now identify the lagging RPC and retain the last heartbeat.

Post-fix live smoke PASS:

• configured/actual duration: 300s / 301.016s
• Binance nonce: 1 -> 7 (6 advances)
• relayer checkpoint coverage: 4/4
• callback events: Binance 7, Polymarket 1
• applied soak epoch interval: 7,200s
• final Gravity block: 1,618

The PR remains draft until a fresh 24-hour run passes.

[ByteYue]

Wave 7 soak update (second formal-run diagnosis and harness fix):

• The post-epoch-fix 24-hour attempt kept the four-validator chain healthy for about 5h36m (last common height 89043, Binance nonce 339, relayers 4/4, Polymarket nonce 1).
• It failed because the harness read latest twice while that exposed block state was still being committed; node1 therefore appeared to disagree with itself by one Binance nonce. This was a snapshot race in the test, not an Oracle/consensus failure.
• Commit 582e7950cd now samples min(latest node heights) - 16, requires all four RPCs to return the same block hash there, and compares NativeOracle/resolver state only at that common historical block. The initial sample waits for the confirmation window to fill. It also fixes the validator-restart block-wait call and records the snapshot hash in heartbeat/summary evidence.

Live regression proof using Binance Futures testnet plus finalized Polygon data:

• configured/actual duration: 300s / 301.041s
• Binance nonce: 1 -> 7 (6 advances; required 3)
• relayer quorum: 4/4 throughout
• node4 forced restart: PASS, recovery 11.164s
• callbacks: Binance 7, Polymarket 1
• common final snapshot block/hash: 1568 / 0x83a745fa191e38d402e935a4beb624a7779f055591e5cf5cacc5dd256b94091c

A fresh formal 24-hour run is starting from this commit. The PR remains draft until that run completes successfully.

[ByteYue]

Wave 7 burn-in update

The previous formal soak stopped after 4,615 seconds when numeric historical `eth_call` requests returned different state snapshots even though every replica first reported the same block hash. The suite now binds every NativeOracle/resolver read to the exact canonical hash with EIP-1898 (`blockHash` plus `requireCanonical`) in commit `c079007a4e`.

Validation after the fix:

• 300-second high-frequency regression: PASS, 135 samples, 7 Binance advances, forced node4 restart recovered in 13.174 seconds.
• Required 30-minute burn-in: PASS, 1,801.352 seconds, 119 samples.
• Binance testnet nonce: 1 -> 33 (32 advances; minimum 24).
• Maximum observed price-update gap: 60.768 seconds.
• Relayer quorum: 4/4 replicas on every sample.
• Callback events: Binance 33, Polymarket 1.
• Static checks: compileall, pytest collect-only, diff check, secret/path scan all PASS.

The PR remains draft while the fresh 24-hour run is pending.

[ByteYue]

Multi-feed update pushed in `d259517172`. The live suite now runs independent NVDAUSDT (1001), BTCUSDT (1002), and ETHUSDT (1003) index-kline tasks through the same feedId-namespaced resolver.

The prior single-feed 24-hour attempt was intentionally stopped after 8,349.273 healthy seconds to add this requested coverage. Post-change gates are PASS: a 305.971s high-frequency run with node4 restart (all feeds 6 advances; BTC/ETH 6 observed price changes each; recovery 11.136s), followed by a 1,803.466s burn-in (all feeds 31 advances; BTC/ETH 31 changes each; 4/4 relayer quorum; callback counts 33/33/33 and Polymarket 1).

The first smoke exposed a cross-contract RPC view transition after restart. The final implementation preserves the strict nonce-to-bucket invariant and uses a progress/resolver/progress seqlock around EIP-1898 hash-bound calls. The rerun and burn-in both passed. A fresh three-feed 24-hour run is starting; the PR remains draft.

[ByteYue]

Three-feed 24h attempt: 12h epoch/restart collision found

The first formal three-feed run stayed healthy for 43,185.741 seconds (about 12h):

• NVDAUSDT, BTCUSDT, ETHUSDT all reached nonce 722;
• all three feeds remained 4/4 replica-consistent and 4/4 relayer-checkpointed;
• observed price changes were NVDA 696, BTC 719, ETH 719;
• Polymarket remained terminal at nonce 1.

The run then failed at the scheduled node4 restart because the restart was aligned to the two-hour epoch boundary. This was not an Oracle-value divergence. During the epoch-8 reconfiguration, node3 timed out and received execution responses out of order: it learned the suffix-block timestamp before the actual reconfiguration-block timestamp, then panicked in BlockInfo::change_timestamp (allow_timestamp_change). The restarted node4 encountered the related reconfiguration-suffix invariant while catching up.

Commit 5c56e1fe77 keeps the restart test but adds a chain-derived five-minute guard around epoch boundaries. Once the configured restart time is reached, the runner defers only the restart; feed, quorum, replica, callback, and stall assertions continue on every heartbeat. This separates the Wave 7 Oracle recovery check from a simultaneous Aptos epoch-reconfiguration stress case. The PR remains draft and the formal 24h run must be repeated.

[ByteYue]

Epoch-guard live regression: PASS

Commit 5c56e1fe77 was exercised against the real Binance Futures testnet and Polygon path:

• configured/actual duration: 420s / 422.694s;
• restart eligible at 60s, deferred for 43 five-second polls, executed at about 300.9s outside the epoch guard;
• node4 full restart recovery: 13.173s;
• NVDAUSDT, BTCUSDT, ETHUSDT each advanced nonce 1 -> 9 (8 advances, minimum 4);
• each pair observed 8 live price changes;
• every heartbeat retained 4/4 replica and relayer agreement;
• callback counts: NVDA 9, BTC 9, ETH 9, Polymarket 1;
• result: PASS.

The repeated formal 24-hour run can now start from this head. The PR remains draft pending that gate.