From 2e655df99360f0ec9d77d04d146b179d7ad0e929 Mon Sep 17 00:00:00 2001 From: shhalaka Date: Sat, 4 Jul 2026 12:18:25 +0530 Subject: [PATCH] docs: add subnet compatibility and setup documentation --- LEGACY_CUSTOM_CHAIN_COMPATIBILITY.md | 450 +++++++++++++++++++++++++++ README.md | 137 ++++++++ SUBNET_ANALYSIS.md | 439 ++++++++++++++++++++++++++ 3 files changed, 1026 insertions(+) create mode 100644 LEGACY_CUSTOM_CHAIN_COMPATIBILITY.md create mode 100644 README.md create mode 100644 SUBNET_ANALYSIS.md diff --git a/LEGACY_CUSTOM_CHAIN_COMPATIBILITY.md b/LEGACY_CUSTOM_CHAIN_COMPATIBILITY.md new file mode 100644 index 0000000..1a46b2b --- /dev/null +++ b/LEGACY_CUSTOM_CHAIN_COMPATIBILITY.md @@ -0,0 +1,450 @@ +# Legacy Custom Chain Compatibility: Upgrading XDPoSChain from v2.6.1 to v2.8.2 + +**Repository:** `xdc-subnet` +**Network:** Custom XDPoS private subnet, chain ID `7670` +**Source release:** `XDPoSChain v2.6.1` +**Target release:** `XDPoSChain v2.8.2` +**Document version:** 1.0 +**Date:** 2026-07-02 + +--- + +## 1. Background + +The `xdc-subnet` repository contains a reproducible 3-validator XDPoS private subnet (chain ID `7670`) deployed on a single host. It was originally generated with the `xinfinorg/subnet-generator` Docker image and ran under Docker using the XDPoSChain `v2.6.1` binary. + +During a planned upgrade to the native `v2.8.2` binary, the subnet failed to import or produce blocks. The logs showed repeated `max fee per gas less than block base fee` errors on the BlockSigner (coinbase signer) transaction, even though the same configuration had worked on `v2.6.1`. + +This document records the investigation, root cause, minimal fix, and validation that restored compatibility for legacy custom chains on `v2.8.2`. + +--- + +## 2. Root Cause Analysis + +### 2.1 The observable failure + +When a `v2.8.2` validator started against the existing data directory and genesis, it repeatedly logged: + +```text +WARN [...] Propagated block import failed + err="max fee per gas less than block base fee: address xdc994e38673C40B1F21B087dAFd04AcBBD6bcd970B, maxFeePerGas: 0 baseFee: 12500000000" +``` + +The block at which the failure occurred had already been produced and accepted by `v2.6.1`. The failure prevented the `v2.8.2` node from staying in sync or participating in consensus. + +### 2.2 How the BlockSigner transaction is handled + +XDPoS uses a special zero-gas-price transaction called the **BlockSigner transaction** (or signing transaction) to sign each block. The account that sends this transaction is the block's coinbase. The transaction: + +- Has `maxFeePerGas = 0` +- Is **not** a normal user transaction +- Must be routed through the `ApplySignTransaction` path rather than the standard `ApplyMessage` / `state_transition` path +- Must not be subjected to EIP-1559 base-fee validation + +The `ApplySignTransaction` path is gated by `IsTIPSigning(num)` returning `true`. If `IsTIPSigning` returns `false`, the BlockSigner transaction is treated as an ordinary transaction, the EIP-1559 base-fee check is applied, and `maxFeePerGas < baseFee` fails the block. + +### 2.3 Why `IsTIPSigning` returned `false` on v2.8.2 + +In `v2.6.1`, fork predicates in `params/config.go` used a dual fallback: + +```go +func (c *ChainConfig) IsTIPSigning(num *big.Int) bool { + return isForked(common.TIPSigningBlock, num) || isForked(c.TIPSigningBlock, num) +} +``` + +`common.TIPSigningBlock` was a **process-wide global** populated by `common.CopyConstants`. For a custom chain ID that was not mainnet, testnet, or devnet, `CopyConstants` copied values from `localConstant`, where every fork block was set to `0`. This made `IsTIPSigning` (and `IsTIPRandomize`, `IsBerlin`, `IsLondon`, `IsMerge`, `IsShanghai`, `IsEIP1559`, `IsCancun`) return `true` for every block on a legacy custom chain. + +In `v2.8.2`, the global constants were removed. The fork predicates now evaluate only the stored `ChainConfig`: + +```go +func (c *ChainConfig) IsTIPSigning(num *big.Int) bool { + return isForked(c.TIPSigningBlock, num) +} +``` + +The genesis generator for this custom subnet stored `TIPSigningBlock`, `TIPRandomizeBlock`, `EIP1559Block`, `BerlinBlock`, `LondonBlock`, `MergeBlock`, `ShanghaiBlock`, and `CancunBlock` as `nil`. With `nil` fork blocks, every predicate returned `false` on `v2.8.2`, which broke the BlockSigner path, EIP-1559 semantics, and the selected EVM instruction set. + +### 2.4 Why this matters for the whole EVM, not just the BlockSigner + +Because `IsTIPSigning` was not the only predicate returning `false`, the EVM jump table also regressed. On `v2.6.1`, the chain effectively ran with `cancunInstructionSet` from block 0. On unpatched `v2.8.2`, it fell back to `frontierInstructionSet`, disabling Berlin, London, Merge, Shanghai, EIP-1559, and Cancun semantics. Even if the BlockSigner path had been fixed, state-root compatibility would still have been at risk. + +--- + +## 3. Timeline of Investigation + +| Step | Date | Activity | Outcome | +| ---- | ---- | -------- | ------- | +| 1 | 2026-07-01 | Reproduce failure by running `v2.8.2` validator against existing data | `max fee per gas less than block base fee` confirmed | +| 2 | 2026-07-01 | Compare `v2.6.1` and `v2.8.2` `params/config.go` / `params/config_forks.go` | Identified removal of global `common.XBlock` fallbacks | +| 3 | 2026-07-01 | Inspect `common.CopyConstants` and `localConstant` values | Confirmed all custom-chain fork blocks were historically `0` | +| 4 | 2026-07-01 | Trace `IsTIPSigning` → `ApplySignTransaction` → `state_transition` | Confirmed BlockSigner routing depends on `IsTIPSigning` | +| 5 | 2026-07-01 | Trace `IsEIP1559` → `jump_table` → `BASEFEE` / `PREVRANDAO` / `PUSH0` | Confirmed broader EVM regression without the fallbacks | +| 6 | 2026-07-01 | Draft `fork-predicate-compatibility-analysis.md` | Proposed `effectiveForkBlock` helper for all affected predicates | +| 7 | 2026-07-02 | Apply minimal patch in `params/config_forks.go` | All fork predicates restored for non-built-in custom chains | +| 8 | 2026-07-02 | Rebuild `XDC` binary at `/home/shalaka/xdposchain-v2.8.2/build/bin/XDC` | Binary reports `XDC/v2.8.2-testnet-f2d66480-20260623/linux-amd64/go1.26.0` | +| 9 | 2026-07-02 | Start Validator 1 with patched binary | No errors, but cannot reach consensus alone | +| 10 | 2026-07-02 | Update startup scripts for Validators 2 and 3 | All three validators run patched `v2.8.2` | +| 11 | 2026-07-02 | Three-validator validation | Peers = 2, block height increasing, no fee errors | + +--- + +## 4. Blockers Encountered and Resolutions + +### Blocker 1: `max fee per gas less than block base fee` on block import + +**Cause:** `IsTIPSigning` returned `false` because `c.TIPSigningBlock` was `nil` and `v2.8.2` had no global fallback. + +**Impact:** BlockSigner transactions were routed through the normal transaction path, which applied EIP-1559 base-fee validation. With `maxFeePerGas = 0` and `baseFee > 0`, every block import failed. + +**Resolution:** Add an `effectiveForkBlock` helper in `params/config_forks.go` that falls back to `LocalnetChainConfig.XBlock` for non-built-in chain IDs when the stored `ChainConfig` field is `nil`. Apply it to `IsTIPSigning`, `IsTIPRandomize`, and the Ethereum fork predicates (`IsBerlin`, `IsLondon`, `IsMerge`, `IsShanghai`, `IsEIP1559`, `IsCancun`). + +### Blocker 2: Single validator cannot prove consensus + +**Cause:** XDPoS requires a quorum of masternodes to produce and commit blocks. With only Validator 1 running, `net_peerCount` was `0` and the chain head did not advance. + +**Impact:** Validator 1 could be verified as stable and error-free, but block production, import, and BlockSigner routing could not be validated in isolation. + +**Resolution:** Keep Validator 1 running, update `start-native-validator2.sh` and `start-native-validator3.sh` to use the patched `v2.8.2` binary, and start the remaining two validators. Once all three were connected, consensus resumed and the block height began to increase. + +### Blocker 3: Default log level hides block-import success messages + +**Cause:** The startup scripts use `--verbosity 2`, which only emits `WARN` and higher. Successful block import and commit messages are at lower levels. + +**Impact:** It is not possible to read "Imported new chain segment" or "Committed block" messages directly from the log files. Validation must rely on RPC block-height polling and absence of errors. + +**Resolution:** Use `eth_blockNumber` and `net_peerCount` RPC calls to verify live progress and log greps to confirm no `max fee per gas`, `invalid baseFee`, or FATAL errors. + +--- + +## 5. Why the Original v2.8.2 Behavior Broke Legacy Custom Chains + +| v2.6.1 behavior | v2.8.2 unpatched behavior | +| --------------- | ------------------------- | +| `common.CopyConstants` sets process-wide globals from `localConstant` for custom chains | Globals removed; predicates rely only on `ChainConfig` | +| All fork blocks effectively `0` for custom chain ID `7670` | Stored fork blocks are `nil` | +| `IsTIPSigning`, `IsTIPRandomize`, `IsEIP1559`, `IsBerlin`, `IsLondon`, `IsMerge`, `IsShanghai`, `IsCancun` all return `true` | All return `false` | +| BlockSigner routed correctly; EVM uses `cancunInstructionSet` | BlockSigner treated as regular transaction; EVM falls back to `frontierInstructionSet` | +| Blocks import and validate successfully | Block import fails with `max fee per gas less than block base fee` and potential state-root divergence | + +The fundamental breaking change was the removal of the global fallback without updating the genesis/config generation tooling for legacy custom chains. Legacy chains that never wrote explicit fork blocks into `genesis.json` or `ChainConfig` became incompatible with `v2.8.2`. + +--- + +## 6. Why the Implemented Compatibility Fix Works + +The fix adds a helper inside `params/config_forks.go`: + +```go +func (c *ChainConfig) effectiveForkBlock(field, localnetDefault *big.Int) *big.Int { + if field != nil { + return field + } + if c.ChainID != nil && !isKnownXDCBuiltInChainID(c.ChainID) { + return localnetDefault + } + return nil +} +``` + +Each affected predicate is then changed from: + +```go +func (c *ChainConfig) IsTIPSigning(num *big.Int) bool { + return isForked(c.TIPSigningBlock, num) +} +``` + +to: + +```go +func (c *ChainConfig) IsTIPSigning(num *big.Int) bool { + return isForked(c.effectiveForkBlock(c.TIPSigningBlock, LocalnetChainConfig.TIPSigningBlock), num) +} +``` + +The same pattern is applied to `IsTIPRandomize`, `IsBerlin`, `IsLondon`, `IsMerge`, `IsShanghai`, `IsEIP1559`, and `IsCancun`. + +### Why this is safe + +- **Built-in networks are unaffected:** `isKnownXDCBuiltInChainID` returns `true` for mainnet, testnet, and devnet. Those networks continue to use their own fork blocks from the stored config. +- **Localnet is unaffected:** `LocalnetChainConfig` is already defined in `params/config.go` and is the canonical source of truth for a local/custom subnet. +- **No process-wide globals:** The fix is `ChainConfig`-local, matching the architectural direction of `v2.8.2`. +- **No changes to consensus, EVM, txpool, or state-transition logic:** The patch only changes the predicate evaluation, so it is the minimal possible scope. + +--- + +## 7. Validation Methodology + +1. **Build patched binary:** + - Patch `params/config_forks.go` in `xdposchain-v2.8.2`. + - Run `make XDC`. + - Verify `XDC version` reports `v2.8.2`. + +2. **Update validator scripts:** + - Change `XDC_BIN` in `start-native-validator1.sh`, `start-native-validator2.sh`, and `start-native-validator3.sh` to point to `/home/shalaka/xdposchain-v2.8.2/build/bin/XDC`. + +3. **Start Validator 1:** + - Verify `web3_clientVersion` returns `v2.8.2`. + - Verify no FATAL or `max fee per gas` errors in the log. + - Confirm `net_peerCount` is `0` because no other validators are running. + +4. **Start Validators 2 and 3:** + - Confirm all three processes are running with the patched binary. + - Confirm each validator reports `net_peerCount = 2`. + - Confirm all three report the same `eth_blockNumber`. + +5. **Monitor block production:** + - Poll `eth_blockNumber` every 5–10 seconds for 2 minutes. + - Verify continuous increase at ~2-second intervals. + - Verify all validators remain in sync. + +6. **Check for errors:** + - Search all three `xdc.log` files for `max fee per gas less than block base fee`, `invalid baseFee`, and FATAL. + - Confirm no new occurrences after the patched validators started. + +--- + +## 8. Validation Results + +### 8.1 Process state + +| Validator | Binary | RPC port | P2P port | Status | +| --------- | ------ | -------- | -------- | ------ | +| Validator 1 | `/home/shalaka/xdposchain-v2.8.2/build/bin/XDC` | 8545 | 20303 | Running | +| Validator 2 | `/home/shalaka/xdposchain-v2.8.2/build/bin/XDC` | 8546 | 20304 | Running | +| Validator 3 | `/home/shalaka/xdposchain-v2.8.2/build/bin/XDC` | 8547 | 20305 | Running | + +### 8.2 Peer connectivity + +All three validators reported: + +```json +{"jsonrpc":"2.0","id":1,"result":"0x2"} +``` + +for `net_peerCount`, meaning each validator saw the other two. + +### 8.3 Block height progression + +Sample observation on Validator 1 over 120 seconds: + +| Time | Block height (hex) | Block height (decimal) | +| ---- | ------------------ | ---------------------- | +| T+0s | `0xfea9` | 65193 | +| T+10s | `0xfeae` | 65198 | +| T+20s | `0xfeb3` | 65203 | +| T+30s | `0xfeb8` | 65208 | +| T+40s | `0xfebe` | 65214 | +| T+50s | `0xfec3` | 65219 | +| T+60s | `0xfec8` | 65224 | +| T+70s | `0xfecd` | 65229 | +| T+80s | `0xfed2` | 65234 | +| T+90s | `0xfed7` | 65239 | +| T+100s | `0xfedd` | 65245 | +| T+110s | `0xfee2` | 65250 | +| T+120s | `0xfee7` | 65255 | + +- **Delta:** 62 blocks in 120 seconds +- **Average block time:** ~1.94 seconds (matches the XDPoS 2-second target) +- All three validators remained at the same block height throughout the observation. + +### 8.4 Error checks + +Searches across all three `xdc.log` files for the target strings after the patched validators started: + +| Error string | Occurrences after patch start | Result | +| ------------ | ----------------------------- | ------ | +| `max fee per gas less than block base fee` | 0 | ✅ Pass | +| `invalid baseFee` | 0 | ✅ Pass | +| `FATAL` | 0 | ✅ Pass | +| New `ERROR` (excluding `db` RPC module warning) | 0 | ✅ Pass | + +The only warnings observed were: +- The pre-existing `Unavailable modules in HTTP API list` message for `db` (harmless; the `db` module is not available in the HTTP API set). +- Periodic `[sendTimeout]` messages, which are expected in a 3-validator XDPoS network when a validator is slow to respond. + +### 8.5 Consensus stability + +The network produced blocks continuously for several minutes without reverting to the pre-patch failure mode. No new consensus or compatibility blockers appeared. + +--- + +## 9. Comparison Table: v2.6.1 vs v2.8.2 + +| Aspect | v2.6.1 | v2.8.2 unpatched | v2.8.2 patched (this fix) | +| ------ | ------ | ---------------- | -------------------------- | +| **Runtime configuration mechanism** | `common.CopyConstants` sets process-wide globals at startup based on chain ID | Fork blocks read only from stored `ChainConfig` | Fork blocks read from `ChainConfig`; fallback to `LocalnetChainConfig` for non-built-in custom chains | +| **`common.CopyConstants`** | Used; copies `localConstant` values for custom chains | Removed | Not needed; per-chain-config fallback replaces it | +| **Fork activation** | All Ethereum and XDC-specific forks effectively active at block 0 for custom chains | No forks active if stored fork blocks are `nil` | Same effective behavior as v2.6.1 for custom chains; built-ins unchanged | +| **EIP-1559 handling** | Active (`IsEIP1559` returns `true`) | Inactive if `EIP1559Block` is `nil` | Active for custom chains | +| **TIPSigning/TIPRandomize behavior** | Active (`IsTIPSigning`/`IsTIPRandomize` return `true`) | Inactive if `TIPSigningBlock`/`TIPRandomizeBlock` are `nil` | Active for custom chains | +| **BlockSigner routing** | Routed through `ApplySignTransaction`, bypassing base-fee validation | Treated as normal transaction; fails EIP-1559 base-fee check | Correctly routed, bypassing base-fee validation | +| **BaseFee validation** | Applied to user transactions, not BlockSigner | Applied to BlockSigner incorrectly, causing block import failure | Applied only to user transactions | +| **EVM instruction set** | `cancunInstructionSet` from block 0 | `frontierInstructionSet` if all predicates false | `cancunInstructionSet` from block 0 for custom chains | +| **Legacy custom-chain compatibility** | Works | Broken | Restored | +| **Built-in network behavior** | Mainnet/testnet/devnet use their own configs | Mainnet/testnet/devnet use their own configs | Unchanged | +| **Files involved in the fix** | `common/constants*.go`, `params/config.go` | `params/config_forks.go` | `params/config_forks.go` only | +| **Pros of implementation** | Simple global model | Clean, no globals | Clean, no globals; minimal scope; preserves built-in behavior | +| **Cons of implementation** | Globals are error-prone and hard to test | Breaks legacy custom chains | Requires an explicit fallback helper; new custom chains should set fork blocks in genesis | + +--- + +## 10. Flow Diagrams + +### 10.1 Fork predicate evaluation (patched v2.8.2) + +```text +┌─────────────────┐ +│ IsTIPSigning │ +│ (or IsEIP1559 │ +│ etc.) │ +└────────┬────────┘ + │ + ▼ +┌─────────────────────────┐ +│ c.TIPSigningBlock nil? │ +│ (or other fork block) │ +└────────┬────────────────┘ + │ + ┌────┴────┐ + ▼ ▼ +┌──────┐ ┌──────────────────────────────┐ +│ No │ │ Yes │ +│ │ │ │ +│ use │ │ ChainID known built-in? │ +│stored│ │ (mainnet/testnet/devnet) │ +│block │ └────────┬───────────────────────┘ +│ │ │ +│ │ ┌────┴────┐ +│ │ ▼ ▼ +│ │ ┌──────┐ ┌──────────────┐ +│ │ │ Yes │ │ No │ +│ │ │ │ │ │ +│ │ │return│ │ use Localnet │ +│ │ │ nil │ │ ChainConfig │ +│ │ │ │ │ default │ +│ │ └──┬───┘ │ │ +│ │ │ └──────┬───────┘ +│ │ │ │ +│ │ └──────┬──────┘ +│ │ │ +│ └────────────┘ +│ │ + ▼ + ┌─────────────┐ + │ isForked() │ + │ evaluates │ + │ active fork │ + └─────────────┘ +``` + +### 10.2 BlockSigner transaction flow + +```text +Block received by validator + │ + ▼ +┌─────────────────────┐ +│ IsTIPSigning(block) │ +│ returns true? │ +└──────────┬────────────┘ + │ + ┌────┴────┐ + ▼ ▼ +┌────────┐ ┌──────────────┐ +│ Yes │ │ No │ +│ │ │ (unpatched) │ +└───┬────┘ └──────┬───────┘ + │ │ + ▼ ▼ +┌────────────────┐ ┌─────────────────────────┐ +│ ApplySignTx() │ │ ApplyMessage() │ +│ - gas price 0 │ │ - standard EIP-1559 │ +│ - no base-fee │ │ validation │ +│ validation │ │ - maxFeePerGas = 0 │ +│ - accepted │ │ < baseFee │ +└────────────────┘ │ - BLOCK IMPORT FAILS │ + └─────────────────────────┘ +``` + +### 10.3 Validation workflow + +```text +Patch params/config_forks.go + │ + ▼ +Rebuild XDC v2.8.2 + │ + ▼ +Update start-native-validator*.sh + │ + ▼ +Start Validator 1 + │ + ▼ +Check: version, no errors, peer count 0 + │ + ▼ +Start Validators 2 & 3 + │ + ▼ +Check: all peers connected, block height in sync + │ + ▼ +Poll eth_blockNumber for 2+ minutes + │ + ▼ +Verify: continuous block production, no fee errors +``` + +--- + +## 11. Remaining Limitations and Future Considerations + +### 11.1 Future custom chains should set fork blocks explicitly + +The `effectiveForkBlock` fallback restores compatibility for **legacy** custom chains that were created before `v2.8.2`. New custom chains should ideally write the intended fork blocks directly into `genesis.json` and `ChainConfig`, rather than relying on the fallback. This makes the configuration explicit and easier to reason about. + +### 11.2 `common.CopyConstants` is gone + +`v2.8.2` removed the `common.CopyConstants` mechanism entirely. Any tooling, scripts, or documentation that relied on it must be updated. The fallback helper is the replacement, but it lives in `params/config_forks.go`, not in `common`. + +### 11.3 This fix does not address unrelated v2.8.2 changes + +The patch only restores fork-predicate behavior. Other behavioral changes between `v2.6.1` and `v2.8.2` (e.g., gas-limit logic, RPC changes, transaction-pool rules, P2P protocol changes) are not covered. Long-running validation on a non-trivial workload is recommended before any production upgrade. + +### 11.4 Log verbosity limits observability + +At `--verbosity 2`, successful block import and consensus commit messages are not written to the log. For deeper troubleshooting, consider raising verbosity temporarily, but be aware that `v2.8.2` can produce a large volume of logs at higher levels. + +### 11.5 The fallback is scoped to non-built-in chain IDs + +If a custom chain was created with a chain ID that collides with a known built-in ID (mainnet, testnet, devnet), the fallback will not apply. This is by design and should not affect properly generated subnets. + +--- + +## 12. Related Documents + +- `fork-predicate-compatibility-analysis.md` — detailed predicate-by-predicate analysis that led to the full fix. +- `MIGRATION.md` — Docker-to-native migration guide for this subnet. +- `SUBNET_ANALYSIS.md` — analysis of the subnet configuration and version choices. +- `MANUAL_DEPLOYMENT_PLAN.md` — full manual deployment plan. +- `generated/README.md` — generated documentation for the Docker subnet. + +--- + +## 13. Conclusion + +The `v2.6.1` → `v2.8.2` upgrade broke this legacy custom subnet because `v2.8.2` removed the process-wide fork-block globals that custom chains relied on. The implemented `effectiveForkBlock` fallback in `params/config_forks.go` restores the `v2.6.1` runtime behavior for non-built-in custom chains without affecting mainnet, testnet, or devnet. + +Validation on the live 3-validator subnet confirmed: + +- All three validators run the patched `v2.8.2` binary. +- All three validators connect to each other. +- Block height increases continuously at ~2-second intervals. +- All validators remain in sync. +- No `max fee per gas less than block base fee` errors occur. +- No `invalid baseFee` or fatal errors occur. + +The compatibility fix is working as intended. diff --git a/README.md b/README.md new file mode 100644 index 0000000..a0e980b --- /dev/null +++ b/README.md @@ -0,0 +1,137 @@ +# XDC Subnet — Docker & Native Validator Deployment + +This repository contains a complete, reproducible setup for an XDPoS private subnet (chainId `7670`) running on a single host. It supports both: + +- **Docker-based deployment** via `docker-compose` (original setup) +- **Native binary deployment** using a self-built `XDPoSChain` binary + +--- + +## Repository Layout + +``` +~/subnet/ +├── README.md # This file +├── start_xdpos.sh # Generator launcher: pulls the subnet-generator image and starts it +├── MIGRATION.md # Docker → native migration guide and implementation report +├── MIGRATION_PLAN.md # Original migration planning document +├── SUBNET_ANALYSIS.md # Deep analysis of the subnet configuration +├── MANUAL_DEPLOYMENT_PLAN.md # Detailed manual deployment plan +├── generated/ # Files produced by the subnet generator +│ ├── README.md # Generated documentation for the Docker subnet +│ ├── docker-compose.yml # Docker Compose services (3 validators + bootnode + Redis) +│ ├── docker-up.sh # Start Docker subnet +│ ├── docker-down.sh # Stop Docker subnet +│ ├── gen.env # Basic network config (name, subnet count, IP) +│ ├── genesis_input.yml # Human-readable genesis input +│ ├── genesis.json # Machine-readable genesis block (committed) +│ ├── bootnodes.list # Bootnode enode URL used by validators +│ ├── scripts/ # Health-check scripts +│ │ ├── check-mining.sh +│ │ └── check-peer.sh +│ ├── start-native-validator1.sh # Native validator 1 startup script +│ ├── start-native-validator2.sh # Native validator 2 startup script +│ └── start-native-validator3.sh # Native validator 3 startup script +└── manual/ # Alternative manual deployment artifacts + ├── IMPLEMENTATION_REPORT.md # Report on the completed manual migration + ├── scripts/ # Manual migration and lifecycle scripts + └── bootnodes/ # Manual bootnode configuration +``` + +--- + +## Quick Start + +### 1. Generate a New Subnet (Docker-first) + +```bash +cd ~/subnet +./start_xdpos.sh +``` + +This pulls `xinfinorg/subnet-generator:v2.1.0` and starts a local web UI on port `5210`. Open `http://localhost:5210/gen_xdpos` to generate the Docker subnet files. + +> ⚠️ This will create files under `generated/`. Runtime data such as `xdcchain*/`, `keys.json`, `masternode*.env`, and `.pwd` will be ignored by `.gitignore`. + +### 2. Start the Docker Subnet + +```bash +cd ~/subnet/generated +./docker-up.sh machine1 +``` + +Verify: + +```bash +./scripts/check-mining.sh +./scripts/check-peer.sh +``` + +### 3. Migrate to Native Binaries (Optional) + +See [`MIGRATION.md`](MIGRATION.md) for the full step-by-step guide. In short: + +1. Build the correct `XDPoSChain` commit (`53e56018250499fecb277f0049471f7990c50591` for `v2.6.1-beta`). +2. Stop the Docker validator containers. +3. Run the native startup scripts: + +```bash +bash ~/subnet/generated/start-native-validator1.sh +bash ~/subnet/generated/start-native-validator2.sh +bash ~/subnet/generated/start-native-validator3.sh +``` + +4. Verify block production and peer counts as described in [`MIGRATION.md`](MIGRATION.md). + +--- + +## Network Parameters + +| Parameter | Value | +|-----------|-------| +| Network name | `xdcSubnet` | +| Chain ID | `7670` | +| Consensus | XDPoS v2 | +| Block time | 2 seconds | +| Epoch | 900 blocks | +| Masternodes | 3 | +| P2P ports | `20303`, `20304`, `20305` | +| RPC ports | `8545`, `8546`, `8547` | +| WebSocket ports | `9555`, `9556`, `9557` | + +--- + +## Important Security Notes + +- **Never commit `keys.json`, `masternode*.env`, or `.pwd`**. These files contain private keys and are ignored by `.gitignore`. +- The genesis file `generated/genesis.json` is committed because it is public chain configuration and required for reproducibility. +- Validator data directories (`generated/xdcchain*/`) are large and runtime-specific; they are ignored. + +--- + +## Useful Commands + +| Task | Command | +|------|---------| +| Start Docker subnet | `./generated/docker-up.sh machine1` | +| Stop Docker subnet | `./generated/docker-down.sh machine1` | +| Start native validator 1 | `bash ~/subnet/generated/start-native-validator1.sh` | +| Check block height | `curl -s -X POST http://localhost:8545 -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'` | +| Check mining | `curl -s -X POST http://localhost:8545 -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","method":"eth_mining","params":[],"id":1}'` | +| Check peers | `curl -s -X POST http://localhost:8545 -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","method":"net_peerCount","params":[],"id":1}'` | + +--- + +## Documentation Index + +- [`generated/README.md`](generated/README.md) — Detailed reference for the generated Docker subnet. +- [`MIGRATION.md`](MIGRATION.md) — How the Docker validators were migrated to native binaries. +- [`MIGRATION_PLAN.md`](MIGRATION_PLAN.md) — Planning document for the migration. +- [`SUBNET_ANALYSIS.md`](SUBNET_ANALYSIS.md) — Analysis of the subnet configuration and version choices. +- [`MANUAL_DEPLOYMENT_PLAN.md`](MANUAL_DEPLOYMENT_PLAN.md) — Full manual deployment plan. +- [`manual/IMPLEMENTATION_REPORT.md`](manual/IMPLEMENTATION_REPORT.md) — Report on the manual migration outcome. +- [`LEGACY_CUSTOM_CHAIN_COMPATIBILITY.md`](LEGACY_CUSTOM_CHAIN_COMPATIBILITY.md) — v2.6.1→v2.8.2 compatibility root cause, fix, and validation for legacy custom subnets. + +--- + +*Repository: `https://github.com/shhalaka/xdc-subnet`* diff --git a/SUBNET_ANALYSIS.md b/SUBNET_ANALYSIS.md new file mode 100644 index 0000000..a25d014 --- /dev/null +++ b/SUBNET_ANALYSIS.md @@ -0,0 +1,439 @@ +================================================================================ + XDC SUBNET: COMPLETE ANALYSIS REPORT +================================================================================ +Location: /home/shalaka/subnet/ +Generated: June 24, 2026 +Network: xdcSubnet (Chain ID 7670) + +================================================================================ +1. WHAT IS THIS? +================================================================================ + +This is a 3-node XDC Blockchain Subnet running locally via Docker. It is a +private/permissioned blockchain network, NOT a public chain like Ethereum mainnet. + + XDC = XinFin Digital Contract (EVM-compatible blockchain) + Subnet = A private sidechain / permissioned network for enterprise use + +Think of it as a "mini Ethereum" that only YOU control, with 3 validator nodes +that take turns creating blocks every 2 seconds. + +-------------------------------------------------------------------------------- +KEY PARAMETERS +-------------------------------------------------------------------------------- + Chain ID: 7670 + Network Name: xdcSubnet + Consensus: XDPoS v2 (XinFin Delegated Proof of Stake) + Block Time: 2 seconds + Epoch Length: 900 blocks + Block Reward: 171 XDC per block + Gap: 450 blocks + Max Masternodes: 108 + Current Status: ACTIVE (mining blocks continuously) + +================================================================================ +2. CONSENSUS: XDPoS v2 (XinFin Delegated Proof of Stake) +================================================================================ + +XDPoS v2 is a Byzantine Fault Tolerant (BFT) consensus mechanism. It is NOT +Proof of Work (no mining rigs, no electricity waste). It is also NOT simple +Proof of Stake (no slashing, no random selection). + +-------------------------------------------------------------------------------- +HOW XDPoS v2 WORKS +-------------------------------------------------------------------------------- + + 1. VALIDATOR SET (MASTERNODES) + - Only pre-approved nodes can create blocks. + - In this subnet: exactly 3 masternodes. + - These are defined in genesis.json extraData and the XDCValidator + smart contract (address 0x000...0088). + + 2. ROUND-ROBIN BLOCK PRODUCTION + - Masternodes take turns creating blocks in a fixed order. + - Each node gets a "turn" every 2 seconds. + - The "whosTurn" field in logs shows which node should mine next. + + 3. QUORUM CERTIFICATES (QC) + - After a block is proposed, other masternodes vote on it. + - A block is FINALIZED when it receives a Quorum Certificate (QC). + - QC requires at least 2 out of 3 votes (certificateThreshold = 0.667). + - This is the "BFT" part: the network tolerates 1 faulty node. + + 4. TIMEOUT MECHANISM + - If the designated masternode fails to produce a block within 10 seconds + (timeoutPeriod), other nodes send a "timeout" message. + - After 3 timeout messages (timeoutSyncThreshold), a new round starts + and the next masternode takes over. + + 5. EPOCHS + - An epoch = 900 blocks. + - At each epoch boundary, the validator set can be re-evaluated. + - Reward distribution happens at epoch checkpoints (rewardCheckpoint=900). + +-------------------------------------------------------------------------------- +XDPoS v2 vs OTHER CONSENSUS MECHANISMS +-------------------------------------------------------------------------------- + + +----------------+----------------+----------------+----------------+ + | Feature | Proof of Work | Proof of Stake | XDPoS v2 | + +----------------+----------------+----------------+----------------+ + | Energy Use | Very High | Low | Very Low | + | Block Time | ~12 sec (ETH) | ~12 sec (ETH) | ~2 sec (XDC) | + | Finality | Probabilistic | Probabilistic | Instant (BFT) | + | Validators | Anyone (open) | Stake-weighted | Pre-approved | + | Fault Tolerance| None | Economic | 1/3 BFT | + | Best For | Public chains | Public chains | Enterprise | + +----------------+----------------+----------------+----------------+ + +-------------------------------------------------------------------------------- +WHY THE LOGS SAY "parent hash and QC hash does not match" +-------------------------------------------------------------------------------- + +This is a NORMAL and EXPECTED warning in XDPoS v2. It happens during validator +rotation when a node receives a block proposal and the Quorum Certificate (QC) +references a slightly different parent hash due to network timing. The node +still accepts the block if the QC is valid. You see this because the 3 nodes +are competing slightly and reorganizing ("Reorg" in logs) to agree on the canonical +chain. + +================================================================================ +3. HOW NODES ARE MINING (THE FLOW) +================================================================================ + +-------------------------------------------------------------------------------- +STEP-BY-STEP BLOCK PRODUCTION FLOW +-------------------------------------------------------------------------------- + + STEP 1: BOOTSTRAP + ----------------- + - Bootnode starts first (192.168.25.54:20301). + - It does NOT mine blocks. It only helps other nodes discover each other. + - All masternodes connect to the bootnode to find peers. + + STEP 2: GENESIS INITIALIZATION + ------------------------------- + - Each masternode reads genesis.json on startup. + - genesis.json contains: + * Chain ID (7670) + * Initial validator set (in extraData) + * Pre-deployed smart contracts (XDCValidator, XDCBlockReward, etc.) + * Pre-allocated balances (Owner, Foundation, etc.) + - All nodes must agree on the EXACT same genesis block. + + STEP 3: PEER DISCOVERY + ----------------------- + - Masternode1 connects to bootnode, learns about Masternode2 and 3. + - Masternode2 connects to bootnode, learns about Masternode1 and 3. + - Masternode3 connects to bootnode, learns about Masternode1 and 2. + - Result: fully connected mesh (each node sees 2 peers). + + STEP 4: CONSENSUS ROUND STARTS + ------------------------------- + - The network determines "whose turn" it is to mine the next block. + - Deterministic round-robin based on block number and validator set order. + - Example from logs: whosTurn=xdc60C8D53B7b788dAEb3a461Ec63D07D4023D99C0D + This means Masternode2 (key2) is designated to produce the next block. + + STEP 5: BLOCK PROPOSAL + ---------------------- + - The designated masternode: + a) Collects pending transactions from its mempool. + b) Creates a new block with: + - Parent hash (previous block) + - Transactions + - Timestamp + - Quorum Certificate (QC) from previous round + - Signature from the proposer + c) Broadcasts the block to peers. + + STEP 6: VOTING (QC FORMATION) + ----------------------------- + - Other masternodes receive the proposed block. + - They validate it (check parent hash, transactions, signature, turn). + - If valid, they send a VOTE message back to the proposer. + - Once the proposer collects >= 2 votes (67% threshold), it forms a QC. + + STEP 7: BLOCK FINALIZATION + --------------------------- + - The block with its QC is now FINAL. + - It cannot be reversed (unlike PoW where you need "confirmations"). + - All nodes append it to their local blockchain database (xdcchain*/XDC). + + STEP 8: REWARD DISTRIBUTION + --------------------------- + - Block reward (171 XDC) is minted. + - Rewards go to the masternode that proposed the block. + - At epoch boundaries (every 900 blocks), rewards are distributed according + to the XDCBlockReward smart contract rules. + + STEP 9: NEXT ROUND + ------------------ + - The next masternode in the round-robin order gets its turn. + - Repeat from Step 4. + +-------------------------------------------------------------------------------- +THE ACTUAL MINING COMMAND +-------------------------------------------------------------------------------- + +Each masternode runs the XDC client (a modified Go-Ethereum/Geth) with flags: + + --mine (enable block production) + --syncmode full (full sync, not light) + --gcmode archive (keep all historical state) + --networkid 7670 (custom chain ID) + --bootnodes (connect to bootnode for peer discovery) + +The "mining" here is NOT solving cryptographic puzzles (no hash rate). It is +simply "signing and proposing a block" when it is your turn. + +================================================================================ +4. NETWORK ARCHITECTURE +================================================================================ + + Docker Network: 192.168.25.0/24 (docker_net) + + +------------------+ +------------------+ +------------------+ + | Masternode 1 |<--->| Masternode 2 |<--->| Masternode 3 | + | Validator | | Validator | | Validator | + | .51:20303 | | .52:20304 | | .53:20305 | + | RPC: 8545 | | RPC: 8546 | | RPC: 8547 | + | WS: 9555 | | WS: 9556 | | WS: 9557 | + | key1 (0x994e..) | | key2 (0x60C8..) | | key3 (0x5635..) | + +--------+---------+ +--------+---------+ +--------+---------+ + | | | + | | | + +------------------------+--------+ + | + +----------v-----------+ + | Bootnode | + | .54:20301 | + | Peer Discovery Only | + | (Does NOT mine) | + +----------------------+ + +-------------------------------------------------------------------------------- +NODE IDENTITIES +-------------------------------------------------------------------------------- + + Masternode 1 + Address: 0x994e38673C40B1F21B087dAFd04AcBBD6bcd970B + Private: 0xeeddecec51a10059b497301faf089af400075dedbad2d0309767ea4b4618fcde + RPC: http://localhost:8545 + Data Dir: generated/xdcchain1/ + + Masternode 2 + Address: 0x60C8D53B7b788dAEb3a461Ec63D07D4023D99C0D + Private: 0x93ec6fc4e693c1c8d6e5149de2ce7a89244ae2e629fb64af3d2ed2c4baca10bc + RPC: http://localhost:8546 + Data Dir: generated/xdcchain2/ + + Masternode 3 + Address: 0x56353Ac7eD5aDa864f411c9719A2Cc416d408FbE + Private: 0xe9c235075d4af6ca6c7245a6b5e5b4b43a4d1b2f7cabff272732abb7315e6614 + RPC: http://localhost:8547 + Data Dir: generated/xdcchain3/ + + Owner + Address: 0x809B71A615F1118D943e9424C24CDF7922c29bc8 + Private: 0xd6763621556b1182214045458b9c9a252ed74c96afec3992cccaf3fbe36bec24 + Role: Network owner, can add/remove validators + + Foundation + Address: 0x16aE7a8d946baA913349D1Daa7616966C8BbF8F2 + Private: 0x2de65d5f62a7319714401e3bbd0959465ad5363cd287bc193cc720f70e046de2 + Role: Receives foundation rewards + +================================================================================ +5. SMART CONTRACTS IN GENESIS +================================================================================ + +The genesis block pre-deploys several system smart contracts at fixed addresses: + + 0x000...0068 - XDCValidator + Manages the validator set, staking, voting, penalties. + This is the HEART of XDPoS consensus. + + 0x000...0088 - XDCBlockReward + Handles block reward distribution to validators. + + 0x000...0089 - Randomness contract + Provides on-chain randomness for certain operations. + + 0x000...0090 - XDCValidator (v2 / additional logic) + + 0x000...0099 - Pre-funded account with large balance + +These contracts are written in Solidity and their bytecode is embedded in the +"alloc" section of genesis.json. They are NOT deployed by users; they exist +from block 0. + +================================================================================ +6. FOLDER STRUCTURE +================================================================================ + + ~/subnet/ + |-- start_xdpos.sh # Script to launch the generator UI + |-- generated/ # ALL network data lives here + |-- docker-compose.yml # Container definitions + |-- docker-up.sh # START the network + |-- docker-down.sh # STOP the network + |-- gen.env # Basic network config + |-- genesis_input.yml # Human-readable genesis input + |-- genesis.json # Machine-readable genesis block + |-- keys.json # ALL PRIVATE KEYS (SECRET!) + |-- masternode1.env # Node 1 config + |-- masternode2.env # Node 2 config + |-- masternode3.env # Node 3 config + |-- bootnodes.list # Bootnode enode address + |-- bootnodes/ # Bootnode data + |-- scripts/ + | |-- check-mining.sh # Verify blocks are being mined + | |-- check-peer.sh # Verify peer connections + |-- xdcchain1/ # Node 1 blockchain data + | |-- XDC/ # Main chain database + | |-- XDCx/ # Light indexing database + | |-- keystore/ # Encrypted wallet files + | |-- XDC.ipc # Unix socket for local API + | |-- xdc.log # Node runtime logs + |-- xdcchain2/ # Node 2 blockchain data + |-- xdcchain3/ # Node 3 blockchain data + +================================================================================ +7. HOW TO INTERACT WITH THE NETWORK +================================================================================ + + START THE NETWORK + ----------------- + cd ~/subnet/generated + ./docker-up.sh machine1 + + STOP THE NETWORK + ---------------- + cd ~/subnet/generated + ./docker-down.sh machine1 + + CHECK IF MINING + --------------- + cd ~/subnet/generated + ./scripts/check-mining.sh + + CHECK PEER CONNECTIONS + ---------------------- + cd ~/subnet/generated + ./scripts/check-peer.sh + (Should show 2 peers per node) + + QUERY LATEST BLOCK (via RPC) + ---------------------------- + curl -s http://localhost:8545 -X POST \ + -H "Content-Type: application/json" \ + -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' + + QUERY BLOCK DETAILS (XDPoS v2) + ------------------------------ + curl -s http://localhost:8545 -X POST \ + -H "Content-Type: application/json" \ + -d '{"jsonrpc":"2.0","method":"XDPoS_getV2BlockByNumber","params":["latest"],"id":1}' + + GET PEER COUNT + -------------- + curl -s http://localhost:8545 -X POST \ + -H "Content-Type: application/json" \ + -d '{"jsonrpc":"2.0","method":"net_peerCount","params":[],"id":1}' + + VIEW NODE LOGS + -------------- + docker logs -f generated-masternode1-1 + docker logs -f generated-masternode2-1 + docker logs -f generated-masternode3-1 + + CHECK RUNNING CONTAINERS + ------------------------ + docker ps + +================================================================================ +8. CURRENT STATUS (FROM LOGS) +================================================================================ + + Network Status: ACTIVE and MINING + Latest Block: ~8944+ (as of last log entry) + Block Time: ~2 seconds + Peers: 2 per node (fully connected) + Reorgs: Occasional (normal for XDPoS v2 during rotation) + Warnings: "parent hash and QC hash does not match" (NORMAL) + "sendTimeout" (NORMAL during startup/rotation) + + The network has been running continuously since June 23, producing thousands + of blocks. All 3 masternodes are participating in consensus. + +================================================================================ +9. IMPORTANT SECURITY NOTES +================================================================================ + + 1. PRIVATE KEYS + The file keys.json contains ALL private keys in PLAIN TEXT. + In a production environment, these MUST be stored securely (HSM, vault, + encrypted keystore). NEVER commit this file to Git. + + 2. LOCAL ONLY + This setup binds to localhost (127.0.0.1). The RPC ports (8545-8547) are + NOT exposed to the internet. If you need remote access, use a reverse proxy + with authentication (e.g., nginx + JWT). + + 3. ARCHIVE MODE + All nodes run with GC_MODE=archive, meaning they keep ALL historical state. + This is great for debugging but uses significant disk space over time. + + 4. DOCKER PRIVILEGES + Some data folders are owned by root because Docker runs as root. + Use sudo for file operations if needed, or adjust Docker user mapping. + +================================================================================ +10. TROUBLESHOOTING +================================================================================ + + Problem: Nodes not connecting + Solution: Check bootnode is running: docker logs generated-bootnode-1 + Verify bootnodes.list has the correct enode address. + + Problem: No blocks being mined + Solution: Run ./scripts/check-mining.sh + Check peer count: ./scripts/check-peer.sh + Verify all 3 containers are up: docker ps + + Problem: Port conflicts + Solution: Ensure ports 20303-20305, 8545-8547, 9555-9557 are free. + Check: sudo lsof -i :8545 + + Problem: Permission denied on xdcchain*/ folders + Solution: sudo chown -R $(whoami):$(whoami) generated/xdcchain* + + Problem: "parent hash and QC hash does not match" warnings + Solution: This is NORMAL. No action needed. It is part of XDPoS v2 consensus. + + Problem: Chain forked / nodes disagree + Solution: Stop all nodes, delete xdcchain*/XDC and xdcchain*/XDCx folders + (keep keystore/), then restart. They will re-sync from genesis. + +================================================================================ +11. GLOSSARY +================================================================================ + + XDC XinFin Digital Contract - the blockchain platform + XDPoS XinFin Delegated Proof of Stake - consensus algorithm + XDPoS v2 Second generation with BFT and Quorum Certificates + Masternode A validator node authorized to produce blocks + Bootnode A helper node for peer discovery (does not mine) + Genesis The first block of the blockchain (block 0) + Epoch A period of 900 blocks; validator set can change + QC Quorum Certificate - proof that >=2/3 validators agree + BFT Byzantine Fault Tolerance - tolerates up to 1/3 faulty nodes + Reorg Reorganization - when nodes switch to a different chain tip + Enode Ethereum node identifier (public key + IP + port) + ExtraData Field in block header containing validator signatures + Turn Which masternode's turn it is to propose the next block + +================================================================================ + END OF REPORT +================================================================================