ha.adoc: document how schema DDL replicates, and the arcadedb.ha.schemaDelta rolling-upgrade constraint (v26.10.1)

[robfrank]

src/main/asciidoc/how-to/operations/ha.adoc explains the replicate-first three-phase commit and arcadedb.ha.quorumTimeout for ordinary data transactions, but it never says how schema DDL reaches a follower. grep over the page finds no occurrence of SCHEMA_ENTRY, "recording session", "schema entry", or DDL. src/main/asciidoc/concepts/high-availability.adoc has the same gap in ==== Replication and the Raft Log.

That gap predates the 26.10.1 work, but three changes in that release make it operator-visible, and one of them is a way to silently diverge a cluster:

• HA: add a bulk schema scope so a DDL script replicates as one Raft entry instead of one per statement arcadedb#6990 (PR #7210): a DDL-only SQL script now commits as one Raft entry by default, holding the database write lock for the whole script.
• HA: every SCHEMA_ENTRY ships the full schema JSON instead of a delta arcadedb#6989 (PR #7211): a SCHEMA_ENTRY can carry a delta instead of the whole schema document, behind arcadedb.ha.schemaDelta (off by default).
• HA: every schema entry triggers a full LocalSchema.load() on the follower, making schema creation quadratic arcadedb#6988 (PR #7212): a follower applies a DDL entry incrementally instead of rebuilding every component.

1. A Schema replication subsection in ha.adoc

Near the three-phase-commit material. It should say that schema DDL travels as its own entry type rather than through the WAL path already documented, that a schema recording session is the entry boundary, and that with arcadedb.schemaBulkDDLScript on (the default) a whole DDL script is one such session - so a long script holds the database write lock for its full duration.

That last point has a consequence worth naming: on the leader an ordinary commit waits out an active recording session for at most arcadedb.ha.quorumTimeout (10s by default) before proceeding anyway. A DDL script that runs longer than that widens the window the ArcadeData/arcadedb#4083 guard covers. A long index build already opens the same window today, so this is not new, but batching makes it a common shape rather than a rare one. Point anyone who needs the old per-statement behaviour at arcadedb.schemaBulkDDLScript=false.

2. An anchored [[ha-schema-delta]] Schema Delta Replication (from v26.10.1) subsection

In the style of the existing <<ha-durable-raft-storage>> (L414) and <<ha-channel-self-healing>> (L366) subsections, because this one carries an upgrade-order constraint that nothing enforces at runtime:

Reading a delta requires no configuration; only emitting one is gated. Upgrade every node in the cluster to v26.10.1 or later first, then turn arcadedb.ha.schemaDelta on. A node running an older version cannot see the delta section - it stops decoding after the sections it knows - and applies a SCHEMA_ENTRY with an empty schema payload: it changes nothing, logs nothing, and its schema diverges from the leader's silently until WAL-version-gap detection or checkDatabase catches it. There is no peer-capability negotiation, so the leader cannot detect this and protect you (ArcadeData/arcadedb#7219). Turning the setting back off is safe at any time.

Two more sentences earn their place here:

• While the setting is on, the leader holds one parsed copy of the schema document per replicated database in heap - tens of MB per database on a multi-MB schema - released when it goes back off.
• The counters that show whether the delta path actually engaged are currently reachable only through RaftReplicatedDatabase.getSchemaDeltasShipped() / getSchemaDocumentsShipped() or arcadedb.ha.logVerbose=2, not through any metrics endpoint. So "I turned it on" and "it is being used" are not the same statement, and an operator has no easy way to check the second.

The ===== HA Settings table on the same page (L635) also needs rows for ha.schemaDelta and ha.schemaIncrementalApply.

3. One sentence in concepts/high-availability.adoc

==== Replication and the Raft Log (L42) describes only the three-phase WAL commit. A sentence noting that schema DDL travels as its own entry type, which since v26.10.1 can optionally carry a delta rather than the whole schema document, keeps the conceptual page honest.

---

Companion to #452, which covers the settings-reference rows for the same three settings. Filed from the post-merge pass over ArcadeData/arcadedb#6988, #6989, #6990.