[Feature Request] Validate available disk space before accepting an offline _split request

[shiv0408]

Is your feature request related to a problem? Please describe

The offline _split API (POST /<index>/_split/<target-index>, ResizeType.SPLIT) performs no disk-space validation at request time. A split can be accepted when the node hosting the parent primary does not have enough free space to complete it; the failure only surfaces later as target shards stuck UNASSIGNED, discoverable solely via _cluster/allocation/explain.

How much space a split actually needs

Let S = the parent shard's primary store size. Splitting that shard requires at least S of additional free space on the node hosting the parent primary — i.e. room for roughly one more full copy of the shard. This floor holds regardless of how many child shards the parent is split into:

1. StoreRecovery.addIndices (server/src/main/java/org/opensearch/index/shard/StoreRecovery.java:220-274) wraps the target directory in Lucene's HardlinkCopyDirectoryWrapper (:237), so writer.addIndexes(sources) hardlinks the parent's segment files into each child. In the common case this adds ~0 physical bytes, since hardlinks share inodes.
2. writer.deleteDocuments(new ShardSplittingQuery(...)) (:257-259) is a hard delete that only writes per-segment .liv live-docs bitsets (~maxDoc/8 bytes). It frees nothing.
3. The subsequent merge — indexShard.getIndexer().forceMerge(false, -1, ...) (:206), which resolves to IndexWriter.maybeMerge() (InternalEngine.java:1701-1703) — rewrites each child's live documents into new, unshared segment files and only then drops that child's hardlinks. Summed across the children this is ~S of newly allocated physical bytes, on top of the parent, which still occupies its own S until the source index is deleted.

Two consequences worth calling out:

• Because hardlinks are refcounted at the inode level, deleting the source index frees nothing while any child still links its segment files.
• The footprint therefore grows after the split reports complete, during step 3 — past the point where any allocation-time disk check is still looking.

Why the existing disk accounting doesn't cover this

• DiskThresholdDecider.getExpectedShardSize (.../decider/DiskThresholdDecider.java:645-681) does correctly estimate each split child at the full parent size S (via IndexMetadata.selectRecoverFromShards, which returns a singleton for split), so the per-child canAllocate check is reasonable in isolation.
• However DiskThresholdDecider.sizeOfRelocatingShards (:130-137) explicitly skips these shards from its accounting. Split children are INITIALIZING with relocatingNodeId == null (ShardRouting.initialize, ShardRouting.java:544-564), so they hit continue under the comment: "the only initializing-but-not-relocating shards with a nonzero expected shard size will be ones created by a resize (shrink/split/clone) operation which we expect to happen using hard links, so they shouldn't be taking any additional space and can be ignored here."
• The ClusterInfo.ReservedSpace backstop does not fire either. Hardlinked files are recorded as reused — StoreRecovery.java:345: index.addFileDetail(dest, l, true); // hardlinked - we treat it as reused — and reused files are excluded from bytesStillToRecover(). Reserved space therefore reads as ~0 (or UNKNOWN, since setFileDetailsComplete() is only called after addIndexes returns, :255-256).
• There is no request-time check at all. MetadataCreateIndexService.validateSplitIndex (.../metadata/MetadataCreateIndexService.java:1890-1893) validates only routing-shard arithmetic and the write block. TransportResizeAction (.../shrink/TransportResizeAction.java:145-231) does fetch store stats, but consumes them only for shrink's max_shard_size calculation (:286-288); for split they are unused.

Net effect: the accounting assumes hardlinks make a split free, so the ~S of additional space the post-split merge genuinely requires is never validated before the split is accepted.

Describe the solution you'd like

1. Add a reusable helper in DiskThresholdDecider that answers whether a given node can accommodate N additional bytes without breaching the configured high watermark (reusing DiskThresholdSettings, ClusterInfo disk usages, and ReservedSpace).
2. Call it from the split validation path (alongside validateSplitIndex) with S = the parent shard's primary store size, so that a _split request is rejected synchronously with a clear error when the node hosting the parent primary lacks ~S of free space — instead of being accepted and silently producing unassigned shards.
3. Keep the check to the 1 × S floor. That is the requirement that always holds, and it is simple to compute from data already available at request time.

DiskThresholdDecider is the natural home for (1) because the same per-node "can this node take another N bytes" question is needed by the in-place/online shard split effort (MetadataInPlaceSplitShardService, SplitShardsMetadata, RecoverySource.Type.IN_PLACE_SPLIT_SHARD) once its allocation/routing layer lands. That work has "Disk, max shards per node/index, awareness validation changes" as an open, unfiled item in the tracking meta-issue #13254, so a shared helper avoids building this twice.

Related component

Indexing

Describe alternatives you've considered

No response

Additional context

• Related meta-issue: [META] In-place Shard Splitting #13254 ([META] In-place Shard Splitting) — lists "Disk, max shards per node/index, awareness validation changes" as an unaddressed task.
• Related RFC: [RFC] In-place Shard Splitting #12918 ([RFC] In-place Shard Splitting)