Perf: batch all_reduce, replace deprecated torch.inverse, optimize hot-path numpy ops, enable pin_memory

[Copilot]

Targeted performance improvements across training loop, data loading, and matrix operations.

Distributed training

• Batch reduce_metrics: Single dist.all_reduce on a stacked tensor instead of N separate calls — eliminates N-1 unnecessary synchronization barriers per invocation
• Reduce sync frequency: reduce_metrics now only called at save/log intervals; per-iteration printing uses local averages

Data loading

• pin_memory=True on all DataLoaders — enables async CPU→GPU transfer paired with existing non_blocking=True in preprocess_data()

Deprecated API replacement

• torch.inverse() → torch.linalg.inv() across 5 files (11 callsites) — deprecated since PyTorch 1.9, linalg.inv is better optimized on modern CUDA backends

NumPy hot-path optimization

• find_closest_ind: Replace 25-line np.digitize + 3 mask branches + concatenation with 5-line vectorized np.searchsorted + np.clip:

def find_closest_ind(train, query):
    idx = np.searchsorted(train, query, side='left')
    idx = np.clip(idx, 1, len(train) - 1)
    left = idx - 1
    right = idx
    return np.where(np.abs(query - train[left]) < np.abs(query - train[right]), left, right)

• np.digitize → np.searchsorted in align.py and ensemble.py — digitize wraps searchsorted internally

Transform object overhead

• pad2square: Replace per-call v2.Pad() construction with direct F.pad, with reshape handling for 6D+ tensors in replicate mode

---

📍 Connect Copilot coding agent with Jira, Azure Boards or Linear to delegate work to Copilot in one click without leaving your project management tool.