Fix crash in trig rasteriser from 16.16 fixed-point overflow

[cerwym]

Problem

At high resolutions (2560×1440, 3440×1440), the software triangle rasteriser crashes with EXCEPTION_ACCESS_VIOLATION in trig_render_md10 — the translucent shadow rendering path.

Root Cause

The triangle setup functions (trig_ll_start, trig_rl_start, etc.) compute (delta_x << 16) and (X << 16) using 16.16 fixed-point arithmetic in signed 32-bit integers. When vertex screen coordinates exceed ±32767 — which happens when rotpers() projects near-camera geometry at high resolutions — the left-shift silently overflows, corrupting the polyscans array and producing out-of-bounds framebuffer writes.

This overflow was always latent in the original Bullfrog x86 assembly (shll $0x10 wraps silently on x86) and was faithfully preserved in the ASM-to-C conversion (PR #1973). It simply never triggered at the original 640×480 resolution.

Fix

• Coordinate guard in trig(): Rejects triangles where any vertex X/Y exceeds ±32767 or where the X or Y span exceeds 32767, preventing overflow before it reaches the setup functions. Covers all ~25 rendering modes.
• OOB safety net in trig_render_md10(): Bounds-checks the output pointer against the poly_screen buffer and logs an error if triggered, preventing the access violation as a defence-in-depth measure.

[Copilot]

Pull request overview

Fixes a crash in the software triangle rasteriser at high resolutions by preventing 16.16 fixed-point overflow during triangle setup and adding a defensive bounds check in the translucent shadow render path (trig_render_md10).

Changes:

• Add coordinate range guarding in trig() to skip triangles that can overflow 16.16 fixed-point setup math.
• Add an out-of-bounds safety check for the output pixel pointer in trig_render_md10() to prevent access violations.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The lower-bound of this OOB check is too permissive: poly_screen is initialized as screenbuf - vec_screen_width, so allowing o >= poly_screen still permits o to point into the pre-buffer row (before vec_screen). To reliably prevent invalid reads/writes, compare against the real first valid pixel (e.g. vec_screen / poly_screen + vec_screen_width) rather than poly_screen itself.

[cerwym]

Done, checks start of buffer instead of poly_stream

[Copilot]

This bounds check runs inside the inner per-pixel loop (and can execute for many pixels when m[colM] is non-zero), which risks a noticeable slowdown in this hot path. Consider hoisting the check to once per scanline (validate the start/end pointers before the loop), and/or making it conditional on a debug/assert build, while still keeping a safe early-return when triggered.

[cerwym]

Done. hoisted out of inner loop