MDEV-40378 heap: roll back key changes when a blob write fails in heap_update()

[arcivanov]

Problem

heap_update() moves all changed key entries to the new key values before writing the new blob chains. When a blob chain write then fails (e.g. HA_ERR_RECORD_FILE_FULL on a full HEAP table), the rollback restores the record bytes and blob chain pointers, but the err: label only undoes key changes for HA_ERR_FOUND_DUPP_KEY — historically the only possible failure once the key loop had run.

The hash/btree entries are left keyed on the new values while pointing at a record holding the old values, corrupting the index:

1. index lookups by the old key value miss the row;
2. CHECK TABLE reports the table corrupt;
3. on debug builds the heap consistency check in ha_heap::external_lock() raises a second error into an already-set diagnostics area, firing a Diagnostics_area assertion on the next statement.

Fix

Widen the err: recovery gate to also run for HA_ERR_RECORD_FILE_FULL, HA_ERR_OUT_OF_MEM and ENOMEM, so a failure raised after the key loop also moves every changed key back to its old value. One recovery path now serves every failure that leaves keys moved to their new values, including any future error source in the key loop itself.

The err: block assumed the failure happened inside the key loop, so that keydef addresses the partially processed keydef. A blob-chain write fails after that loop has run to completion, and therefore arrives with keydef == keydef_end. Reading info->errkey and keydef->algorithm from there addresses share->keydef[share->keys]; since sizeof(HP_KEYDEF) (888) far exceeds the key segments and blob descriptors that follow the keydef array, that read runs past the end of the HP_SHARE allocation, and the rollback sweep then dereferences a garbage keydef->seg in hp_rec_key_cmp(). The recovery therefore distinguishes the two failure sites: with keydef == keydef_end there is no partly updated key to repair and none to name in info->errkey, and the sweep starts at the last keydef instead. The same branch also covers a table with no keys at all (share->keys == 0), where the sweep has nothing to do.

info->errkey is initialized to -1 on entry to err:, so a failure that is not a key error can never expose a stale key number from an earlier operation.

The original errno is captured before the recovery and restored after it, so a rollback write_key failure (which hp_rb_write_key() reports as HA_ERR_FOUND_DUPP_KEY with a stale errkey) cannot mask it.

Test

New heap.blob_update_key_rollback exercises:

• hash index, single-row UPDATE;
• BTREE index, single-row UPDATE;
• two changed indexes (both rolled back);
• an index on an unchanged column (rollback must leave it untouched);
• a partial multi-row UPDATE (invariant-only assertions, no platform-dependent fill point);
• a table with no indexes at all (the keydef == keydef_end branch with share->keys == 0).

Each asserts table consistency after the failure via CHECK TABLE and index lookups. Verified to crash the unfixed server (DA assertion / index-corrupt) and pass with the fix; full heap suite passes.

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[gkodinov]

Thank you for your contribution. This is a preliminary review.

From the remarks in the Jira I'm assuming these are problems found with the main contribution. I still think it's best if this goes to main. But, after discussing with my colleagues, I'm approving it as is and leaving this to the final reviewer to reply.

[arcivanov]

No, this code is not on main, it's in preview 13.1. I literally can't PR this into main.