llama-finetune broken on modern transformers: cascading failures in SET_ROWS, FLASH_ATTN_EXT, and graph allocation

[pestopoppa]

Summary

llama-finetune fails on all modern transformer architectures (LLaMA 3.2, Qwen2.5, SmolLM2) due to three interconnected issues in the backward pass infrastructure. This affects builds b7405 through b7717 (current master).

Environment

• Build: b7717 (537d424), Linux x86_64, GCC 13.3.0
• Model: Llama-3.2-1B-Instruct-f32.gguf

Failures

#	Trigger	Location	Cause
1	Default	ggml.c:6928	GGML_OP_SET_ROWS not in allowed view-ops, no backward impl
2	--flash-attn	ggml.c:ggml_compute_backward()	GGML_OP_FLASH_ATTN_EXT missing backward case
3	Disable FA	ggml.c:6763	Graph node overflow from standard attention

Reproduction

./build/bin/llama-finetune \
  -m Llama-3.2-1B-Instruct-f32.gguf \
  -f train.txt -c 256 --epochs 1

Related

- #15279: Same view_src assertion (closed COMPLETED)
- #15090: Same graph overflow (closed STALE)

[pestopoppa]

Follow-up: Discovered existing FLASH_ATTN_BACK implementation

While investigating potential fixes, I found something interesting:

ggml_flash_attn_back() exists but is explicitly disabled:

// ggml/src/ggml.c:5335
struct ggml_tensor * ggml_flash_attn_back(...) {
    GGML_ABORT("TODO: adapt to ggml_flash_attn_ext() changes");
    // ... full implementation follows but never executes ...
}

The CPU implementation also exists at ggml-cpu/ops.cpp:8352-8684 (ggml_compute_forward_flash_attn_back_f32) but is unreachable.

Questions for maintainers

Before assuming this could be revived:

1. Why was the GGML_ABORT added? Is there a known incompatibility between the old FLASH_ATTN_BACK and the new FLASH_ATTN_EXT tensor layout?

2. Are there correctness concerns? The external attempt at Pints-App/llama.cpp#1 was closed due to correctness issues—is the existing implementation known to have similar problems?

3. Are there plans to update this? PR CPU/CUDA: Gemma 2 FlashAttention support #8542 mentioned "work on the LLM backwards pass in the coming months" (from ~8 months ago).

Understanding the history here would help us determine whether reviving this code is feasible or if there are deeper architectural issues we should be aware of.

[ggerganov]

@pestopoppa These AI comments/issues/PRs are really not helping atm and are only causing heated discussions. I try to keep moderation at the very minimum as it is not something I enjoy doing. So given there were already a few complaints about your approach, my advice is to take a break from using the llama.cpp repo for at least a few weeks and reconsider your approach if you really wish to contribute in a meaningful way. If these generated attempts continue, I will have to block you.

[pestopoppa]

Sure. If it makes you and others happier I'll abide. Fwiw it's not like I'm one-shot yoloing single-prompt stuff into here. Behind the scenes are many many hours of processing, attempting to understand why certain things work and other don't, feedback drafting of what gets pushed, etc. At the end of the day, if somethign is wrong in my bug reports it's because of my misunderstanding, not because of the AI approach. Had I done everything 'by hand' I would've reached the same exact misunderstanding, just over a much longer period of time (which I physically do not have). I believe I have stumbled on bugs that were meaningful (some fixes were even accepted), and contributed optimizations that others have even started integrating in their own work. THis is the very opposite of "AI slop" imo, and true to the spirit of the contribution policies of this repo. That said, I apologize if I've unbalanced the repo's equilibrium. That certainly was not my intention. I never demanded that my PRs/ bug fixes would be accepted. If so irritating, why not just ignore me? Is there anopther venue where they would be better suited? Regardless, I have appreciate all feedback received. This is a great repo

[JohannesGaessler]

I personally don't particularly care whether or not something is generated by a language model or not, the thing I care about is making efficient use of the very limited time of maintainers. With the current state of language models the code they produce is simply too low-quality for anything complex. No, it doesn't matter that you've extensively tested the code.

When it comes to this particular Github issue that we are commenting on, while I do think it's plausible that the finetune example broke for LLaMA 3 when I wasn't looking, I would want a bug report to be concise and with a high concentration of useful information. The bug report as you posted it is too noisy and I don't want to spend my time reading through a wall of text that doesn't follow the provided Github issue template and lacks the corresponding debugging information (particularly a full log and a git bisect).

[pestopoppa]

I would want a bug report to be concise and with a high concentration of useful information. The bug report as you posted it is too noisy and I don't want to spend my time reading through a wall of text that doesn't follow the provided Github issue template and lacks the corresponding debugging information (particularly a full log and a git bisect).

Thank you. I 100% agree. This is exactly the kind of feedback I was hoping to get. I would've been more than happy to adjust my methodologies and exposition to prioritize reviewer demands when submitting anything to the repo. The last thing I want to do is spam you guys. Like I said, my objective was to contribute meaningfully.

P.S. I've updated the original submission text in an attempt to satisfy your request