About scalability to larger base models (e.g., Qwen3-8B)

[bxren]

Hi, thanks for the great work.

I have a question about scalability across model sizes, especially for larger base models such as Qwen3-8B. In my domain-specific experiments, I observed consistent improvements on models like Qwen3-0.6B and Qwen3-1.7B after adding the module, including gains on benchmarks such as MMLU. However, when I moved to Qwen3-8B, I no longer observed similar improvements.

So I wanted to ask:

Have you tried Qwen3-8B or other similarly strong base models?
Did you observe that the method becomes less effective as the base model gets stronger/larger?
Do you have any suggestions on how to make the method work better on larger models:
different insertion layers,
different learning rates,
partial unfreezing of the backbone,
larger memory/table size,
or different training recipes?

[Alichua]

Hi bxren,

Thanks for your experimental details and questions.

Your observation is accurate and aligns with Engram's design: Engram functions as a supplementary memory module, activated by index-based lookups rather than forcing information into the model like LoRA. It only affects computations related to the retrieved patterns. This design means Engram's effectiveness depends on the base model's knowledge gaps.

Benchmarks like MMLU test general knowledge, that a larger model like Qwen3-8B has likely already learned during pre-training, reaching its performance ceiling for that task. Since Engram's role is to fill knowledge gaps, its impact is naturally limited when those gaps are small.

The same principle may applied to your domain-specific experiments. If Qwen3-8B already covers your domain well, the additional optimization from Engram will be marginal.

So in short, yes, it's expected that for small-scale & non-specialized domain experiments, Engram becomes less effective as the base model grows stronger.

If your goal is to leverage larger models, the optimal path aligns with the scenario described in the DeepSeek-Engram paper: utilizing large-scale data combined with a super-large Engram vocabulary is necessary to reach its performance comfort zone.

Within the scope of TinyEngram, as for methods to make your training more effective, we recommend referring to our parameter tuning logs. If your experimental scale is similar to ours, these records should serve as a helpful reference.

[DDDMUC]

Perhaps you can try mixing LoRA and Engram.

[bxren]

Perhaps you can try mixing LoRA and Engram.

Interesting! Have you done any experiments or found improments on lora and Engram in larger models?

[Alichua]

Mixing LoRA and Engram can be a "promising" solution, as LoRA itself is a safe and stable SFT method. However, there are some logical issues to address:

1. Entanglement: There is coupling in the training and activation pathways of the two methods. This might can be resolved by sequential training (e.g., first training backbone + LoRA, then adding Engram to create backbone + LoRA + Engram).
2. Interference: Engram is triggered selectively via hash lookups and inherently possesses anti-catastrophic forgetting properties, acting like a "free" bonus. In contrast, LoRA is globally activated. Having both active simultaneously may interfere with this stability property.