[Idea] Why can pruning or quantization improve domain-specific performance over the base model? #31

Research question

Why can a model perform better than the uncompressed base model on some domains after pruning or quantization?

Record this as a research idea: beyond measuring capability preservation, investigate whether compression can improve domain-specific performance, when this happens, and what mechanism explains it. No specific result or causal explanation is established by this issue.

Hypotheses to test

• Reduced interference / implicit regularization: pruning may remove components that contribute irrelevant or conflicting signals for a target domain. Quantization may perturb these contributions differently; the two methods should not be assumed to share a mechanism.
• Calibration-driven specialization: domain-specific calibration may favor preserving target-domain computations, potentially improving that domain while sacrificing others.
• MoE routing and expert contributions: compression may change expert output magnitudes, their relative contributions, and downstream routing. Are gains associated with particular experts, layers, or FFN intermediate dimensions?
• Output behavior rather than underlying capability: changes in confidence, answer margins, verbosity, or formatting may improve benchmark scores without improving domain reasoning.
• Evaluation variation: apparent gains may come from sample uncertainty, decoding randomness, numerical/runtime differences, or calibration/evaluation overlap.

These are candidate explanations, not conclusions.

Initial validation plan

• Compare the same base checkpoint against pruning-only and quantization-only variants, without recovery training or SFT in the initial comparison. Sweep pruning ratios and quantization bit widths separately.
• Fix evaluation data, prompts, tokenizer, decoding settings, and runtime where possible; isolate unavoidable backend changes. Save per-example predictions and distinguish base-wrong → compressed-correct from base-correct → compressed-wrong cases.
• Report per-domain deltas with paired uncertainty estimates. Repeat calibration sampling and stochastic generation where applicable; account for selecting apparent winners across many domains/settings.
• Compare in-domain, out-of-domain, and mixed-domain calibration with disjoint evaluation data, and check whether target-domain gains coincide with losses elsewhere.
• For reproducible gains, inspect expert routing, expert/FFN contributions, and answer margins. Use targeted restoration or ablation of implicated components to test causality, with random-pruning or matched-perturbation controls where feasible.
• Manually inspect improved and degraded examples, including formatting-sensitive failures, to distinguish reasoning improvements from scoring effects.

Desired outcome

A reproducible map of which domains benefit, under which compression settings, followed by evidence that supports or rules out specific mechanisms. In particular, determine whether pruning and quantization produce similar-looking gains for the same reason or through different mechanisms.