VCP-Net: Sparse Chinese LM with Soft-Shrinkage Activation

144M params, pure conv (no attention), soft-shrinkage spike activation. Val CE 2.28.

Architecture

Token + Pos Embed → LayerNorm → [VCPBlock × 12] → LayerNorm → VocabHead VCPBlock = Pre-LN → VCM(dilation=d) → SoftSpike(V_th) (×2) VCM = CausalDWConv(k=3) + GatedMLP(4x) → spike

Soft-Shrinkage Spike

`python

Forward: soft shrinkage — no surrogate gradient

out = sign(v) · max(0, |v| - V_th)

Backward: constant gradient for active, zero for silent

grad_v = 1 (|v| ≥ V_th) or 0 (|v| < V_th) `

Results

T=1 (Single-step Spike) — Best Overall

Epoch	Train CE	Val CE	Spike%	Notes
1	27.78	3.55	18.5
2	3.23	3.03	34.5
3	2.90	2.78	35.1
4	2.67	2.65	36.5
5	2.53	2.60	35.2
6	2.44	2.49	35.1
7	2.27	2.28	36.4	← best @ lr=0.00033
8	2.23	2.28	36.9	converged

• Best Val CE: 2.28, Spike rate 35% (genuinely sparse)
• Data: 144K Belle Chinese, batch=48, gradient checkpointing

T=4 (LIF Multi-step SNN)

Epoch	Train CE	Val CE	Spike%	Notes
1	4.06	2.81	93.2
2	2.50	2.34	93.3	← best
3	2.17	—	95.4	plateaued

• Converges 4× faster per epoch, but 95% spike = effectively dense
• Ceiling lower than T=1 due to loss of sparsity regularization

Key Insight

T=1 beats T=4 because sparsity (35% spike) acts as built-in regularization, enabling deeper convergence on small data. T=4's LIF dynamics saturate the membrane, removing the sparsity advantage.

Ablation: Hard Threshold vs Soft Shrinkage

Method	Best Val CE	Spike Rate
Hard threshold + surrogate gradient	~5.0	63%, unstable
Soft shrinkage (ours)	2.28	35%, stable

Comparison with GPT-2 Baseline

Same 50K Belle data, similar parameters:

Epoch	GPT-2 (102M)	VCP-Net (144M)
1	4.45	3.55
2	4.38	3.03
3	4.45 (overfit)	2.78

VCP-Net outperforms attention-based GPT-2 by 38% on small data — sparsity prevents overfitting.

Failed Approaches

1. Hard threshold + surrogate gradient: Gradient death at high V_th, CE plateau at 5.0
2. Manual homeostasis: Fought against CE gradient, never converged
3. Optimizer reset on resume: Fresh Adam at lr=0.001 overshoots weights
4. OOM with large models: Solved by gradient checkpointing

Usage

`python from vcp_net import VCPNet from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('bert-base-chinese') model = VCPNet(vocab_size=tokenizer.vocab_size, d_model=768, n_layer=12) model.load_state_dict(torch.load('best_model.pt')['model_state_dict']) model.eval()

input_ids = tokenizer('今天天气', return_tensors='pt')['input_ids'] output = model.generate(input_ids, max_new_tokens=30) print(tokenizer.decode(output[0])) `

Files

File	Purpose
cp_block.py	SoftSpike, CausalDWConv, VCM, VCPBlock
cp_net.py	VCPNet with contrastive training
cp_block_t1.py	T=1 snapshot backup
cp_net_t1.py	T=1 backup
rain_webtext_100m.py	Training script
�blation.png	Hard vs soft shrinkage comparison

License

MIT