Issues

training: on-policy batched generation left-pads without attention mask (pad positions not skipped)

egesabanci/sped#108 · opened

on Jun 25, 2026

benchmark: max_new_tokens=32 is too short — per-step overhead dominates speedup measurement

egesabanci/sped#107 · opened

on Jun 25, 2026

training: gradient checkpointing always on (Unsloth) — add toggle for short-sequence throughput

egesabanci/sped#106 · opened

on Jun 25, 2026

training: LoRA target modules include MLP by default — make configurable (attention-only option)

egesabanci/sped#105 · opened

on Jun 25, 2026

training: KL divergence recomputes teacher log_softmax twice (minor)

egesabanci/sped#104 · opened

on Jun 25, 2026

training: validation runs full SpeculativeDecoder.generate — add lightweight acceptance proxy

egesabanci/sped#103 · opened

on Jun 25, 2026

inference: draft always loaded 4-bit in serve — add --draft-dtype for bf16 (avoids dequant overhead)

egesabanci/sped#96 · opened

on Jun 25, 2026

inference: no torch.compile / CUDA graphs on draft hot path (kernel launch overhead)

egesabanci/sped#94 · opened

on Jun 25, 2026

inference: emergency fallback does full target forward instead of reusing verify logits

egesabanci/sped#93 · opened

on Jun 25, 2026

inference: metrics timing breakdown never recorded — _phase() never called in _speculate

egesabanci/sped#92 · opened

on Jun 25, 2026

inference: DraftTree (EAGLE-2 tree attention) implemented but not wired into speculation loop

egesabanci/sped#91 · opened

on Jun 25, 2026

inference: per-step CPU→GPU tensor creation from Python lists (generated_ids, draft_ctx_ids)

egesabanci/sped#90 · opened

on Jun 25, 2026

inference: draft proposal without KV cache does K full forwards (O(K×L))

egesabanci/sped#88 · opened

on Jun 25, 2026

Navigation Menu

Repository navigation

Issues

training: on-policy batched generation left-pads without attention mask (pad positions not skipped)

benchmark: max_new_tokens=32 is too short — per-step overhead dominates speedup measurement

training: gradient checkpointing always on (Unsloth) — add toggle for short-sequence throughput

training: LoRA target modules include MLP by default — make configurable (attention-only option)

training: KL divergence recomputes teacher log_softmax twice (minor)

training: validation runs full SpeculativeDecoder.generate — add lightweight acceptance proxy

inference: draft always loaded 4-bit in serve — add --draft-dtype for bf16 (avoids dequant overhead)

inference: no torch.compile / CUDA graphs on draft hot path (kernel launch overhead)

inference: emergency fallback does full target forward instead of reusing verify logits

inference: metrics timing breakdown never recorded — _phase() never called in _speculate

inference: DraftTree (EAGLE-2 tree attention) implemented but not wired into speculation loop

inference: per-step CPU→GPU tensor creation from Python lists (generated_ids, draft_ctx_ids)

inference: draft proposal without KV cache does K full forwards (O(K×L))