Hello,
Your paper says "finer granularity in quantization results in more overhead but less information loss," but I do not understand why.
Could you explain why per-tensor quantization results in faster inference than per-token quantization?
Thank you.
Hello,
Your paper says "finer granularity in quantization results in more overhead but less information loss," but I do not understand why.
Could you explain why per-tensor quantization results in faster inference than per-token quantization?
Thank you.