Skip to content

About

No description, website, or topics provided.

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

10 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 

Repository files navigation

SpeculativeNanoGPT: Accelerated Inference with Speculative Decoding

🚀 Overview

SpeculativeNanoGPT implements speculative decoding for accelerating inference in NanoGPT-style language models. By using a smaller "draft model" to propose tokens and a larger "target model" to verify them, this technique speeds up text generation while maintaining quality.

✨ Features

  • Speculative Decoding: Efficient draft-and-verify algorithm.
  • Dual Model Architecture:
    • 19M parameter draft model (fast).
    • 124M parameter target model (high-quality).
  • Performance Comparison: Speed and time comparisons between standard and speculative decoding.
  • Customizable Parameters: Supports temperature and top_k for diverse text generation.

🧠 Model Architecture

Uses a simplified GPT architecture with:

  • CausalSelfAttention: Efficient multi-head attention.
  • MLP: Two-layer feed-forward network with GELU.
  • GPT: Stacks Transformer blocks, shared token embeddings, and a language modeling head.

Two configurations are provided: GPTConfig124M and GPTConfig19M.

💾 Pre-trained Models

Download the pre-trained models from Hugging Face:

Place these .pth files in the root directory.

Example Output

    Prompt: "love in the air"
    Normal Generation (124M):
    Speed: 7.18 tokens/sec
    Time: 13.36 seconds
    
    Speculative Decoding (19M -> 124M):
    Speed: 10.67 tokens/sec
    Time: 9.09 seconds
    Speedup: 1.48x
    Acceptance Rate: 77%

⚙️ Customization

   max_length: Max tokens to generate.
   temperature: Controls randomness.
   top_k: Filters top K tokens.
   num_draft_tokens: Number of tokens for the draft model.

🙏 Acknowledgements

    Andrej Karpathy’s NanoGPT: Base for the GPT implementation.
    Speculative Decoding Researchers: For pioneering the technique.

About

No description, website, or topics provided.

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages