Add nanogpt example

[skyw]

No description provided.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[greptile-apps]

Confidence Score: 4/5

[Low impact] Adds example training scripts for nanoGPT.

The PR still needs the previously reported CUDA waits addressed before merging.

Findings

1.  CPU waits for GPU results ▶

Summary

Adds a nanoGPT example with optimizer selection, training, checkpoint resume, data preparation, and text generation.

• The latest changes restore the saved context length before loading training data.
• Sampling now rejects character metadata for direct GPT-2 use and checks character mappings.
• No new findings were accepted.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    Text[Input text] --> Prepare[prepare.py]
    Prepare --> Data[Token files and metadata]
    Data --> Train[train.py]
    Model[Upstream nanoGPT model] --> Train
    Optimizers[Selected optimizer and auxiliary AdamW] --> Train
    Train --> Checkpoint[Checkpoint]
    Checkpoint --> Train
    Checkpoint --> Sample[sample.py]
    Sample --> Output[Generated text]

Loading

_{Reviews (2) · Last reviewed commit: "address comment" · Reviewed by Greptile}

[greptile-apps]

Restart forgets saved context length

Resuming a checkpoint with a nondefault block_size fails unless the user repeats that flag. The documented run saves block_size=128, but a restart defaults to 256. The model uses the saved value while TokenData uses the default, so the context-length check aborts. Restore block_size from the checkpoint before creating TokenData.

[greptile-apps]

CPU waits for GPU results

On CUDA, accumulated_loss.item() makes the CPU wait for GPU work during logging. Evaluation does the same with total.item(). Sampling copies generated tokens with tolist(), and muon_hyperball setup compares a CUDA norm with zero.

The repository requires all device-to-host synchronizations to be flagged as high priority, including logging and inference. Avoid unnecessary waits and batch the required output transfers before merging.

Rule Used: Always flag device-to-host synchronizations (e.g., comparing a CUDA tensor to a Python scalar, .item() calls, or any operation that moves data from GPU to CPU) as high priority issues, regardless of context (training loops, inference, distributed o... (source)

[greptile-apps]

Character mapping breaks GPT-2 sampling

--init_from=gpt2 --data_dir=/tmp/shakespeare_char loads GPT-2 but encodes prompts with character IDs. Those IDs mean different things to GPT-2. Decoding a generated ID absent from itos also raises KeyError.

Reject character metadata for direct GPT-2 sampling, and check metadata against the loaded model before using it.