LLM Knowledge System

A topic-based map of this repo. This section is organized by knowledge domains rather than learning phases.

Visual Map

flowchart TD
    A[LLM Knowledge System]

    A --> B[Foundations]
    A --> C[Architecture and Scaling]
    A --> D[Adaptation and Alignment]
    A --> E[Inference and Serving]
    A --> G[Model Case Studies]

    B --> B1[SVD / dtypes / AdamW]
    B --> B2[Attention: MHA / MQA / GQA]
    B --> B3[RoPE / SwiGLU]

    C --> C1[FlashAttention / MLA]
    C --> C2[DeepSeekMoE]
    C --> C3[TP / PP / EP / FSDP]
    C --> C4[Gradient Checkpointing / Mixed Precision]

    D --> D1[LoRA / QLoRA / DoRA]
    D --> D2[Specialized LoRA Variants]
    D --> D3[SFT / RLHF / DPO / PPO / GRPO]

    E --> E1[Speculative Decoding]
    E --> E2[Continuous Batching / PagedAttention]
    E --> E3[AWQ / GPTQ / TensorRT-LLM]
    E --> E4[Hallucination Mitigation]

    G --> G1[DeepSeek-V2]
    G --> G2[DeepSeek-V3]
    G --> G3[DeepSeek-V3.2]

    G1 -. combines .-> C1
    G1 -. combines .-> C2
    G2 -. combines .-> C1
    G2 -. combines .-> C2

Loading

The diagram gives a high-level overview; the sections below act as the detailed index.

Domain	Focus	Core Topics
Foundations	Math, optimization, losses, normalization, and Transformer building blocks	SVD, dtypes, AdamW, learning rate schedulers, Sigmoid, GELU, LayerNorm, RMSNorm, BatchNorm, GroupNorm, MHA/MQA/GQA, RoPE, SwiGLU
Architecture & Scaling	Efficient training and large-scale model design	FlashAttention, MLA, DeepSeekMoE, TP/PP/EP/FSDP, Gradient Checkpointing, Mixed Precision Training
Adaptation & Alignment	Task adaptation and preference learning	LoRA family, SFT, RLHF, DPO, PPO, GRPO
Agent Systems	Retrieval, memory, tool use, API interfaces, and task orchestration	Agent Basics, Memory Systems, RAG Systems, OpenAI API Interfaces
Inference & Serving	Latency, memory, and deployment efficiency	Speculative Decoding, Continuous Batching, Quantization, TensorRT-LLM, Hallucination Mitigation
Model Zoo	Language models, vision models, vision-language models, and robotics systems	DeepSeek, Gemma, DINOv2, DINOv3, CLIP, SigLIP, SmolVLM, robot policies, embodied datasets, and real-time policy inference
Book Notes	Chapter-by-chapter foundations with compact definitions, equations, and examples	Modern Robotics; Reinforcement Learning: An Introduction

1. Foundations

• Math and numerical basics: SVD, dtypes, Memory Estimation, AdamW, Learning Rate Schedulers
• Activation basics: Sigmoid, GELU
• Normalization basics: LayerNorm, RMSNorm, BatchNorm, GroupNorm
• Attention mechanisms: SVD + Attention, MHA, MQA, GQA
• Position and FFN blocks: Sinusoidal Position Embedding, RoPE, SwiGLU

2. Architecture & Scaling

• Efficient attention: FlashAttention, MLA
• Sparse architecture: DeepSeekMoE
• Training memory and numerics: Gradient Checkpointing, Mixed Precision Training
• Distributed training: TP, PP, EP, FSDP

3. Adaptation & Alignment

• PEFT: LoRA, QLoRA, DoRA, Specialized LoRA Variants
• Supervised and preference alignment: SFT, RLHF, DPO, PPO, GRPO

4. Inference & Serving

• Decoding acceleration: Speculative Decoding (Medusa/Lookahead)
• Serving systems: Continuous Batching & PagedAttention, TensorRT-LLM & Multi-LoRA Serving
• Compression and reliability: Post-Training Quantization (AWQ/GPTQ), Hallucination Mitigation at Inference

5. Agent Systems

• Core concepts: Agent Systems Basics
• Memory design: Memory Systems for Agents
• Retrieval grounding: RAG Systems
• Reusable task methods: Skill Systems
• Tool architecture: Tool Registry and Function Calling
• API interface format: OpenAI API Interface Format
• Protocol layer: Model Context Protocol (MCP)

6. Model Zoo

• Language models: DeepSeek-V2, DeepSeek-V3, DeepSeek-V3.2, DeepSeek-R1
• Vision models: DINOv2, DINOv3
• Vision-language models: CLIP, SigLIP, PaliGemma, SmolVLM, Gemma 3, Gemma 4, DeepSeek-VL, DeepSeek-VL2
• Robotics datasets: LIBERO, Open X-Embodiment
• Robot policies: ACT / ALOHA, Diffusion Policy, Fast-WAM, MolmoAct2, Octo, RT-1, RT-2, OpenVLA, SmolVLA, WALL-OSS, pi0, pi0-FAST, Hi Robot, Human-to-Robot Transfer, MEM, pi0.5, pi0.5-KI, pi0.6*
• Robotics inference: Real-Time Chunking

7. Book Notes

• Robotics: Modern Robotics: Mechanics, Planning, and Control
• Reinforcement learning: Reinforcement Learning: An Introduction

---

File Structure

Images are grouped by model or topic under assets/; notes link to the corresponding folder.

.
|-- assets/
|   ...
`-- docs/
    |-- Agent_Systems/
    |   |-- Agent_Basics.md
    |   |-- MCP_Protocol.md
    |   |-- Memory_Systems.md
    |   |-- OpenAI_API_Interface_Format.md
    |   |-- RAG_Systems.md
    |   |-- Skill_Systems.md
    |   `-- Tool_Registry_and_Function_Calling.md
    |-- Activation_Layers/
    |   |-- GELU.md
    |   |-- Sigmoid.md
    |   `-- SwiGLU.md
    |-- Attention_Machanisms/
    |   |-- FlashAttention.md
    |   |-- GQA.md
    |   |-- MHA.md
    |   |-- MLA.md
    |   |-- MQA.md
    |   `-- SVD_Attention.md
    |-- Books/
    |   |-- Reinforcement_Learning/
    |   |   `-- Reinforcement_Learning_An_Introduction.md
    |   `-- Robotics/
    |       `-- Modern_Robotics.md
    |-- Inference_Optimization/
    |   |-- continuous_batching.md
    |   |-- hallucination_mitigation.md
    |   |-- quantization_inference.md
    |   |-- speculative_decoding.md
    |   `-- tensorrt_multilora.md
    |-- Math/
    |   |-- Memory_Estimation.md
    |   |-- SVD.md
    |   `-- dtypes.md
    |-- Model_Zoo/
    |   |-- Language_Models/
    |   |   |-- DeepSeek_R1.md
    |   |   |-- DeepSeek_V2.md
    |   |   |-- DeepSeek_V3.md
    |   |   `-- DeepSeek_V32.md
    |   |-- Robotics/
    |   |   |-- Datasets/
    |   |   |   |-- LIBERO.md
    |   |   |   `-- Open_X_Embodiment.md
    |   |   |-- Inference/
    |   |   |   `-- RTC.md
    |   |   `-- Policies/
    |   |       |-- ACT.md
    |   |       |-- Diffusion_Policy.md
    |   |       |-- Fast_WAM.md
    |   |       |-- MolmoAct2.md
    |   |       |-- Pi_Hi_Robot.md
    |   |       |-- Pi_Human_to_Robot.md
    |   |       |-- Pi_MEM.md
    |   |       |-- Octo.md
    |   |       |-- OpenVLA.md
    |   |       |-- Pi_0.md
    |   |       |-- Pi_0_5.md
    |   |       |-- Pi_0_5_KI.md
    |   |       |-- Pi_0_6.md
    |   |       |-- Pi_0_FAST.md
    |   |       |-- RT_1.md
    |   |       |-- RT_2.md
    |   |       |-- SmolVLA.md
    |   |       `-- WALL_OSS.md
    |   |-- Vision_Models/
    |   |   |-- DINOv2.md
    |   |   `-- DINOv3.md
    |   `-- Vision_Language_Models/
    |       |-- CLIP.md
    |       |-- DeepSeek_VL.md
    |       |-- DeepSeek_VL2.md
    |       |-- Gemma_3.md
    |       |-- Gemma_4.md
    |       |-- PaliGemma.md
    |       |-- SigLIP.md
    |       `-- SmolVLM.md
    |-- MoE/
    |   `-- DeepSeekMoE.md
    |-- Norm/
    |   |-- BatchNorm.md
    |   |-- GroupNorm.md
    |   |-- RMSNorm.md
    |   `-- LayerNorm.md
    |-- Optimizer/
    |   `-- AdamW.md
    |-- PEFT/
    |   |-- DoRA.md
    |   |-- LoRA.md
    |   |-- QLoRA.md
    |   `-- Specialized_LoRA.md
    |-- Parallelism/
    |   |-- EP.md
    |   |-- FSDP.md
    |   |-- PP.md
    |   `-- TP.md
    |-- Position_Embeding/
    |   |-- RoPE.md
    |   `-- Sinusoidal_Position_Embedding.md
    |-- Preference_Alignment/
    |   |-- DPO.md
    |   |-- GRPO.md
    |   |-- PPO.md
    |   |-- RLHF.md
    |   `-- SFT.md
    |-- Scheduler/
    |   |-- Cyclical_and_Restart.md
    |   |-- LLM_Training_Recipes.md
    |   |-- Metric_Adaptive.md
    |   |-- Scheduler_Basics.md
    |   `-- Warmup_and_Decay.md
    `-- Training_Optimization/
        |-- Gradient_Checkpointing.md
        `-- Mixed_Precision_Training.md

---

Learning Resource Recommendation

• Datawhale / happy-llm
• Datawhale / hello-agents
• Datawhale / all-in-rag
• Hugging Face / LeRobot
• Physical Intelligence / Home