ViT / Vision transformers are similar to language transformers but split the image into "tokens" of image patches that are typically 16x16 pixels for the input.
VLM is a Vision-Language Model — a model that can process both images and text together, rather than just one modality (Example LLaVa).
- 2020 Original Vision transformer ViT paper An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- 2021 Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
- 2021 MAE Masked autoencoders
- 2021 DINO
- 2022 iBOT
- 2023 LLaVa VLM paper Visual Instruction Tuning
- Stable Diffusion paper using U-net 2022 - High-Resolution Image Synthesis with Latent Diffusion Models
- DiT Diffusion transformer paper 2022 Scalable Diffusion Models with Transformers
- Stable Diffusion 3 paper w/ transformers 2024 - Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
- Stability AI - Stable Diffusion
- Black Forest Labs - Flux
- Hugging Face trending models: text to image image to image
- Google and OpenAI have dropped separate diffusion models (Nano Banana and Dall E) and use native, multimodal models where the image is tokenized into the same representation space as text.
- Midjourney