Diffusion Language Models Explained: How Google's Diffusion Gemma Works
Educational article by the **MindStudio Team** (blog of the MindStudio platform, multi-model workflow orchestration) explaining **modèles de langage par diffusion** (*Diffusion Language Models*) through the case of **Diffusion Gemma**, Google's first **open weights** implementation (2B parameters, derived from Gemma 2). The thesis: whereas **autoregressive** models (GPT-4, Claude, standard Gemma) generate text **token by token, left to right** (causal attention, each token fixed once produced), **diffusion** models start from a **masked/noised** sequence and **refine it iteratively** (masked diffusion / *absorbing diffusion*), with **bidirectional attention**: the model can **revise any position at any step**. Consequences: high **parallelism** (a 500-token text would require 50-100 denoising steps instead of 500 sequential passes), natural **infilling** and **constrained generation** (template filling, code completion with surrounding context), and built-in **revision** capability. But at the current scale (2B), Diffusion Gemma **does not match** the large autoregressive models (GPT-4o, Gemini 1.5 Pro) on reasoning, instruction-following, and general knowledge: the gap is "closing" without being closed. The inspiration comes from image generation (Stable Diffusion, DALL-E left autoregression behind years ago); whether the same principle holds for text remains an open question. Diffusion Gemma is distributed on Hugging Face (Google DeepMind), AI Studio, and Vertex AI.
MindStudio Team