Diffusion Language Models Explained: How Google's Diffusion Gemma Works

Diffusion Language Models: How Diffusion Gemma Works

Educational article by the MindStudio Team (blog of the MindStudio platform, multi-model workflow orchestration) explaining modèles de langage par diffusion (Diffusion Language Models) through the case of Diffusion Gemma, Google's first open weights implementation (2B parameters, derived from Gemma 2).

By MindStudio Team / June 12, 2026 / Source mindstudio.ai ↗ / Reading 2 min / .md / / Auto-verified translation

#modèles de langage par diffusion#Diffusion Gemma#Google DeepMind#masked diffusion#absorbing diffusion#autoregressive generation#bidirectional attention#causal attention

The MindStudio Team publishes an explainer (June 12, 2026) on modèles de langage par diffusion, drawing on Diffusion Gemma, the first open weights implementation of this architecture, from Google.

The starting point is a clash of paradigms. The autoregressive models that dominate today (GPT-4, Claude, standard Gemma) generate text sequentially, one token at a time, left to right, via causal attention. Each output depends on all preceding tokens: generation cannot be parallelized across positions, and each token is fixed once produced — the model cannot revisit its choices.

Diffusion models proceed differently: they start from a noised/masked sequence and refine it iteratively toward a coherent output (masked diffusion, or absorbing diffusion). The forward pass progressively masks tokens; the model learns to reconstruct them; inference reverses the process over several adjustable denoising steps. Attention is bidirectional: the model sees the whole sequence in both directions and can update any position at any step — "changing its mind" about earlier tokens. The metaphor: writing a draft and then revising it, rather than a final copy produced word by word.

Diffusion Gemma: 2 billion parameters, Transformer base derived from Gemma 2, released in early 2025, weights on Hugging Face (Google DeepMind), also on AI Studio and Vertex AI. Key adaptations: removal of causal masking, noise conditioning, and simultaneous prediction of distributions over all masked positions.

Advantages: parallelism (a 500-token text would require 50-100 steps rather than 500 sequential passes, hence potentially much higher speed on long outputs), natural infilling and constrained generation (templates, code completion with surrounding context, rewriting while preserving the beginning/end), and built-in revision.

The article states a clear limitation: at the 2B scale, Diffusion Gemma does not match the best autoregressive models (GPT-4o, Gemini 1.5 Pro) on reasoning, instruction-following, and knowledge — the gap is "closing" without being closed. For conversational use, multi-step reasoning, or token-by-token streaming, autoregressive models remain the better choice.

The lineage traces back to images: Stable Diffusion and DALL-E left autoregression behind years ago; the open question is whether the same principle holds for text. Diffusion Gemma, through its open weights, makes it a testing ground for controllable generation.

Key takeaways

Core idea. two paradigms for text generation. Autoregressive = token by token, left→right, causal attention, each token fixed once produced. Diffusion = starts from a noised/masked sequence and refines it iteratively, bidirectional attention, can revise any position at any step.

Diffusion Gemma. 1st open weights implementation of a diffusion LLM; Google; 2 billion parameters; Transformer base derived from Gemma 2; trained via masked diffusion on large-scale text; released early 2025; weights on Hugging Face (Google DeepMind).

Changes vs. standard Gemma 2. removal of causal masking, addition of noise conditioning (current noise level as input), simultaneous prediction of distributions over all masked positions (instead of just the next token).

Mechanism. masked diffusion (absorbing diffusion) — the forward pass progressively masks tokens, the model learns to predict from the masks, inference reverses the process; number of denoising steps is adjustable.

Speed argument. an autoregressive model = 500 sequential passes for 500 tokens; a diffusion model 50-100 steps updating all positions in parallel → "potentially much faster", especially on long outputs.

Clear limitation. at the current scale (2B), Diffusion Gemma does not beat GPT-4o or Gemini 1.5 Pro on reasoning / instruction-following / knowledge. Gap is "closing", not closed.

Strong use cases. constrained generation (templates, mandated phrases, global structural constraints), infilling (document editing/revision, function-body completion with context), parallel workloads (lower-cost batching), research/experimentation (fine-tuning enabled by open weights).

Stay autoregressive when. general multi-step reasoning, complex instructions, nuanced judgment at scale, conversational chatbots/assistants, Q&A with token-by-token streaming, benchmark quality is the priority.

Lineage. inspired by images — Stable Diffusion, DALL-E abandoned autoregression; the open question: does the principle hold for text?

Related. post-Transformer architecture debate, race for open models (see GLM-5.2 fact sheet), inference economics (parallelism = cost/latency), context engineering and agentic code completion.

Diffusion Language Models: How Diffusion Gemma Works

Key takeaways

Key figures

Attributed claims

Diffusion Language Models: How Diffusion Gemma Works

Summary

Key takeaways

Key figures

Attributed claims