How diffusion models work

Take a photo and add a tiny amount of random noise. It still looks like the photo. Add a little more, and a little more, hundreds of times, and eventually you have static that looks like nothing at all.

Key idea: A diffusion model is a neural network trained to undo one small step of that noising. If it can take a slightly noisy image and guess what the noise was, you can subtract that guess and get a slightly cleaner image. Repeat from pure static, and the model "finds" an image that was never in the training data.

The forward process is fixed. Nothing is learned here.

• Start with a training image.
• At each of many steps (often around 1,000 in training), add a small amount of Gaussian noise: random values drawn from a bell-curve distribution.
• A noise schedule decides how much noise each step adds. By the last step the image is indistinguishable from pure noise.

Because the noise is added by a known formula, the training program can jump straight to any step: "give me this image at step 400" is one calculation, not 400. That makes training efficient.

Training repeats one simple lesson millions of times:

1. Pick a training image and a random step number.
2. Add the right amount of noise for that step, and remember exactly what noise was added.
3. Give the network the noisy image and the step number. It predicts the noise.
4. Compare the prediction with the real noise (a loss such as mean squared error) and adjust the weights a little by backpropagation and gradient descent.

Key idea: The network never learns to draw a sofa directly. It learns what noise looks like, at every noise level. That is an easier, very well-defined task, which is a big reason diffusion training is stable.

To make a new image:

1. Start from an image of pure random noise. A seed decides that starting noise, so the same seed, prompt and settings give the same image again.
2. Ask the network to predict the noise, and remove part of it.
3. Repeat for a number of sampling steps.

The original methods used as many steps as training (around 1,000). Modern samplers get good images in roughly 20 to 50 steps, and distilled models in 1 to 4. Key idea: Every step is a full pass through a large network, so the number of steps is the main reason diffusion is slower than a GAN.

A diffusion model on its own makes a random plausible image. To follow a prompt:

• A separate text encoder turns the prompt into a list of numbers (an embedding) that captures its meaning.
• The denoising network receives that embedding at every step, usually through cross-attention layers, so its noise predictions pull the image towards the description.
• Classifier-free guidance compares the prediction with and without the prompt and pushes further in the prompt's direction. A higher guidance scale follows the prompt more closely, but images become less varied and can look overcooked.

Most current systems also work in a compressed latent space rather than on pixels. An autoencoder compresses training images into a small latent grid; generation denoises in that grid and the decoder expands the result to pixels at the end, which makes every step far cheaper. That is a hybrid design, covered in the HL topic on hybrid models.