The Diffusion Idea: Learning to Undo Noise

Those traces and blind spots begin to matter in a rather surprising training exercise. To learn how to make images, a diffusion model is first shown how images can be taken apart.

Imagine starting with a photograph of a red bicycle outside a blue café on a rainy Parisian street. At first, every detail is clear: the bicycle’s frame, the café’s awning, the wet pavement catching the light. Now sprinkle visual static over the photograph. A little noise softens the details. More noise obscures the spokes and blurs the awning. Add enough, and the original scene is overwhelmed by speckles.

The noise values are random, but their strength is controlled. A noise schedule determines how much corruption to apply, producing noisy versions of training images at different stages, from barely disturbed to almost entirely obscured. The model receives a noisy example and information about its noise level, then learns to estimate the noise that was added—or, equivalently, to predict another quantity that helps move the example toward a less noisy state. It repeats this task across many images and noise levels, adjusting its internal parameters when its estimate is off.

The model is not simply shown a clean bicycle photograph and asked to memorize it. It is given a constructed puzzle: given this degree of corruption, what part looks like noise, and what underlying visual structure could remain? Because the exercise is repeated on different images, the model encounters edges, textures, shapes, and relationships among visual features in many contexts.

A useful analogy is learning to recognize a picture as it is progressively obscured, then learning what must be removed to reveal it again. But the analogy has limits. The model does not necessarily watch a smooth film of each photograph dissolving, nor does every system manipulate ordinary pixel images directly. Some systems add noise to a compressed latent representation instead. The particular prediction target can also vary, while the broad training idea remains similar: create controlled corruption and teach a model to estimate a quantity useful for reversing it.

This corruption is artificial, but purposeful. The model’s training examples provide the starting images; the noise-making procedure manufactures the puzzles. Rather than needing a vast collection of photographs at every possible level of blur and static, the training process can create noisy versions as needed. A single image can supply many such learning examples.

This exercise teaches the model how to make predictions about corruption at different levels. It is still a training task on examples derived from existing images; how those learned predictions are used to generate an image is a separate step.

A noisy image is not useful to a model merely because it can be shown one. It must learn what to predict. Depending on the system, the training target may be the noise that was added, a cleaner version of the image, or another quantity that helps define the reverse process. In each case, the model’s prediction is compared with a known target, and its internal parameters are adjusted to improve future predictions.

The corruption level, image, and added noise can all vary. A successful prediction must respond to the particular noisy input and its noise level, rather than replay a fixed restoration sequence for a remembered photograph. Training therefore teaches a general denoising prediction across images and noise levels—not a prescribed sequence for restoring one particular picture.

That prediction can draw on recurring regularities: edges often continue smoothly, wheels tend to form circles, and objects occupy characteristic arrangements. These are statistical patterns in the training material, not guarantees that every wheel will be round or every bicycle will have two of them. Nor does the model recover a unique original image from arbitrary noise: many different images can be consistent with learned patterns, and generation can produce different outcomes.

“Learning to clean a picture” is a helpful analogy, but only up to a point. Imagine trying to recognize a street photograph through a pane of glass speckled with rain. You might infer that a dark curve belongs to a bicycle wheel and that a pale patch is part of a shopfront. The model’s task has a similar direction: use the structure in a noisy representation to make a prediction. But it does not look at the picture as a person does, nor does it necessarily produce a visibly cleaner picture at every internal step. It performs numerical computations on a representation of the image, using patterns learned during training.

The model also does not usually receive the answer in the form of a little label saying “here is the bicycle; here is the rain.” Its prediction may be evaluated across a large numerical representation, and the training signal rewards estimates that match the intended target. Repeated correction shapes a network whose predictions are useful for reversing corruption. No single photograph teaches the whole skill; it emerges from many examples and many small adjustments.

There is an important boundary here. In this training exercise, each corrupted input is derived from an image that already existed in the training data. The model is learning predictions for reversing corruption; it has not yet been asked to generate an image from a random starting point. That change of starting point is the focus of the next section.

Why should a restoration lesson become a creation method? With enough added noise, very different training images approach the same simple distribution of random values. A bicycle photograph and a mountain landscape become statistically almost indistinguishable. We can sample fresh noise from that distribution without choosing either original image.

Training has taught the model useful predictions at many points along the corruption process. A sampling procedure uses those predictions to take successive steps back toward the distribution of images. At high noise levels, many outcomes remain possible; later steps work with the structure already taking shape. The process learns a route back to plausible images, rather than a route back to one particular original. That is the bridge from restoration to generation. ^[2]

During training, the model learned to undo corruption in images that began as real photographs. Generation changes the starting point. Instead of handing the model a damaged photograph, we give it a field of random values—noise with no bicycle, café, or street hidden inside like a picture under a bedsheet. Noise is a starting representation, not a hidden picture.

The model then applies its learned denoising process in a sequence of updates. At each step, the network predicts a numerical quantity, such as the added noise, from the current representation and noise level. The sampler uses that prediction to calculate the next state, sometimes adding fresh randomness along the way. The next state becomes the starting point for another update. After many such steps, the result can settle into a coherent image: perhaps a red bicycle outside a blue café, rain stippling the pavement and Parisian buildings receding down the street.

The particular arrangement takes shape through the generation process, guided by learned regularities: bicycles have frames and wheels; cafés have façades and windows; wet streets reflect light. Randomness helps determine which possible scene emerges. The learned denoising skill makes some patterns more likely than others, turning an initially unstructured starting point into something that resembles an image. These updates follow the model and sampling procedure; they are not a general-purpose image-quality judge.

A useful analogy is sculpting a statue from a rough block. Each pass removes material that does not fit the emerging form, until a recognizable shape appears. But the analogy is imperfect: noise is not a block of pixels waiting to be uncovered, and the model does not simply reveal a complete bicycle that was secretly present from the start. Its updates construct a pattern from statistical relationships learned during training. The analogy captures the gradual refinement, not a hidden image being exposed.

Nor should we imagine a neat little film in which every intermediate stage is visibly clearer than the last. Some generation processes move through abstract representations, and a given update may alter the overall pattern rather than sharpen a recognizable object. Early states might look like texture or color without obvious meaning; later states may acquire edges, forms, and composition. The path is a sequence of computational changes, not necessarily a series of pictures a person would describe as “getting clearer.”

This is the central leap: a model trained to reverse corruption can start from pure random noise and repeatedly move toward image-like structure. The bicycle, café, and rainy street are new outcomes shaped by that learned process—plausible because the model has absorbed visual regularities, but not guaranteed to be accurate in every detail. A wheel may wobble into an impossible angle; a window may become a doorway. A different random starting point can lead to a different result, while a prompt can guide the process without blueprinting every pixel. For now, the important point is how a recognizable scene can emerge at all. The next question is how words can steer this journey toward this bicycle and this café rather than some other scene.