From Still Images to Moving and Sounding Worlds

A still image can make one instant look convincing. Video must make many instants belong to the same world.

Imagine our red bicycle outside the blue café on a rainy Parisian street. In a single frame, the bicycle may look beautifully photographed. But let the camera move, or let a cyclist ride past, and new questions appear. Is it the same bicycle in the next frame? Do its wheels turn as it moves? Does the café stay put while the viewpoint shifts? Do raindrops fall consistently, rather than suddenly streaking upward because the model found that visually plausible for one frame?

A video is not simply a pile of good pictures. It is a sequence whose pictures must agree.

That agreement is called temporal coherence: objects keep their identity, positions change in connected ways, and events unfold rather than popping into existence. If the bicycle’s handlebars change shape between frames, or a puddle jumps from one side of the street to the other, each frame might still look plausible by itself. The sequence, however, gives the game away.

This is why motion raises the difficulty. In a still picture, the model can suggest that a wheel is spinning through blur or angle. In video, the wheel’s appearance should change in ways that fit its rotation, the bicycle’s speed, and the camera’s view. A person who steps behind a parked car should reappear on the other side—not become a different person, or vanish into the cinematic equivalent of a trapdoor.

The model therefore has to capture relationships across time as well as relationships within a frame. It needs useful representations of appearance, motion, and change: what remains stable, what moves, and what causes one moment to follow another. More frames also mean more material to generate and coordinate, so keeping a long sequence consistent can be harder than making a short one.

There is no single recipe for doing this. Some video systems use diffusion-like methods, learning to refine noisy representations of sequences; some use flow-based approaches that learn a route between distributions; others generate elements autoregressively, step by step. Hybrid systems can combine techniques. In each case, the challenge is not merely to create a plausible frame, but to make the sequence hang together. Video diffusion research demonstrates how image diffusion architectures can be extended to sequences. ^[9]

The connection to image generation is close: learned patterns and generative objectives still matter. The added demand is continuity. A convincing video must not only show a red bicycle in the rain. It must let that bicycle remain itself while the rain falls and the world moves around it.

A moving bicycle asks a model to preserve the identity of an object across time. Music poses a related challenge, but its ingredients are different. A sound is not a picture that happens to change: it unfolds as pressure waves, and a musical phrase depends on timing, pitch, tone, and the relationships among notes.

During training, an audio model can encounter recordings represented as waveforms, or as transformed descriptions of sound such as spectrograms. A spectrogram maps sound energy across time and frequency, turning some features of audio into a visual-like grid. In either case, the model can learn recurring patterns: how a drumbeat tends to be spaced, how notes form a melody, how instruments sound, or how one musical passage often leads into another. Captions or other text may connect those patterns to words such as “soft piano,” “fast,” or “ominous.” These associations are learned from examples; they are not a guarantee that the system understands music as a listener does.

The prompt “a gentle piano melody with a steady beat” gives a broad direction, rather like “a rainy Paris street” does for an image. But music must satisfy that direction over time. A beat that begins steadily and then drifts may undermine the whole piece. A melody can sound locally plausible note by note while wandering without a satisfying phrase or ending. The challenge is not only to produce sounds that fit, but to arrange them into a structure that holds together: repetition with variation, a sense of development, and continuity from one moment to the next.

That continuity has several scales. Individual sound waves must combine into a clear tone; notes must form phrases; phrases must relate across a longer passage. A model that generates audio in small steps needs some way to keep earlier context relevant as it continues. Too little continuity, and the result can feel like a collection of musical fragments. Too much repetition, and the piece can become a loop that has mistaken persistence for composition.

There is no single method behind generated music. Some systems generate audio tokens autoregressively, predicting one discrete unit after another, much as a language model predicts text tokens. MusicGen is one documented example using compressed audio tokens. ^[10] Others use diffusion-like methods to refine a noisy audio representation, as illustrated by Stable Audio Open, ^[11] or flow-based methods that learn a transformation from a simpler starting distribution toward audio. Hybrid systems may combine these approaches or use different methods for different stages. The details matter: sound has far more fine-grained temporal structure than a typical image, so an image-generation method cannot simply be transferred unchanged and expected to work as a music machine.

The useful connection is broader. Image, video, and music models can all learn patterns from examples and use a prompt or other condition to steer generation. But each medium makes its own demands. For music, plausibility is not just the sound of a note; it is the feeling that the next note belongs, and that the piece is going somewhere.

An image-generation model is not retrieving a finished picture hidden inside a prompt. It has learned visual patterns from examples, and the prompt steers generation without specifying every detail. In diffusion, the model repeatedly refines a noisy starting representation; other systems can reach an image by different routes. In either case, the learned patterns help shape what appears.

For a bicycle on a rainy Paris street, those patterns may guide the model toward wheels, a frame, wet pavement, and reflected light. They do not ensure that every object is counted correctly, that the street makes sense, or that the bicycle is physically possible. The prompt points generation in a direction; it does not certify the result.

Video and music share the broad idea of learned, guided generation, but they must sustain structure over time in ways a still image does not. What looks coherent in an image is therefore a sign that its elements fit together plausibly, not that the depicted scene is accurate.