Words Become Numbers

Before a model can work with a prompt, the text must be cut into pieces it can process. These pieces are called tokens. A token might be a whole word, part of a word, a punctuation mark, or a small stretch of characters. The cuts are made by a tokenizer—the system that turns text into the units the model receives.

Take the prompt: “Explain why rainbows appear.” A tokenizer might divide it into five tokens: “Explain,” “ why,” “ rainbows,” “ appear,” and “.” In many systems, spaces are attached to the word that follows them, so the first token after “Explain” may actually include a leading space. The exact division depends on the tokenizer; there is no single universal set of token boundaries.

Now consider “unbelievable.” It could be treated as one token, or split into smaller familiar pieces, perhaps “un,” “believ,” and “able.” A rare name, a new technical term, or a misspelling may be broken into even smaller fragments. Meanwhile, a common word may fit neatly into one token. Punctuation and digits count too: a comma may be its own token, and a number may be split in ways that do not match how a person would read it.

That is why tokens are not quite words. People naturally divide language into words and sentences because that is useful for reading. A tokenizer divides text according to the system’s learned inventory of useful chunks. Its boundaries may land in the middle of a word, or treat a familiar combination as a single piece. Think of it less as slicing a sentence at every word-space and more as packing text with a set of reusable tiles—some large, some small, and occasionally an awkward sliver around an unfamiliar name.

The model does not receive the prompt as a line of letters on a screen. Each token corresponds to an ID: a number that identifies that piece in the tokenizer’s vocabulary. So “Explain why rainbows appear.” becomes a sequence of IDs, one for each token. Those IDs are not secret definitions of the words; they are labels the system can use to look up numerical representations. The word “rainbows” does not arrive with a tiny picture of a rainbow attached.

Tokenization matters in practical ways. A long response uses many tokens, but the count is not simply the number of words. Unusual words and character strings can take several tokens; common phrases may be more compact. This is one reason a model can treat a familiar phrase differently from an odd spelling, even when a person can easily guess what the writer intended. These are general tendencies, not universal rules: the precise segmentation depends on the tokenizer.

The prompt supplies the current material for the model to work with: text becomes a sequence of tokenizer-dependent pieces and then IDs. Those pieces do not automatically settle ambiguity or provide information that was never included. For now, the important shift is simple: the prompt that looks like a stream of words to us becomes a sequence of discrete pieces to the model. Those pieces are its starting material. To make anything of them, the system needs a way to represent how they relate—not just which numbered tile came first.

A numbered tile tells the model which piece arrived. It does not, by itself, tell the model that rainbow has something in common with prism, or that rain is more closely related to umbrella than to trombone. For that, each token needs a richer numerical representation.

A useful way to picture one is as a location on an invisible map. Imagine that every token is assigned coordinates—not just north and east, as on a paper map, but along many dimensions at once. Those coordinates are called an embedding: a numerical representation that gives the model a starting point for working with a token.

On this imaginary map, tokens that appear in similar settings can end up near one another. Words such as doctor, nurse, and clinic may share some coordinates because they often occur in related contexts. Rain, cloud, and umbrella may cluster in other useful ways. The model does not need a human to draw boundaries around “weather words” or “medical words.” During training, its representations are shaped by the patterns that help it predict what comes next.

The map is not a dictionary. An initial embedding gives a token a numerical starting representation, not a final interpretation. As the model processes the surrounding text, later representations can reflect context. For example, the word bat takes on different significance in a sentence about baseball than in one about a cave.

Nor is there a literal picture hidden inside the machine. The map analogy captures one useful idea: relationships can be represented by patterns of distance and direction. But the real space has far more dimensions than we can draw, and its coordinates do not correspond neatly to labels like “animal,” “past tense,” or “near a river.” A two-dimensional sketch would flatten away most of the structure—like trying to show every street, hill, and subway line in a city on a child’s treasure map.

The coordinates are useful not because each one has an obvious human meaning, but because together they help the model distinguish and connect patterns. A token’s initial representation can carry clues that make some continuations more fitting than others. As the model processes context, later representations can be transformed to reflect relationships among information in the prompt; the initial embedding remains a starting point, not a complete interpretation.

So the model’s input is no longer merely a row of numbered pieces. Each piece has a numerical starting representation in a vast, learned landscape of relationships. The landscape is mathematical rather than visual, and it was not planned like a chart by a lexicographer. It emerged because certain arrangements of numbers helped the model do its central job: predict language in context.

A token’s place on the model’s invisible map is only the beginning. To turn that starting representation into useful predictions, the model repeatedly transforms it. The machinery performing those transformations is shaped by an enormous collection of learned numbers called parameters.

Imagine a vast control room lined with adjustable knobs. Each knob can nudge the flow of information in some way: amplify a signal, soften it, or alter how one pattern combines with another. A modern language model has a great many such settings—often billions of them. The image is helpful, but don’t picture a labeled knob marked “past tense” or “recipe for pancakes.” Parameters do not usually correspond to tidy facts or rules that can be pointed to one by one.

Instead, their effects are distributed. A pattern such as “questions about boiling water often call for a temperature” may depend on many settings working together, alongside the model’s learned representations and the context supplied in the prompt. No single knob need contain that idea. It is more like a melody produced by an orchestra: the tune belongs to the arrangement of many notes, not to one instrument alone.

During training, the model makes predictions and receives a signal about how far they missed the mark. A learning process uses that signal to adjust the parameters, a little at a time, so future predictions are more useful. Across countless examples, the settings settle into a configuration that captures many recurring relationships in language and the world described by language. Training builds these learned settings; an ordinary prompt later supplies material for the model to process using them, rather than ordinarily changing them.

That does not mean the model builds a clean encyclopedia inside itself. Its parameters shape how incoming patterns are transformed and what continuations become likely. They can support a useful response without storing a fact as a sentence in a hidden drawer. This is one reason its knowledge can be uneven: a familiar association may be strongly supported, while a detail that sounds plausible but appeared rarely—or is simply not established—may be handled poorly.

The knobs analogy has another limit: a conversation generally does not adjust the model’s learned parameters. Those settings were shaped during training; a prompt supplies the current text for the model to process without ordinarily changing those settings.

So far, we have a landscape of numerical representations and a huge set of settings that transforms them. The prompt supplies the current clues; the parameters are patterns built earlier through training. The next question is how training adjusts those settings—and how a game of predicting what comes next can shape such a capable system.