Context Changes What Matters

How does information from one part of a prompt reach another? One important answer is attention: a calculation that lets each token representation draw on information from other tokens in the context, with different degrees of influence.

Imagine a table conversation. Someone says, “Maya told Leah that she had won.” To interpret she, a listener looks back at the names and the surrounding context. In an LLM, attention calculations let the representation of she draw on information from earlier tokens, with their relative influence depending on patterns in the sentence. The sentence may remain ambiguous—perhaps either person won—but the model has a way to use available clues rather than treating every token as an isolated item.

The “selective listening” analogy describes an effect, not an experience inside the model. Attention is not an inner spotlight or a feeling of focus, and particular words do not literally listen to one another. It is a set of numerical operations: the system compares patterns in representations, calculates how information from other tokens can contribute, and combines information accordingly.

Attention works across many relationships, not just a single word reaching back to find a clue. A token representation may draw on nearby words, distant details, or several pieces of information at once. Different calculations can contribute to different relationships, such as linking a pronoun to a name or connecting a description to the thing described. These are not tidy, human-readable rules tucked into separate boxes; they are patterns of computation learned during training.

A model’s output depends, then, not only on which tokens are present but on how their representations interact. Attention gives the network a flexible way to use connections within the prompt. It does not guarantee that the model will interpret every relationship correctly; it provides a mechanism for bringing context to bear. When a prompt contains several clues, the important question is not merely whether the model received them, but how one clue influences another.

The model can let one clue influence another only if both clues are available in its current context. A conversation with an AI can feel like one continuous exchange: you mention a detail several screens ago, then expect it to matter now. But a model has a bounded capacity for the text it can process at once. This limit is called the context window.

Think of it as the model’s current desk. What is on the desk can be considered together: your latest question, earlier messages, and perhaps a document you supplied. What is outside the desk is not automatically present just because it once appeared in the conversation. The context-window limit belongs to the model; deciding what text from a longer exchange to put into that window is a separate job that an application or system may perform. Different systems handle long exchanges in different ways—for example, by keeping recent portions, compressing earlier material into a summary, or retrieving selected passages. These strategies vary: keeping or summarizing conversation history is application-level handling, while retrieval brings selected material into the current context. In every case, the model can use only what is actually brought into its current context.

Suppose you tell it, “My train arrives at 6:10, and I need to meet Lena at the north entrance,” then ask, “Which entrance should I go to?” If those details remain in the context, they can shape the answer: the north entrance. The model does not need a special memory of your travel plans; the relevant words are on the desk, where attention can connect them to your question.

But a long conversation can push earlier messages out of view, or leave them represented only by an imperfect summary. Ask the same question much later, and the model may not have the original detail available. It might ask you to repeat it, give a generic answer, or confidently choose an entrance without a sound basis. That is not necessarily forgetfulness in the human sense. It is a limit on what information is available for this particular round of processing.

A large context window does not mean every detail inside it will matter equally. Imagine a desk piled high with papers. The needed note may be present, yet easy to overlook among pages of unrelated material. Attention provides ways for the model to connect useful pieces, but it is not a guarantee that the right piece will be found or weighted correctly. A buried instruction, a subtle exception, or a detail mentioned only once may have less influence than a recent, repeated, or strongly phrased clue.

This helps explain why adding more background can sometimes make an answer worse rather than better. More text offers more possible clues, but also more competing material and more opportunities for relationships to be missed. The practical limit is not simply “How much can fit?” It is also “Can the model use the important parts well?”

The context window is therefore a boundary around the model’s working information, not a measure of everything it has ever learned. The model’s learned parameters shape how it processes text; the context window holds material available while it answers now. Keeping that distinction in mind makes a long chat easier to understand: continuity depends on which details are still on the desk, and how effectively the model connects them to the question at hand.

Section 1 introduced attention as a way token representations can draw on information from other tokens. Here, we can follow that process through a small prompt:

The fisherman sat on the bank.

At first, bank could point to more than one thing: a river’s edge, perhaps, or a financial institution. The model does not look up the word in a tiny dictionary and select one definition. Instead, the prompt is broken into tokens—pieces that may be whole words or parts of them—and each token is turned into a numerical representation. Those representations carry learned patterns about how pieces of language tend to relate. The prompt supplies the current clues; it does not automatically settle an ambiguity.

Now add a few words:

The fisherman sat on the bank and watched the current.

The new clue, current, changes the pattern of relevance. During processing, information associated with fisherman and current can contribute to the representation of bank. From training, the model has encountered many patterns in which fishermen, currents, and banks belong to rivers. Together, these clues make the river-edge interpretation more likely.

Change the ending:

The fisherman sat on the bank and checked his balance.

Now balance pulls the surrounding pattern in another direction. It often appears near money and accounts, so the financial sense of bank becomes more plausible—even though fisherman still points toward a river. The model’s response reflects the combined clues, not a single word acting as a switch.

Those relationships are not all stored as neat definitions. The model’s learned parameters—the many adjustable settings shaped during training—help it form and transform representations across layers. A layer may combine clues in one way; later computations can build on that result. The useful behavior emerges from these interacting calculations. There is no single “river-bank file” that the model opens, nor one calculation that settles meaning forever.

That is why the prompt is more than a bag of words, and why adding context can change an answer. Meaning-relevant behavior comes from patterns meeting one another: the tokens present, the relationships learned during training, and the influence of particular parts of the current prompt. A longer prompt may put more clues on the desk, but the answer still depends on which clues influence one another—and how.