Brave Search API

More Search API pages

API glossary

Context window

A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

A context window is the maximum amount of text (measured in tokens) that a language model can consider at one time. It spans everything the model is working with in a single conversation: the system instructions, the user’s prompt, any conversation history, any retrieved documents, and the response the model is generating. Once the combined text exceeds the window, the model can no longer “see” the overflow, so the oldest or least relevant material has to be dropped, summarized, or retrieved on demand.

In short: the context window is the model’s working memory for a single conversation.

How a context window works

A model doesn’t read characters or words directly; it reads tokens, and the window is a fixed token budget for the whole exchange:

  1. Tokenization: Input text is broken into tokens (sub-word units), and the window is counted in tokens rather than words or characters.
  2. Shared budget: The window is split across the system prompt, conversation history, retrieved context, and the generated output, so more of one leaves less room for the others.
  3. Attention pattern: In standard full-attention transformers, tokens in the active window can attend broadly across that window. Some long-context architectures instead use sparse, local, or sliding-window attention patterns, so attention is not always unrestricted all-to-all across the full context.
  4. Overflow handling: When content would exceed the limit, the application must truncate, summarize, or selectively retrieve so the most relevant material fits.
  5. Position effects: Models often use information at the start and end of a long window more reliably than material buried in the middle, an effect sometimes called lost in the middle.

The defining point is that a larger context window lets a model take in more at once, but it does not guarantee the model uses every part equally well: relevance and placement still matter.

Context window vs. memory and retrieval

  • Long-term memory persists across sessions and lives outside the model; the context window holds only what is present in the current conversation.
  • Training knowledge is baked into the model’s weights and is always available; the context window is filled fresh at request time and is where new or private information must be placed.
  • Retrieval (RAG) selects only the most relevant passages to fit inside the context window; the context window is the limited space those passages compete for.

Where the context window matters

  • Long conversations: As a chat grows, older pieces of the conversation eventually fall outside the context window unless they are summarized or stored, which is why assistants can appear to “forget” earlier details.
  • RAG and document Q&A: Retrieved chunks have to fit alongside the prompt and the answer, so chunk size and the number of chunks retrieved are both bounded by the context window.
  • Long-document tasks: Summarizing or analyzing large files depends on either a large enough context window or a strategy for feeding the document in parts.
  • Cost and latency: More tokens in the context window generally mean higher cost and slower responses, so filling it efficiently matters as much as its raw size.

The context window is a budget to manage, not just a number to maximize: the goal is to fit in the right information, rather than all information.

Token, tokenization, retrieval-augmented generation (RAG), chunking, prompt, large language model (LLM), long-term memory, lost in the middle, inference cost.