Brave Search API

More Search API pages

API glossary

Tokenization

A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

Tokenization is the process of breaking text into smaller units called tokens, the pieces a system actually processes. For a language model, tokens are usually sub-word fragments that the model reads as numbers rather than whole words. In classic search, tokenization instead splits text into the terms used to build and query an index. Either way, it is the step that turns raw text into units a system can work with.

In short: tokenization splits text into tokens, which are the sub-word pieces that an LLM processes, or the terms a search index matches on.

How tokenization works

For language models, common approaches use sub-word tokenization:

  1. Choose a tokenizer: A model uses a specific tokenization scheme. Common approaches include byte pair encoding (BPE), WordPiece, and SentencePiece variants, which all build a fixed vocabulary from sub-word units.
  2. Split the text: Input is broken into tokens from that vocabulary, allowing common words to become single tokens, while rare words, code, and emoji are split into several tokens.
  3. Map to IDs: Each token is converted to a numeric ID, since the model processes numbers and does not directly see letters.
  4. Count: The number of tokens determines how much of the context window the text uses and, on most APIs, how much a request costs.

In search indexing, tokenization is simpler and different: text is split into terms, often around whitespace and punctuation, so documents and queries can be matched. Search analyzers may then apply normalization steps (for example, lowercasing, stemming, or lemmatization), which are distinct from token boundary splitting.

The defining point is that tokenization is model-specific and method-specific: the same text can produce a different token count depending on the model, the provider, and whether it is being processed by an LLM or a search index. Never assume token counts transfer between systems.

Tokenization vs. chunking and embeddings

  • Tokenization splits text into the smallest units a model or index uses; chunking splits a document into larger passages for retrieval. Tokens are small fragments; chunks are whole passages.
  • Tokenization produces discrete token IDs; embeddings turn tokens or chunks into continuous vectors that capture meaning. One is a lookup, the other a learned representation.
  • Sub-word tokenization (for LLMs) optimizes for a fixed model vocabulary; search tokenization (for indexing) optimizes for matching terms between queries and documents.

Where tokenization matters for developers

  • Cost and limits: API requests are typically billed per token and capped by the context window, so token counts directly drive both price and what fits.
  • Latency and budgeting: More tokens mean more to process, both of which add to latency, so trimming prompts and retrieved context is one way to optimize for faster response times.
  • Keyword search: Building or querying a lexical index depends on consistently tokenizing documents and queries, which is the basis of keyword matching in hybrid search.
  • Search APIs in LLM pipelines: Content returned by a search API for grounding consumes tokens once it enters the model’s context, so token-efficient, pre-extracted results help an answer pipeline stay within budget and cost less to run.

Tokenization is invisible until it is not: it quietly determines what a model can read, what a search index can match, and what each request costs.

Token, context window, chunking, embeddings, large language model (LLM), byte pair encoding (BPE), keyword search, hybrid search, latency, search API.