Brave Search API

More Search API pages

API glossary

Latency

A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

Latency is the delay between a request and the response to it—basically how long a user or system waits for results. In AI search and language-model applications, it usually means the time from sending a query to receiving an answer, and it is a core measure of how responsive and usable a system feels.

In short: latency is wait time. But in LLM systems this wait time is split into distinct stages, and time-to-first-token often matters as much as time-to-full-response.

How latency works

For a large language model, the total wait breaks into stages:

  1. Network and queueing: The request travels to the server and may wait if the system is busy before any work begins.
  2. Prefill: The model reads and processes the entire prompt, including any retrieved context. This step scales with prompt length, so longer contexts directly increase time to first token.
  3. Time to first token (TTFT): The moment the first piece of the response appears, which is the metric users feel most because it signals the system is working.
  4. Generation: The model produces the rest of the answer token by token, at a rate often measured as time per output token.
  5. End-to-end latency: The full duration from request to final token, which is roughly TTFT plus total time to generate a response.

The defining point is that latency is not one number: a system can begin responding quickly yet take a while to finish, or vice versa, so the most meaningful measure will depend on the user experience being optimized for.

Latency vs. throughput and quality

  • Latency is how long a single request takes; throughput is how many requests or tokens a system can handle per unit of time. Optimizing one can come at the expense of the other.
  • Time to first token measures responsiveness and how fast output starts; end-to-end latency measures completion, and how fast it finishes. Both matter, for different tasks.
  • Faster responses and higher quality often trade off, since larger, more capable models tend to be slower to respond.

Where latency matters

  • Context length: A larger context window (one full of prompts and retrieved text) will increase prefill time and time to first token.
  • Model size: Bigger models generally produce higher-quality output but respond more slowly.
  • Retrieval and multi-step work: Retrieval-augmented generation (RAG) adds a lookup step, and agentic search will chain together many calls to the model, so each added step compounds the total wait.
  • Search APIs: When an application calls a search API to fetch live results, often as the retrieval step that feeds an LLM, the API’s own response time adds directly to the user’s total wait, so a low-latency search API is essential to keeping an AI answer pipeline responsive.
  • Mitigations: Streaming the response, trimming unnecessary context, caching, and using smaller or faster models all reduce actual or perceived latency.

In AI search and chat, latency is often the difference between a system that feels “fast” and one that users will abandon because of perceived slowness. This is why getting the first tokens out quickly can matter as much as the final answer’s quality.

Throughput, context window, large language model (LLM), time to first token (TTFT), streaming, inference, retrieval-augmented generation (RAG), agentic search.