Brave Search API

More Search API pages

API glossary

Crawl budget

A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

Crawl budget is the amount of crawling a search engine or bot will do on a given site within a period of time (in effect, the set of URLs it both can and wants to fetch). Often, this entails the combination of two factors: crawl capacity (how much a site’s server can handle without slowing down) and crawl demand (how much the crawler actually wants to fetch, based on a site’s popularity and how often its content changes). For large or frequently updated sites, crawl budget determines how quickly new and changed pages get discovered.

In short: crawl budget is how much a search crawler will fetch from your site; on larger websites, crawl budget can also determine how fast your content gets discovered and refreshed.

How crawl budget works

Several forces can work together to set crawl budget:

  1. Crawl capacity limit: The crawler calculates how many simultaneous connections it can use and how long to wait between fetches, based on how the site responds. Fast, error-free responses raise the limit; slow responses or server errors lower it.
  2. Crawl demand: The crawler fetches more from sites and pages it judges popular, fresh, or frequently updated, and less from stale or low-value URLs.
  3. Effective budget: Actual crawling is roughly the lower of the two. Even a fast server won’t be crawled heavily if demand is low, and high demand gets throttled if the server struggles.
  4. Wasted budget: Time spent on duplicate, low-value, or near-infinite URL spaces (faceted navigation, endless parameters) is essentially a portion of your crawl budget that was not spent on the pages that matter.
  5. Monitoring: Site owners track crawl activity in the Crawl Stats report in Google Search Console and in their raw server logs.

The defining point is that crawl budget is a constraint the crawler imposes to protect both its own and the site’s resources. You influence it mainly by serving content quickly and making your valuable URLs easy to find while keeping junk URLs out of the way.

Crawl budget vs. indexing and crawl rate

  • Crawling is the act of fetching pages, and crawl budget caps how much of that happens. Indexing is the separate step of deciding which fetched pages to store and rank. A page can be crawled without being indexed.
  • Crawl rate limit (or crawl capacity) is only the speed ceiling: how fast a crawler can fetch without straining the server. Crawl budget is the broader outcome of that ceiling combined with crawl demand.
  • Traditional crawl budget refers to search crawlers such as Googlebot. In practice, sites also handle traffic from AI crawlers such as GPTBot and ClaudeBot that consume shared server capacity, which is related operationally but not a redefinition of the SEO term.

Where crawl budget matters

  • Large sites: Sites with thousands or millions of URLs, such as e-commerce stores, marketplaces, and news archives, where not every page can be crawled often.
  • Frequently updated content: News and inventory pages that need quick re-crawling to stay current in results.
  • Site migrations and launches: Promptly discovering newly published or moved URLs depends on having a portion of your budget available.
  • Operational bot load: AI crawler traffic can consume server resources and reduce capacity for search crawling, so deciding which bots may crawl what (through robots.txt or tools such as Cloudflare AI Crawl Control) can support crawl-efficiency planning.

Optimizing your crawl budget

Crawl budget optimization is the practice of managing how efficiently search engine bots spend their limited time and resources discovering your website’s content. By eliminating technical waste and streamlining site architecture, you help crawlers prioritize crawling and discovery of your highest-value pages. Key optimization strategies can include:

  • Clean up robots.txt: Block search engine bots from wasting time on low-value or near-infinite URLs, such as internal search results, admin areas, and faceted e-Commerce filters.
  • Eliminate redirect chains and 404s: Fix broken links and collapse multi-hop redirects into a single direct path so crawlers don’t hit dead ends or burn requests.
  • Manage duplicate content: Use canonical tags or consolidate identical pages to stop bots from repeatedly processing the same text under different URLs.
  • Boost server speed: Enhance your Time to First Byte (TTFB) and leverage caching; a faster server allows search engine bots to safely process more pages in less time.
  • Prune your XML sitemap: Keep your sitemap updated by including only clean, canonical, and indexable URLs.
  • Streamline internal linking: Establish a flat site structure where your most critical pages sit no deeper than three clicks away from the homepage, keeping them highly accessible to crawlers.
  • Manage AI crawler traffic: Use robots.txt or tools such as Cloudflare AI crawl control to selectively block AI crawlers (e.g. GPTBot, Claudebot) that consume server resources without contributing to search visibility.

Most small sites do not need to think about crawl budget; they can often keep their sitemap current and let crawlers do their work. But crawl budget becomes critical as a site grows and/or content changes more frequently.

AI web crawler, robots.txt, Googlebot, indexing, crawl rate, server logs, sitemap, answer engine optimization (AEO), pay-per-crawl.