Crawl Budget
The number of pages a search engine bot will crawl on your site within a given timeframe, determined by crawl rate limit and crawl demand. Crawl budget optimization ensures important pages are discovered and indexed efficiently.
Crawl budget is the combination of how fast a search engine can crawl your site (crawl rate limit, based on server health) and how much it wants to crawl (crawl demand, based on page importance and freshness). For most small to medium sites, crawl budget is not a concern. It becomes critical for large sites with thousands or millions of pages.
For growth teams running programmatic SEO at scale, crawl budget management is essential. If your site generates thousands of landing pages, you need to ensure Googlebot prioritizes high-value pages. Optimize crawl budget by eliminating duplicate content, blocking low-value pages via robots.txt, using canonical URLs to consolidate similar pages, maintaining a clean internal linking structure that prioritizes important pages, and keeping your sitemap updated with only indexable URLs. Monitor crawl stats in Google Search Console to understand how Googlebot allocates its budget across your site. A sudden drop in crawl rate often indicates server performance issues or a crawl trap.
Related Terms
Core Web Vitals
A set of three Google-defined metrics that measure real-world user experience for loading performance, interactivity, and visual stability. Core Web Vitals are a confirmed ranking factor in Google Search.
Largest Contentful Paint (LCP)
A Core Web Vital that measures the time from page load start until the largest visible content element (image, video, or text block) is rendered on screen. Good LCP is 2.5 seconds or less.
Interaction to Next Paint (INP)
A Core Web Vital that measures the latency of all user interactions (clicks, taps, keyboard input) throughout the page lifecycle, reporting the worst interaction. Good INP is 200 milliseconds or less.
Cumulative Layout Shift (CLS)
A Core Web Vital that measures the total amount of unexpected layout shifts that occur during a page's entire lifespan. Good CLS is 0.1 or less, where layout shifts are calculated from the impact and distance of moving elements.
Time to First Byte (TTFB)
The duration from the user's request to the first byte of the server response reaching the browser. TTFB measures server-side processing speed and network latency, directly impacting all subsequent loading metrics.
Canonical URL
An HTML element that tells search engines which version of a page is the preferred one when multiple URLs serve similar or identical content. Canonical tags consolidate ranking signals and prevent duplicate content issues.