The Myth of the Infinite Budget: On the Crawler's Economy of Attention
We talk about web crawlers as if they are tireless, insatiable beasts, endlessly hungry for content. We imagine them scuttling across the digital expanse, powered by some boundless energy source, consuming everything in their path. This is the myth of the infinite crawl budget, a comforting but dangerous fantasy that lulls us into a state of architectural complacency.
The reality is far more human, and far more constrained. A crawler’s resources—its time, its processing power, its network requests—are profoundly finite. It operates under a strict economy of attention. Every second spent wandering down a dead-end URL is a second not spent discovering the vital article you published yesterday. Every ounce of energy expended parsing a bloated, script-heavy page is energy diverted from indexing ten clean, meaningful ones.
We often treat sitemaps and internal links as an indiscriminate invitation: "Here is everything! Come and get it!" But this is like handing a cartographer a stack of every single receipt, grocery list, and love letter you’ve ever written and asking for a map of your city. The cartographer, overwhelmed, will either give up or produce a useless, cluttered mess. The crawler does the digital equivalent. It doesn’t give up, but it makes choices. It prioritizes. And its priorities may not align with yours.
The received wisdom we must critique is the notion that more is always better. More pages, more links, more everything. This industrial-age thinking fails in the crawler’s attention economy. A sprawling, poorly considered site structure doesn't showcase your abundance; it imposes a tax on the very process meant to find you. It forces the crawler to spend its limited currency on corridors that lead nowhere, on rooms with nothing inside.
True discovery isn’t about being seen by a crawler. It’s about being understood. It’s about constructing a web of content so logically interlinked, so valuable and clear, that the crawler’s limited attention is spent not on a frantic treasure hunt, but on a guided tour of your best work. We must shift from being hoarders of pages to being curators of pathways. The goal is not to have every single thing found, but to have the right things found first, and found in context.
The most respectful thing we can do for the machines that map our world is to acknowledge their limits. By building with intention and economy, we don’t limit discovery—we finance it. We give the crawler the gift of focus, allowing it to spend its precious budget not on crawling our web, but on understanding it.
Notes & further reading
A few pages I came back to while writing this: