The Librarian's Dilemma: On Choosing Which Shelves to Dust
Imagine the grandest library you can. Its stacks are endless, a labyrinth of knowledge. Now imagine the head librarian, tasked not with reading every book, but with a more fundamental chore: keeping the dust at bay. Their time is finite. They cannot wipe every shelf every day. So, they must choose. Which corridors are most frequented? Which tomes, though perhaps less popular, are of such critical importance they demand daily attention? This is not a story about reading; it is a story about prioritization. This is the daily reality of a web crawler.
We often speak of a website as a single entity, but to a crawler, it is a vast and sprawling archive. Its ‘crawl budget’ is the librarian’s limited hours in the day. It is the finite amount of time and resources a search engine allocates to explore your particular library. The central question, then, is not ‘Will my site be crawled?’ but rather, ‘Which parts of my site will be crawled, and how often?’
The crawler makes its choices based on signals we often provide without realizing it. It notices the well-trodden paths—the pages linked from your homepage, the ones that gather incoming links from other respected libraries, the ones that change their content with a regular and meaningful cadence. These shelves get dusted often. The crawler trusts their importance.
But what of the back rooms? The archive of last year’s promotional content, the thousand-page deep catalog of obsolete products, the duplicate versions of pages created by faulty filters? These are the dusty, disorganized annexes that the librarian must reluctantly navigate. Every minute spent wiping down these forgotten shelves is a minute not spent in the bustling new releases section. A site cluttered with low-value, infrequently updated, or poorly interlinked pages forces the crawler into a dilemma: exhaust its budget on the archives or risk missing a vital new volume in the main hall.
The power, then, lies not in begging for more of the librarian’s time, but in curating a better library. It is in building clear, logical shelves (a robust information architecture) and using a precise map (a lean, accurate sitemap) to highlight the essential wings. It is in sealing off the rooms filled with broken furniture (by removing or properly signaling outdated content with 410s) so the librarian doesn’t waste time trying to clean them. By making the valuable and vital easier to find, we guide the limited attention of the crawler to the very heart of what we have to offer, ensuring our most important stories are the ones that get told.
Notes & further reading
A few pages I came back to while writing this: