The Librarian's Shelving Cart: On the Quiet Work of Triage and Placement
In the quiet aisles of a great library, long after the patrons have gone home, a cart stands laden with returned books. This is not a chaotic pile but a carefully ordered one, sorted by a librarian who understands that the journey back to the shelf is as critical as the information within the books themselves. This process of triage and placement, of deciding what goes where and when, is a profound, if unexpected, lesson for how we think about a website’s pages being discovered by a search engine’s crawler.
The librarian does not, and cannot, reshelve every book at once. Some are bestsellers, requested daily, and must be returned to their prime locations immediately. Others are obscure academic texts, important to the collection’s depth but rarely sought; their journey to a high, remote shelf can wait. The cart itself is a form of crawl budget—a finite capacity for work that must be allocated with intention. The librarian is the algorithm, making constant, quiet judgments about priority, value, and the overall architecture of the available knowledge.
We can view our websites through this same lens. The crawl budget is that shelving cart. A crawler arrives, its capacity limited, and it must decide what to ‘pick up’ and process. The high-priority pages—our home page, key product pages, vital articles—are the bestsellers. They must be easily accessible, found quickly, and returned to the ‘shelf’ of the index promptly. They are the books placed right by the entrance.
But what of the older blog posts, the archived press releases, the deeply nested support documents? These are our academic texts. They hold value and contribute to the richness of our domain, but they do not require the same immediate, daily attention. A wise librarian, and a wise webmaster, doesn’t waste the cart’s limited space on them every single day. They might use a sitemap to say, “These exist, in this section, when you have time.” They might employ sensible robots.txt directives to prevent the crawler from getting lost in the labyrinthine stacks of infinite URL parameters, the literary equivalent of a dusty, disused basement.
The goal is not to hide these pages, but to acknowledge that a system’s resources are finite and must be directed with purpose. A library where every book is deemed equally urgent is a library in chaos. A website where every URL screams for the crawler’s immediate attention is a website that may see its truly important content languish on the cart, waiting behind a pile of low-value clutter. By embracing the librarian’s art of thoughtful triage, we don’t neglect parts of our collection; we honor the whole by ensuring the right things are found at the right time.
Notes & further reading
A few pages I came back to while writing this: