The Virtue of Shallowness: In Defense of the Surface Crawl

It is a foundational tenet of our field that a deeper crawl is a better crawl. We are told to relentlessly optimize, to hoard crawl budget like misers, and to guide a crawler’s unblinking eye into the deepest catacombs of our sites. The goal is to leave no page unturned, no unlinked orphan stranded. We are architects of comprehensiveness. But I want to propose a different, quieter approach. What if, for the health of a site and the sanity of its curator, a shallow website is not a failure, but a deliberate and virtuous design?

The prevailing wisdom is obsessed with discovery, with pushing past the sixth click, the tenth, the twentieth. We’re driven by a fear of the unseen. We worry about that perfect, poetic piece of content languishing in a sub-sub-folder, never to be found by a search engine, and thus, never to be found at all. This fear fuels a complex industry of tools and audits, all aimed at proving that our digital estates are fully mapped and cataloged.

But this exhaustive mapping comes at a cost. A deeper crawl spreads a crawler’s finite attention thinner. Pages of diminishing value—archived log files, ancient user profiles, experimental dead ends—compete for resources with the vital, current content that forms the core of a site’s purpose. In trying to illuminate every corner, we risk dimming the light on what truly matters. The crawl budget is not just a resource to be maxed out; it is a spotlight to be focused.

This is where the counterintuitive idea takes root: instead of engineering for maximum depth, engineer for elegant shallowness. Design a site that is intentionally, proudly shallow. A structure where nothing of consequence is more than three or four clicks from the root. This isn’t about hiding content; it’s about prioritizing it. It forces a brutal and beautiful clarity. It asks the essential question: if this piece of content is important enough to exist, is it not important enough to be easily accessible?

A shallow architecture is a declaration of intent. It says, "This is who we are, and this is what we offer." It has no room for the digital detritus that accumulates over years. It compels a kind of mindful minimalism, where every page must justify its place in the hierarchy or be culled. The crawler, in turn, responds to this clarity. It can spend its precious time reinforcing the index of your best work, rather than fruitlessly spelunking into archives of the irrelevant. It’s not about preventing discovery; it’s about guiding it with a firm and confident hand. Sometimes, the most powerful way to be found is to have less that needs finding.

Notes & further reading

A few pages I came back to while writing this: