The Curator's Waste: On the Essential Inefficiency of Being Found

We are taught, relentlessly, to streamline. In the world of discovery, this translates to a singular gospel: efficiency. Make your site fast. Prune your duplicate content. Craft the perfect sitemap. Hoard your crawl budget like a miser, directing the spider’s precious attention only to where it absolutely must go. This logic is impeccable, clean, and in its own sterile way, profoundly limiting. It treats the crawl as a production line, where waste is the ultimate sin. But what if waste—thoughtful, deliberate, generous waste—is not a bug in the system, but a feature of a living web?

Consider the advice to eliminate all "orphaned" pages, those dead-end documents that receive no internal links. The efficient curator sees a leak in the crawl budget, a page draining resources without strategic value. They would delete it or, at minimum, wall it off with a noindex tag. But an orphaned page is not necessarily a useless page. It might be an old, detailed research note, a forgotten essay that answered a question for one person five years ago. It exists. By the ruthless logic of internal linking, it doesn’t "deserve" to be found. Yet, through an external link from a forum, a citation in a scholarly paper, or a social media post now lost to time, someone might still be looking for it. To prune it is to sever that thread, not because it was frayed, but because it didn’t originate within your own neatly bound book.

This obsession with internal linking creates a kind of intellectual feudalism. Only pages that swear fealty to the site’s central hierarchy merit protection and visibility. Everything else is left to die in the cold. It creates a web that is increasingly inward-looking, where discovery is purely a function of architecture, not of accident, reference, or tangential curiosity. We are building walled gardens and calling them efficient websites. The old web, the one we sometimes romanticize, was riddled with these orphans—and they were often the most interesting places to be.

The same applies to the fear of the "deep" page. We’re told to keep everything within three clicks of the homepage, as if user and crawler alike possess the attention span of a gnat. But depth implies specialization. A page that is five clicks deep isn’t necessarily buried; it is curated for someone on a very specific quest. The journey through those layers is a process of filtering, of moving from the general to the profoundly particular. To flatten that structure for the sake of crawl efficiency is to disrespect the intent of the seeker, who may well prefer the narrow path to the crowded plaza.

Efficiency seeks to eliminate all friction, all redundancy, all unanswered doors. But friction is where thought happens. Redundancy is resilience. An unanswered door is an invitation to knock. Perhaps the health of a website, like the health of an ecosystem, isn’t measured by its lean throughput, but by its tolerable surplus—its capacity for the unexpected connection, its hospitality to the passerby who arrived through a hidden gate. Stop treating your site like a machine optimized for a single known output. Start tending it like a landscape where getting gloriously, productively lost is still possible. Let a little crawl budget go to waste. You might be surprised by what, and who, it finds.

Notes & further reading

A few pages I came back to while writing this: