The Forager's Plenty: On the Problem with Always Being Fed

For decades, the mantra of the web has been findability. We are told, in a thousand different ways by a thousand different experts, that a page’s primary purpose is to be discovered. To be crawled, indexed, and presented to the seeking user. Entire industries are built on optimizing this journey, on smoothing the path from server to search result. But I want to question this fundamental goal. Is being found—constantly, effortlessly, algorithmically—always the ultimate good? Or, in our quest to make every morsel of content instantly available, have we lost something essential about the act of discovery itself?

This received wisdom hinges on a simple equation: if a page exists, it should be found. We deploy sitemaps like detailed dinner menus and fret over crawl budget as if it were a scarce ration. We structure our data with meticulous care, laying out a perfect trail of breadcrumbs for the robotic forager. The goal is a kind of frictionless consumption, where the user’s hunger for information is satisfied with a single, efficient query. The web becomes a well-stocked pantry, its contents fully inventoried and perpetually at hand. This seems like an unassailable victory for order and accessibility.

Yet, this system creates a peculiar kind of emptiness. The metaphor of the pantry is apt, but it misses a crucial point: the food found there is often pre-packaged, predictable, and stripped of context. When discovery is a transaction, it loses its soul. It becomes mere retrieval. There is no thrill in being handed exactly what you asked for. The real magic of exploration, the kind that built the early web, was in the stumble, the unexpected link, the recommendation from a friend, the deep dive into a forgotten directory. It was the difference between a curated grocery run and foraging in a wild forest. You might not find the specific mushroom you envisioned, but you might instead discover a berry you never knew existed, growing in the shade of a concept you’d never before considered.

A World of Known Unknowns

Our modern crawl-centric web, for all its power, excels at answering questions we already know how to ask. It thrives on "known knowns." It struggles profoundly with the "unknown unknowns"—the things we don’t know we’re looking for because we lack the language or even the awareness of their existence. This is the territory of the accidental find, the serendipitous connection. By prioritizing absolute findability, we have systematically engineered serendipity out of the equation. The crawler’s logic is one of completeness; it wants to map the entire forest. But in doing so, it leaves no room for the unmarked, personal trails that lead to the most surprising clearings.

Perhaps, then, the greatest flaw in our obsession with being found is that it makes the web less interesting. It flattens the landscape. When every page is optimized for a specific query, it ceases to be a place you wander and becomes a destination you target. The journey is reduced to a loading bar. Instead of trying to feed every page to the crawler, perhaps we should be more comfortable with the idea that some things are better found by accident, through curiosity and human connection, rather than by algorithmic decree. A little obscurity isn’t always a failure; it can be a feature, preserving the web’s capacity for wonder.

Notes & further reading

A few pages I came back to while writing this: