The Gardener and the Angler: On Two Approaches to Cultivating Discovery

We often talk about web crawling as a technical process, a matter of bots and budgets. But beneath the protocols and pings lies a fundamental philosophical choice about how we believe the web should be explored. I’ve come to see this choice not in server logs, but in two contrasting figures: the Gardener and the Angler.

The Gardener approaches their domain with a plan. They till the soil of their server, plant seeds in neat rows with clear markers, and provide a meticulous map—the sitemap—of every plot. Their work is one of structure and intention. They believe that for a crawler to understand the whole, it must first be shown the blueprint. Every internal link is deliberately placed, every hierarchy carefully considered. The Gardener’s goal is to create a perfectly organized ecosystem where nothing is left to chance and every valuable page is presented for easy indexing. It is an act of cultivation, of controlled growth.

The Angler operates on a different principle: patience and potential. They cast a single, well-baited hook—a powerful, central page—into the vast ocean of the web and wait. They are not focused on providing a map of the entire lake bed. Instead, they trust in the current. Their strategy is to create a page so compelling, so richly interlinked by others, that it naturally attracts the crawlers that swim by. From that single point, the crawler is meant to discover the rest of the domain by following the natural, organic lines of connection. The Angler doesn’t force the path; they suggest it, relying on the innate behavior of both crawlers and humans to find their way.

These approaches are not mutually exclusive, but they represent a clear tension. The Gardener seeks to eliminate discovery risk through explicit instruction, while the Angler embraces it, betting on the quality of a single point to generate its own gravitational pull. One is a top-down strategy of control; the other is a bottom-up strategy of influence.

Most successful endeavors likely employ a blend of both. A sitemap (the Gardener’s tool) ensures the foundational structure is visible, while a truly link-worthy cornerstone article (the Angler’s bait) attracts the organic traffic that reinforces it. The key is understanding that crawling is not just a technical function but a dialogue. You can choose to speak in clear, direct sentences, laying everything out in a structured format. Or you can choose to tell a single, captivating story so good that listeners—both human and bot—will inevitably ask to hear more.

Notes & further reading

A few pages I came back to while writing this: