The Gardener and the Undergrowth: On Cultivating a Site for Discovery

In formal garden design, there’s a concept known as the ‘borrowed landscape.’ The idea is to subtly incorporate elements from the wilder, untended territory beyond the garden’s fence—a distant hill, a stand of mature trees—to create a sense of depth and connection to a larger world. It’s a deliberate blurring of the line between the meticulously curated and the naturally occurring. This principle, I’ve come to think, holds a profound lesson for how we think about a website’s relationship with web crawlers. We are not just building tidy, walled-off sitemaps; we are cultivating an ecosystem where discovery can bloom from the edges.

Too often, we approach site architecture like 17th-century French formalists, insisting on rigid, symmetrical parterres of internal links and perfectly pruned XML sitemaps. Every path is predetermined, every page given its ordained place. This creates a clean, comprehensible garden for the search engine’s crawler to stroll through. But it also risks sterility. It tells the crawler, “This is all there is. Look no further.” The ‘borrowed landscape’ of the wider web—the unexpected inbound links from obscure forums, the tangential references in a niche blog’s comment section, the forgotten but still-relevant deep link from a decade-old resource list—these are the wild hills we fail to incorporate into our design.

The Value of the Unplanned Path

A skilled gardener knows that some of the most interesting growth happens at the margins, where cultivated plants self-seed into surprising new arrangements. On a website, this is the ‘undergrowth’ of organic, user-driven discovery: the related-post plugin that suggests an old article based on semantic similarity, not hierarchy; the tag cloud that creates unexpected connections; the search function that surfaces a deeply buried tutorial because it perfectly answers a long-tail query. These features don’t just serve users; they create new, unplanned pathways for a crawler to follow. They allow the crawler to ‘borrow’ context from the interior wilderness of your own content, discovering connections you didn’t explicitly architect.

The lesson is one of managed permeability. Our job isn’t to build an impenetrable fortress of perfect information architecture, nor is it to let the site run wild into an impenetrable thicket. It’s to cultivate a core structure so strong and clear that it can afford to have soft, permeable edges. It’s about ensuring that the internal ‘undergrowth’—those autogenerated links, related content modules, and robust internal search indices—is healthy and pointing to valuable content, not just generating crawl traps of infinite parameter spaces.

Ultimately, we must see our sites not as finished blueprints but as living plots. We plant the seeds of core pages and nurture them with clear navigation. But then we must step back and observe how visitors and crawlers actually move through the space, where new trails are being worn in the digital grass. Our sitemaps and robots.txt files are our pruning shears and trellises—essential tools for guidance, not the design itself. The true design welcomes a little of the wild in, understanding that the most complete map of a territory is drawn not just by its cartographer, but by all the creatures who walk through it.

Notes & further reading

A few pages I came back to while writing this: