The Streetlamp's Radius: On the Light We Cast for Discovery

There’s a streetlamp on the corner near my house that I pass every evening. Its light doesn’t flood the entire block; it creates a perfect, distinct circle of visibility on the pavement below. Inside that circle, every crack in the concrete, every fallen leaf, is rendered in sharp detail. Just beyond the circle’s edge, the world dissolves into shadow, its details obscured and uncertain. It’s a simple, everyday phenomenon, but it strikes me as the most honest metaphor for how a search engine’s crawler discovers the pages of a website.

We often talk about a website as if it’s a single, luminous entity, willing itself to be seen. But to the mechanical spider trawling the web, a site is more like a vast, dark neighbourhood. It doesn’t have floodlights. It only has the light cast by the streetlamps we’ve erected—our links. Each hyperlink, whether on our own site or on another, is one of these lamps. It illuminates a very specific radius around it, making the pages it points to visible and, crucially, making the links *on those pages* visible as well. The crawler moves from lamp to lamp, navigating the illuminated paths we’ve built.

The Unlit Alleyways Between Our Pages

The problem, of course, is the darkness between the lamps. In our websites, these are the pages that have no internal links pointing to them. An old blog post, buried in an archive. A product page for a discontinued item. A utility page created for a single, specific campaign. These pages sit in the digital shadows, mere steps away from a well-lit path, but completely invisible to the crawler because no light falls upon them. They exist in the database; they have a URL. But without a link, they are like houses in an unlit alley—present on the map, but unreachable and undiscoverable in the practical geography of the crawl.

This is where our sitemap.xml file comes in. We often think of it as a master list, a comprehensive census of every page we own. And it is. But a sitemap is less like turning on the sun and more like handing a list of addresses to the crawler. It’s a suggestion, a polite tip. The crawler may indeed visit those addresses, but it will be a fleeting, tentative visit. Without the sustained light of internal links signalling a page’s ongoing importance and centrality to the neighbourhood, that page remains on the periphery. Its discovery is a bureaucratic formality, not a natural result of being part of a well-connected community.

The ethical question for us, as the architects of these digital spaces, is about where we choose to install our lamps. Do we concentrate all the light on our newest, shiniest pages, leaving valuable but older content in the dark? Or do we thoughtfully string lights along the paths that connect our most important ideas, ensuring that discovery is a continuous journey, not a series of disconnected, bright spots? We must tend to the radius of our light, ensuring the most valuable parts of our web are not just discoverable, but meaningfully connected. Because in the end, a crawler, like a person walking home at night, will always follow the light.

Notes & further reading

A few pages I came back to while writing this: