The Scavenger and the Architect: On Two Ways to Build a Web

We often speak of web crawlers as if they are a single, monolithic force, a patient and inexorable tide that washes over the internet's shores. But in the quiet, methodical work of discovery, two distinct philosophies emerge, each with its own logic and its own consequences. One is the way of the Scavenger; the other, the way of the Architect.

The Scavenger’s approach is one of emergent discovery. It begins with a single, trusted seed—a known URL—and from there, it follows the trails. It reads the signposts left in the HTML: the links. Each new page offers a new set of pathways, a new network of connections to be followed. This is a process of foraging, of building a map not from a grand plan but from the paths actually taken. The resulting web is organic, a reflection of how pages are genuinely connected. It finds the forgotten corners and the unexpected hubs that human design might overlook. It trusts the lived structure of the site over any intended one.

The Architect, by contrast, works from a blueprint. Its primary tool is not the link but the sitemap, an XML file laid out like a formal set of plans. This document is a deliberate declaration: "These are my pages. This is their hierarchy. This is how often they change." It is a top-down view, an attempt to communicate directly with the crawler, bypassing the messy, interlinked architecture of the live site. The Architect’s web is clean, efficient, and intentional. It represents the ideal state, the structure as the builder wishes it to be seen.

Neither approach is inherently superior, but each casts a different shadow. The Scavenger risks getting lost in the weeds, spending its precious crawl budget on infinite loops of tag pages or endless comment threads—the digital equivalent of following breadcrumbs into a thicket. It can miss isolated pages that are never linked from anywhere, leaving beautifully constructed rooms with no doors.

The Architect, for all its clarity, can become an island unto itself. A sitemap that drifts out of sync with the reality of the site creates a map to phantom pages and broken promises, eroding a crawler’s trust. It represents a vision, but it cannot replicate the rich, contextual tapestry that a link provides—that vote of confidence from one page to another.

The most resilient presence on the web understands that discovery is not a choice between these two methods but a conversation between them. It lays down the Architect’s clear plans via sitemaps while also building a landscape rich with the Scavenger’s pathways—meaningful internal links. It invites the crawler to both read the blueprint and wander the halls. For a page to be truly found, it must exist both in the formal directory and in the lived-in world.

Notes & further reading

A few pages I came back to while writing this: