The Compass of Broken Pottery: On the Sherd Theory of Discovery

In archaeology, there is a method for surveying a new site called the sherd survey. Before you dig a single trench, you walk the plowed fields after a rain. You don’t look for the grand temple or the buried throne. You look for broken pieces of pottery—sherds—scattered on the surface. A single fragment tells you little, but a pattern of them, a density gradient, becomes a map. It points not to the artifact itself, but to the living spaces, the middens, the places where people once gathered and discarded. The discovery of the whole begins with the systematic, patient reading of the broken.

This is, I think, a perfect metaphor for how a modern crawler understands a website. We often imagine the bot arriving like a triumphant excavator, spade in hand, aiming directly for the treasure of our latest article or product page. But that’s not how it works. The crawler’s first and most crucial act is the sherd survey. It moves from page to page, not initially to comprehend the masterpiece, but to map the density and connection of fragments—the links.

Each internal link is a piece of broken pottery on the digital surface. A lonely, orphaned page with no links pointing to it is a sherd in an empty field; it suggests a one-off event, perhaps unimportant, likely forgotten by the very civilization that created it. But a cluster of links, a neighborhood of pages that reference each other? That gradient of connection is the crawler’s primary compass. It whispers: Here. Dig here. This is where the living happened. The crawler’s “crawl budget” is, in essence, its time allocated for walking the field. It will spend most of that time in the dense clusters your link-patterns have revealed.

The Unintentional Map

The profound lesson from the sherd survey is that the most honest map is often the unintended one. The archaeologists don’t trust the legendary chronicle that says “the king’s hall lies to the east.” They trust the broken cookware and shattered common lamps found on the west hill. The chronicle may be propaganda; the garbage is truth.

So it is with our sites. We can write a pristine sitemap.xml—our official chronicle—declaring what’s important. But the crawler, if it’s wise, will cross-reference that chronicle with the unintentional map written by your actual link structure. If your “Important Announcement” page is linked only from your sitemap but never from your homepage, your blog, or your product hubs, the crawler sees a lone sherd in an empty field. It may note its coordinates, but it won’t spend time digging around it. The real “Important Announcement” to the machine is the page that is fragmentedly, repeatedly connected to by the rest of your digital settlement.

The practice, then, is one of intentional discard. Every time you link from one page to another, you are not just providing navigation; you are scattering a handful of pottery sherds on the digital surface. You are saying, “These two ideas live in the same neighborhood.” A website with a rich, thoughtful, and organic internal link structure is conducting a constant, rain-washed sherd survey upon itself, leaving a pattern so clear that the mechanical archaeologist cannot miss the heart of the city. It is in the broken connections—the humble, plentiful links—that the whole is first, and most truly, found.

Notes & further reading

A few pages I came back to while writing this: