The Unspoken Geometer: On the Ancient Algorithm of the Spider's First Step
Before a crawler maps a domain, before it budgets its requests or parses a single sitemap, it must solve a problem of pure geometry. Its first instruction is not a URL, but a seed. And from that singular point, it must decide how to move through a theoretical space it cannot yet see. This initial, silent calculation is a tradition inherited not from computer science, but from a far older practice: the geometric proofs of the ancient Greeks.
Consider the crawler at its moment of inception, poised at the homepage. It is like a geometer with a single point. Euclid’s first postulate states that a straight line can be drawn from any point to any other point. But the crawler has no other points. Its world is one of potentiality. So it does what the geometer would do: it creates a circle. It sends out a request, not in a line, but in all directions at once, tracing a circumference of links from that initial seed. It defines its world not by what it knows, but by the radius of its first connection.
This radial expansion is the web’s primordial algorithm. It is how discovery must begin—with an assumption of connection, a faith that from one point, others will be derived. The crawler doesn’t know if the links it finds are the ‘important’ ones; it simply knows they are there, and therefore they must be visited. This is the digital embodiment of a geometric truth: from a single axiom, an entire universe of theorems can be built.
The Elegance and the Burden of the First Principle
This elegant, necessary first step is also the source of the crawler’s greatest burden. For the geometer, the circle is a perfect, theoretical form. For the crawler, it is a practical nightmare. That first radial sweep captures everything: the vital ‘About Us’ page, the monthly blog posts, the link to the shipping policy, and the forgotten, decade-old ‘Summer Picnic 2013’ photo gallery. The geometry is blind to value; it only sees connection.
And so, the tradition of the sitemap is not a replacement for this ancient, radial method, but a correction to it. It is a human attempt to introduce hierarchy into the crawler’ perfect, flat circle. By providing a guided list, we are essentially handing the geometer a pre-drawn diagram, saying, “Here are the points and the lines that truly matter. You can still draw your circles, but consult this first.” We are trying to impose a kind of intellectual cartography onto a process that is, at its heart, profoundly and beautifully geometric.
The next time you study a site’s crawl stats, look for the ghost of that first circle. You’ll see it in the pages discovered one hop from the seed, the ones found not through shrewd strategy but through simple, radial proximity. They are the testament to a tradition of discovery that values existence over importance, connection over curation. It is the unshakable first principle from which the entire visible web is proved.
Notes & further reading
A few pages I came back to while writing this: