The Prospector's Claim: On the First Web Page to be Indexed

We spend so much time worrying about how our pages are found that we rarely ask a more fundamental question: what does it mean to be the 'first' to be found? In the vast, unmapped wilderness of the early web, how did a search engine even know where to begin? The answer isn't a technical specification; it’s a story of a single, fragile claim staked in the digital void.

Before Googlebot and its countless peers, there was the World Wide Web Wanderer. Created by MIT student Matthew Gray in the summer of 1993, the Wanderer was one of the very first web crawlers. The web then was a small town, a collection of a few hundred known servers, but it was expanding at a bewildering rate. Gray’s creation had a simple, almost poetic purpose: to measure the web’s growth. It would wander from link to link, counting what it found, a cartographer sketching the very first, blurry edges of a new world.

The Groundbreaking Non-Event

There had to be a starting line. The Wanderer, like any explorer, needed a home base, a place to launch its expeditions. That place, the first URL ever intentionally fetched by an automated crawler for the purpose of indexing, was Gray’s own creation: the homepage for the Wanderer project itself. Think about that for a moment. The entire concept of search engine discovery, the multi-trillion-dollar industry of being found, arguably began with a piece of code looking at itself in the mirror.

This wasn't a grand, orchestrated index of human knowledge. It was a test, a proof of concept. The Wanderer started its journey by verifying its own existence. From that solitary page, it would then venture out, following the hyperlinks it found to other servers, adding them to its list, and slowly, patiently, weaving the first crude web of what was out there. The crawl budget was the entire web, and it was small enough that a single, slow-moving spider could realistically hope to see most of it.

This origin story feels profoundly different from our modern understanding of crawling. Today, we submit sitemaps and worry about crawl efficiency, treating discovery as a negotiation with an immense, impersonal system. The Wanderer’s first act was one of self-reference, an act of bootstrapping a world into being. The signal it chose to send was, first and foremost, a beacon to itself: 'I am here, and from here, I will go everywhere.'

There’s a quiet humility in this beginning that we’ve lost. The first indexed page wasn’t a corporate homepage vying for attention or a news site fighting for prominence. It was a tool’s own instruction manual, a seed from which the forest would grow. Every page that is found today, every piece of content that surfaces in a search result, is a distant descendant of that initial, self-referential loop. The prospector’s claim wasn't on a vein of gold, but on the very idea of a map. And from that single, modest stake, everything else followed.

Notes & further reading

A few pages I came back to while writing this: