The Stalled Engine and the First Map: On Berners-Lee's 'What's New?'
In the popular imagination, the early web was a kind of digital wilderness, waiting for search engines to emerge as its necessary cartographers. This narrative, however, overlooks a quiet, human-scale struggle that came first: the fundamental inability to know what was even out there to be indexed. Long before Googlebot, the first great discovery problem was solved not by a crawler, but by a list. And the man who built the web itself was also the first to realize its discovery engine had stalled.
By late 1992, Tim Berners-Lee’s World Wide Web was growing, but its engine of growth was sputtering. The protocol was brilliant, but it lacked a fundamental social layer. How would a user, having explored their own bookmarks and the links within them, find the new, the interesting, the relevant? The web was a universe with no public signposts. In response, Berners-Lee created and maintained, by hand, a page on the CERN server titled simply "What's New?" It was a chronologically ordered list of new websites, with brief descriptions. For a time, it was the primary way the web discovered itself.
The Exhaustion of the Human Registry
Berners-Lee’s "What's New?" page was, in essence, the first centralized sitemap for the entire web. But it was a sitemap curated by a single, overwhelmed individual. As the web expanded from dozens to hundreds of sites, the task became unsustainable. He would receive emails (the submission form of the day) from webmasters announcing their new "home page," vet them for appropriateness, and manually add a line to the HTML document. This was a crawl budget of one human brain, and it was rapidly being exhausted.
The failure of this model was its greatest lesson. Berners-Lee saw that a human-maintained registry was a bottleneck antithetical to the web's decentralized nature. In his own words, it was "impossible to keep up." This exhaustion directly spurred the development of the first automated tools. The "What's New?" page proved the desperate need for discovery, thereby creating the space for the first primitive web crawlers—like Matthew Gray’s Wanderer in 1993—and later, automated directories like Yahoo!, which began as Jerry and David's manually curated list but soon cried out for its own automated search.
Today, we fret over XML sitemaps and crawl budgets, sophisticated signals meant to guide incredibly efficient but impersonal bots. It's worth remembering that the origin point was a plain HTML page, maintained by the inventor, that ground to a halt under its own success. It teaches us that discovery isn't a secondary feature of a network; it is the network's circulatory system. Without it, everything still exists, but in isolated pockets, unknown and fading into silence. Berners-Lee didn't just give us the links; his 'What's New?' page highlighted the empty space where the engine of discovery needed to go, a quiet admission that even the architect needed a better map.
Notes & further reading
A few pages I came back to while writing this: