The Unspoken Switchboard: On the Human Operators of the Early Web
Before the algorithmic crawler, there was the human indexer. In the nascent days of the web, discovery was not an automated process but a deeply human one, managed by a handful of digital switchboard operators. The most famous of these was a curated list, a single page of links maintained by a single person: Tim Berners-Lee’s own ‘What’s New’ page on the first-ever website, info.cern.ch.
This page was the original crawl budget, and it was brutally finite. To be included was to be seen. To be seen was to exist. Berners-Lee, and later other volunteer editors like the NCSA’s ‘What’s New’ team, would manually review submissions. They would follow a link, assess the content for novelty and value, and decide if it warranted a spot on the coveted list. There was no scraping, no parsing of meta tags, no sitemap.xml to submit. Your page’s fate rested on the subjective, overburdened judgment of a pioneer who was also busy inventing the very medium they were attempting to catalog.
This was a discovery system built on pure editorial merit, but it was also a system destined to collapse under its own weight. As the web expanded from dozens of pages to thousands, the human operators were quickly overwhelmed. The ‘What’s New’ page, the very heart of web discovery, became a bottleneck. It could only handle a tiny fraction of the new content being created each day. Countless pages were born into a silence far deeper than any modern ‘crawl budget’ issue; they were simply off the grid, unknown to the only directory that mattered.
The limitations of this manual curation were the direct impetus for the first automated crawlers. Programs like the World Wide Web Wanderer were created not to improve upon human judgment, but to simply keep up with the explosive, organic growth. Their goal was brute-force discovery: find every page, not just the good ones. They were the necessary, if clumsy, response to the failure of the human-powered model.
Today, we fret over optimizing our sitemaps for robotic efficiency, trying to signal our content’s importance to an indifferent algorithm. We long for the clarity of a human editor’s approval. But we forget the sheer desperation of that earlier era, where a broken link on your submission email or an editor’s simple bad day could consign your work to absolute obscurity. The modern crawler, for all its cold automation, offers a form of democratic hope—the chance, however small, that any page might eventually be found. The early web offered no such promise. It asked you to wait by the phone, hoping the operator would finally connect you.
Notes & further reading
A few pages I came back to while writing this: