The Lamplighter's First Rounds: On Illuminating the Web Before the Algorithm

Before algorithms could stride across continents of data in milliseconds, there was a humbler, more manual kind of discovery. We often imagine the early web as a digital frontier, a lawless expanse, but for it to function at all, it first needed a kind of public utility. It needed lamplighters. The most famous of these was a man named Tim Berners-Lee, and his first creation, in 1990, wasn’t the World Wide Web as we know it, but something far more intimate: a program he called the "WorldWideWeb" browser-editor, which contained a small, crucial file named WORLD_WIDE_WEB.RTF.

This document was the first sitemap. It was not an automated index generated by a spider, but a hand-curated register, painstakingly maintained by the inventor himself. Berners-Lee, in his role as the original lamplighter, would walk the nascent digital lanes of the CERN network, noting each new "site"—often just a single page on a colleague's NeXT computer. He would then return to his own workstation and update the .RTF file, adding a new title, a new path, a new description. This file was the central directory, the one place where everything known to exist was recorded. To "crawl" the web at this point was, quite literally, to consult Tim’s list.

This act feels almost impossibly distant from today’s automated discovery processes. There was no crawl budget to optimize because the entire known universe of hypertext could be traversed in an afternoon. The challenge wasn't managing scale, but creating a reliable system of illumination in the first place. The lamplighter’s round was a ritual of care, ensuring that each new point of light was connected to the whole, that no valuable resource was left flickering in obscurity. The integrity of the entire network depended on the diligence of this single, human curator.

We can see the ghost of this manual process in the modern sitemap protocol. The XML sitemap we submit to search engines today is a formalized, automated version of Berners-Lee’s WORLD_WIDE_WEB.RTF. It is our way of raising a hand to the algorithmic lamplighters and saying, "I am here. This is what I have built." But the fundamental trust relationship is the same: we are providing a trusted guide to our own territory, hoping it will be used to illuminate our pages for others.

The lesson from these first rounds isn't about efficiency; it's about intent. The early web was lit by hand because it was built on a principle of shared knowledge. The lamplighter didn't own the light; he serviced it for the benefit of everyone on the street. In an age where discovery is dominated by immense, inscrutable algorithms, there’s value in remembering this origin. It suggests that the most effective way to be found is not always to game a system, but to build something worth adding to the map, and to clearly, honestly, signal its presence—just as the first architect of the web did, one manually typed entry at a time.

Notes & further reading

A few pages I came back to while writing this: