The Unmourned Indexer: On the Forgotten Souls Who Built the First Web Catalogues

Before the algorithm was a god, it was a person. Before Google’s crawlers could parse the semantic web, human eyes were parsing screen after screen of raw HTML, making judgements that would determine whether a page lived in obscurity or found an audience. In the earliest days of the web, indexing was a craft, not a computation. And the craftsmen, the first indexers, were figures like Martijn Koster.

You may not know his name, but you likely know his creation, or at least the echo of it. In 1993, Koster created ALIWEB, the Archie-Like Indexing for the WEB. At a time when the web was a sprawling collection of academic papers and nascent personal sites, discovery was a monumental problem. There were no automated crawlers scouring the links. Instead, Koster proposed a system of courtesy and consent: webmasters would submit a simple text file, a precursor to the sitemap, describing their own site’s contents. ALIWEB would then collect these files and build a central index.

It was a library where every author wrote their own index card. This was the unspoken pact of the proto-web: you had to actively declare your presence. There was no passive discovery. A page that wasn’t submitted was, for all practical purposes, a page that didn’t exist. The crawl budget was measured not in server resources, but in human initiative.

Koster and his contemporaries were not just technicians; they were librarians of a frontier town, creating the first card catalogues for a collection that was doubling in size every few months. They made editorial decisions about categorization, about what constituted a relevant description. They were the gatekeepers of the signal in the noise, but they saw themselves as facilitators. The goal wasn’t to rank or penalize, but simply to make findable what was willing to be found.

ALIWEB was soon eclipsed by automated crawlers like the WebCrawler and, eventually, the behemoths we know today. The human indexer was rendered obsolete by a machine that could work tirelessly, following every link, building a map of the web from the ground up. The elegance of the voluntary sitemap was replaced by the brute force of the spider. The craft of indexing became the science of crawling.

We seldom mourn this transition, because the automated crawler is so much more efficient. But we lost something in the handover. We lost the clarity of intent. When a webmaster submitted a site to ALIWEB, they were making a conscious statement: “This is what I have made. This is what it is about.” Today, a page is discovered, interpreted, and categorized by a black box that may or may not grasp its creator’s intent, often judging it by metrics of popularity and freshness that have little to do with its inherent value.

The ghosts of the first indexers haunt our modern SEO strategies. The sitemap.xml file is a direct descendant of Koster’s text file, a nod to that original system of self-declaration. But now it’s a whisper in a hurricane of automated activity. The lesson from these forgotten pioneers isn’t about a specific technology; it’s about philosophy. They understood that discovery required a mutual agreement between publisher and searcher, a shared vocabulary. In our race to optimize for the machine, we risk forgetting that the ultimate destination of every crawled page is still a human mind, waiting for a signal it can understand.

Notes & further reading

A few pages I came back to while writing this: