The Library's Ghost: On AltaVista and the Artisanal Crawl
Before the world wide web was seen as a resource to be strip-mined for data, it was treated as a library to be catalogued. And in the mid-1990s, the head librarian, for a glorious moment, was a piece of software run by a company called Digital Equipment Corporation. They named it AltaVista, and its creation story speaks to a time when the act of crawling was not just a brute-force logistical operation, but a considered, almost artisanal craft.
The central figure in this story is Paul Flaherty, a researcher at DEC. In 1995, he had an idea: to demonstrate the raw power of his company’s Alpha server by applying it to the biggest, messiest dataset he could find—the exploding internet. This wasn't a project born in a marketing meeting to capture ad revenue; it was an engineer’s proof of concept, a showcase of technical prowess. The goal was simple, almost pure: build the biggest, fastest, and most comprehensive search engine the web had ever seen.
To do this, AltaVista’s crawler, Scooter, had to be something special. Today, a web crawler is a distributed, globally-balanced swarm, an army of bots working in concert. Scooter was different. In its initial, most storied incarnation, it was said that a single, powerful machine—one of those formidable Alpha servers—did a tremendous amount of the work. Imagine that: one diligent, digital ghost, flitting through the nascent web, page by page, link by link. This wasn’t an unseen tide; it was a single, incredibly focused current.
This approach had a profound effect on discovery. Because Scooter was so fast and efficient for its time, it could afford to be comprehensive. It didn’t need complex algorithms to prioritize a ‘crawl budget’ because its budget, relative to the web’s size, was immense. Pages weren't discovered through a labyrinth of prioritization signals; they were found by a crawler that aimed to simply visit everything. If a page was linked, Scooter would likely find it. This brute-force comprehensiveness was its defining feature and its greatest selling point. It gave users a sense that they were searching the entire web, not just a curated slice of it.
AltaVista’s brief reign represents a poignant moment in the history of discovery, a time before the web’s map became too vast for any single cartographer, no matter how skilled. The artisanal crawl, powered by a single machine’s ambition, was ultimately a lost cause. The web grew exponentially, and the model shifted from one master librarian to the distributed, prioritization-obsessed models we know today. We traded the ghost in the library for an entire spectral infrastructure. But for a while, the most powerful tool for finding things on the web was operated with an engineer’s idealism, a testament to the belief that with enough processing power, you could, in fact, build a complete index of human knowledge. It was a beautiful, impossible dream.
Notes & further reading
A few pages I came back to while writing this: