The Forgotten Architect: On Koster's Crawl and the First Rules of the Road
Long before ‘crawl budget’ was a term of art in a million SEO strategy decks, it was a simple, pressing question in a university lab: how do you walk the web without breaking it? In the web’s first dawn, its scale was unimaginable, its structure a wild frontier. The first search engines weren’t commercial giants but academic projects, and their architects were pioneers writing the rules of engagement as they went.
One of the most pivotal, yet often overlooked, of these figures is Martijn Koster. In the early 1990s, while working on the World Wide Web Worm, one of the very first search engines, he encountered a problem we still grapple with today. His crawler, a simple but ambitious program, would traverse the nascent web by following every single link it found. It was a brute-force approach for a brute-force age.
But the web, even then, was fragile. Servers were often little more than desktop machines tucked under desks, their bandwidth measured in kilobits. Koster’s crawler, in its enthusiastic quest, could—and did—overwhelm them. It was the original ‘unwelcome guest,’ consuming precious resources and often crashing the very sites it was trying to index. He realized that unfettered crawling wasn't just inefficient; it was unethical. It violated the unwritten social contract of the early web.
In 1993, he authored a seminal document known as the Standard for Robot Exclusion. It wasn't a protocol enforced by any governing body, but a polite suggestion, a set of voluntary guidelines. It proposed a simple text file, `robots.txt`, that a site owner could place at its root. Inside, they could write directives—‘User-agent: *’ followed by ‘Disallow: /’—to ask automated visitors to please keep out.
This was the birth of the crawl budget as a negotiated agreement. It wasn’t about a search engine unilaterally deciding how much time to spend on a site; it was a site owner having a say in the process. It was a handshake between the mapper and the territory. Koster’s work acknowledged that discovery is not a right but a privilege, granted by the host.
Today, the `robots.txt` protocol is a fundamental pillar of the web, used by every major crawler. We talk about directives and crawl delays with the clinical language of modern analytics, but it’s worth remembering its origin. It wasn’t conceived in a boardroom to maximize indexation efficiency. It was crafted by a conscientious architect in a collaborative spirit, to prevent his creation from being a nuisance. It was the first, crucial rule of the road, establishing that for the web to be crawled, it first had to be respected.
Notes & further reading
A few pages I came back to while writing this: