The Persistent Scout: On Marking the Trail with Last-Modified
We often talk about directing crawlers with the grand gestures—the sitemap and the robots.txt. These are the official maps we draw, the proclamations we post in the town square. They are essential, of course, but there is a quieter, more persistent conversation happening with every visit, a conversation conducted in the subtle language of HTTP headers. Among these, one of the most eloquent and underutilized signals is the humble Last-Modified header.
Think of your website not as a static monument, but as a landscape that changes with the seasons. Some parts, like an old forest path, remain the same for years. Others, like a market square, see new stalls and activities every week. The Last-Modified header is the way you mark the date on each of these trails. It's a note left for the persistent scout—the web crawler—telling them precisely when the terrain was last altered. Without this note, the scout has no choice but to traverse the entire path every time, checking every stone and tree, using up their valuable time and energy, what we call 'crawl budget,' on paths that haven't changed in a decade.
The Practical Ritual
Implementing this is a matter of server-side diligence. The goal is to ensure that for every page, image, PDF, or other resource a crawler might request, your server returns a valid Last-Modified header in its response. The value should be a timestamp, an exact moment in time when the content was last substantively altered. This isn't about trivial fixes to a typo; it's about a meaningful update, a new section added, a product listing changed.
If you use a static site generator, this is often handled for you automatically, as the system can use the file's last-modified timestamp from the server's file system. For dynamic sites, driven by a database, the responsibility falls on your application. You must track when a piece of content—a blog post, a product page—was last updated and program your server to output that timestamp in the header. Modern web frameworks have built-in ways to set HTTP headers; it's a matter of querying your content's 'last updated' field and sending it along.
The payoff for this small effort is significant. When a crawler that has visited your site before returns, it presents the Last-Modified timestamp it cached from its last visit in a header called 'If-Modified-Since.' Your server then has a simple decision to make. If the content hasn't changed since that date, it can send back a very light '304 Not Modified' response. This response contains no body, only the signal that the old content is still valid. The crawler saves the immense bandwidth and processing time of downloading and re-parsing the entire page, and you save server resources. It's a handshake of efficiency.
This practice turns the crawl from a repetitive, brute-force survey into an intelligent, incremental update. You are not just publishing content; you are maintaining a dialogue with the mechanisms that discover it. By faithfully marking the trail with Last-Modified, you guide the scout's attention to what is new and relevant, ensuring their effort, and your server's, is spent not on rediscovering the old, but on charting the new.
Notes & further reading
A few pages I came back to while writing this: