http://simonwillison.net/2008-06-09T09:21:24+00:00Simon Willison2008-06-09T09:21:24+00:002008-06-09T09:21:24+00:00https://simonwillison.net/2008/Jun/9/official/#atom-tag

<p><strong><a href="http://googleblog.blogspot.com/2007/07/robots-exclusion-protocol-now-with-even.html">The X-Robots-Tag HTTP header</a></strong></p> News to me, but both Google and Yahoo! have supported it since last year. You can add per-page robots exclusion rules in HTTP headers instead of using meta tags, and Google’s version supports unavailable_after which is handy for content with a known limited shelf-life. <p>Tags: <a href="https://simonwillison.net/tags/google">google</a>, <a href="https://simonwillison.net/tags/http">http</a>, <a href="https://simonwillison.net/tags/robots-txt">robots-txt</a>, <a href="https://simonwillison.net/tags/xrobotstag">xrobotstag</a>, <a href="https://simonwillison.net/tags/yahoo">yahoo</a></p>