http://simonwillison.net/2007-09-22T00:36:17+00:00Simon Willison2007-09-22T00:36:17+00:002007-09-22T00:36:17+00:00https://simonwillison.net/2007/Sep/22/nextthingorg/#atom-tag

<p><strong><a href="http://www.nextthing.org/archives/2007/03/12/robotstxt-adventure">robots.txt Adventure</a></strong></p> Interesting notes from crawling 4.6 million robots.txt, including 69 different ways in which the word “disallow” can be mis-spelled. <p>Tags: <a href="https://simonwillison.net/tags/andrew-wooster">andrew-wooster</a>, <a href="https://simonwillison.net/tags/crawling">crawling</a>, <a href="https://simonwillison.net/tags/robots-txt">robots-txt</a></p>