A UTF-8 BOM Turned a Disallow-All robots.txt Into Allow-All in My Crawler
In August, a page from a small Danish website turned up on the "recently indexed" admin page of my search engine. That site's robots.txt is two lines long and couldn't be clearer: