I allowed every crawler, and half of them still backed off
My robots.txt blocks nothing. Every page carries a meta robots tag permitting indexing. Google crawls the site happily and indexes almost everything within a few days of publication.
And when an AI assistant tried to read one of my articles, it was refused.