Often it is respected, but the resulting problem is platforms conflate things with the questionable AI scraping crawlers to blackmail websites into participating in feeding AI.

For example, Googlebot if enabled won't just list you for search, but will also scrape your contents for Google's AI. Edit: see https://arstechnica.com/tech-policy/2025/07/cloudflare-wants-google-to-change-its-ai-search-crawling-google-likely-wont/ as source. I imagine LinkedinBot, given it's microsoft, will feed some other AI of theirs as well on top of the previews.

Until regulation steps in to require AI bots to separately ask for crawling permission, or to actually get a proper license for reuse of the contents, this situation isn't going to improve.

Googlebot if enabled won’t just list you for search, but will also scrape your contents for Google’s AI.

False.

Absolutely true. They'll buy the data they want from some shitty crawler running from some data broker in some far-flung and lawless part of the world, hallucinate the actual source, and pretend they had no idea their "data partner" wasn't respecting robots.txt if they have to, which they won't ever have to do because it's literally impossible to detect and prove and realistically unenforceable.

This is a company that removed it's company motto of "Don't be evil" because it found it too "limiting". Don't be naive.

Googlebot if enabled won’t just list you for search, but will also scrape your contents for Google’s AI.

False.

So. If I can add something here for everyone's benefit

No search engine really obeys robots.txt

Their publicly acknowledged crawlers do, but they have other crawlers that aren't know that ignore the file.

Google knows every inch of your site, allowed or not.

See, just because a search engine says it doesn't know, doesn't mean it hasn't crawled.
Just doesn't display the results based on your settings.

What did he think a crawler is? Why was he surprised that not allowing companies to use his data lead to them not using his data? Looks like he has another surprise coming when he notices that search engines no longer index his blog.

See here: https://arstechnica.com/tech-policy/2025/07/cloudflare-wants-google-to-change-its-ai-search-crawling-google-likely-wont/ If you have a source that says it's false, I'd be curious.

I feel like most casual users would not make the connection of "crawlers" to link previews that they talk about it the article.

Sure, if you understand that robots.txt includes all robots then sure. But that is not how general news media has been talking about robots.txt.

Ok. That quotes a tweet by Cloudflare's CEO. IDK what his qualifications are, but his conflict of interest is obvious enough. Real quality journalism there.

Here's Google technical documentation on its crawlers: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers

So what's the quote from the documentation that backs up your claim? The line "perform other product specific crawls" seems extremely vague by design.

I'm not really sure what you are asking here. Did you notice that you can scroll down and see a list of their crawlers?

Nothing on this page seems to contradict the article. But if I simply missed the part that does, I'd be happy to learn.