I was wrong about robots.txt
Technology
23
Beiträge
8
Kommentatoren
5
Aufrufe
-
Huh. So in this case, the file actually is respected. Refreshing
Kinda, but also not really. Any major tech player that has billions to lose will make a show of respecting robots.txt when presenting that information to third parties, lest they be exposed by basic journalism.
However, they also have separate networks in R&D that sweep the net all the time and do not care about such restrictions. It's theatre.
And they're still happy to punish people that have the gall to publicly decline their crawlers. Basically they can eat their cake and have it too.
-
But the article later does back it up: "Although Cloudflare singled out Google, other search engines that view AI search features as part of their search products also use the same bots for training as they do for search indexing."
In any case, I'm okay with admitting neither you nor me can look inside Google to see they're doing. But the claims are out there, I didn't make them up, whether they're true or not. Thank you for the certainly interesting Google crawler info link.
But the article later does back it up
The CEO of Cloudflare did not assert that. I was surprised that he would claim such a thing, and that should have made me read more carefully. Elon Musk notwithstanding, neither incompetence nor conspiracy theorizing are common at that level, publicly anyway.
You can believe whatever you like, of course. Freedom of opinion is nothing if not the right to be wrong.
-
But the article later does back it up
The CEO of Cloudflare did not assert that. I was surprised that he would claim such a thing, and that should have made me read more carefully. Elon Musk notwithstanding, neither incompetence nor conspiracy theorizing are common at that level, publicly anyway.
You can believe whatever you like, of course. Freedom of opinion is nothing if not the right to be wrong.
Right, but the article does. Anyway, I'm moving on. Thanks for the discussion.
