It's always a cat-n-mouse game.

I feel like at some point it needs to be active response. Phase 1 is a teergrube type of slowness to muck up the crawlers, with warnings in the headers and response body, and then phase 2 is a DDOS in response or maybe just a drone strike and cut out the middleman. Once you've actively evading Anubis, fuckin' game on.

These crawlers come from random people’s devices via shady apps. Each request comes from a different IP

I mean, we really have to ask ourselves - as a civilization - whether human collaboration is more important than AI data harvesting.

Eventually we'll have "defensive" and "offensive" llm's managing all kinds of electronic warfare automatically, effectively nullifying each other.

If this isn't fertile grounds for a massive class-action lawsuit, I don't know what would be.

I think the best thing to do is to not block them when they're detected but poison them instead. Feed them tons of text generated by tiny old language models, it's harder to detect and also messes up their training and makes the models less reliable. Of course you would want to do that on a separate server so it doesn't slow down real users, but you probably don't need much power since the scrapers probably don't really care about the speed

There once was a dream of the semantic web, also known as web2. The semantic web could have enabled easy to ingest information of webpages, removing soo much of the computation required to get the information. Thus preventing much of the AI crawling cpu overhead.

What we got as web2 instead was social media. Destroying facts and making people depressed at a newer before seen rate.

Web3 was about enabling us to securely transfer value between people digitally and without middlemen.

What crypto gave us was fraud, expensive jpgs and scams. The term web is now even so eroded that it has lost much of its meaning. The information age gave way for the misinformation age, where everything is fake.

I think every company in the world is telling everyone for a few months now that what matter is AI data harvesting. There's not even a hint of it being a question. You either accept the AI overlords or get out of the internet. Our ONLY purpose it to feed the machine, anything else is irrelevant. Play along or you shall be removed.

Is that really true? I guess I have no reason to doubt it, I just hadn't heard it before.

I'm ashamed to say that I switched my DNS nameservers to CF just for their anti crawler service.
Knowing Cloudflare, god know how much longer it'll be free for.

That's actually a major plot point in Cyberpunk 2077. There's thousands of rogue AI's on the net that are constantly bombarding a giant firewall protecting the main net and everything connected to it from being taken over by the AI.

Yes. A nonprofit organization in Germany is going to be launching drone strikes globally. That is totally a better world.

Its also important to understand that a significant chunk of these botnets are just normal people with viruses/compromised machines. And the fastest way to launch a DDOS attack is to... rent the same botnet from the same blackhat org to attack itself. And while that would be funny, I would also rather orgs I donate to not giving that money to blackhat orgs. But that is just me.

Wasn't this called black ice in Neuromancer? Security systems that actively tried to harm the hacker?

I love catching bots in tarpits, it's actually quite fun

Most of these AI crawlers are from major corporations operating out of datacenters with known IP ranges, which is why they do IP range blocks. That's why in Codeberg's response, they mention that after they fixed the configuration issue that only blocked those IP ranges on non-Anubis routes, the crawling stopped.

For example, OpenAI publishes a list of IP ranges that their crawlers can come from, and also displays user agents for each bot.

Perplexity also publishes IP ranges, but Cloudflare later found them bypassing no-crawl directives with undeclared crawlers. They did use different IPs, but not from "shady apps." Instead, they would simply rotate ASNs, and request a new IP.

The reason they do this is because it is still legal for them to do so. Rotating ASNs and IPs within that ASN is not a crime. However, maliciously utilizing apps installed on people's devices to route network traffic they're unaware of is. It also carries much higher latency, and could even allow for man-in-the-middle attacks, which they clearly don't want.

whos the defendent, specifically?

Honestly, man, I get what you're saying, but also at some point all that stuff just becomes someone else's problem.

This is what people forget about the social contract: It goes both ways, it was an agreement for the benefit of all. The old way was that if you had a problem with someone, you showed up at their house with a bat / with some friends. That wasn't really the way, and so we arrived at this deal where no one had to do that, but then people always start to fuck over other people involved in the system thinking that that "no one will show up at my place with a bat, whatever I do" arrangement is a law of nature. It's not.

Capitalism is grand, innit. Wait, not grand, I meant to say cancer