Millions of websites to get 'game-changing' AI bot blocker
Technology
28
Beiträge
23
Kommentatoren
13
Aufrufe
-
The problem is that the biggest service Cloudflare provides is DDoS protection, and doing that requires that you have more bandwidth available than your attacker. Having enough bandwidth to withstand modern botnet powered DDoS attacks is ridiculously expensive (and it's also a finite resource, there's only so much backbone infrastructure). Basically it's economically infeasible to have multiple companies providing the service Cloudflare does. You might be able to get away with two companies doing so, but it's unlikely you could manage more than that without some of them starting to go bankrupt.
I wonder if it would be a good investment for a country to have their own then down the line expand to sell the same service to others
-
The problem is that the biggest service Cloudflare provides is DDoS protection, and doing that requires that you have more bandwidth available than your attacker. Having enough bandwidth to withstand modern botnet powered DDoS attacks is ridiculously expensive (and it's also a finite resource, there's only so much backbone infrastructure). Basically it's economically infeasible to have multiple companies providing the service Cloudflare does. You might be able to get away with two companies doing so, but it's unlikely you could manage more than that without some of them starting to go bankrupt.
when a critical service is not economical for more than one business to do (natural monopoly), that's when govt should be stepping in.
-
I wonder if it would be a good investment for a country to have their own then down the line expand to sell the same service to others
It's OurFlare, comrade.
-
This post did not contain any content.
This is not about stopping bot-scrapers, it's about charging them.
-
Until the AI companies find a way around it. Love the idea so hopefully it causes at least 3 days of struggle for the AI crawlers.
Having said that... Can someone else put this in place so we do not have Cloudflare hosting everything where we would just be one intern away from a global outage. Please? Pretty please?
Proof of work seems to be working pretty well for many websites.
-
when a critical service is not economical for more than one business to do (natural monopoly), that's when govt should be stepping in.
Which govt? I'm not comfortable with the idea of the current US govt having control over this sort of service.
-
This post did not contain any content.
I can't wait to be denied access to websites because of it. Even more than I already am, that is.
-
This post did not contain any content.
To that end the company is developing a "Pay Per Crawl" system, which would give content creators the option to request payment from AI companies for utilising their original content.
So Cloudflare is not as much "saving the Internet", as just becoming a middleman between LLM training companies and content creators. Which I believe has a potential of being a true goldmine in the future.
-
To that end the company is developing a "Pay Per Crawl" system, which would give content creators the option to request payment from AI companies for utilising their original content.
So Cloudflare is not as much "saving the Internet", as just becoming a middleman between LLM training companies and content creators. Which I believe has a potential of being a true goldmine in the future.
Corps are gonna corp.
-
To that end the company is developing a "Pay Per Crawl" system, which would give content creators the option to request payment from AI companies for utilising their original content.
So Cloudflare is not as much "saving the Internet", as just becoming a middleman between LLM training companies and content creators. Which I believe has a potential of being a true goldmine in the future.
Can you DRM a crawl ?
-
This is not about stopping bot-scrapers, it's about charging them.
Hopefully people will price their content out of reach of the bot-scrapers, effectively stopping them.
-
Which govt? I'm not comfortable with the idea of the current US govt having control over this sort of service.
are you comfortable with a single corporation having control over this sort of service? the current government is obviously not ideal but that shouldn’t stop us from regulating monopolies.
-
Until the AI companies find a way around it. Love the idea so hopefully it causes at least 3 days of struggle for the AI crawlers.
Having said that... Can someone else put this in place so we do not have Cloudflare hosting everything where we would just be one intern away from a global outage. Please? Pretty please?
GitHub - TecharoHQ/anubis: Weighs the soul of incoming HTTP requests to stop AI crawlers
Weighs the soul of incoming HTTP requests to stop AI crawlers - TecharoHQ/anubis
GitHub (github.com)
-
Until the AI companies find a way around it. Love the idea so hopefully it causes at least 3 days of struggle for the AI crawlers.
Having said that... Can someone else put this in place so we do not have Cloudflare hosting everything where we would just be one intern away from a global outage. Please? Pretty please?
Yeah this will have absolutely no impact to gathering training data.
I assumed it was to block ai agents crawling it during requests, which they’d be unlikely to bypass in the web ui.
But no company spending millions on training will hesitate to have an agent appear as a regular desktop user to scrape data.
-
Can you DRM a crawl ?
You can if you're Cloudflare.
-
Yeah this will have absolutely no impact to gathering training data.
I assumed it was to block ai agents crawling it during requests, which they’d be unlikely to bypass in the web ui.
But no company spending millions on training will hesitate to have an agent appear as a regular desktop user to scrape data.
Does cloudflare still look at the agent? I thought they have more reliable data points.
-
Does cloudflare still look at the agent? I thought they have more reliable data points.
I meant an ai agent not the browser agent. All data points can be spoofed and if not they’ll pay a human to scrape before they pay for content.
-
This post did not contain any content.
So... Proprietary Anubis?
-
This post did not contain any content.
I didnt read "bot blocker" wrong, thats for sure..
-
I meant an ai agent not the browser agent. All data points can be spoofed and if not they’ll pay a human to scrape before they pay for content.
Okay, fair enough, I thought you meant just the user agent. Trouble with having a bot make it look like an actual user is looking at the data, is that it's slow and inefficient. Trouble with paying humans to scrape the data is that it's slow and inefficient. These companies want to ingest data ridiculously fast because there's so much of it. If all else fails, they'll resort to paying the content creators. But only if it's data they really do think gives their model a competitive edge in some metric and they can't pirate it. E.g I can see them paying for scientific research they can't get from libgen, but not some rando's blog post or local news website.
