> Another change that will apply on September 15 is that multi-purpose crawlers (specifically those that combine Search with Training) will be allowed/blocked according to all of their behaviors, in line with our call for transparency for website owners. Since the defaults will be enforced by the most restrictive applicable rules, multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training (either through the new options to manage AI traffic, or through the legacy Block AI bots service).

It's kind of exhausting seeing Cloudflare playing both sides of the arms race.

I just can't imagine bringing myself to use their technology to build agents and build AI products when they're also doing things like this.

> This also lines up the incentive model we want to foster. Losing trusted status across the more than 20% of web domains that sit behind Cloudflare is a deterrent with teeth. Trust becomes something you can carry with you, and something you can lose.

And even more so, LLM language aside, fun and fascinating to see them flagrantly calling out their position here as if it's a positive.

The reasoning behind this is also flawed: blocking "bots" and "AI" means that our AI agents working for us are unable to do their work for us, because of knee-jerk bot-blocks.

What would force their hand?

It’s more likely they’ll strike undisclosed agreements with major sources of discussion like reddit etc.

That’s not to say getting new information as a way of context-providing is not going to happen but that’s not scraping.

As far as I can tell, Google is the only one sending me visitors. And the other big AI players might do so in the future.

Another option would be "No anonymous bots". So at least if a bot would want to crawl my site, they would have to identify themselves. Since the rise of the AI bots, I am getting hurt badly with insane amounts of requests from residential IPs that mimic real humans. The only difference being they don't make me any money. Only produce costs.

By the way, how is the situation over at Amazon's Cloudfront? Do they offer something that helps? Anyone here with them?

I dont get this. The question is are they a bot or a human. It doesnt matter what they are doing I dont want bots on my site.

I want pay-per-crawl. I clicked the link for it a year ago, got presented with a "request access" button, I "requested access" and obviously since I'm nobody I heard absolutely nothing. Now they're touting the link again, I checked, still that same "request access" button. I have no idea if anyone even has access to this feature.

I don't care about all this other stuff, I want the AI crawlers to pay me cash. Because boy do those fuckers want to crawl me. I'll gladly double the size of my gerrit/jenkins servers to keep up with the load if these stupid bots want to pay to crawl every jenkins build artifact and every changeset source file on the server, as they really seem to want to do.

There is no technical mechanism whereby it is actually possible to allow people to read your webpage and not use it for other things. You can’t give responses that say “this is ok for indexing but not for training”. Anyone trying to sell you this sort of technology is lying.