As an actual content provider I see this as an opportunity. We pay our journalists real money to write real stories. If AI results haven't started affecting our search traffic they will start to soon. Up until now we've had two choices: block AI-based crawlers and fall completely out of that market, or continue to let AI companies train off of our hard-won content and take it as a loss that still generates a little bit of traffic. Cloudflare now offers a third option if we can figure out how to use it.
Dissing on Cloudflare is the new thing, and I get it. They're big and powerful and they influence a massive amount of the traffic on the web. Like the saying goes though, don't blame the player, blame the game. Ask yourself if you'd rather have Alphabet, Microsoft, Amazon or Apple in their place, because probably one of them would be.
> If AI results haven't started affecting our search traffic they will start to soon. Up until now we've had two choices: block AI-based crawlers and fall completely out of that market, or continue to let AI companies train off of our hard-won content and take it as a loss that still generates a little bit of traffic
You have another option, one that iFixit chose: poison[1] the data sent to AI crawlers, you may even use GenAI to generate the fake content for maximum efficiency.
I do hate it whenever somebody says that line to me, because it's up to the player to choose if they want to play, and that automatically puts them in a certain bucket.
I believe the game is rigged from the get-go. Nobody should be able to get that big without having a level of accountability that matches their size, and our current economic system doesn't support that. That's why X can go one way with content moderation, Meta another, etc. and whole countries get pissed off. That's why I hate the game. The players have scaled past it.
Web infrastructure is headed in that direction more and more too. I personally think that for all their reach and influence Cloudflare does a great job protecting the internet, but that can change at any time and it would be in nobody's control but Cloudflare's. For now I'm glad it's them and not AWS or Alphabet. I don't know how I'll feel in five years.
Not dissing any company; just pointing out a real concern to be considered, in this freshly disrupted and rapidly evolving environment.
We all know that someone is going to try to slip one past the regulators, and they're probably on HN, and we know from the past that this can pay off hugely for them.
Maybe, this time, the HN people who grumble about past exploiters and abusers in retrospect, can be more proactive, and help inform lawmakers and regulators in time.
And for those of us who don't want to be activists, but also don't want to be abusers -- just run honest businesses -- we're reminded to think twice about what we do and how we do it, when we're operating in what seems like novel space.
I hear this pretty often. I am curious what do you think Cloudfare should do?
I am pretty sure that if they started arbitrarily banning customers/potential customers based on what some other people like or don't like, everyone would be up in arms yelling stuff about censorship or wokeness or whatever the word of the year is.
As an example, what if I'm not a DDoS-for-hire, but just a website that sells some software capable of launching DDoS attacks? Should I be able to buy Cloudfare protection? Should a site like Metasploit be allowed to purchase protection?
> As an example, what if I'm not a DDoS-for-hire, but just a website that sells some software capable of launching DDoS attacks? Should I be able to buy Cloudfare protection? Should a site like Metasploit be allowed to purchase protection?
Would you say this nuance is a major issue on the other big cloud providers? Your own grey-area example of Metasploit is hosted on AWS without any objections. Yet the other cloud providers make a decent effort to turn away open DDoS peddlers, whenever I survey the highest ranked DDoS services it's usually around 95% Cloudflare and 5% DDoS-Guard.
I think Cloudflare should make the bare minimum effort to kick services which are explicitly offering illegal DDoS attacks, given that their current policy of not doing anything unless legally compelled to is demonstrably enabling the overwhelming majority of DDoS providers to stay online, which has terrible optics when they're in the business of mitigating those attacks.
Whatever slippery slope excuses they give, somehow AWS, Azure, GCP, Fastly, Akamai and so on have managed to solve the impossible problem of turning away DDoS providers without imposing Orwellian censorship in the process.
I distinctly remember Cloudfare being accused here of hosting spammers and selling protection against them a decade ago. Then suddenly the name became associated with positive things only, and the whole thing have been memory-holed.
> Website owners can block all web scrapers using AI Audit, or let certain web scrapers through if they have deals or find their scraping beneficial.
You don't have to make any deals, or participate in the marketplace, "block all" is right there.
And if you are not using Cloudflare, you are going to be abused. This is a sad fact, but I have no idea why you are blaming Cloudflare and not AI companies.
Well, as long as Cloudflare pays you to be "abused" (by which we mean, spending more money on bandwidth) it should be no problem for many of the site owners.
IMHO, this kind of thinking is only cynicism iff you're only looking for your angle to profit, and someone is peeing on your parade, every time they boorishly mention irrelevant, imaginary concerns like "ethics", "legality", or "Geneva Convention".
Interesting, as my theory for why cynicism is so common nowadays is that it's a coping mechanism for people who understand perfectly well what's happening in the modern world
You don't need to be a cynic if you have a grasp on reality If your truly understand something you are capable of evaluating it on a case by case basis without resorting to pathos.
It's extremely rare to truly understand the agenda, motivation and consequences of complex enterprise initiatives like this one by Cloudflare. Not to mention it can be pivoted.
So conjectures and hypothesis are not boring, but welcome in a discussion forum like HN.
It's pretty easy to see how cloudflare arrived in a situation where they are in a position to create this sort of marketplace without resorting to conspiracies about them trying to take over the internet.
They solved the very real problem of DDOS which consequently put them in a position to be a middleman between internet traffic between consumers and producers. Now they are expanding their business to take advantage of this privileged position they have.
> This time, Cloudflare has formed a "marketplace" for the abuse from which they're protecting you, partnering with the abusers.
When the original comment has a statement like this, it's a clear sign there is no potential for constructive discussion. Their understanding of markets has to be completely warped if they think a market existing constitutes partnering with one side of the exchange.
The term "abuse" in this description is both confused and confusing. Websites are trying to meter out a public resource, which is something they're unable to do by themselves. Cloudflare is offering to help them, for a fee. Once the practice is metered, it isn't abuse anymore. It's just using the public service, which the website owner deliberately operates.
This time, Cloudflare has formed a "marketplace" for the abuse from which they're protecting you, partnering with the abusers.
And requiring you to use Cloudflare's service, or the abusers will just keep abusing you, without even a token payment.
I'd need to ask the lawyer how close this is to technically being a protection racket, or other no-no.