Without a market CloudFlare wouldn't be able to ruin the internet. You can thank all of the incessant AI bots for that. I can't even browse GitHub anymore without logging in.
Don’t forget that the main issue isn’t the initial data scrape but rather the fact that prompting an llm agent like claude code can amplify into 10 requests as it tries to answer your question.
Neither Cloudflare nor their competitors were aywhere near as prevalent before the AI bots. Farnell and Renesas outright blocked me, DigiKey makes me sit through their bullshit at nearly every request, issue trackers have the slowdown widgets etc. etc. etc. As late as early this year I never had these problems.