Cloudflare to block AI crawlers by default, charge micropayments

Wading into the ongoing battle between web publishers and AI chatbots, the infrastructure giant that serves up 15-20% of the web is setting up some new rules.

Starting Tuesday, all new accounts on the content caching service will block AI bots — the crawlers that scrape a website’s content for AIs — by default.

— AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate, said Matthew Prince, co-founder and CEO of Cloudflare, quoted by CNBC.

Several big players signed up
You will then have to go to their new dashboard to specifically allow, block or charge them.

Several publishers, like Conde Nast, TIME, The Associated Press, The Atlantic, ADWEEK, and Fortune have already signed up to block AI bots entirely, reports TechCrunch.

New web business model
They are, like many others, concerned that AI responses are breaking their business models as compared to the traditional ten blue links of Google.

Before AI, Google would crawl website’s and provide snippets in their search results — but would crucially send traffic back to the publisher.

In the AI space, the chatbots will scrape websites for relevant information and return it in a full reply in a chatbot interface. So they scrape, but don’t return any users.

The idea of charging for scraping is a big, bold idea and might work someone as large as Cloudflare — but it’s still in beta and it’s anyone’s guess wether the AI labs will support it just yet, unless you are a huge player that can set your own terms.

Read more: Cloudflare: Content Independence Day, Cloudflare: A more technical review, Cloudflare: Pay per crawl, writeups at CNBC and TechCrunch.