According to Cloudflare, 52% of crawler requests in June 2026 came from AI training bots. Site owners can now choose: allow search engines, refuse AI training.
Who reads your website most often? According to a Cloudflare report published on 1 July 2026, probably not humans. 52% of crawler requests in June 2026 came from bots collecting data to train AI models — up from about 22% in spring 2025. Overall, more than half of internet traffic is now non-human.
Three kinds of bots to tell apart
| Kind | Purpose | Benefit to you |
|---|---|---|
| Search | Indexes pages for search results. | Visitors arrive from Google/Bing. |
| Agent | An AI assistant opens a page on a user's request. | May bring prospective customers, depending on the assistant. |
| Training | Collects content to train AI models. | Usually no visits in return. |
The controls now available
| Date | Change |
|---|---|
| 1 Jul 2025 | Cloudflare began blocking AI crawlers by default for new sites and introduced *Pay Per Crawl* (announcement). |
| 1 Jul 2026 | The single switch was split into three — Search, Agent, Training — on all plans including Free (changelog). |
| 15 Sep 2026 | New domains automatically refuse Training and Agent traffic on pages with ads, while Search stays allowed. |
Cloudflare also offers an option to refuse AI training while still allowing Googlebot and Bingbot for search (on mixed-use crawlers).
What about llms.txt?
llms.txt is a proposed file summarising a site for AI models. Google states that Google Search does not use it — adding one neither helps nor hurts rankings (Google's guide). What crawlers still respect is robots.txt and the settings at your CDN provider.
Impact on small websites
- Server load: bot surges can slow your site and use up your hosting bandwidth.
- Skewed statistics: unfiltered bot visits make analytics look higher than reality.
- Content used without visits: articles and product photos can be summarised by AI without readers ever opening your site.
Decide your position
- Decide: may your content be used to train AI? May AI assistants read your pages?
- Configure it in
robots.txtand/or your CDN dashboard. Don't block search engines if you still want to be found on Google. - Check your server logs or hosting analytics to see which bots visit most.
- Review your decision every few months — AI bot policies change fast.
Sources
- The agentic internet: bot report — Cloudflare — blog.cloudflare.com
- AI traffic options changelog (1 Jul 2026) — Cloudflare — developers.cloudflare.com
- Content Independence Day: no AI crawl without compensation — Cloudflare — blog.cloudflare.com
- Accountable mixed-use AI crawlers — Cloudflare — blog.cloudflare.com
- AI optimization guide — Google Search Central — developers.google.com
Written from the sources above as of 31 October 2026. Third-party policies and figures can change — check the original sources for the latest.
All news