Cloudflare is requiring AI companies to separate their web crawlers from search bots by September 15 or face automatic blocking on publisher sites. The move aims to ensure content creators are compensated for material used in AI training.
Cloudflare has issued a deadline for artificial intelligence companies to distinguish between web crawlers used for search indexing and those deployed for AI model training. Companies failing to comply by September 15 will be blocked by default across many publisher websites using Cloudflare's services.
The policy addresses growing tensions between AI developers and content creators over unauthorized use of published material. Publishers argue that AI companies scrape their content without permission or compensation to train large language models and other AI systems.
Under Cloudflare's framework, AI crawlers would need separate identification from search engine bots. This allows publishers to permit search indexing—which drives traffic—while blocking AI training operations. Publishers can then choose to allow AI access only through paid licensing agreements.
Cloudflare's move targets the fundamental business model dispute in AI development. Companies like OpenAI, Google, and others rely heavily on web-scraped content to train their systems. Meanwhile, publishers including news outlets and content creators argue they deserve compensation or control over how their work is used.
The deadline gives AI companies three months to restructure their crawling infrastructure. Cloudflare will implement automated blocking of non-compliant crawlers across its network, which protects millions of websites globally.
This policy represents one of the first major platform-level interventions in the AI-content creator dispute. Rather than taking sides, Cloudflare is creating technical infrastructure for publishers to enforce their own preferences.
The move could pressure AI companies to negotiate licensing deals with publishers or develop alternative training methods. Some AI firms may already use separate crawlers for different purposes and could easily comply. Others may need significant technical changes.
Cloudflare's approach offers a middle ground between blanket AI bans and unrestricted scraping, but its effectiveness depends on widespread adoption and enforcement across the internet.
Smaller language models have reached performance levels competitive with much larger systems, shifting the economics of AI development. The trend suggests efficiency gains are making compute-intensive giants less necessary.
Google has released Gemini-3.5-Transcribe, a new AI model specialized in converting audio to text. The model aims to improve transcription accuracy across multiple languages and audio conditions.
Google has released Gemini Omni 1.1 Flash, an updated version of its multimodal AI model designed for developers. The new release focuses on improved performance and accessibility across text, image, audio, and video inputs.
Anthropic and AI chip startup MatX ended discussions over a roughly $7 billion acquisition. MatX is now pursuing a fresh funding round at a $4 billion valuation.