Google is testing how websites handle AI agents through a new experimental category called "Agentic Browsing" in its Lighthouse analysis tool, checking for llms.txt files that guide autonomous AI interactions.
Google has introduced an experimental "Agentic Browsing" category to Lighthouse, its web performance analysis tool, to measure how well websites prepare for AI agents accessing their content.
The new feature checks for llms.txt files—a standard file that websites can place in their root directory to provide instructions and guidelines for large language models and AI agents. Similar to robots.txt, which directs search engine crawlers, llms.txt helps AI systems understand how to interact with a site's content appropriately.
Lighthouse, which already audits websites for performance, accessibility, and SEO, now extends its scope to assess readiness for agentic browsing—the automated traversal of web content by AI systems without human direction.
This development reflects growing recognition that AI agents will increasingly browse the web independently. As LLMs become more capable of autonomous decision-making, website owners face new considerations about how their content is accessed and used by machines.
The introduction of this audit category signals Google's interest in establishing best practices for AI-agent interactions. By integrating the check into Lighthouse, a tool widely used by developers, Google encourages broader adoption of standardized guidance files.
Websites can use llms.txt to specify which sections of their site AI agents can access, rate limits, preferred data formats, and other interaction guidelines. This provides site owners with control over how their content is processed by automated systems.
The feature remains experimental, indicating Google is still refining how it evaluates agentic browsing readiness. As the tool evolves, additional metrics may be added to the category.
This move underscores the shifting landscape of web access patterns, where AI agents represent an increasingly significant portion of traffic. Website owners now must consider not only human visitors and traditional search engine crawlers, but also autonomous AI systems accessing their content.
An NHS watchdog has warned that AI systems transcribing patient consultations are making dangerous errors, including incorrectly documenting drug names and serious diagnoses that doctors sometimes fail to catch.
Andrew Bailey, governor of the Bank of England and chair of the Financial Stability Board, has warned G20 members that advanced AI models threaten global financial stability. Bailey cited risks of cyber-disruption spreading across jurisdictions in a letter to international finance ministers and central bank governors.
Simon Willison breaks down the mechanics of ChatGPT's operation in a detailed technical analysis. The explanation clarifies how the AI processes language and generates responses.
As autonomous vehicle companies expand robotaxi services, transportation workers face mounting job displacement concerns. The shift raises questions about workforce transition and economic impact on millions of professional drivers.