The UK's AI Security Institute has found that standard AI benchmarks systematically underestimate what AI agents can actually accomplish by imposing artificial computational constraints. When token budgets are increased tenfold, success rates on software engineering tasks jump roughly 25 percent.
A comprehensive study by the UK's AI Security Institute examined seven widely-used AI benchmarks and discovered a consistent pattern: they measure agent performance under artificially limited conditions that don't reflect real-world capabilities.
The research specifically tested the impact of increasing token budgets—the amount of computational resources available to AI systems during evaluation. On software engineering tasks, this adjustment alone produced a 25 percent jump in success rates.
Key Findings
The gap between benchmark results and actual capability is substantial. According to AISI analysis, true progress at the AI frontier is approximately 60 percent steeper than previous measurements indicated. This discrepancy matters most for newer models, which benefit disproportionately from additional computational resources.
The implications are significant for both capability assessment and safety evaluation. If benchmarks systematically underestimate what AI agents can do, organizations may be drawing incorrect conclusions about agent limitations and appropriate safeguards.
What This Means
The findings suggest that current evaluation methodologies need revision. Standard benchmarks provide useful comparative data but may not accurately represent agent performance under realistic computational conditions. Researchers and developers relying on these benchmarks for capability claims should account for this systematic underestimation.
The study highlights a methodological blind spot in AI evaluation: the assumption that benchmark constraints reflect natural constraints on agent performance. In practice, computational budgets in deployment scenarios often differ significantly from those used in standard evaluations.
This work comes amid broader scrutiny of AI evaluation practices. As AI systems become more capable and integrated into critical systems, accurate measurement of their abilities becomes increasingly important for risk assessment and responsible deployment decisions.
The UK's AI Security Institute's findings underscore the need for more comprehensive evaluation frameworks that test AI agents under varied computational conditions rather than relying on standardized but potentially misleading benchmark scores.
Skild AI has released S1, a robotics foundation model capable of learning novel tasks from a single video demonstration without requiring fine-tuning. The model can execute commands with 10-minute planning horizons.
Anthropic has enabled shared memory across Claude chat and Cowork, allowing the AI to retain context about projects and preferences without repeated briefing.
A new tool called Claudette lets users remove casual, clickbait-style language from Claude's responses. The open-source project addresses complaints about AI-generated text mimicking viral content formats.