Analysis of pretraining progress from 2019 to 2025 reveals that improvements in data quality and curation, rather than architectural innovations, account for most gains in compute efficiency.
A breakdown of six years of AI advancement shows a clear pattern: the field's rapid progress in pretraining efficiency has relied more heavily on data improvements than on model architecture changes.
The analysis, discussed on the Dwarkesh Podcast, separates the drivers of AI efficiency into two categories. Model improvements encompass advances in neural network architecture, training algorithms, and optimization techniques. Data improvements include better dataset curation, filtering, quality control, and training data selection strategies.
From 2019 to 2025, the data improvements category emerged as the dominant force. Companies and researchers have increasingly focused on refining what their models learn from rather than fundamentally redesigning how models process information.
This finding has significant implications for the AI industry. It suggests that raw compute power, while still important, becomes more effective when paired with thoughtfully prepared training data. Organizations can achieve better efficiency by investing in data pipelines, annotation quality, and dataset construction rather than pursuing ever-larger model architectures.
The trend also reflects practical realities in AI development. Scaling model size faces physical and economic constraints—larger models require exponentially more computation and memory. Data improvements, by contrast, offer more incremental optimization opportunities without these hard limits.
Industry leaders have recognized this shift. Major AI labs have expanded their data operations teams and invested in techniques like synthetic data generation, active learning, and adversarial data filtering. These approaches maximize learning efficiency per unit of compute.
The analysis provides context for understanding how AI labs have managed to deliver increasingly capable models despite facing diminishing returns from raw scale. Rather than hitting a wall, the field has pivoted toward smarter data utilization.
As compute costs stabilize and competition intensifies, the emphasis on data efficiency is likely to continue. Organizations that excel at dataset construction and curation may gain competitive advantages over those pursuing purely architectural innovations.
Research shows large language models can develop novel social biases as they adapt and explore during operation. The findings challenge assumptions that model biases remain static after training.
Researchers at frontier AI companies are publicly raising concerns about the safety risks of their own technology. These internal warnings deserve attention despite coming from potentially biased sources.
Indian workers are using iPhones to generate training data for humanoid robots, fueling a global race for real-world AI datasets. The practice highlights a growing paradox in automation: humans building the tools designed to eliminate their own jobs.
Meta unveiled Muse, a personal AI agent powered by Muse Spark 1.3 that performs tasks on users' behalf. The service offers a free tier with 100M tokens per week, plus $20 and $100 monthly subscription options.