DATA, NOT MODELS, DROVE AI EFFICIENCY GAINS

Analysis of pretraining progress from 2019 to 2025 reveals that improvements in data quality and curation, rather than architectural innovations, account for most gains in compute efficiency.