Internal Data Repetition Destroys Language Models

A study of how internal data repetition — duplicated subsets within a training corpus, as opposed to repeating whole epochs — harms language models under Chinchilla-era, compute-optimal training. The counterintuitive finding: repeating a moderately-sized subset a moderate number of times does more damage than either extreme, and the peak damage grows as a power law in model size — so bigger models are hurt disproportionately. On FineWeb-Edu-Dedup, a 344M-parameter Qwen3-style model loses up to 67% of its FLOP-equivalent compute to the worst repetition pattern.

The effect is explained through a misspecified linear-regression framework and shown to generalize beyond language models, making it a caution for data-curation and dedup pipelines. From the Koyejo (Stanford CS) and Donoho (Stanford Statistics) groups — authors include Rylan Schaeffer and Joshua Kazdan — with co-authors at Tel Aviv University and IMC Trading.