Scaling Domain Data Repetition in LLM Pretraining

ByteDance Seed / Tsinghua study (work done at Seed; project lead Xintong Hao, corresponding author Jingzhao Zhang) of how many times to repeat scarce high-quality domain data as pretraining scales. Its central move is methodological: prior cross-scale repetition studies hold the training-data size fixed while growing the model, which makes larger models overfit sooner and pushes the optimal repetition count toward 1. Under the regime that actually matters in practice — a fixed tokens-per-parameter ratio (TPP), where the token budget grows with model size — the conclusion flips: the optimal repetition count mildly increases with model size.

The sweep covers four high-quality domains (code, math, Wikipedia, medical) mixed into unique web data, with the unique domain fraction α ∈ {1/40, 1/20, 1/10} of the token budget and repetition counts 1–7, across several model sizes at a constant TPP (>100; absolute sizes and TPP are not disclosed). Findings: the optimal count is strongly negatively correlated with a domain's final validation loss (low-loss domains tolerate and benefit from more repeats, high-loss domains overfit earlier); it is largely insensitive to the amount of unique data; and domain matters most, then model size, with α barely moving it. A theoretical model frames this as a trade-off between knowledge acquisition and noise fitting, and the paper also shows the learning-rate schedule sets how much repetition a model tolerates before degrading. Practical recipe: sweep repetition counts on a small proxy model at the target TPP (any representative α works) and transfer the result conservatively — a count that doesn't overfit the proxy is unlikely to overfit the larger model.