UltraX

"Refining Pre-Training Data at Scale with Adaptive Programmatic Editing": instead of filtering or rephrasing, a lightweight refinement model predicts structured edit operations (insert/delete/modify) that are executed deterministically on pretraining text — cheaper than full synthetic rewriting, more surgical than filtering. Reports the best average across five corpora at 1B scale with over 2% gains on fewer tokens. Full code, model (UltraX-0.6B-Preview), and data released (Apache-2.0 with a no-unchanged-redistribution clause). Continues the Ultra-FineWeb data-refinement lineage.