Over-Tokenized Transformer

Argues that a Transformer's input and output vocabularies should be decoupled and scaled independently. The Over-Tokenized Transformer (OT) pairs Over-Encoding (OE) — expanding the input vocabulary with hierarchical n-gram tokens so each position carries more context — with Over-Decoding (OD), a larger output vocabulary that supplies finer-grained next-token supervision.

The central finding is a log-linear relationship between input vocabulary size and training loss: bigger input vocabularies reliably lower loss across model scales. Scaling the input vocabulary 128× lets a 400M model match the training loss of a 1B baseline at essentially no extra compute — roughly free capacity from tokenization alone. Validated up to a 7B-scale (OLMoE, 1.3B active) sparse MoE. Accepted at ICML 2025.