Scaling FP8 Training to Trillion-Token LLMs

"Scaling FP8 Training to Trillion-Token LLMs" (ICLR 2025 Spotlight): the first FP8 training run at trillion-token scale, exposing late-training instabilities — SwiGLU-induced outlier amplification — that are invisible in the short runs prior FP8 work relied on. Joint with Intel but university-led by Daniel Soudry.